最便宜的模型不等于最便宜的系统：AI 定价应看产出而非投入

I see a world where the smartest model is actually the cheapest. People are thinking about outcomes in AI and they're looking at 20 30 50 and they're saying that's ludicrous. That's crazy. That is underestimating by an order of magnitude how massive a transformation this is going to be. 8 and 10 billion is the new 1 billion. Today we have CTO and co-founder of Factory Eno Reyes. He is one of the most articulate and insightful thinkers about the value stack of AI that I've interviewed.

Factory is one of the leading companies that specialize in autonomous software development and ENO is incredible in this show today. There's going to be a lot of notes taken in this discussion.

Two of the largest companies that provide models today have explicitly said we are going to go after every single one of these industries and businesses that we provide intelligence for. Calling open- source models Chinese models is a scop by the Frontier Labs to basically trick people into thinking that they're scary and otherize them. In 3 years, 99% of workflows are going to be done on open models.

Ready to go.

Eno dude, it is so good to have you in the studio. I obviously had Matan in the studio. I was chatting to Keith Boy over the weekend about you. So, thank you so much for joining me, dude.

No, thank you for having me. I'm super pumped to be here. Now, I hate background stories. I'm sure you listen to podcasts. It's like, how did you get into this? And you were like, I'm bored of this. But downstairs, I asked you, how did you get into technology and fall in love with it? And your story was like heart-wrenching and compelling. And so, I have to ask it, how did you first fall in love with computers, do you think?

Yeah, I I think that the beginning was my parents. So both my parents went to art school and they were always in love with technology and how that sort of intersected with creativity and media and I think that where that started especially with my dad was he was born in San Francisco in the late60s and he was 6 years old and he got actually hit by a bus uh and that changed the trajectory of his life in a pretty crazy way where you know he wasn't playing sports, he wasn't able to do as much of that like traditional 1960s kids stuff.

Instead, he went to technology and computers like the early days of uh when you know you barely had screens and were instead sort of tinkering. And he spent most of his life basically embracing technology as a way to extend his own reach beyond like what I'd argue his physical body could do. It's so interesting how life delivers you a hand, so to speak, and how consequential that hand is to kind of how you are today.

100%.

Um, it's very hard to transition from a father being hit by a bus to margins, right? Like only a venture capitalist could do that in such a swift transition.

Well, some margins in AI might make you feel like you've been hit by a bus. So,

I mean, yes, absolutely. and we we chatted before you said the cheapest model isn't necessarily the cheapest system and I read this when I was doing the work over the weekend and I was like huh can we just unpack that the cheapest model isn't necessarily the cheapest system what does that mean yeah I you know it really comes down to this idea that when you're thinking about price you should not be thinking about the inputs to the price you should be thinking about the outputs so I think about the price of the outcome so let's take software as an example How much does a code review cost is far more interesting than how much do the tokens inside of that code review?

And so uh you take a very sophisticated model if it's able to do that code review uh immediately without making any mistakes, getting the right outcome right away, searching the right phrases uh and you use you know a thousand tokens or whatever it will be significantly more than that a million versus you use a cheap model and it spends time. running it uses 50 million tokens and ultimately you know that price difference of the full outcome uh makes the higher quality model cheaper.

Uh now that's not how all tasks go but for many of the most uh intelligent demanding tasks. I see a world where the smartest model is actually the cheapest. I totally hear you there. And it kind of goes against that a token is a token theory, but will we then have millions of specialized models with every company having specialized models operating on their own data because to your point there, it'll be able to work much more efficiently.

Yeah, I think that there's a real world where the speciation of models increases very rapidly. This is sort of the world where the fireworks and the people who help make models possible. I think win because the alternative is you only have a very few specialized providers that have models. Now in our view there is going to be probably a difference between the commodity task executors. So this is just like your everything model.

Uh and that we think will be dominated by open models. And then you have businesses that will say well you know we do a lot of commodity tasks but there's a couple of very high volume specialized tasks that only we do. And for those your commodity model won't be good enough. Your frontier model will be too expensive. And so they'll want something in between where they can take a commodity model and make it good enough via post training.

And then ultimately they'll run that and they probably will be the only consumer of it. So they won't even give it to the rest of the world. They'll just keep it entirely internal. Uh so I think that that will lead to a lot of models not like a you know it won't be millions but it'll definitely be quite a lot. When you look at the post-training required when you look at the implementation required I look at that and I think that company structures and teams today are simply not equipped to do that.

How will we solve for that? Is this just a moving into an incredibly services and implementationheavy world where we have these insane AI teams coming into every how do we solve that?

Yeah, and I think that this is one of those things where right now the recipe for post-training and building models does live primarily in the heads of a specialized few group of people. Um, but that's also how software development was like 20 years ago. So in my mind, the same tools that are currently democratizing access to software are actually the tools that will be used to democratize access to intelligence in general.

So I see a world where uh you're a company, an enterprise, and you say today, well, we don't have the the knowledge or the skill set to build our own specialized models. Well, very shortly already to a certain extent, you can uh open up a platform, go into their, you know, web page, click a couple buttons, describe the task you care about, point it towards those workflows that happen in your business today, and outcomes a model.

And that model will be really good. Uh, and so I think that right now the there's like a couple of companies that claim that like recursive self-improvement and model training will be only their domain. And I think in reality many businesses will have access to that technology via software services that other companies sell.

You said about kind of focusing on the outcomes and not the inputs.

That is a great idea when it's a verifi verifiable output. code review when there is ambiguity to something which could be uh marketing conclusion did it come through X channel or Y channel uh my girlfriend's a lawyer different legal notes ambiguous some people like it one way some people like how do you think about the importance of verifiability in determining outcome quality

yeah I mean verifiability is ultimately the single most important property of success with current AI systems and I think that The way that we'll sort of progressively address this is by building new ways to verify the work t