AI的下一个突破：在工作中学习

Here’s the big research bet the labs are making currently: if we train AIs to accomplish millions of verifiable tasks across thousands of diverse RL environments, then we’ll basically have built AGI. Because such training will create these general problem solving skills (like how to make progress on an open ended task for weeks on end in the face of errors, mistakes, and ambiguity).

The people optimistic about this vision would say that anything we might consider a fundamental deficits with the current learning paradigm—for example, data inefficiency and lack of continual learning—can be steamrolled by just scaling training more, just as all the supposed “fundamental” research problems in natural language processing collapsed against the flood of compute thrown into LLMs.

Yes, these models are 1/1-millionth as sample efficient as humans during training. But training a one-time cost amortized across billions of user sessions. What matters is how smart, general, and sample efficient the model is within a session, and that’s clearly been improving as we do more RL training. AIs are able to solve more and more ambitious problems across longer and longer time spans - anybody who’s been using these models for coding knows that.

Similarly, continual learning—as defined as the model’s weights getting updated from deployment—may simply not be necessary. Again, because if in-context learning gets so good across longer and longer horizons, then we don’t need to distill back to weights to get on-the-job learning. People often say that their employees are not net productive until six months or more on the job, so clearly online learning is necessary for competence. But what if you could just fit those six months into the context window? There’s been tons of architectural innovations on the transformer which dramatically increase the length of context you can store. With a couple more years of progress, why couldn’t we have arbitrarily large context windows?

Grindability is just as important as verifiability

To address whether this will work, I want to first take a detour and ask a question about the current nature of AI progress that I find confusing and interesting. Why has progress on computer use been so slow?

Computer use is so clearly verifiable (did the desired Etsy item get ordered, is everything I need corporate for my event booked, have my taxes been submitted). So isn’t it weird that computer use has been making much slower progress than coding and math and other verifiable domains? There’s many reasons for this, I’m sure, among them the fact that the models are exposed to far less high quality multimodal data during pretraining, and that video consumes the context window far faster.[1]

But one reason that I think it quite underrated, and also which reveals the canyon walls against which the river of AI progress will only slowly chip away at, is that it is not enough for a domain to be verifiable. It also has to be very grindable—in the sense that you can run lots of parallel rollouts against a deterministic and replayable simulator. If you’re trying to make a model better at coding, you can create an environment that has a software repo with some missing feature that you’ve tasked the AIs with creating, and then you have a thousand parallel agents just go at the problem, each with their identical copy of the container.[2]

But this doesn’t work with computer use—at least not trivially. You can’t have a thousand agents go try the same checkout flow on Amazon.com. Because Andy Jassy will find and detect your bots and shut your ass down.

You can solve this by making clones of Slack, Gmail, and all the other common applications and websites. But at least currently, this is a very labor-intensive and unscalable way to build environments. Of course, once AIs get good enough at coding to themselves build these clones with extremely high fidelity, then I’m sure computer use will make a ton of progress. And you’re also killing two birds with one stone with this kind of procedure, because getting AIs to rebuild whole complex applications from scratch is a great RL objective for coding as well.

But while computer use itself may soon be solved, its current lethargy tells us the following: that unless you can build a very replayable training target for a domain, the models will struggle to make much progress. The reason this is true is, of course, that the models are incredibly sample inefficient during training. This is the point I was making in my last monologue.

In computer use, we might be able to make up for this sample efficiency deficit by building these farmable deterministic simulators. But for so many different other kinds of skills an AGI would need to learn, we simply can’t do this.

How would we train an AI to build a business? How would you make an AI that’s really good at winning court cases? Or having a profitable day trading in the markets? Or helping a candidate win an election? The rollout requires interacting with the world and cannot be recreated simply within the datacenter. And the outer loop verification may take months or years of real world actions to elicit, and cannot be re-observed by perturbing the model’s actions thousands of times in parallel so that you can isolate what exactly the model did that actually worked.

Dealing with such reset-free non-stationary environments is a known open problem in RL. I’m not pointing out anything new. But I really do want to emphasize that because of the idiosyncratic and sparse nature of the data in most domains in the world, you need sample efficiency in order to get proficient.

If AIs are to develop all the skills that humans have, and even skills that no humans have, then they need to be able to learn from information revealed in unstructured, unverifiable, and ambiguous ways from scarce amounts of real world interaction. Because in many domains, the relevant training information simply doesn’t exist in any other way.

What is the RL environment to make an AI as good at politics as Lyndon Johnson, or as good at building a space launch business as Elon Musk?

Will RLVR alone generalize?

The labs are betting that RLVR will generalize to all these other domains. If you train in enough containerized, reproducible environments, you will develop a very general agent that can make and execute plans, and learn rapidly from new information, and even pick up new skills, all within a session.

If you drop this endlessly RLVRed AI into Texas politics in 1948, it could give you better advice than LBJ about winning the Senate seat; And if you gave it 100 million dollars in 2002 and let it cook, it could build SpaceX for you.

Whether RLVR generalizes that well is an open empirical question: if labs went from spending billions of dollars on RL environments to a trillion dollars, would you get a fully general, human-like intelligence operating within the context window?

Dario gave a telling quote during our podcast together, which I think hints that RLVR generalization is not this infinitely strong. When he was explaining why model performance tends to degrade at long context, he said:

There’s the context length you train at and there’s a context length that you serve at. If you train at a small context length and then try to serve at a long context length, maybe you get these degradations.

Maybe I’m reading too much into it but he seems to be saying that short-horizon RL training doesn’t necessarily generalize to long-horizon RL performance. And if we can’t generalize from short to long horizon, how are agents supposed to generalize from lots of white collar task training, to, say, getting dropped into the real world and building a business from scratch as well as Sam Walton?