Google如何教LLM控制机器人并引发行业热潮

It’s day three of Robot Week! You can click here to get 25% off an annual subscription.

Most people first heard about large language models after OpenAI introduced ChatGPT in 2022. But in the minds of many AI researchers, the key breakthrough came two years earlier with the release of GPT-3.

With 175 billion parameters, the OpenAI model was more than 100 times larger than its predecessor, GPT-2. It was trained on a massive 300 billion tokens. And as a result, it generalized far better than previous models. For the first time, a single model could perform a wide variety of tasks — from translating between languages to answering trivia questions — without task-specific training.

It took OpenAI two more years to develop the techniques that transformed this raw “base model” into a user-friendly chatbot like ChatGPT. And then it took a couple more years to develop the techniques — like long-context reasoning, tool use, and context management — that transformed those early chatbots into the powerful agents we have today.

In short, there was a long road from GPT-3 in 2020 to Claude Code in 2025. But for those who knew where to look, the potential of LLMs was already clear in 2020.

The robotics world is now traveling a similar path. Its “GPT-3 moment” came in July 2023, when Google announced a model called RT-2. To create it, Google trained a multimodal LLM to directly generate robot actions. RT-2 wasn’t Google’s first transformer-based robotics model — the company released a predecessor called RT-1 a few months earlier, for example — but RT-2 was massively larger than earlier models. RT-1 had 35 million parameters. The RT-2 models had billions of parameters.

And as with GPT-3, size mattered. The RT-2 team reported its model showed “significant improvements to generalization over objects, scenes, and instructions.” They added that the new model exhibited “a breadth of emergent capabilities inherited from web-scale vision-language pretraining.”

For example, researchers placed a can of Coca-Cola on a counter alongside framed photos of Snoop Dogg, Tom Cruise, and Taylor Swift. They then prompted the robot to “move coke can to Taylor Swift.” The robot grabbed the can and moved it toward Swift’s photo.

At the time, Karol Hausman was a member of the RT-2 team. In a March interview, he described this as a moment of “huge, huge excitement” because “the robot models had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connect it to the image of Taylor Swift, and then connect it to the right motion that would move the Coke can to the picture of Taylor Swift, all from Internet data.”

“That was the moment where it clicked for us that it could actually work — where you could bring in a lot of prior knowledge from LLMs, from the Internet, and connect it to robot motions,” Hausman said.

Google dubbed RT-2 a vision-language-action (VLA) model. Both Google’s approach and the term VLA quickly became industry standards. But as impressive as RT-2 was, it also had significant shortcomings — shortcomings the industry has been working to remedy over the last three years.

The RT-2 breakthrough kicked off a robotics boom that’s been underway ever since. Big companies in both the US and China have poured resources into robotics. Numerous robot startups have been created, and several have raised hundreds of millions of dollars in venture capital. And the models powering most of these robots are based on the basic architecture Google pioneered back in 2023.

The origins of RT-2, the first VLA model

The robot Google used to train RT-2. (Image courtesy of Google)

Google invented the transformer in 2017 and had been experimenting with large language models ever since. The company had also been working on robotics for many years. So combining LLMs and robots was an obvious research direction.

In March 2023, Google announced PaLM-E, a 12-billion-parameter model that was optimized for robotics (the “E” stood for “embodied”). PaLM-E was a vision-language model (VLM) — meaning an LLM trained to understand images as well as text. It had been trained to generate natural-language robot commands like “move the blue block to the left.”

But PaLM-E couldn’t control a robot directly. Google’s robots didn’t have enough onboard computing power to run a VLM as large as PaLM-E. So PaLM-E ran in the cloud, and it was designed to work with a second, smaller model that would run on the robot. This second model would translate PaLM-E’s English instructions into low-level robot commands.

The RT-2 team’s plan was simple: delete the smaller model and instead train PaLM-E to directly control the robot.1 RT-2 — like PaLM-E — was too big to run directly on a robot. So the team ran the model in a Google data center and had it send commands to the robot over the network.

Like any LLM, RT-2 worked by prompting. Google would send RT-2 a prompt like “What action should the robot take to move coke can to Taylor Swift?” along with an image from the robot’s camera.

RT-2 would respond with a sequence of numbers like “1 128 91 241 5 101 127 217.” The robot would interpret this as a command to move the robot’s gripper to certain x-y-z coordinates (like x=128, y=91, and z=241), rotate the gripper to a certain angle (roll=5, yaw=101, pitch=127), and open (or close) the gripper to a certain position (217).

Then RT-2 would get the same prompt again, but with a fresh image. The model would generate another sequence of numbers representing a new target position for the robot arm. The robot would move its arm another few inches. Then the whole cycle would repeat again. It might take dozens of iterations to complete a task like “move coke can to Taylor Swift.”

To transform PaLM-E into RT-2, Google had to teach the model how to generate low-level robot instructions. That required a different kind of training data.

To collect that data, Google built three test kitchens and purchased 13 robots. Over the course of 17 months, human workers teleoperated the robots as they performed tasks — picking up objects, opening drawers, placing objects in the drawers, and so forth — more than 130,000 times.

Training PaLM-E on this data gave RT-2 surprisingly broad capabilities. Robots could manipulate objects they hadn’t seen before. They could operate in new kitchens. And they could complete tasks on counters that were cluttered with “distractor objects” that weren’t needed for the assigned task.

Five roboticists left Google to co-found Physical Intelligence

Karol Hausman was excited by the RT-2 breakthrough, but he also concluded that Google wasn’t the right place to develop the technology.

“It became clear that the way to accomplish this is to create an organization whose sole purpose is to solve physical intelligence,” Hausman said in March. “It can’t be solved as priority number 20 in another organization.”

So Hausman became the CEO of a startup called Physical Intelligence. He was joined by four other members of Google’s RT-2 team and two others from outside Google.

According to Hausman, the team sought out “investors that are fully aligned with this starting as a research company and not being oriented around short-term revenue.”

“If we do this right, this is going to completely change the world and it’s going to be the most valuable business of all time,” Hausman said. “But you need to have the patience to let us do it the right way.”

There was a lot to do. RT-2 was a big improvement over previous robotic models, but it was still far less capable than the average human. Over the last two years, the Physical Intelligence (PI) team has been working hard to close that gap. The company has been remarkably transparent, publishing at least 10 papers describing their work. For this story, I read all the PI papers I could find — along with 20 more from other companies and academic labs.