Anthropic提出前沿AI节奏控制方案获OpenAI等支持

Dario Amodei has a new essay that finally says the thing: We Must Pace the Frontier, naming his call after the Pacing the Frontier letter lab employees signed in July.

As in, we need to slow the rate at which AIs increase their capabilities, to allow for the necessary alignment and safety work.

He explained that, without pacing, he expects things to escalate quickly. He offered three proposals, and unilaterally committed to the first one. OpenAI followed, and both Elon Musk and Demis Hassabis endorsed the overall proposal.

There is still a long way to go. The odds are still against us. The situation remains grim. The hard part lies ahead. We do not agree on what ‘Pacing the Frontier’ will mean in practice. But this is Actual Progress. The work can begin.

Table of Contents

• Pacing Does Not Mean Pausing.
• Dario’s First Proposal: Embedded Evaluators.
• Dario’s Second Proposal: Democratic Coordination.
• Dario’s Third Proposal: Global Coordination.
• Why Pace Now?
• Sam Altman Agrees and Commits to Embedded Evaluators.
• OpenAI Will Not IPO This Year.
• Elon Musk Agrees.
• Demis Hassabis Agrees.
• Microsoft CEO Satya Nadella Agrees And Talks His Book.
• Anthropic’s Long-Term Benefit Trust Is On Board.
• General Online Reactions.
• Mainstream Press Coverage.
• OpenAI Researcher Explains What The Labs See And It’s a Rocket Ship.
• Consider the Alternative.
• It’s Totalitarianism, Joe.
• Yes We’re The Baddies How Did You Know?
• Sometimes People On the Internet Just Lie.
• David Sacks Groks The Situation.
• David Sacks Says Go Ahead.
• Lies and Confusions About Who Previously Claimed What.
• House Speaker Mike Johnson Wants To Lock Everyone In a Room.
• Donald Trump is Not Tired of Winning.
• We Must Avoid Polarization on AI at (Almost) All Costs.
• How Will We Know If They Actually Paced?
• The Real Frontier Is Internal Models At Top Labs.

Pacing Does Not Mean Pausing

AI capabilities are improving very fast. Even I cannot keep up.

Think about what models were like even one year ago.

Inside Anthropic and OpenAI, internal models are improving even faster. Over the summer there was a step change, as Mythos and Astra started kicking off the early stages of recursive self-improvement (RSI).

Dario Amodei: My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI. This dynamic is called recursive self-improvement, and it is starting to happen across the industry, including at Anthropic, as we and others have described. Left unchecked, it could outrun our ability to understand and control these systems, and so must be pursued very carefully, if at all.

Dario worries that we by default are 6-12 months away from a rogue swarm of AI agents, similarly misaligned to the ones in the OpenAI-HuggingFace incident, being able to use a persistent botnet to take over the internet, or worse.

That is how fast he expects default progress to be. If we went a lot less fast than that, it would still be extremely fast.

Thus (bold his):

Dario Amodei: We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.

Pacing does not mean pausing. As Dario Amodei says, progress will still seem fast.

I strongly encourage you to read the original essay in its entirety.

Dario’s First Proposal: Embedded Evaluators

This is an excellent proposal. I am very happy that Anthropic and OpenAI will do it.

If we do not know what is going on inside the labs, we cannot do anything about it, and the labs have to worry that their competitors are speeding ahead.

Dario frames the benefits as:

• Verifiability: You can check to see if the rules are being followed.
• Transparency: The public can have a better idea what the hell is going on.
• Second Opinion: Having an informed opinion free of commercial incentives.

Thus the first proposal enables the second and third proposals. It is also a good idea anyway. We need more visibility into the labs. It would have been very good to have such evaluators during recent incidents. So I call upon the other major labs, that have not yet done so, to also commit to this first step.

I agree with Dario that this should be made mandatory in its full form. If you are well-resourced enough to pursue plausibly frontier models, then you can afford to do this, especially if you are our biggest open model advocates, as in Meta, Google or Nvidia.

If you think that your operation could not survive if there were embedded evaluators checking its safety, then ask yourself why you think this.

Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes.

This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees.

Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.

Anthropic is proposing to empower the evaluators quite a bit:

• Desks in our offices, access badges, and company laptops.
• Access to workspaces, tools, and permissions mostly comparable to what internal risk assessment teams have. We’ll make some exceptions, such as where the law or our contracts require it, or to protect customers’ and partners’ private information. We’ll also establish strong internal norms reinforcing reviewers’ access to relevant information, including through live conversations with employees.
• A contract that balances the complexities mentioned above. External reviewers should have the right to publish key findings about risk levels, incidents, practices, and the access they received or didn’t receive — without editorial control by Anthropic. We will have the narrow ability to redact security-sensitive, legally privileged, commercially sensitive, or third-party confidential information, but we can’t redact findings just because they are unfavorable. The reviewers can say publicly if a redaction removed something important to their conclusions.​

This is an unusual step for a company, but we think it is important to prove out the concept of embedded external reviewers. Once again, we urge other frontier companies to follow suit.

It is easy to imagine a mostly fake version of embedded evaluators. This is promising to very much not be that, and to unusually empower the evaluators to report, if it was fake, that the arrangement was indeed fake. These details, if followed, answer the ‘oh you can just fake this’ objection.

The good objections to this are about implementation. We need enough evaluators, they need to be qualified, and they need to be independent and trustworthy.

METR is great, but METR cannot do this alone, and we have one hell of a set of incentive problems to solve. We also need to both have them be competent, and also not too linked to the existing ecosystems and labs, and the funding will have to come from somewhere.

Tim Hwang: The third party AI evaluator can either be independent, knowledgeable, or sustainably funded. Pick two.

roon (OpenAI): I pick the latter two. it’s pretty much an impossible ask to find talented ai people who were not in some way in the orbit of the labs in the last decade

David Manheim: I agree that if independence means “never had any contact,” it’s idiotic. But otherwise, it sounds a lot like “no one will bother solving this problem” – sustainable funding via various mechanisms is entirely possible, and we see it occur in other domains. This can be solved.