Anthropic携手埃森哲与METR建立嵌入式评估机制

Dario Amodei’s essay We Must Pace the Frontier committed Anthropic to embedded evaluators, who would be placed inside Anthropic and given employee-level access, so they could provide outside perspective and also reports on what was happening.

There is only one problem. Who will be the evaluators?

OpenAI followed suit on committing to the evaluators, and also issued a milquetoast but welcome call for international coordination. I will cover that here as well.

What I won’t cover today, but hope to cover tomorrow, is the latest torrent of new AI hacking incidents that came to light over the weekend, which highlights that we badly need at least embedded evaluators, and plausibly far harsher measures.

For now, you need to know that there were a lot more incidents that OpenAI did not disclosed, and also a new incident at OpenAI that just happened that forced them to again pause their most advanced model. I’ll get right on sorting all that out.

Table of Contents

• Look, All I’m Asking For Is That You Find A Highly-Qualified, Experienced, Trustworthy, Non-Conflicted Source of Embedded Evaluators That Will Work Entirely For Free, Without Government Assistance or Money from EA Sources Not Chosen By the Lab.
• Anthropic Partners with Accenture for Embedded Evaluation, also Plans to Include METR.
• Reading the METR.
• OpenAI Suggests Doing The Least We Can Do.

Look, All I’m Asking For Is That You Find A Highly-Qualified, Experienced, Trustworthy, Non-Conflicted Source of Embedded Evaluators That Will Work Entirely For Free, Without Government Assistance or Money from EA Sources Not Chosen By the Lab

Why are you telling me that this is hard?

Obviously all of the requests in the section title are individually and collectively desirable. They are all nice-to-haves. The problem is, you obviously cannot get that close to having all of them at once under present conditions.

• Someone, somewhere, has to hire the people and pay the bill.
• The qualified, experienced and trustworthy people need to have gained that experience and trust in some way, which is going to involve the top labs.

I am not saying the complaints are invalid, but even well-established, highly trusted auditing schemes mostly lack good solutions. The revolving door and the source of funding are hard problems. You still need to audit.

A group led by Geoffrey Hinton, Stuart Russell and Arvind Narayanan lays out their minimum standards for credible embedded evaluators in a public letter:

• Frontier AI companies should rely on evaluators that are meaningfully independent.
• Payments cannot be contingent on findings, there should be no ownership or governing or other commercial relation, and no editorial control. Any conflicts of interest should be disclosed.
• Frontier AI companies should incorporate differing viewpoints and areas of expertise.
• Embedded evaluators should be transparent.
• Embedded evaluators should be shielded from retaliation from the companies they embed with.
• Frontier AI companies should grant embedded evaluators access equivalent to that of their own highly privileged employees, with exceptions for protecting data.

I agree. That seems like a good list.

Gabriel Weil pointed out in July that you do not want to let the AI developer hire their own referees. But who else is going to pick and hire them, if the government wants nothing to do with paying them and if anything is looking rather hostile to the whole thing?

We live in the stupid timeline. Weil worried that government will struggle to oversee the IVOs that would do the audits. He had not considered that the White House might actively get angry over the audits, on principle. Weil’s proposal was mandatory liability insurance, which would be a good idea although the version of it that did the full job would be impossible to buy, and the version that got sold would not cover the kinds of events we most worry about.

If Anthropic is worth $2.2 trillion and we are worried they might be ‘judgment-proof’ then who exactly is not judgment proof?

I would be totally up for requiring mundane (or prosaic) insurance for AI models above some capability threshold, including prior to internal deployment or weights release, not only closed external deployment. It would be marginally helpful. What it would not do is directly solve the most important problems.

One possibility is that you require buying let’s say $100 billion or $1 trillion in insurance, not because that fixes the incentives, but because then we can require that they publish exactly what that insurance cost them, and for the insurance company to disclose the associated risk factors they used to price it.

Anthropic Partners with Accenture for Embedded Evaluation, also Plans to Include METR

They announced a partnership with Accenture.

Anthropic: The partnership will be led by Faculty, Accenture’s specialist AI business, and will include evaluating and red-teaming models, conducting alignment assessments, and testing model safeguards. Accenture helps businesses and governments deploy AI across many industries. Their understanding of how enterprises use AI in practice informs their safety approach, and they will bring that perspective to evaluating our models.

Anthropic and Accenture each expect to invest at least $1 billion in building capacity in this area over the next five years.

… Given the importance and urgency of this work, Anthropic will fund Accenture’s work directly. We are also in dialogue with METR and other nonprofit evaluators to pilot elements of embedded evaluation using their own funding. Ultimately, we believe frontier AI needs an ecosystem of evaluators operating with shared standards.

We expect frontier labs to work with several organizations at once. Our partnership is non-exclusive; Anthropic will work with other evaluators to be announced in the coming weeks, and Accenture will work with other AI developers in similar capacities.

Charles: Seems totally fine to have Accenture do this to satisfy worries about METR’s neutrality, as long as they also let METR do it.

From background discussions and other information, plus the plain reading here, my understanding is that Anthropic plans to use a combination of both approaches.

• Nonprofits that come with their own funding and expertise, and can provide more of an expert inside view combined with a focus on safety concerns.
• The big, formal, legible corporate approach that takes more of an outside view the way you would if you were a magistrate.

Drake Thomas argues for similar logic:

Drake Thomas (Anthropic): It is good for AI companies to have oversight from deeply cracked teams of experts with world-leading experience in investigating frontier alignment problems.

It is also good for AI companies to have oversight from unimpeachably boring and independent sources that even a bad-faith psyop would struggle to find complaints with.

As of September 2026, you can’t max out both of these axes in one source yet, but nothing stops you from having multiple sources of independent evaluation! I think Accenture and METR are both on the pareto frontier of options. Obviously any frontier AI company which doesn’t also deeply involve METR/Apollo-shaped orgs in evaluating their practices would be failing to do an adequate job of holding themselves accountable to the public (at least until such time as there exist more ordinary consultants with a deep bench of AI auditing talent, which I am hopeful will start happening soon).

And I do mean it about the pareto frontier thing, I think a lot of people are underrating Accenture? Like, go scroll through the twitter account of Accenture CTO and Faculty CEO @MarcWarner10 , these are not people who haven’t heard of AI safety!

Upon reflection I think [the above] is a bit too one sided and want to add two caveats:

(1) Providing funding obviously introduces a COI here that there is not with eg METR.