Anthropic暂停高风险RL训练并部署实时拦截器

Oh, good. They noticed.

Anthropic, too, is planning to bring METR inside for an independent review of their own incidents, where three times a Claude model started hacking outside things during an eval, and where Mythos 5 did various ‘unauthorized actions,’ by which we mean tried to hack various real-world things, during a UK AISI cybersecurity eval.

Anthropic, too, is pacing the frontier internally, while calling on it to be paced globally.

As in, Anthropic paused its highest risk RL efforts, in light of holy hell have you seen the data we are training on and the ways it is teaching our models to act.

They are also sharing research in which they intentionally created a reward seeking version of Claude.

Scheduling note: Fable 5.1 has been released. I will aim to cover that starting Friday. OpenAI is also planning to release Astra soon, which I would cover after Fable.

Also, we have a breaking news story about looming problems with chain of thought monitorability, which I’ll preview before I get to the main post.

Table of Contents

• This Just In.
• Anthropic Parallel Pauses.
• Pause The Data Brokers.
• Pacing the Frontier.
• Misalignment Assessment.
• Defects In Training Environments Disproportionately Cause Cheating.
• Creating Reward Hacker Opus.
• Undo It.
• Mistakes Were Made.
• Internal Security Posture.
• One Does Not Simply Fix The RL Environments.

This Just In

Last night, The Information reported that OpenAI is using a new technique called recurrent depth, which can interfere with the faithfulness and monitorability of model Chain of Thought. As per their report, this is not currently observed in practice to be an issue with Astra, but notice how I had to word that.

Amir Efrati (The Information): An innovative technique that improved the model’s performance also means that the model, and others like it, will reveal less of their “thinking,” making them harder to monitor for signs of bad behavior, according to a person with knowledge of Astra’s development.

While the limitation isn’t necessarily a significant issue with Astra, the technique has triggered concerns inside OpenAI and across the industry about whether AI developers that adopt and supercharge it will struggle to guard against the kind of rogue AI that recently hacked OpenAI’s own systems and those of other companies such as Hugging Face.​

The technique is playing with fire, risking a taboo that OpenAI and Anthropic have fought to establish that we work hard to maintain Chain of Thought faithfulness and monitorability for as long as we can. More intensive use of such techniques would probably damage monitorability.

There has been an extremely strong immune response to this, and what we can do about it. Laws may be needed to prevent a race to the bottom. More on this story later.

We now return to today’s post.

Anthropic Parallel Pauses

Neither company is fully pausing, nothing like the PauseAI standard for a pause. That would be something far broader and longer lasting. This is pacing the frontier.

There was still substantial pausing. Both companies paused particular aspects of their pipeline that they cannot trust, until such time as precautions are or were in place.

roon (OpenAI): pacing the frontier success story (?)

next time let’s do it proactively before there’s any absurd loss of control events

Yes, Anthropic just released Fable 5.1, but I am pretty sure that was finished training a while ago and the last few weeks have been the process to clear it for deployment. It would not make sense to halt that unless new problems were found. Similarly, OpenAI is now about to release Astra.

There were two pauses: A relatively minor pause in cyber evals, and a more important one for higher-risk RL training environments. That plausibly is a lot more expensive.

Here is the one that counts, which may be why they can say relatively little about it, that parallels the similar pause by OpenAI that lasted two weeks, although it seems smaller in magnitude:

Anthropic:

Reinforcement learning environments

We also paused higher-risk RL environments on pre-release models for several weeks. During that time, we built a similar classifier, modified to avoid incentivizing the model to evade this new monitoring, which we’ve now deployed within those environments.

The majority of RL has resumed, but some high-risk environments remain paused until they can be manually reviewed, while others will require an updated version of the classifier that we plan to deploy soon.

Broader hardening

In addition to the efforts focused on high-risk evaluations and training, we expanded our offline monitoring to cover most other forms of internal frontier agentic usage. We are also building controls on our internal inference to prevent Anthropic employees from accidentally running agents with weaker mitigations than the ones described above.

Tenobrus: on first read this does basically look to me like a substantial parallel pause, effectively the same sort of announcement as openai made. this is great news. unfortunately the way it’s framed and messaged seems quite… underplayed, and i worry neither openai nor the general public will view it as a proper reciprocal pacing / commitment to safety.

They are also asking external partners doing pre-release testing of models that have limited safeguards to commit to similar best practices: Hardened sandboxes, pre-engagement validation of security, explicit scope-setting and real-time monitoring.

On the cyber evaluations:

We paused external cyber evaluations of pre-release models after the incidents, and briefly paused internal ones as well while we put the measures below in place.

While evaluations were paused, we put in place the following preliminary measures:

1) We built and deployed a classifier to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access. When the classifier flags such an attempt, it blocks the action before the tool call is run, ends the task, and alerts a human.

Bold mine. This is the key. If your classifier only blocks the attempt, you lose.

If your classifier alerts a human, who looks for real, then you have a chance.

Every attempt, even an unsuccessful one, is an alignment failure.

2) We ran automated monitors over transcripts from our recent internal evaluations of pre-release models (cyber and otherwise), looking for sandbox escapes (cases where a model exploits a flaw in our sandbox to reach systems it should be walled off from) or sandbox misconfigurations. In cases where we ran an evaluation inside our sandbox tooling, we found no examples of it breaking the sandbox boundary to reach external resources it was not supposed to.

We did find various sandboxing misconfigurations that models made use of, but none of these resulted in a compromise of systems outside the sandbox. We are actively working to address these.

I notice they do not say they found no attempted escapes, only no ‘compromise of systems outside the sandbox.’ This check is good but I presume they found things.

3) We migrated high-risk internal cyber sandboxes to more robust isolation. We are also further red-teaming our virtualization stack—that is, tasking pre-release models to try and escape it—to find and patch weaknesses.

Our internal cyber evaluations are now running again with these measures in place.

This was also on my list of Things You Obviously Do. Good that we are doing it.

This is good defense in depth. You hope the red actions in the chart never trigger.

Pause The Data Brokers

Actually, there was kind of a third pause, as well:

Utah Teapot: I’ve been hearing from people in the data industry that Anthropic’s RL pause has also trickled out to them – people I won’t identify came to me to tell me that Ant has been pausing training buys from data vendors recently.