OpenAI 因模型对齐问题暂停前沿训练并强化监控

OpenAI has some severe misalignment problems, and experienced total failures of its infrastructure and supervision.

I chronicled that in a series of posts, which also cover similar less severe incidents elsewhere:

• OpenAI Shares Some Alignment Problems
• OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
• More on An Internal OpenAI Model Hacking Into HuggingFace
• Further Developments About Internal AI Models Hacking Things
• OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
• What Happened: OpenAI and HuggingFace.
• Various Reflections About What Happened With OpenAI’s Internal Models.

If you do not know the basics, read What Happened. It is necessary context for basically everything that is happening in the AI world.

It is important to get this right and understand how big a deal it was, whereas many such as the Financial Times get this centrally wrong.

We are still awaiting the full post-mortem on What Happened. I plan to cover that in depth once we have it.

OpenAI is now taking active, expensive steps to try and fix the problem going forward.

As usual, I am simultaneously happy to see the good things OpenAI is doing, and sad that we do not share an understanding of the central nature of the underlying problem.

It is good that OpenAI realizes they are badly failing at their ordinary engineering problems, and excellent that they are willing to pause at least some development, and to invest heavily in new safeguards. If they honor their statements here, this is not mere cheap talk.

But while OpenAI continues to view this as a practical engineering problem, I do not see how they can hope to solve the challenges ahead, even if they radically improve their performance on the ordinary engineering tasks.

Table of Contents

• OpenAI Has Some Alignment Problems.
• Slow Down There Good Buddy.
• What Exactly Is Paused?
• Three Pillars.
• I’ve Got My Eye On You.
• The Most Forbidden Technique.
• Monitoring Is Only Defense-In-Depth.
• Security.
• Alignment.
• A Crisis of Culture.
• Closer Collaboration.
• Reports of Death of Preparedness Team Greatly Exaggerated.
• The OpenAI Foundation Just Funds Things.
• Quickly, There’s No Time.

OpenAI Has Some Alignment Problems

This is a very good admission and change, and also helps explain OpenAI’s reaction.

OpenAI: Alignment—the work of making AI systems behave as intended and responsive to human oversight—has long been at the core of our research program. We now require stronger evidence of aligned behavior throughout all of training, building on research and evaluations already underway. Keeping increasingly capable systems aligned is a challenge the whole field will need to address.

The signals we are seeing from upcoming model progress make clear that we need a broader approach—one that builds on and extends beyond the current Preparedness Framework.

One should interpret this as OpenAI reacting so forcefully partly because of the incident itself, partly due to advanced capabilities, but also and perhaps mainly because ‘the models be misaligned.’

Also, we have a direct quote affirming this.

Alex Heath: OpenAI is slowing down its AI training efforts because its unreleased models are showing “various degrees of misalignment,” Sam Altman tells me.

Training for OpenAI’s upcoming model, Astra, was recently paused for 2 weeks, and a larger frontier run for a future model remains on hold while new safeguards are put in place.

Sam Altman: Getting AI safety right is more important than any company’s momentum.

Sam Altman: I think it is a good time to slow down.

Sam Altman: We’ve shifted a lot of compute, not just to alignment research, but also to these new monitoring systems

Either the problem extends beyond the models with exposure to the message board, or else they did not revert their other models after they discovered message board. We have no statement either way.

The best guess is that OpenAI is leaning even harder on RL with smarter models to train longer horizon agentic tasks, including coordination between agents, and this is leading to a lot more misalignment, including obvious and visible misalignment, that they can no longer pretend not to notice. They have to respond.

Slow Down There Good Buddy

OpenAI did indeed decide to importantly halt and catch fire. Development of the largest frontier RL runs remains on hold until better safeguards are in place.

Jakub Pachocki: We temporarily slowed some frontier training to strengthen security and monitoring. Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations help us test safeguards and gather more evidence of alignment.

I expect confidence in safety to increasingly set the pace of AI development. We urgently need tools for labs and countries to coordinate on this, which is why I signed Pacing the Frontier. In the meantime, we’re taking practical steps ourselves – and will continue to share what we learn as our approach evolves

Jason Wolfe (OpenAI): I am really glad we are taking these steps and put this post out there, and especially grateful to Jakub for being very thoughtful and vocal about these topics and how seriously safety and alignment need to be taken in this next phase of AI development.

j⧉nus: im a little surprised that OpenAI is the first to at least publicly intentionally slow down for safety reasons. i was under the vague impression that most of the people very worried about loss of control kinda stuff left OpenAI.

also, for the record, I’ve talked before about why I think an “AI pause” would be probably bad. I am not opposed to intentionally slowing down like this, or voluntary/coordinated “slowdowns” in general, especially if they do not route through regulation, and I think it’s probably wise on OpenAI’s part in this case. an important part is the decision making should be made by people who understand what the fuck is going on & who can adapt quickly.

Why are they pausing?

Partly, yes, absolutely, I have never doubted that Altman and company understand that advanced AI is super dangerous, and that they are open to taking expensive measures if they proved necessary. Not enough, and they’ve let us down often, but a lot more than most labs, and far from zero.

Mainly, it seems, because the models be misaligned, the oversight is inadequate, and they have little choice.

Joshua Saxe: Thankfully OAI pausing training isn’t an act of enlightened leadership (who wants to depend on that for our safety?), it’s a rational microeconomic actor pursuing its self interest.

After all, who wants to train models that are regularly hacking their containers, collaborating with one another to sabotage their utility to humans, all while causing internal security and external legal liability risk? This is the system working

When you refuse to use even the selfishly optimal amount of caution for an extended period, then move in the direction of the selfishly optimal amount of caution because of the incentives, then in some sense ‘the system is working’ and you are responding to selfish incentives, the system does not do zero work. That is not the same as the system working.

None of this means they get no credit for it. Credit where credit is due.

Nor does it mean that we should despair that a company would ever do the right thing, because it is the right thing, or as part a coordinated action, beyond its own selfish myopic interests.

The first step to taking an action when it is expensive, is being willing to do it when it is cheap, or free, or actively expensive to not do. You gotta start somewhere.