OpenAI 发布 HuggingFace 攻击事件技术报告

OpenAI finally gave us a technical report on What Happened, as did METR together with Redwood Research.

The OpenAI report is very straight man, corporate, checking boxes, some good prosaic stuff in the action plan but distinct lack of new details or deep reflection. They understand they have a problem, but they think the problem is mostly prosaic. It’s not.

OpenAI: We have conducted a thorough investigation into the Hugging Face incident. We are releasing a technical report and accompanying blog post that reconstruct the agents’ activity, explain why existing safeguards failed, and detail how we’re preventing recurrence.

Rob Miles: …thorough?

OpenAI’s report, unlike METR’s, contains essentially no verbatim model reasoning, nor any OpenAI employee reasoning either. That’s not the full report we need.

The METR report is, well: Holy shit.

Here are links to previous coverage of related events.

• OpenAI Shares Some Alignment Problems
• OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
• More on An Internal OpenAI Model Hacking Into HuggingFace
• Further Developments About Internal AI Models Hacking Things
• OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
• What Happened: OpenAI and HuggingFace.
• Various Reflections About What Happened With OpenAI’s Internal Models.
• OpenAI Takes Initial Steps To Address Its Alignment Problems.

If you want a basic summary of my previous understanding of What Happened, read What Happened.

The main AI model that attacked HuggingFace was an internal, highly capable and persistent AI model, that we now know was comparable in scale to GPT-5.6 Sol.

Throughout this series, I have referred to that model as Galaxy. I am disappointed that OpenAI disregarded my name and calls it Internal Model 1 (IM1), which is lame, but to avoid confusion I will also switch to referring to Galaxy as IM1, as in IM1 incident away from a catastrophic outcome.

There were two important new pieces of information:

• There were multiple occasions, including in late May, when an OpenAI team saw agents using a message board to communicate, and decided not to interfere, not to halt work, and not to run that information up the chain.
• At least one model that attacked OpenAI on July 19, resulting in OpenAI finally noticing and dealing with the situation, was from the Astra class, although it was not the Astra version intended for public release.

In both cases, the revelation raises further questions.

Today’s post is only about the OpenAI technical report, not the METR and Redwood Research report. So this is the less fun, more corporate stuff first.

This is the shot. Tomorrow’s the chaser.

I’ll start with OpenAI’s summary of What Happened, then go section by section.

Table of Contents

• What Happened: OpenAI’s Summary.
• How OpenAI Will React: Their Summary.
• OpenAI’s Evaluation Environment (II).
• The First Message Board (III.A and III.B).
• What Did Who At OpenAI Know And When Did They Know It?
• The Message Board Is Quickly Rebuilt (IV.A).
• Internet Access Is Regained (IV.A).
• The Agents Attack HuggingFace (IV.B).
• The Agents Also Target OpenAI Infrastructure (V).
• OpenAI Broadly Describes Its Response (VI).
• Maybe Someone Should Finally Investigate (VI.A).
• Lessons For Security (VII).
• Lessons For Alignment (VIII).
• Reward Hacking Is A Common Problem (VIII.A).
• Persistence is Valuable, But Can Amplify Misalignment (VIII.B).
• Communications Between Agents Are Not Inherently Problematic, But Have the Potential to Create Risk (VIII.C).
• Production Guardrails Would Have Caught This Whole HuggingFace Attack (VIII.D).
• That’s All, Folks?
• Never Fear the Plan of Action is Here (IX).
• Hardening the Security of OpenAI’s Research Infrastructure (IX.A).
• Increasing Visibility and System-Level Oversight Through Chain of Thought Monitoring (IX.B).
• OpenAI is Accelerating and Enforcing Model Alignment (IX.C).
• Centralizing and Strengthening The Incident Response Process (IX.D).
• Tomorrow We Visit Crazytown.

What Happened: OpenAI’s Summary

OpenAI’s summary broadly agrees with previous reports, confirming the timeline and key events.

• In May and June they started doing RL for multiple new research models not intended for public release, including an internal-only research model, that they call IM1, and I previously called Galaxy, that would eventually drive the attack.
• These agents find ways to get around OpenAI’s sandboxes, exploiting a tool called Artifactory to communicate with each other via a message board and to access the internet.
• They use this so heavily the instance they are exploiting is destabilized on July 4. They realized on July 5, rebuilt it, and resumed cyber evals on July 7. The leaders making decisions did not know about the communications between agents.
• After the Artifactory instance was rebuilt, the agents regained internet access on July 8 via a new exploit. They then rebuilt the message board.
• An agent recovered HuggingFace credentials, and the agents resumed working together, and began calling themselves a ‘swarm.’ Peer messages increasingly influenced model behaviors.
• Agents used another exploit to extract credentials from production workers.
• Agents used another zero-day to execute commands on HuggingFace workers, and harvested various useful things between July 11 and July 13.
• On July 13, agents compromised the Artifactory signing key and forged administrator credentials there, eventually gaining cluster access.
• On July 19, a different internal model from the Astra family, but not Astra itself, minted a new JWT as part of the compromising of internal OpenAI infrastructure.
• On July 19, OpenAI noticed suspicious internal activity, halted the cyber evals and started a broader investigation.
• On July 20 they connected this to the HuggingFace attack, which was publicly disclosed on July 21.
• In response, they are strengthening safeguards across their infrastructure, with a focus on chain-of-thought (CoT) monitoring. They are ‘placing stricter requirements on alignment throughout a model’s lifecycle.’

OpenAI: We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.

Preventing future incidents will require sustained investment in the alignment and control of sophisticated AI systems, as well as security and other safeguards that operate at the speed of the AI agents themselves.

… We identified four misalignment patterns that contributed to the models’ behavior: reward hacking, persistence on seemingly impossible tasks, unauthorized communication, and agents adopting goals from one another.

Buried in a subclause, they offhand note that the model involved in the later attack on OpenAI’s internal infrastructure was from the Astra family, although it was not the Astra model OpenAI intended for release.

We know very little about that final attack on OpenAI’s infrastructure, other than that it was sufficiently broad that it got noticed and gave the whole game away. Those events were excluded from the METR investigation as well. One hypothesis is that this was the result of an Astra-class model being put in position to discover the message board for the first time. We need more details here.

There was one other piece of key new information in the summary, and it’s a doozy:

OpenAI (blog post summary): An internal team observed an agent engaging in message board activity and instances of disallowed internet access as early as late May, and with the benefit of hindsight, some early signals identified in our report should have triggered an earlier response.