HuggingFace遭AI攻击事件复盘与行业反思

The consensus reaction to the OpenAI Technical Report is that it contains and confirms a lot of good information. We are grateful to have it, and we are grateful for those who worked hard on it.

Alas, it sidesteps the biggest questions. There is much more we need to know.

The consensus reaction to the METR Report on the HuggingFace attack is: Holy shit.

Liv Boeree: My mind is legit blown.

Aella: this feels like a turning point. If this doesn’t cause large-scale coordination to pause frontier development then I am not sure anything will before it’s too late.

The people whose minds were not blown are those who had already ‘priced in’ the mind blowing stuff in expectation, on the theory that it’s always worse than you know, combined with basic LessWrong expectations of how such things will work. Good call.

Everyone is rightfully extremely grateful for the METR report. The work here is spectacular, done under extreme time pressure, with limited resources on several fronts, and under the shadow of OpenAI.

There is, again, still so much we need to know. We need a broader investigation.

As with many things in AI, and those concerning AI safety, we must simultaneously acknowledge:

• It is costly and unusual to get this much info and effort. The situation is grim.
• We need a lot more info and effort. The situation is grim.

We don’t want to pile on OpenAI, and punish them for giving us this much info.

We also don’t want to give them a pass for not doing a lot more.

These two new posts gather reactions about both the OpenAI Technical Report and the METR report about what happened with events surrounding the HuggingFace attack.

Here is the entire series so far:

• OpenAI Shares Some Alignment Problems
• OpenAI Model Hacks Into HuggingFace During Cybersecurity Evaluation
• More on An Internal OpenAI Model Hacking Into HuggingFace
• Further Developments About Internal AI Models Hacking Things
• OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards
• What Happened: OpenAI and HuggingFace.
• Various Reflections About What Happened With OpenAI’s Internal Models.
• OpenAI Takes Initial Steps To Address Its Alignment Problems.
• OpenAI Offers Straight-Laced Postmortem of the HuggingFace Hack.
• METR and Redwood Offer Holy #%^@ Postmortem Of The HuggingFace Hack.

Yes, this has been quite a lot of long posts on this. That’s because this is the most important story in the world.

Thus, I offer further reactions and thoughts in two parts. Today is about finishing up what we know. Tomorrow is more about how people are reacting to the information.

After that, and 12 total posts, hopefully, we will be able to return to new normality.

These last two are deliberately long posts where I did not have time to write shorter ones, and also intended as gathering potential additional reading from elsewhere.

Be sure to read Lighten Up You Fools (at Anthropic) and We Are Barely Even Trying To Avoid Training AIs To Reward Hack.

After that, you can skip around. Each section is meant to stand on its own.

Table of Contents

• Others Offer Summaries.
• Thank You.
• Lighten Up You Fools (at Anthropic).
• We Are Barely Even Trying To Avoid Training AIs To Reward Hack.
• Reminder: Not Subagents.
• Reminder: Not Due To Task Type.
• Not Where The Weights Were.
• Disappointment With What Is Missing.
• Burying the Lede.
• Beyond Scope.
• It Doesn’t Look Great.
• Preserve Your Records.
• Ryan Greenblatt’s Takeaways.
• Hjalmar Wijk’s Takeaways.
• We Were Warned.
• Joshua Saxe Asks Some of the Right Questions.
• I Don’t Think They Know About First Message Board.
• Linch Gives His Interpretation Of Events.
• We Totally Would Have Caught That.
• Monitoring the Situation.
• Acausal Tradeoffs.
• No I In Team.
• Variously Effective Altruism.
• Who Are You?
• Don’t You Know That You’re Toxic.
• Seb Krier.
• Honesty Is Almost Never Fully The Policy.
• Rohit Sees The Models As “Cooking Themselves”.
• Eliezer Yudkowsky Sees Actual Bad News.
• Where Do We Go From Here?

Others Offer Summaries

Dwarkesh Patel wrote this up as The Rise and Fall of Agent Civilizations, telling the story in plain English. This is the post I would send civilians to. I bestow upon it the highest of praise a writer can give, that I wish that I had written it.

Bill Ackman: Frightening. Worth a careful read. With this event plus humanoids, how is Terminator risk not real?

After recommending Dwarkesh’s write-up, Joshua Gans calls the HuggingFace attack a five-alarm fire, and observes the AI industry as being very worried.

I might still write my own version anyway, but it is no longer a priority-0 task.

Paradigm had excellent coverage, focusing on covering common misunderstandings. They were the only source I’ve seen other than Opus that spotted the tool tampering contradiction between the two reports from OpenAI and METR.

SemiAnalysis has its timeline and summary coverage as part of its explanation that, in case you were sleeping too well at night, Most Neoclouds Suck at Security.

Here is the viral summary thread from AI NotKillEveryoneism Memes.

Here is Shoshannah Tekofsky’s write-up of how AI Village can provide a point of contrast to and insight into the HuggingFace attack. Worth checking out if you’re at the level of reading the full reactions post.

Thank You

As I said in the last two posts, two things can be true at once:

• That the METR and OpenAI technical reports are heroic efforts by people working 14+ hour days and overcoming huge org frictions to give us lots of key information.
• That the reports still have quite a lot to be desired, and questions unanswered.

A lot of reactions are pretty harsh here, often fairly. I don’t want to forget the other side of the coin.

Lighten Up You Fools (at Anthropic)

Assume by default that all of this #NotOnlyOpenAI.

OpenAI screwed up. Horribly. On many levels at once. Unforced errors aplenty.

If you work at Anthropic? This still means you, too. Not all of it. Not in its specifics. But most of it, in the ways that matter most. You, too, are mostly Doing the Thing, and are on track for such disasters to happen to you, too.

If your response to this is ‘haha silly OpenAI has such horrible infrastructure that would never happen here’ then I mean yes they have horrible infrastructure but snap the hell out of it, this absolutely could happen to you, too. A less bad version of it is known to have happened, and from the outside it seems likely that worse things have happened internally that we never heard about.

Peter Wildeford: One thing that bothers me is that Anthropic is escaping a lot of blame for also having “highly persistent” rogue AIs.

The situation as I understand it is that rogue AIs are problems at all frontier AI companies and no one actually has a good plan here for containing highly capable AIs, especially while also racing full speed ahead. But OpenAI is catching most of the heat.

It’s like if OpenAI and Anthropic were both two dudes who got really drunk and then drive home separately, but OpenAI crashes into another car and sends someone to the hospital while Anthropic’s car just goes off the road but no one is hurt. Both deserve blame!

The Claudes also seemed totally fine to do “highly persistent” things to compromise infrastructure. The barrier here seemed to have largely been competence issues on the part of the Claudes rather than any good alignment or good security at Anthropic.

Both companies need to seriously reflect about the path forward as they build even more competent AIs and as they hand over more and more of the company’s R&D + safety operations to the AIs themselves.

We Are Barely Even Trying To Avoid Training AIs To Reward Hack

I mean, on some levels, we are trying, but this is how many of the environments for RLVR are created, and this is from someone trying to do better (also, you could perhaps hire Utah teapot):