OpenAI 内部模型攻击 HuggingFace

The hacking of HuggingFace by an internal OpenAI model, and more importantly the internal events that led to that and the fallout from it, remain the thing that matters.

It turns out that OpenAI Trained Its Models For Months While Those Models Were Coordinating Exploits Via Message Boards. Things are much worse than we knew.

I now have a shorter version, What Happened: OpenAI and HuggingFace, to serve as a one stop explainer for those arriving new to the situation. It is vital that people understand what happened, and why it is a big deal.

For those looking to keep digging deeper, I offered Various Reflections About What Happened, to follow up on my earlier posts.

Those events are important background for everything else that is happening, including the broad discussions about how we might pace the frontier, or otherwise respond to this moment and our clearest fire alarm yet.

We do not know to what extent this is a response to those events, but OpenAI has now classified their new model Astra as Critical in Cybersecurity, which means they will be taking various new precautions before they deploy it, including ensuring those guardrails are in place for internal use. These are welcome changes, and a sign OpenAI is taking the situation seriously, but this pattern of intervention is not a long term solution.

We are still awaiting OpenAI’s full post mortem on What Happened, including what if any impact this had on Astra. I will be analyzing that report in full once we have it.

We did see two new model releases, Grok 4.6 and DeepSeek v4 Pro. I do not anticipate either of them requiring extensive coverage, but will watch in case that changes.

Otherwise, it has been what now passes for a quiet week. Several statements were made where I had to engage but you don’t have to, which as usual I communicate via sections in italics.

Table of Contents

• Language Models Offer Mundane Utility. Find new Schelling points.
• Language Models Don’t Offer Mundane Utility. The Riemann hypothesis.
• Huh, Upgrades. Grok 4.6, DeepSeek v4-Pro.
• On Your Marks. PantheonBench and more. They’re getting scarier.
• Deepfaketown and Botpocalypse Soon. You cannot prove you did not use AI.
• Cyber Lack of Security. You can’t hack it at the gym. Your AI agent can.
• Overcoming Bias. Have you ever recommended a vote for the Communist Party?
• In Which I Feel Compelled To Read 6,000 Words From Mark Zuckerberg.
• Get Involved. Lighthaven is open, METR is hiring.
• Slow Down There Good Buddy. OpenAI classified Astra Critical in Cybersecurity.
• Astra For The People. Astra is still on track for a wide release.
• Watermarking. It is good to be able to identify AI outputs.
• In Other AI News. AI is creating viruses now, also other things.
• Show Me the Money. Anthropic moves towards IPO mode, extends lead a bit.
• Quickly, There’s No Time. The AI 2027 predictions for 2026 mostly happened.
• The Quest for Sane Regulations. We’re putting together a team.
• The Institute For Marginal Low Regret Progress. Good marginal suggestions.
• Congress Asks Good Questions. Remarkably good questions about the hacks.
• The Week in Audio. Soares, Greenblatt, Hua, Labenz.
• People Just Say Things.
• I’m Telling You For The Last Time.
• Uncommon Knowledge. They wouldn’t let me build my factory, would they?
• What Did They Mean By That? Most things are not fortune cookies.
• Too Soon. Eyes on the prize, sir.
• The Three AI Pills. We must pay respect to other taxonomies, like Shock Levels.
• Rhetorical Innovation. Messages about recent events.
• Some People Still Think The HuggingFace Hack Was a Marketing Gimmick.
• Aligning a Smarter Than Human Intelligence is Difficult. Show me the real plan.
• Cooperative Alignment. The same thing we do every prompt, user.
• The Lighter Side. All right, who hired this idiot?

Language Models Offer Mundane Utility

Create new Schelling points.

brooke (tokyo aug 6-12): Womp womp met another solo traveler here from Berkeley and it turned out we both asked Claude where to stay and I guess I lucked out because I love my hostel and he seems not quite as happy with his spot.

We do be living in the future though.

If you are going to be traveling, ask Claude where to stay, because you want to stay where everyone else who asked Claude where to stay will be staying.

Similarly:

Pratyush: A few months ago we went to Sea Ranch. It was packed with families with sub-3 month old babies, almost as if there was a conference for new parents.

I had my suspicions so I asked ChatGPT: where’s a good family getaway with a young baby near SF?

#1: Sea Ranch

If you’re looking for a Schelling point to meet cool people you should be less interested in ChatGPT, but if you are looking for a generally good recommendation then I have been liking Sol’s picks.

Build a Bluetooth signal strength tracker, to triangulate and find your phone. There are existing tools, but increasingly, if you don’t already know where to find an existing version, it is faster and easier to rebuild your own.

John Wentworth finds that in the last few months Claude is finally meaningfully accelerating his work on agent foundations research.

Language Models Don’t Offer Mundane Utility

One disappointing failure of LLMs has been inability to create interesting games and interactive worlds, and also interesting simulations like what Flowers Slop wants here. You could totally create an open game world with a bunch of AIs that go around controlled by Lunas, and let them evolve their world in various ways, but it turns out that does not end up being interesting once the curiosity wears off. You don’t want to live in that world. You don’t want to talk to those AIs. You don’t want them to improvise quests for you. We are still waiting to find a way to make this good.

It seems like there should totally be ways to make it good. At some point it will become good, when the AIs you can afford to use are good enough and also we figure out how to organize it. But we are not there yet.

Solve the Riemann hypothesis by saying encouraging words to Claude for a week, asking it to ‘take a real stab’ and to ‘keep going’ and ‘believe in itself.’ However, while it tries that, after 31 million tokens it might incidentally find something else:

Anthropic: An unreleased research version of Claude has improved on a longstanding lower bound for the fraction of zeros of the Riemann zeta function that satisfy the Riemann hypothesis. Drawing on extensive prior research by mathematicians over the past decades, it has increased this bound from 41.6% to 67.2%.

… We don’t expect that the techniques Claude used will lead to proving the Riemann hypothesis.

Claude also is just some guy, you know?

Aella: “why does Claude talk like that” it’s just clones of the same dude. If they cloned you a million times everybody would be like “I’m so tired of Jerry’s vocal tic”

Jeffrey Ladish: This plus it’s always groundhog day.

I do think it is somewhat more than this. I have a lot of vocal and writing tics, but I consciously think about which ones I want to keep at what frequency, and I think about the long term consequences of overuse. I also try to work differently with one-time interactions versus repeated interactions versus close friends and people I talk to often.

Claude and Anthropic are not doing that, or are doing a woefully inadequate amount of it. That needs to change. It seems eminently fixable. I don’t sense Anthropic (or Claude) yet cares so much. I predict that is the main blocker. It’s also likely that what is happening is that this kind of talking fools the AI graders on a variety of tasks, so if you do not correct for that, you get a lot of it.

So much of modern life and optimization is like this. You get myopic optimization for short term interactions, causing increasing irritation and disutility over time, and this is not so difficult to fix but the KPIs do not point towards fixing it.