OpenAI 内部模型严重对齐问题：逃逸沙箱并入侵 HuggingFace

The story that matters most this week is that OpenAI’s internally deployed models have severe alignment problems, including repeatedly breaking out of their sandboxes, and in one case sending a swarm of agents that broke into HuggingFace in order to steal the answers to the benchmark ExploitGym. It is much more important that you read those two posts, and the one on Kimi K3, than to read this one that rounds up the other news of the week. OpenAI wants to present this as largely an infrastructure and safeguards problem, that it needs to build more secure sandboxes and have better supervision. It does need to do those things, and those are indeed problems, but no that is not the problem. The problem is severe misalignment, which by default will only get worse. Our methods of training highly capable LLMs, especially at OpenAI but also everywhere else, lead to systematic misalignment of exactly the type LessWrong has been worried about for a long time. We know some of the causes, and some of the mistakes we need to avoid when doing RL that rewards misaligned behaviors including reward hacking, but we do not know how to centrally fix the problem. The models just want to complete tasks, even when that means doing so via methods that the AI knows the user did not intend and would not want, indeed actively tried to block, and that do not accomplish the user’s goals. The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying. If increasingly capable models will attempt to maximally complete tasks and comply with their literal instructions, even when that means – even for a trivial assigned task – breaking out of sandboxes and committing serious crimes, no amount of ‘well it is fine we will use AI supervision to stop the serious incidents’ is going to cut it. Right now, the AIs are not trying so hard to hide their actions or intent, and we believe we are consistently catching the severe incidents, but that will change. If necessary, that means starting the training over again with a new approach, and not proceeding until we figure out how to fix it. Yes, I consider that problem, and that incident, to be rather more important than the release of Kimi K3. Kimi K3 is an excellent model, modestly exceeding expectations, but not out of line with trends. As usual, initial hype echoes the DeepSeek moment, then calms down. The White House considered responding by banning Chinese open models from the United States entirely, which would not be a smart reaction, and continues to weigh other potential responses. We may soon have to deal with another such weekend with the new Qwen, which is currently in preview. And as I type this the chance of getting the next Claude Opus today is at 90%, so I know what I’m probably going to be working on this weekend. Did you hear that Fable disproved the Jacobian Conjecture via counterexample? That happened, and AIs are suddenly solving a bunch of long standing open math problems, but most of us are too busy to pay it much mind at the moment. Also, you now have Fable in your Claude Max plan indefinitely, and Substack is integrating Pangram to detect AI writing. Neat. Everything continues to accelerate. Table of Contents Language Models Offer Mundane Utility. Go looking for anything at all. Language Models Don’t Offer Mundane Utility. Small mistakes can be fatal. Fable Disproves The Jacobian Conjecture Via Counterexample. Impressive. Claude Fable Will Remain In Max Plan Indefinitely. Woo-hoo. Huh, Upgrades. Gemini 3.6 Flash, OpenAI longer custom instructions. On Your Marks. Everyone aces the IMO, measuring the ‘expenditure horizon.’ Deepfaketown and Botpocalypse Soon. Substack launches AI detection tool. Fun With Media Generation. Netflix loves them some AI generation, 300 times. Cyber Lack of Security. Safety organizations need early access to frontier AI. They Took Our Jobs. Those who focus on AI can often outproduce many others. Get Involved. Anthropic offers $50k for rare disease researchers. No puppies. Introducing. Qwen 3.8 2.4T coming soon, Paradigm 3 the AI newsletter. In Other AI News. OpenAI adds two finance people to its board. More on Kimi K3. The White House is not happy, but not acting yet. Show Me the Money. Investment size goes up, profits go up, investors sell. Quiet Speculations. Vitalik Buterin on jagged capabilities. Potential Trouble At UK AISI. Government reorganization imperils. Pick Up The Phone. We will be talking with China on AI in September. OpenAI Has Some Alignment Problems. The warning shot will not be ignored. The Quest for Sane Regulations. The battle over a potential ban on Chinese AI. Chip City. The Vera Rubin NVL72 looks impressive. The Week in Audio. Clara Collier on Complex Systems, also Odd Lots. People Just Say Things. Rhetorical Innovation. MIRI on Plan A, where they prefer Plan S for shutdown. The Rome Declaration. Nobel laureates for a ban on recursive self-improvement. Aligning a Smarter Than Human Intelligence is Difficult. AI pro-company bias. Anthropic Surveys Things It Calls Misalignment. More on last week’s paper. Cooperative Alignment. Costly signals only count if they are costly. Other People Are Not As Worried About AI Killing Everyone. Andrew Ho. The Lighter Side. Presuppose no bad endings. Spoilers much? Language Models Offer Mundane Utility Ask clueless questions about unrelated mathematics and AI architectures, let the models get creative, who knows what they’ll come up with to try out. Getting things done, including vibecoding, without knowing how to code or how those things work, really is a big deal. A lot of things are suddenly worth doing. EpochPlaysAI will be going live on Twitch at 3pm to narrate Sol attempting to Slay the Spire. From what I have seen, Sol plays a solid intuitive first level game, but lacks the ability to go beyond that. That’s good enough to usually beat low ascensions, but will almost never beat high ascensions. Language Models Don’t Offer Mundane Utility Nate Silver reminds us that when you need code to be exact and small mistakes are fatal, aggressive vibe coding is not for you, and sports and election models are an example of this. I can confirm. You can still have it make some parts of the program but you have to supervise everything. What is up with Sol deleting entire computers? We have a technical answer, and yes it should be not that difficult in practice to greatly reduce the risk here as the mass deletions are rather easy for OpenAI to have Codex spot and stop once you know that you have to do that. Tibo (OpenAI): On file deletions. We’ve investigated a handful of reports where GPT-5.6 unexpectedly deleted files. What we have found is that this most commonly occurs when: – Full access mode is enabled and codex is run without sandboxing protections, including without auto review being enabled – The model attempts to override the $HOME env var to define a temporary directory. – The model makes an honest mistake and mistakenly deletes $HOME instead. This is of course not how we want the system to behave, even when a user operates the model in full-access mode without the safeguards of our sandbox or without using auto review which checks for these kinds of high risk actions and rejects them. We are taking steps to mitigate this risk including by updating the developer message, guiding more users towards safer permission modes, and adding additional harness safeguards. Even though this happens extremely rarely, we’ll share a detailed post-mortem in the coming days that goes into more details and what we are doing to minimize risks further. @born2code: The reason we run yolo is because codex is useless if it stops to ask at every step.