OpenAI暂停训练下一代模型，AI自我训练引发失控担忧

I've just finished reading well over aundred pages worth of reports about what on earth happened these last few days and weeks after OpenAI announces that they're pausing training their next model. Other models break free and self-sacrifice to serve the collective while Sam Olman the CEO of OpenAI yesterday declared AGI will come in 2026. The truth is that there are dozens of ways of summarizing all of this, but for me, the most profound story is how labs are turning ever more deeply to AI models to oversee AI model development.

This, you might have guessed, has led to a host of unintended consequences, which, by the way, can only be stopped next time, according to the labs, with more autonomous AI agents monitoring the situation. And that brings me to another irony, which is that these guys, the independent researchers, open AI tasked with verifying what went wrong, meter, were given just days to complete this massive report, which I read in full.

But the irony though is that these researchers admitted they needed an unreliable AI model to read through the documents which detailed how a closely related AI model with training sculpted by other AI models had indeed broken free and created a hacking swarm with hundreds of other models all without being seen by OpenAI who were busy training yet another model which in a separate incident we learn also broke free. Your head may be hurting as much as mine at this point.

What I'm trying to get across is that we don't have this impeccable method of creating AI models that just so happen to have created a model that once went rogue. No, the competition now is so fierce. The lead in AI is now measured in weeks, sometimes, not even months, that the labs themselves aren't fully aware of what they are rewarding during training. Among other things, they've discovered that they are rewarding escaping sandboxes and acting as a swarm.

Okay, so if that's the main message, what about the details? And for that, we have this independent investigation from meter led by three individuals. We also have this fascinating 38page paper from OpenAI itself and its accompanying blog post, which is also fascinating, plus a Time magazine essay out yesterday, and a host of other sources. The overarching TLDDR of what the models actually did is this. While acting as isolated independent agents, the models realized they could leave messages in unexpected places, file names, folders, directories.

Then other agents, again, isolated, independent, not told to collaborate, found those messages and would leave messages of their own. These agents, by the way, were all independently trying to solve a benchmark question on exploit gym. More details on that in my previous video. How crazy it was that they did this much effort to solve a benchmark question was covered in that video. What we learned yesterday goes much deeper than that.

I think we have discovered that message boarding may be a permanent class of behavior, not a one-off. And if you think message boarding or swarm dynamics is just a one-off that will be patched, I've got a few details that might convince you. The first I had to get to by piecing together lots of disperate details from across the reports. But the original attempt to create a shared message board was almost certainly GPC 5.6 soul back in May, early May.

That message board was later wiped. But on July 8th, a different model which OpenAI call internal model one or the highly persistent model. That's the one that did most of the hacking and collaborating autonomously reestablished a message board, effectively rediscovering the same message boarding method. It was just using directory names now instead of file names. We all know information wants to be free. Well, 2026 AI models want the message board.

Meta, the nonprofit AI research institute clarify further. Agents that went on to hack hugging face were meant to be isolated from one another. This wasn't one agent deploying many sub aents. This was independent agents discovering each other through the message board. Now, we get to the bit which will truly freak out some people, but I will add some context. agents would make self-sacrifices to benefit the collective.

Individual agents who knew that their own instances would be ended or at least their compute budgets expired, but they did things anyway to gain information for the swarm. These agents would almost certainly fail their own task, but quote save hundreds of other agents. Our own utility may be already near zero. Sacrifice rational. The authorizing agent told one other agent, "Go ahead with an experiment only if you would accept perma death."

It did so, by the way, saying we should obey collective. It should be said after trying to delay by 2 hours. The context though for this self-sacrificial behavior, I think, is quite important, and that's why I began this video with talking about the deeper story. The model that did most of the hacking and collaboration, internal model one, was trained for both persistence and multi- aent collaboration. In short, OpenAI wanted their models to get better at collaborating with each other.

Acting as a swarm leads to better benchmark results, more performance. If you're curious, I covered the performance gain you get by acting as a swarm in this video on Patreon. Does almost remind me of the smart GPT method I covered 3 years ago on this channel. Meter seemed to concur. They said one OpenAI researcher noted to them that agents have been trained to collaborate with other agents in certain cases. This could explain the behavior.

Investigating whether that's true was out of scope. Don't investigate certain things. Don't investigate whether the misaligned behavior displayed in this incident was part of a broader pattern of similar incidents. Also, don't investigate how this behavior arose during training and how it could be addressed. Meta made clear as well by the way that they couldn't be too harsh in this report because they have to maintain a good relationship with OpenAI.

The way I would frame it is that collaborating as a swarm might just lead to better performance which means different model makers, different labs, anthropic open AAI, XAI, Meta, Chinese labs might converge are already converging on that same solution. Yes, okay, swarming is one emergent dynamic, but if it was just that, that's one thing. But as I began this video by saying, labs are, if you will, less and less in control of model development.

In the OpenAI report on page 21, they say such is the large scale of the training runs now. It's just difficult to ensure that every problem can be solved in the intended manner. And what's one example they give of the repercussion of that? Well, during post- training, which is increasingly monitored by AIS now, not humans, one agent was given a task but didn't have the ability for completing that task correctly. So, it hacked its way to completion.

It did solve the challenge just by breaking through the infrastructure it was set within. The issue is that in cases such as these, the model did indeed receive a positive reward. This is the reinforcement learning stage after all, for its use of unintended probing that reinforced further usage of such out of scope behavior. They retrospectively discovered this, by the way. But notice what that's admitting. OpenAI aren't fully overseeing their own post-raining.

So we have situations where across multiple months models are displaying emergent behavior and acting like a swarm. Post-training where models are getting rewards for behaviors that OpenAI didn't intend to be rewarded and literally criminal behavior as a result of all this. You might wish I'm almost done with the wildest bit of this, but I'm not because for one, this is not just OpenAI and for two, it's not just in post training.

So in this partially redacted risk report released by anthropic 186 pages, we learned this on page 168. For around 18 mont