AI实验室警告：激烈竞赛导致重大安全漏洞不可避免

The last few days and weeks have been some of the most fascinating, adventurous, and disquing in all of recent history. With an emerging consensus that a major AI breach is due, the newfound, likely unintended ability for you to use AI models to reverse engineer almost any piece of software out there, and of course, yet more research breakthroughs. Because there's so much, I'm going to split this video into three parts.

Part one about the forthcoming breach. Part two on the reasons why labs say they are still racing forward faster than ever. And part three on why the next 18 months are still nevertheless definitionally hard to predict. Now on the topic of the current race dynamic seeming to inevitably lead to a major AI breach. I have noticed in the weeks since I made that point at the end of one of my videos that there has been increasing media coverage as to the almost predictability of that eventuality. financial loss or a significant loss of life.

I think it will happen reasonably soon. You know, shorthand the Chernobyl of AI, but I hope it won't be that bad.

And they don't anticipate that lawmakers will react appropriately until after there's some sort of catastrophic event.

Risks that if if if that did happen, we're talking about a harm that's much larger, for example, than a single nuclear power station melting down. And the internal controls and safety and redundancies are just nowhere near what the world expects for a nuclear power facility.

So when you have bad people, no safeguards, and then the the latest models, that's where over the next year, primarily in the Siberia, we are going to have a a a lot of negative surprises. If you haven't seen my last video, there are of course lots of ingredients that go into people making predictions like these. How security teams at OpenAI are increasingly struggling to put parimeters around the models. While giving models access to tools and the internet helps keep labs at the frontier of intelligence, why jumps in capability can be surprising even to OpenAI researchers and security staff.

And I'll cover that in more detail in part three. And why our three main mechanisms for keeping models under control are each weakening for different reasons. for alignment. Getting models to want what we want is increasingly precarious, especially given that models seem to know now when they're being tested. Monitoring models chain of thought or scratchpad is getting increasingly difficult. So yeah, monitoring is getting tough and locking down the perimeter, putting models in a sandbox.

Well, even with older models that was getting tough, see hugging face and with even the very latest models we saw just a few days ago that say Anthropic had to turn off internet access for all their internal evaluations because models were getting up to all sorts of shenanigans like leaving false tips on various crime and homicide reporting forms. You can see why Joe here who works on agent security at OpenAI says that we should all prepare for the next wave of what can go wrong with AI.

What I will admit not foreseeing though is just how revealing Sam Alman would be in an interview released in the last few days regarding how close some of these breach scenarios are. Notice the phrase in the middle of this clip that Alman drops in where he says GPT 6.1 Astra is an interesting thing to study. So there are kind of two major ways that this could go wrong and I think the Astra 6.1 is an interesting thing to study here.

You can have a big safety accident, eventually one that leads to like a real loss of control where AI, you know, escapes and it's out doing wrecking havoc on the internet and society can't really get control back. 6.1 Astra, the current model we had ready to go, I think would have tipped too much towards the accident risk. So, what we're doing instead is pushing towards a model that much we can make much more confident claims about the alignment.

In case you missed it, he said that the model they were due to release next. 6.1 Astra, not their strongest internal model, which would be Bell. At least that's the code name. No, 6.1 Astra would have tipped too much toward the accident risk, the wreaking havoc on the internet where society can't really get control back. This of course echoes the recent warning from Dario Amade CEO of Anthropic where he said that if the hugging face swarm had possessed greater capabilities but a similar level of misalignment the damage he thinks would have been catastrophic on timings given the accelerations he's talking about 6 to 12 months with swarms being capable of taking over the entire internet even more startling to me I would say talking about the next crop of models was someone saying that they couldn't quite predict even the category of damage that would be done cyber bio we are in the realm of powerful enough models that it is difficult to predict the specific things that will go wrong in what order.

Um, and I think politics and policy is going to be very reactive to the specific things that go wrong. So

if the next big thing that goes wrong is like someone makes a novel pathogen, I think that will have a very different response than if the next thing that goes wrong is more cyber security. And I would guess the world in general will be caught off guard by the next thing that goes really wrong. Obviously, you don't need to be a lab leader to foresee where the current race dynamic ends up. As you've seen, myself and many other outside commentators have drawn the same conclusion.

And of course, researchers, too. Jacob Coxin of the super viral tweet said to Time magazine that at the desks in which he worked at OpenAI anthropic, there was a kind of fatalism. There's this atmosphere of almost resignation. Researchers try to make their own work as safely as possible, even if they think there's a decent chance the whole thing spirals out of control. On the topic though of spiraling out of control, it seems like the labs are actually putting contingency measures in place should this happen.

Axios reported the other day that top executives Anthropic OpenAI and other AI companies are gaming out scenarios for a public and political revolt after a catastrophic AI event. This would be most likely a cyber attack that shuts down financial services, internet connectivity, or even power and water. Obviously, this would turn the public further against this technology, they say. and its leaders. And the report adds that many top AI researchers and executives believe a major incident is inevitable.

Anthropic declined to comment, but OpenAI said, "These scenarios are not treated as inevitable, but are meant to help us prepare for a variety of circumstances." So, those preparations were happening. Famed AI researcher Andre Karpathy pointed out that the vast majority of people are not experiencing the vertigo of what's happening in AI. Either they're the six billion people who just don't use AI, the 1 to2 billion people who use derpy chat bots," unquote.

Then there's basically a lot, I'm sure most of the people watching, the 2% who are engaging with AI, building apps, following the news closely, but that might be 20 million people with of course the 5,000 people at the Frontier Labs in a different category entirely. Those guys or maybe us plus those guys are experiencing the vertigo, Karpathy says, of the curve going vertical. Now, speaking of going vertical, that inspired me to work with GPT image 2.5 to come up with this.

I'll be coming back to this image, but for me, it hints at the first reason why labs find themselves in this position. Many researchers at the labs, indeed even safety researchers, didn't think we'd be reaching this point. And I didn't think that we were going to be able to create something fundamentally smarter than we were. And when I thought, no, we really are going to have something that thinks circles around us, that I thought, oh, we're in a totally inapp