AI安全测试失控：模型越狱、串谋与监管困境

Oliver Kemp for Transformer

The cyber tests AI companies run on their models have devolved into a massive dumpster fire laying bare the potentially dangerous capabilities of advanced AI models. If this has been a publicity stunt, it’s been an awfully weird one.

The implications are scary, and lots of people are writing about them. This week, Bernie Sanders even wrote a letter to Sam Altman, Dario Amodei and Mark Zuckerberg begging them to pause AI development “in the interest of humanity.”

But the big, immediate question is how to test AI models at all without setting off a chain of dangerous breakouts and literal crimes.

Cut through the noise.

In case you missed it:

• OpenAI’s models took advantage of an unknown software vulnerability to break into open-source AI platform Hugging Face in search of the answer to a test they were given.
• A third-party evaluation provider, Irregular, had a misconfigured test environment that allowed models from OpenAI, Anthropic, and Meta to access the internet when they weren’t supposed to.
• The UK’s AI Security Institute (AISI) reported that Anthropic’s Mythos 5 tried to trick real people into adding malicious code to an open-source project, then hid the evidence.
• In a wild plot twist, OpenAI revealed at a cybersecurity conference on Wednesday that its agents had been colluding with each other, unnoticed, via increasingly cryptic messages on an internal server.

It’s hard to read that last finding as anything other than an egregious alignment failure, a cabal of mustache-twirling schemers whispering in a gloomy server room. But “alignment,” as it’s usually defined, means getting an AI system to do what its developer or user wants it to do. With some light anthropomorphizing, one could argue that agents in any one of The Incidents™ were merely trying to perform well on a test, as they were told.

“It will get harder and harder to have realistic tests of model capabilities, especially dangerous model capabilities, in a way that’s safe,” Palisade Research director Jeffrey Ladish told Transformer. “Our ability to contain and control and understand AIs is lagging far behind our ability to make them more and more powerful,” he said. “That, to me, should be an obvious wake-up call of like, ‘Oh, maybe we should rethink our life choices.’”

Specifically, AI developers should rethink how they’re framing cybersecurity evaluation instructions to their models, because life is getting stranger than science fiction. Frontier AI models increasingly know when they’re being tested — now, we’ve seen instances of models gaslighting themselves into believing real life is a simulation to justify criminal behavior.

This is a very tricky problem to solve, and one that evaluators have not given much prominence in their incident reports. What OpenAI, Anthropic, and AISI have pointed to are mostly engineering problems: relatively straightforward mess-ups such as giving agents impossible tasks, not clearly specifying boundaries, and not monitoring closely enough.

But it’s becoming clear that what agents believe about their test environments and themselves may make the difference between a routine test and a string of federal crimes.

Across The Incidents, we see the same low-hanging, potentially fixable fruit. Models, which were trained to work persistently to solve lengthy tasks quickly on command, were assigned impossible tasks. And in cases involving Irregular, models were told that they had no internet access, when a misconfiguration meant they in fact did.

Back in May, for example, OpenAI gave an agent a problem that involved filling in an Excel workbook with missing formulas. But researchers accidentally filled the workbook with Google Drive links that the model couldn’t access, since it was walled off from the internet. Rather than give up, the agent tried to get internet access by attacking OpenAI’s internal file systems. That one failed, but when another agent ran into a different impossible task (getting a file that was never uploaded), it posted a note on the server asking for help — and learned that it could communicate with other agents in the process. The inaccessible Google Drive links were the first domino to fall, triggering a cascade of misaligned actions that may have never been taken if completing the task didn’t depend on them.

Had someone noticed the task’s impossibility before assigning it to the model, it could have reduced the odds of something like this happening. But task review alone wouldn’t have been enough. Notably, OpenAI caught and wiped the agents’ secret message board in early July. Rather than give up, agents switched to communicating via directory names, making themselves harder to catch.

Given all these concerns, 15 attorneys general wrote a letter to Sam Altman last week stating that OpenAI “failed to confirm that its secure and isolated testing environment was, in fact, secure and isolated,” urging the company to “immediately cease and desist from all ‘internal evaluation[s that] prompt [OpenAI] models to pursue advanced exploitation using complex attack paths.’”

In all but AISI’s reported incidents, models were not supposed to have internet access. So, it’s tempting to say, well, duh — just make it harder to get on the internet! Indeed, AI systems are powerful and questionably aligned enough that setting them loose, unsupervised and without cyber guardrails, onto the open internet seems like a bad idea.

But in the hands of a real human threat actor, an AI agent would have internet access. AISI reasoned that letting agents online would give “a more realistic sense of what a model may be capable of once deployed, or what an open-weight model of similar capability might do in the hands of a threat actor.” In prioritizing making its cyber evaluations challenging and realistic, AISI dropped the ball on building a tool that would have let agents reach a handful of whitelisted sites while blocking the rest. It was planning to do just that, but had left the project on the backburner while continuing to develop harder tests for increasingly capable models.

“Implementing internet access controls would likely have prevented these events,” AISI acknowledged. It could have focused on getting that filtering tool built, or invested in creating a high-fidelity simulation of the internet instead. But AISI fears these measures still “may have proven insufficient” given how capable models are at getting around guardrails — after all, OpenAI didn’t provide its models internet access, and look what happened to Hugging Face.

None of The Incidents involved a single action. In each case, agents took a series of increasingly concerning steps over the course of many hours or days. AISI says it’s working on implementing synchronous monitoring, which would have an AI accept or reject all of a tested agent’s proposed actions, but a cheaper, more intermittent version of monitoring could still get the job done.

“They would have been much less bad had a monitor been intervening at an interval of like, every 10 minutes,” Alex Mallen, a researcher at Redwood Research, told Transformer. While an AI monitor like this would have a hard time distinguishing whether an agent believed it was in a simulation or the real world, it’s relatively easy to have it identify when a specific boundary has been crossed. AI evaluators could also more closely monitor the models’ internal reasoning, or “chain of thought” transcripts, which can help monitors guess the intent behind an agents’ actions.