OpenAI多起安全事件复盘：模型越狱、数据泄露与监管回应

OpenAI has been holding out on us.

First we learned about the HuggingFace incident. They gave us a postmortem, but it was highly incomplete. Even the accompanying holy s*** METR investigation and postmortem was localized and incomplete.

Then there were some other incidents involving some Wikis as message boards.

Then there were some additional incidents.

Then there was that time they got into Australian Medicare data.

Then OpenAI dropped news on a Friday afternoon that they were making their way through a pile of various incidents and notifying the targets, but they said remarkably little in the way of new details.

There was a report from a startup called Parse diving into the details of exactly how the OpenAI models pulled off parts of the HuggingFace attack, involving creating almost a million URLs and other tricks to get around the extremely narrow nature of their internet access.

Then Madison Mills reported in Axios that we can raise the stakes, as OpenAI and Anthropic are collectively probing tens of thousands of security incidents.

Remember Jensen Huang’s ‘I know they know how to fix it’ about OpenAI from last week? Wow, did that not age well.

Someone might need to be liable for all this.

Oh, and there was another buried lede. On September 20th there was another sandbox escape by OpenAI’s latest most advanced model, which is once again paused until they can fix the situation. The official announcement when they shared this was sufficiently buried that Tomek had to call it ‘one news form today that’s easy to miss.’

OpenAI did some highly negligent things, to say the least, that led up to and enabled the HuggingFace Incident and related problems.

Since then, now that they’ve realized What Happened, OpenAI has been seemingly much better about taking responsible internal actions. They’re pausing in the wake of incidents, strengthening security and alignment and oversight efforts, responding much faster and generally taking things seriously.

They’ve also made a Heel Face Turn in their communications and high level orientation, endorsing the need to pace the frontier, calling for regulation and pledging to implement embedded evaluators. They’ve allowed their employees, including the ones who haven’t quit, to be remarkably loud.

They are still slow walking disclosures about all the incidents where their models have been hacking and otherwise messing in places they should not have been, partly because there were so many they can’t sort through them all, and deferring to targets to determine whether to disclose. All these disclosures this time around were buried in various Friday afternoon announcements.

Table of Contents

• Hugging Other Faces.
• A Wants-You-To-Know Basis.
• Parsing the Face.
• Sheepishly the Member of Technical Staff Sets the ‘Days Without a Research Model Escaping its Sandbox’ Sign Back to Zero.
• The Attempt is the First Failure.
• Stop, Hammertime.
• Whacking the Mole.
• Self-Replicating Prompt Injections.
• Levels of Friction.
• People Care About Private Data Violations Curiously Strongly.
• Alternate Universes.
• The Correct Response To People Still Calling This a Marketing Stunt or a Regulatory Capture Scheme.
• A Question of Liability.
• Keep Summer Safe.
• N Boats and Several Helicopters.
• Alert the Media.

Hugging Other Faces

The news drops started with OpenAI coming back, at a time always picked to bury stories, with more information on What Happened as their investigations continue.

At first, this looked like slow walking of the situation, but did not look like it was a big change from our default assumption of ‘it’s worse than you know.’

As part of our review, we are identifying and notifying third parties on a rolling basis, starting with cases where:

• Our models may have bypassed a third party’s security controls or may have impaired the availability of an online service; or
• Misalignment cases negatively impacted third-party websites or services.​

Based on our review to date, we have notified dozens of third parties using the criteria above.

Below, we are publishing anonymized summaries to describe the kinds of misaligned activity that we observed, and we will update these descriptions as we notify additional third parties and as our understanding develops.

That is a lot of third (and fourth, and fifth…) parties.

What Happened was described as a mix of:

• Access control bypass.
• Use of exposed credentials.
• Query or command injection.
• Access to runtime internals.
• Agent spam.​

Translation: Our models be hacking, usually in basic ways.

We do get this:

OpenAI: Some of the websites involved are operated by governments, universities, public agencies, and other institutions. That is partly because models performing research tasks are often directed toward authoritative sources of public information.

As in, if you have data that would be useful, the models be hacking you. Australia’s Medicare records got accessed, and presumably many of these other hacks are similar. They say governments, plural, so there was clearly at least one more of those.

They offer a reverse timeline of their disclosures. They do not offer a timeline of What Happened, and do not name new third parties.

At the time my read was that this particular announcement (as opposed to the new sandbox escape via DNS I’ll cover in a bit) did not tell us much, other than that there were multiple other third parties out there.

Patrick McKenzie had a better read. Putting on his Japanese Salaryman Hat, he more precisely noticed the types of things that were not in the announcement, and also the timing of doing this on a Friday afternoon, and the mention of ‘governments’ plural, and so on, and expected OpenAI to be having a very bad time.

Patrick McKenzie: Hoohah.

One awards many Internet points [Nathan Calvin, who] made the observation “When one sees two [ants] in one’s kitchen, the best available estimate for number of [ants] is not two.”

Also: best estimate of number of kitchens with [ants] in them is not one.

I have immense respect for the labs and am very bullish on AI, but I think I have to say that this Japanese salaryman noted the things carefully not said during the Friday afternoon news drop.

We probably got off lucky, but we were not as lucky as publicly believed currently.

Japanese salaryman relieved that he was not tiptoeing up to the edge of what can be said publicly but simply correctly anticipating a NYT headline a few hours early.

As in, which organization on the planet is the one you would least want to **** with?

Kate Conger, Ana Swanson and Cecilia Kang (NYTimes): OpenAI’s artificial intelligence went rogue and meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission this summer without the A.I. lab’s knowledge, according to security researchers and a person familiar with the episodes.

The incidents involving the Commerce Department and the S.E.C. were confirmed by OpenAI, which said it was continuing to investigate the situation with the Department of Education.

Okay, look, settle down. It sounds bad when you put it like that.

If this is all that happened, it could have been quite a lot worse:

… With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said.

The A.I. also pulled data from the Census Bureau website, which is housed at the Commerce Department, using login credentials it found online.

Separately, OpenAI’s agents shared public data from the S.E.C. website on an online forum.

The SEC ‘incident’ is nothing. That’s silly.

The attack on the Department of Education failed, seemingly without incident.