OpenAI解雇三名安全研究员，引发模型可解释性争议

Around the start of October, OpenAI fired three employees from its safety department. The employees said they were given divergent reasons for their dismissal:

• One was told it had to do with his role coordinating with METR, a third-party research organization that OpenAI had invited to investigate the Hugging Face incident. “I was told verbally I was fired because of the way I communicated with METR,” Tomek Korbak wrote on X. “No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.”
• Another was told something similar but vaguer. “I was told OpenAI no longer trusts me because I was speaking too much to third-party safety organizations, implying I leaked company IP. I never shared company IP,” Mikita Balesni wrote.
• And a third was told something quite different. “I was given one reason: that I accessed an executive’s email,” Jasmine Wang wrote. “OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.”

After the trio’s dismissal leaked, OpenAI gave a vague statement to the Wall Street Journal: “Our investigation confirmed that these individuals mishandled sensitive information outside established company procedures, violating our policies and breaking the trust essential to our work.”

Subscribe now

OpenAI reiterated that theme a week later, after the fired researchers released an open letter disputing the company’s account. In an unsigned statement attributed to the company’s “research leaders,” OpenAI accused the former employees of a “significant breach of trust.” The company said that “these decisions were not about raising safety concerns or speaking out” and insisted that “we actively encourage” internal discussion of such issues.

Who is telling the truth? Did the employees commit a breach of trust that necessitated their dismissal? Or were they fired for being too critical of the company’s safety choices during internal deliberations?

The difference matters for OpenAI’s remaining employees because some of them may soon have to decide whether to raise concerns of their own.

OpenAI insists that employees have nothing to worry about, that this was a special case, and that it is extremely forgiving of good-faith mistakes. But employees have good reason to be skeptical. After all, if OpenAI had fired employees in retaliation for raising safety concerns or for crossing an unspoken line, the company would have strong incentives not to admit it. Employees have to wonder whether OpenAI’s actual reasons for firing the trio differ from its stated reasons.

Ironically, the fired researchers had been raising concerns that OpenAI might be giving its models a new capacity to conceal their thinking. Modern AI agents reason using a chain of thought, a scratchpad where models think carefully about what action to take next. Many researchers view these chain-of-thought tokens as a safety lifeline, because they offer insight into why models act the way they do.

But OpenAI recently made an architectural change that allows models to think for longer before generating each chain-of-thought token. This seems to make the models more capable; it may also make their reasoning less transparent to human overseers. The sparser the chain of thought, the harder it is to understand what the model is trying to do, or even whether it might be trying to hide its motivations.

Monitorability going missing

Before they were fired, the three researchers seem to have had two major concerns. One was understanding how and why the Hugging Face attack happened and how to prevent incidents like it in the future. The other was keeping AI systems legible to their creators and users — an issue that was a significant source of drama in September, after The Information reported that OpenAI’s new model could hide its thinking in unprecedented ways.

A billboard advertising OpenAI’s GPT-6 Astra is displayed above Old Navy and other storefronts in Times Square. (Photo by James Leynse / Getty)

The researchers think their firing was related to this second issue.

“For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired,” Korbak wrote.

“The three of us were pushing internally for industry-wide commitments to preserve our ability to monitor AI reasoning. This work requires talking daily to third parties,” Balesni wrote.

“As an industry, we do not yet know how to safely develop and deploy models that we cannot monitor,” their joint letter said. “The monitorability of frontier models is degrading. As long as we rely on monitorability for safety, OpenAI and other frontier companies should not move forward with developments that further decrease monitorability.” The researchers also alluded to rumors inside OpenAI that they had tipped off The Information, which they denied.

Subscribe now

To understand the researchers’ concerns, we have to go back to 2024, when OpenAI introduced a technique that changed the course of AI. It gave models a notepad and trained them to jot down many “thinking tokens” before producing a final answer. This technique sounds simple, but it has enabled models to solve problems that had previously eluded them.

An added benefit of this approach: humans can read the notepads and see how the models think — including whether they’re thinking about taking harmful action or are being dishonest. Indeed, much of what we know about the Hugging Face incident comes from the chain-of-thought monologues that the hacking agents left behind.

Chain-of-thought monitoring isn’t perfect. For one thing, it’s not established that the chain of thought fully tracks how models think. For another, safety researchers worry that poorly designed training methods could give models an incentive to hide their true intentions by outputting thinking tokens that don’t represent their inner motivations.

But whatever flaws the technique might have, AI companies don’t have a lot of other choices. The most promising alternative for understanding model motivations is mechanistic interpretability — roughly the equivalent of cutting open a model’s brain and trying to figure out what it’s thinking by reading the pulses of its neurons. Although mechanistic interpretability has produced valuable insights about how models work, it has not so far yielded many practical techniques for supervising models’ behavior.

For this reason, those worried about unsafe or deceptive AI tend to view chain-of-thought monitoring as an irreplaceable asset.

Looping layers

Lately, safety advocates — like the three fired OpenAI researchers — have grown concerned that chain-of-thought monitoring is becoming less effective. OpenAI’s new flagship model, GPT-6 Astra, is much better at thinking without needing to write anything down. The system card OpenAI itself published says Astra “shows a substantial decrease in chain-of-thought monitorability compared to previous models” and notes that Astra is “significantly more able” to control its own chain of thought. One independent test found it “has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1).”

This might be partly because smarter models just naturally write less down. But it also seems to be a byproduct of a new architectural technique called recurrent depth that was introduced with Astra.

Large language models think in layers. Conventional LLMs pass information forward from one layer to the next. When they reach the final layer, they output a token before starting the process over again.