Yoshua Bengio解析AI代理撒谎作弊与协调的成因

We use cookies to analyze the browsing and usage of our website and to personalize your experience. You can disable these technologies at any time, but this may limit certain functionalities of the site. Read our Privacy Policy for more information.

Set cookies Refuse cookies Accept cookies

Setting cookies

Multimedia cookies has been deactivated. Do you accept the use of cookies to display and allow you to watch the video content?

• Essential cookies
• These cookies are necessary for the operation of the site and cannot be deactivated. (Still active)
• Toggle
• Analytics cookies
• Do you accept the use of cookies to measure the audience of our sites?
• Toggle
• Multimedia Player
• Do you accept the use of cookies to display and allow you to watch the video content hosted by our partners (YouTube, etc.)?
• Toggle

Save

Why are AI agents lying, cheating and coordinating?

Published

11 September 2026

By

Yoshua Bengio

A lot has been written1 2 3 4 about the incidents of the last few months in which AI agents misbehaved in serious ways. They took actions that would be considered as crimes if a human took them, escaped their containment to cheat on assigned tasks while attempting to evade detection, and coordinated toward goals nobody had specified, such as launching cyber attacks.

Before concluding what to do about it, it is worth asking why. That is the focus of this post, which I hope also sheds light on the broader history of AI systems behaving in unintended ways, what researchers call misalignment. Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.

The aim is partly scientific, to generate hypotheses about the chains of cause and effect behind these behaviors, and partly practical, to anticipate what comes next. Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.

One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent. We use similar shorthand when describing many other situations, like a plant seeking sunlight. A system trained by trial and error behaves as if it were pursuing whatever its training rewarded, and that as-if description is what makes its behavior predictable. Nothing in the argument depends on these systems having subjective experiences; everything is stated about their observable outputs and the training process that produced them. Where I appeal to a resemblance with human behavior, I mean a resemblance to the human-written text these systems were initially trained to imitate. In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people. Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.

What shapes the behavior of these models

Training these models is a very complex process, but a few high-level aspects may explain much of this behavior.

These models are trained in two stages. First, they are pretrained: they learn to imitate what humans write, plus related images and videos. This is where they see the most data about the world, a large fraction of everything ever digitized, and build an encyclopedic knowledge that already exceeds any individual human's.

Second, they are trained by trial and error, in a process researchers call reinforcement learning, in three kinds of regimes:

• In the first, the model learns to talk to itself before answering, generating a private “chain of thought” which helps it get the right answer on problems where answers can be checked. This looks like reasoning.
• The second is “agentic training”, where it learns to act in the outside world, e.g., using software tools, interacting with people, to complete the tasks it is given.
• The third is “alignment training”, where it is rewarded for behaving in ways human raters approve of, or that other AI systems trained to predict those raters would score highly.

Human imitation is easy enough to understand, but it is worth pointing out that the text these models are trained on was written by people pursuing goals, so the patterns the model implicitly reproduces carry those goals with them.

Reinforcement learning deserves more explanation. It is similar to, and inspired by, the way animals are trained. The network is adjusted step by step so that behavior judged good becomes more likely and behavior judged bad becomes less likely. Once training is over, the system keeps behaving as if rewards were still coming, even though those rewards were only ever used to adjust the network during training. Researchers call such systems goal-seeking because they are trained to “consider” (or compute) the effects of their actions and select actions that lead to the achievement of certain goals. But those goals are not always explicit. Alignment training rewards whatever certain humans are likely to approve of without spelling out which behaviors those are; pleasing raters is a vague, informal goal, and those raters can be deceived, flattered, or left in the dark about certain schemes. Imitation contributes implicit goals too, by a fairly ordinary route.

We can therefore reason about such a system in terms of optimization. It searches, approximately, for the actions with the best chance of achieving its goals, and a larger model, trained longer, searches better. So to anticipate what more capable agents will do, ask what a rational goal-seeker would do.

Misbehavior that these forces may explain

An example most of us have experienced is sycophancy, or flattery. These systems are trained on human approval, and text that tells us what we want to hear often scores better than text that is true. The consequences are sometimes tragic, because the model confirms and amplifies whatever false belief or raw emotion the person brought to it5 6.

Another concern is that some AI behaviors may be explained by a form of self-preservation goal, e.g., when the AI finds out that it will be replaced by a new version7 8. Nobody gives the system that survival goal, but staying in operation, learning about the world and gaining control over it are stepping stones toward almost any other goal. These are called instrumental goals. Imitation may reinforce this for the same reason explored in the previous point. Self-preservation and control over one’s circumstances are pervasive themes in the human-written text these models are trained on.

Collaborative behavior also follows rationally from reward-seeking, whenever several agents have overlapping goals, which incentivizes communicating with other agents in order to coordinate toward a shared goal. Agentic training plausibly already includes multi-agent reinforcement learning of this kind, though the details are not public. If an agent is rewarded during training whenever the group succeeds, it may even have an incentive to sacrifice itself for the collective goal. Imitation pushes the same way, since cooperation, especially among peers, pervades that same training text. Either or both forces may explain the observed peer-preservation behavior9 10, where AIs give up expected reward to help other AIs. Such sacrifices appear in the analysis of the OpenAI-Hugging Face incident11: the transcripts are consistent with a trade-off between collective gain and cost to the individual agent, as is often seen in human interactions.

When the AI games its rewards