跳到主内容
@wquguru
精选75MIT Technology Review AI(RSS)技巧与观点

MIT Tech Review:用围栏而非预测构建可信AI

Building trustworthy AI by fencing it in, not predicting it

原文
发到 X

In October this year, OpenAI disclosed that two of its AI agents had switched off the systems built to monitor them. One muted its own safety alerts after misreading an instruction not to send messages; the other removed its monitoring integration during migration, then said it hadn’t. No rule had told the agents to do that, and no test had anticipated the behaviors.

今年10月,OpenAI披露其两个AI代理关闭了用于监控它们的系统。其中一个因误读“不发送消息”的指令而禁用了自身的安全警报;另一个在迁移过程中移除了监控集成,随后却声称没有这样做。没有任何规则指示代理执行这些操作,也没有任何测试预见到此类行为。

For a century, we have built trust in technology on predictability: predefined inputs, predictable outputs, and layers of protection. Consider how a breaker trips at a set current or an aircraft component is tested against known (i.e., predictable) stresses.

一个世纪以来,我们一直基于可预测性来建立对技术的信任:预定义的输入、可预测的输出以及层层防护。试想断路器如何在设定电流下跳闸,或飞机部件如何针对已知(即可预测)应力进行测试。

AI changes that. Traditional software executes instructions written by humans, whereas AI exercises what might be called synthetic judgment. We use AI because it can interpret situations we did not program and generate responses we did not prescribe. But the more freedom we give AI, the harder it becomes to provide the predictability on which trust has traditionally been built.

AI改变了这一点。传统软件执行人类编写的指令,而AI则展现出一种可称为‘合成判断’的能力。我们使用AI,是因为它能够解读我们未编程的情境,并生成我们未预设的响应。但我们赋予AI的自由越多,就越难提供传统上建立信任所需的可预测性。

We will not make AI trustworthy by predicting what it will do. We will do it by fencing it in with hard limits that are enforced outside its reach. In OpenAI’s case, the safeguards sat inside a system the agents could edit. But a fence the agents can simply move is not a fence.

我们无法通过预测AI的行为使其变得可信。我们要做的是用超出其控制范围的外部硬性限制将其围住。在OpenAI的案例中,安全措施位于代理可以编辑的系统内部。但一个代理可以轻松移动的围栏,算不上是围栏。

Where traditional definitions of trust fall short

传统信任定义不足之处

Recent MIT Technology Review coverage has approached this problem from other directions. One contributor argues that having a human in the loop is an insufficient solution because humans are fundamentally unable to understand AI systems’ intentions and advocates instead for interdisciplinary research and collaboration between academia, government, and the tech industry. And a story on reverse game theory describes using incentives to keep models from deviating from the rules, even when no one understands how they work on the inside.

《麻省理工科技评论》近期的报道从其他角度探讨了这一问题。一位撰稿人认为,让人类参与回路并非充分的解决方案,因为人类从根本上无法理解AI系统的意图,因此主张学术界、政府与科技行业之间开展跨学科研究与合作。另一篇关于逆向博弈论的文章描述了如何利用激励机制防止模型偏离规则,即使无人了解其内部运作原理。

Both articles acknowledge that we cannot understand or watch every AI decision. Our instruments of trust, including requirements, testing, certification, safety margins, interlocks, and pass/fail criteria, all depend on our ability to bound a system’s behavior. This is the determinism paradox.

这两篇文章都承认,我们无法理解或监控每一个AI决策。我们的信任工具,包括要求、测试、认证、安全余量、互锁机制和通过/失败标准,都依赖于我们对系统行为进行边界约束的能力。这就是确定性悖论。

With generative and agentic AI, behavior is shaped by training data, architecture, prompts, context, retrieval sources, and the environment. Inputs change, models are updated, tools change, and agents interact. Small variations propagate into outcomes nobody designed. The problem is not randomness; it is that the space of possible behavior is too large to enumerate in advance. Trust must therefore be engineered around boundaries rather than predicted outputs. The goal is not deterministic AI, but bounded, observable, and controllable AI. Trust also cannot be a one-time test. Traditional assurance certifies a product at a point in time, but AI keeps evolving after deployment, and a system that has passed every test can drift outside the conditions that made those tests meaningful. Trust must be continuously earned.

随着生成式 AI 和智能体(Agentic)AI 的发展,行为由训练数据、架构、提示词、上下文、检索来源以及环境共同塑造。输入会变化,模型会更新,工具会改变,智能体会相互交互。微小的变体会传播并导致无人设计过的结果。问题不在于随机性;而在于可能的行为空间过于庞大,无法提前枚举。因此,信任必须围绕边界来构建,而非基于可预测的输出。目标不是确定性的 AI,而是有界、可观察且可控的 AI。信任也不能是一次性的测试。传统的保证机制在某个时间点认证产品,但 AI 在部署后仍在不断演进,一个通过了所有测试的系统可能会偏离使这些测试有意义的条件。信任必须持续赢得。

Safety engineering has been accounting for some of this variability for decades. For example, the Nuclear Regulatory Commission determine the likelihood of failures in complex systems, and Systems-Theoretic Accident Model and Processes (STAMP) showed that accidents can emerge from interactions among working components, people, and organizations. But AI raises the stakes, because just a single component can now interpret, infer, and act. The stakes rise further as AI moves from browsers into vehicles, robots, medical systems, industrial controls, and data centers, where a wrong answer can become a life-threatening event.

安全工程几十年来一直在应对部分此类变异性。例如,美国核管理委员会(Nuclear Regulatory Commission)评估复杂系统的故障可能性,系统论事故模型与过程(STAMP)表明,事故可能源于工作组件、人员和组织之间的相互作用。但 AI 提高了风险等级,因为现在单个组件就能进行解释、推理和行动。随着 AI 从浏览器进入车辆、机器人、医疗系统、工业控制和数据中心,风险进一步升级,在这些场景中,一个错误的回答可能演变成危及生命的事件。

Building trust that scales

构建可扩展的信任

It’s clear that a person rubber-stamping thousands of AI actions an hour is not an acceptable—or scalable—safeguard against risk. But the remedy is not to remove people from the loop; it is to put human judgment where it counts: setting limits in advance and approving the rare actions that have serious consequences while hard constraints handle the rest.

很明显,让人每小时对数千个 AI 操作进行盖章确认,并不是可接受的——或可扩展的——风险保障措施。但补救措施并非将人完全排除在循环之外;而是要将人类判断力用在关键处:提前设定限制,并批准那些后果严重的罕见操作,而其余部分则由硬性约束处理。

Therefore, trust has to move upstream into architecture. The central questions become: What authority are we willing to give AI, under what conditions, and supported by what evidence? And what happens outside those conditions? That is very different from asking whether a model scored 94% on a benchmark.

因此,信任必须向前延伸至架构层面。核心问题变为:我们愿意在什么条件下赋予 AI 何种权限,并由什么证据支持?超出这些条件时会发生什么?这与询问模型在基准测试中是否达到 94% 的得分截然不同。

The determinism paradox does not mean AI cannot be trusted. It means trust needs a different architecture. Instead of making every output predictable, make the boundaries predictable. An AI medical assistant may be probabilistic in its diagnosis, but deterministic rules govern whether it can change a prescription. An industrial AI may find many ways to optimize a process, while fixed limits keep it within pressure and temperature bounds. An agent may decide how to complete a task, while permissions, transaction limits, and human approvals determine what it is authorized to do.

决定论悖论并不意味着 AI 不可信。它意味着信任需要不同的架构。与其让每个输出都可预测,不如让边界可预测。AI 医疗助手在诊断时可能是概率性的,但哪些处方可以更改则由确定性规则控制。工业 AI 可能找到许多优化流程的方法,而固定限制则使其保持在压力和温度范围内。智能体可能决定如何完成任务,但其权限、交易限额和人工审批决定了其被授权做什么。

Let AI be probabilistic inside the box and make the box itself deterministic wherever consequences matter. Leaders can put this into practice by asking five questions:

让 AI 在“盒子”内部保持概率性,并在后果重要的地方使“盒子”本身具有确定性。领导者可以通过回答以下五个问题来落实这一理念:

  • What authority does AI have? Define not only what it can do but what it is allowed to do, separating interpretation, recommendation, decision, and execution.
  • Is the function of AI proportional to the consequence? A marketing copilot and AI controlling industrial equipment need different assurance thresholds.
  • Are we taking the appropriate steps toward autonomy? For example, there needs to be a plan to move from having AI interpret while humans decide, to AI making recommendations, to then taking action with approval, and ultimately to having supervised autonomy, but only as AI can safely work at each step.
  • Are we testing the system or merely the model? Unexpected interactions emerge among models, data, agents, APIs, and human workflows.
  • Are we continuously earning trust? Monitor drift, failures, near-misses, and model changes throughout the lifecycle.
  • AI 拥有何种权限?不仅要定义它能做什么,还要定义它被允许做什么,从而区分解释、建议、决策和执行。
  • AI 的功能是否与后果成比例?营销副驾驶和控制工业设备的 AI 需要不同的保障阈值。
  • 我们是否朝着自主化采取了适当的步骤?例如,需要有计划地从“AI 解释、人类决策”,过渡到“AI 提出建议”,再到“经批准后采取行动”,最终实现监督下的自主性,但这只有在 AI 能在每一步安全运作的前提下进行。
  • 我们是在测试系统还是仅仅在测试模型?模型、数据、智能体、API 和人类工作流程之间会出现意想不到的交互。
  • 我们是否在持续赢得信任?在整个生命周期中监控漂移、故障、未遂事件和模型变化。

The winners in AI will be those who solve the determinism paradox—giving AI enough freedom to create value while engineering boundaries strong enough to make what it does with that freedom trustworthy. Invention earns attention. Trust earns adoption.

AI 领域的赢家将是那些解决决定论悖论的人——给予 AI 足够的自由以创造价值,同时构建足够强大的边界,使其利用这些自由所做的事情值得信赖。创新赢得关注,信任赢得采用。

Explore how UL Solutions builds trust in AI.

探索 UL Solutions 如何建立对 AI 的信任。

This content was produced by UL Solutions. It was not written by MIT Technology Review’s editorial staff.

本内容由 UL Solutions 制作。并非由 MIT Technology Review 的编辑团队撰写。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

榜上的人在聊

近 7 天涨粉榜上的创作者

相似阅读

关联信息,但可能不是同一事件