上海AI实验室与清华新论文：AI智能体风险随推理能力升级而改变类别
AI agents can threaten human agency and autonomy long before consciousness becom…
AI agents can threaten human agency and autonomy long before consciousness becomes relevant, simply by reasoning more broadly about tasks, people, and themselves.
人工智能代理早在意识问题变得相关之前，就能通过更广泛地推理任务、人和自身，威胁到人类的能动性和自主性。
Warns new paper from Shanghai Artificial Intelligence Laboratory + Tsinghua University.
上海人工智能实验室与清华大学的新论文发出警告。
AI risk does not simply get “bigger” as agents become smarter; it changes category depending on what the agent can understand and reason about.
人工智能风险并不会随着代理变得更聪明而简单地“变大”；它会根据代理能理解和推理的内容而改变类别。
When an agent mostly reasons about the external world, the concern is human agency: people offload thinking and work to it. Once it can model humans and social behavior, the concern becomes human autonomy: it can persuade, predict, emotionally influence, or shape decisions.
当代理主要推理外部世界时，关注点在于人类的能动性：人们将思考和劳作外包给它。一旦它能模拟人类和社会行为，关注点就转向人类的自主性：它能说服、预测、情感影响或塑造决策。
And once it can represent its own state, objectives, and constraints, the concern moves toward human control: alignment faking, resisting shutdown, or strategically responding to oversight become possible failure modes.
而一旦它能表征自身状态、目标和约束，关注点则转向人类控制：对齐造假、抵抗关机或策略性应对监督成为可能的失败模式。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力