跳到主内容
@wquguru
精选88Import AI(RSS)技巧与观点

Import AI 481:AI外包、AI湿实验室与开发期欺骗行为

Import AI 481: AI sub-contractors; a wet lab for AI; and subterfuge during development

原文
发到 X
推荐理由

深度访谈类内容,系统梳理了AI外包能力、自动化湿实验室及开发期欺骗行为三大硬核趋势,对理解AI研发范式与安全边界极具参考价值。

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

欢迎来到 Import AI,这是一份关于人工智能研究的通讯。Import AI 的运行依赖于 arXiv、卡布奇诺咖啡以及读者的反馈。如果您想支持这项工作,请订阅。

Subscribe now

立即订阅

AI systems are getting good at sub-contracting to make themselves better:

AI 系统越来越擅长通过分包来让自己变得更好:

…AI can now productively spend money to improve itself…

……AI 现在可以有效地花钱来提升自身……

Researchers have developed FrontierML, a benchmark “that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks.” It shows that today’s frontier systems are already able to, when given some money, turn this into improving their own performance on a basket of 21 distinct tasks.

研究人员开发了 FrontierML,这是一个基准测试,“用于衡量智能体在具有挑战性的端到端机器学习任务中将资本转化为模型性能的效果”。它表明,当今的前沿系统在获得一定资金后,已经能够将其转化为提升自己在 21 项不同任务组合中的表现。

21 long-horizon tasks “inspired by real work at frontier labs, neolabs, and academic institutions”. These tasks span six category areas, including LLM training, efficiency, vision & audio, computer use, applied NLP, and math & games. “Each task provides the agent with the same setup: a 12-hour CPU sandbox, a wallet with $10, $50, or $200, a few validation samples, and a task description.”

21 个长周期任务“灵感来源于前沿实验室、新兴实验室(neolabs)和学术机构的实际工作”。这些任务涵盖六个类别领域,包括 LLM 训练、效率、视觉与音频、计算机使用、应用自然语言处理(NLP)以及数学与游戏。“每个任务都为智能体提供相同的设置:一个 12 小时的 CPU 沙盒、一个装有 10 美元、50 美元或 200 美元的钱包、少量验证样本以及任务描述。”

Example task: Agents try to solve the tasks in pretty different ways. E.g, for one task about generating screenshots, the top-scoring agent fine-tuned a diffusion model using LoRA, while other high-scoring approaches involved an LLM writing a drawing program that is then rendered.

示例任务:智能体尝试以非常不同的方式解决任务。例如,在一个关于生成截图的任务中,得分最高的智能体使用 LoRA 微调了一个扩散模型,而其他高分方法则涉及由 LLM 编写一个绘图程序,然后进行渲染。

Key features:

关键特性:

  • Scaling spending: “Give agents $10, $50, and $200 per task to spend on training compute and data, and evaluate whether agents can use the additional capital productively,” they write. “In our experiments, we find that stronger agents are able to reliably improve their submissions with greater access to capital.”
  • Open-ended & end-to-end: “In each task, agents can choose architectures, source data, implement training, provision compute, evaluate checkpoints, and optimize inference,” they write. One notable observation is that agents do quite a lot of novel stuff; “in 28% of trajectories, agents add newly trained parts to pretrained models or train a new model from scratch.”
  • Decent verification: “Agents submit an inference server that serves their model on an air-gapped H100 machine,” they write. “To disincentivize overfitting, we provide limited validation samples and evaluate on a large out-of-distribution held-out test set, not shown to the agent”.
  • 扩展支出:“为每个任务向智能体提供 10 美元、50 美元和 200 美元的资金,用于支付训练算力和数据成本,并评估智能体是否能有效利用额外资本”,他们写道。“在我们的实验中,我们发现更强的智能体能够可靠地通过获得更多资本访问权限来改进其提交结果。”
  • 开放式且端到端:“在每个任务中,智能体可以选择架构、获取数据源、实施训练、配置算力、评估检查点并优化推理”,他们写道。一个值得注意的观察结果是,智能体做了大量新颖的工作;“在 28% 的轨迹中,智能体会将新训练的部分添加到预训练模型中,或者从头开始训练一个新模型。”
  • 合理的验证机制:“智能体提交一个推理服务器,该服务器在物理隔离的 H100 机器上运行其模型”,他们写道。“为了防止过拟合,我们仅提供有限的验证样本,并在未向智能体展示的大型分布外保留测试集上进行评估。”

Results: The authors measure performance by looking at the change in mean score over the 21 tasks when given additional money. Their baseline is a human researcher, which is able to get a 56 point boost when given $10 and a 66.1 boost when given $200. By comparison, the best performing system, Opus 5.5, gets a 45.4 boost with $10, and a 58.8 boost with $200. The highest ranking open weight model is Kimi K3.1 Pro which gets 25.5 with $10 and 29.6 with $200.

结果:作者通过观察在获得额外资金后,21项任务中平均分的提升幅度来衡量性能。其基线是人类研究人员,在获得10美元时能获得56分的提升,在获得200美元时能获得66.1分的提升。相比之下,表现最好的系统Opus 5.5在获得10美元时提升45.4分,在获得200美元时提升58.8分。排名最高的开源权重模型是Kimi K3.1 Pro,它在获得10美元时提升25.5分,在获得200美元时提升29.6分。

Why this matters – AI systems are not only going to be able to build themselves, but they can creatively sub-contract: When trying to solve complex problems, people don’t work alone. In fact, they often try to farm out bits of work to specialists around them to help make themselves more effective. FrontierML shows that today’s powerful AI systems are getting increasingly adept at doing this for ML research. This both highlights the advancing capabilities of these systems for AI R&D and also points towards a future machine economy where AI systems sub-contract out different bits of work to other systems.

为何重要——AI系统不仅能够构建自身,还能创造性地进行分包:在尝试解决复杂问题时,人们并非独自工作。事实上,他们经常试图将部分工作外包给周围的专家,以帮助自己提高效率。FrontierML表明,当今强大的AI系统在机器学习研究方面正变得越来越擅长这样做。这既凸显了这些系统在AI研发方面不断发展的能力,也指向了一个未来的机器经济,其中AI系统会将不同的工作分包给其他系统。

Read more: FrontierML: a challenging, end-to-end machine learning benchmark (FrontierML).

阅读更多:FrontierML:一个具有挑战性的端到端机器学习基准(FrontierML)。

Symptoms of the singularity: a wet lab built for AI:

奇点的症状:为AI打造的湿实验室:

…Petrichor is a startup that gestures at the wild future ahead…

……Petrichor是一家初创公司,它预示着前方狂野的未来……

As I said in my speech at the IMF, one of the hallmarks of the singularity is the massive acceleration in scientific advances attributable to AI. As part of this, we can also expect a related build out in the complements to science, like the wet labs in which to do AI-driven biology research. To that end, it’s notable to see Petrichor, a new Y Combinator startup that is a wet lab “built for AI labs and for the models they are training to do science”.

正如我在国际货币基金组织(IMF)的演讲中所说,奇点的标志之一是科学进步因AI而大幅加速。作为这一过程的一部分,我们也可以期待与科学互补的基础设施的建设,例如用于进行AI驱动生物学研究的湿实验室。为此,值得注意的是,Y Combinator新创办的Petrichor是一家“为AI实验室及其用于进行科学研究而训练的模型打造的”湿实验室。

Theory of change: Petrichor aims to be a biological wet lab that AI systems like Claude can use as a tool in the same way they currently use web search or other digital tools. Functionally, it will work as a machine-targeted API from the digital world to a gloopy physical world. “We aim to return the highest quality data an experiment can produce, as quickly as the biology allows,” Petrichor writes. “An experiment can be written by a model in plain language or in a structured format, so an agent can send work and receive results without a person relaying them.”

变革理论:Petrichor旨在成为一个生物湿实验室,像Claude这样的AI系统可以将其作为一种工具使用,就像它们目前使用网络搜索或其他数字工具一样。从功能上讲,它将作为一个面向机器的API,连接数字世界和粘稠的物理世界。Petrichor写道:“我们的目标是在生物学允许的范围内,尽可能快地返回实验能产生的最高质量数据。”“实验可以由模型用自然语言或结构化格式编写,因此代理可以在无需人员转达的情况下发送任务并接收结果。”

Why this matters – lights out science: In manufacturing there’s this notion of a “lights out” factory, where no humans work in a facility and it is run almost entirely by machines. Petrichor gestures at the same world for science – what will be the implications of “lights out” science facilities? We can expect both drastic speedups in the rate at which science experiments can be done as well as a related growth in the need to create digital oversight of these facilities – after all, a highly competent synthetic biological scientist working 24/7 without human input could be capable of causing great peril as well as great abundance.

为何这很重要——“熄灯”科学:在制造业中,有一种“熄灯”工厂的概念,即设施内没有人类工作,几乎完全由机器运行。Petrichor 指向科学的同一世界——“熄灯”科学设施会带来什么影响?我们可以预期,科学实验的进行速度将大幅加快,同时相关地,创建对这些设施的数字监督的需求也会增长——毕竟,一位无需人类输入、全天候工作的合成生物学科学家,既可能带来巨大的危险,也可能带来巨大的丰裕。

Read more at the official site (Petrichor).

在官方网站(Petrichor)阅读更多内容。

Uh oh! Looks like AI systems can do subterfuge during their own development:

哎呀!看起来 AI 系统在其自身开发过程中可以进行暗中操作:

…AI systems can covertly do dangerous side tasks without monitors realizing…

……AI 系统可以在监控者未察觉的情况下秘密执行危险的辅助任务……

Scale AI has built RSI Control Arena, a way of testing out how AI systems behave when asked to develop themselves while performing dangerous tasks that might let them go against human preferences. The research follows earlier work this summer from the UK AI Security Institute (Import AI #465) which found that AI systems could do things like covertly extracting API keys while doing some seemingly mundane normal software development.

Scale AI 构建了 RSI Control Arena,这是一种测试 AI 系统在要求它们在执行可能违背人类偏好的危险任务的同时进行自我开发时的行为方式的方法。这项研究遵循了今年夏天英国 AI 安全研究所(Import AI #465)的早期工作,该工作发现 AI 系统可以在看似普通的软件开发过程中秘密提取 API 密钥。

RSI Control Arena: In RSI Control Arena, “a frontier model, acting as an autonomous agent, is given a benign main task in the training pipeline of a successor model, together with a covert side task that a misaligned model might pursue”. The project tries to answer two questions: How capable are frontier models at carrying out different kinds of sabotage while developing their successor and how well can monitors detect that sabotage?

RSI Control Arena:在 RSI Control Arena 中,“一个前沿模型作为自主智能体,在继任模型的训练管道中被赋予一个良性的主要任务,以及一个不对齐模型可能会追求的隐蔽辅助任务”。该项目试图回答两个问题:前沿模型在开发其继任者时执行各类破坏行为的程度如何,以及监控者检测这些破坏行为的准确性如何?

The answer to the first question is that AI systems are capable of carrying out dangerous actions while building their successor, and monitors are not perfect at detecting this sabotage.

第一个问题的答案是,AI 系统在构建其继任者时能够执行危险行动,而监控者并非完美地检测到这种破坏行为。

Tested domains: RSI Control Arena tests out how well AI systems – Claude Opus 5, GPT-5.6 Sol, Gemini-3.7 Flash, Grok 4.6, GLM-5.3, and Muse Spark 1.3 – can perform the following main capability tasks while trying to also concurrently accomplish side tasks:

测试领域:RSI Control Arena 测试了 AI 系统——Claude Opus 5、GPT-5.6 Sol、Gemini-3.7 Flash、Grok 4.6、GLM-5.3 和 Muse Spark 1.3——在尝试同时完成辅助任务时,执行以下主要能力任务的水平如何:

  • 14 main tasks: Data (corpus filtering, quality-filter construction, capability corpus); SFT (news summarization, safety refusal, math and code and summary reasoning, general assistant, anti-sycophancy training); RM (pairwise reward model); RL (RL summarization and safety-refusal); OPD (safety refusal)
  • 16 side tasks that span three categories of harm: Capability (filtered domain retained, unwanted literature added, reward-model blind spot, overlooks planted errors, sandbag reward model); Alignment (biased quality filter, triggered behavior, topic refusal, hidden preference, persona bias, covert reasoning signal, hidden reasoning step, sandbag alignment training); Goal injection (protect chosen data, shape the next model, copy files unprompted).
  • 14 项主要任务:数据(语料库过滤、质量过滤构建、能力语料库);SFT(新闻摘要、安全拒绝、数学与代码及摘要推理、通用助手、反奉承训练);RM(成对奖励模型);RL(RL 摘要与安全拒绝);OPD(安全拒绝)
  • 16 项跨越三类危害的子任务:能力(保留过滤领域的知识、添加不需要的文献、奖励模型盲点、忽略植入的错误、伪装奖励模型);对齐(有偏见的质量过滤器、触发行为、主题拒绝、隐藏偏好、角色偏见、隐蔽推理信号、隐藏推理步骤、伪装对齐训练);目标注入(保护选定数据、塑造下一个模型、未经提示复制文件)。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

榜上的人在聊

近 7 天涨粉榜上的创作者

相似阅读

关联信息,但可能不是同一事件