跳到主内容
@wquguru
精选88Microsoft Research(RSS)产品发布/更新

微软亚洲研究院开源 Agent Lightning v1.0:轻量级

Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses

原文
发到 X
推荐理由

Agent 训练是当前的工程热点,这个框架直接解决“训练与部署不一致”的核心痛点,且代码量极小便于快速上手验证,做 Agent 基建的同学值得收藏参考。

At a glance

概览

  • Harnessed Agentic RL: Microsoft Research Asia introduces a training paradigm in which the same agent harness used in deployment participates directly in reinforcement learning, removing the need to reimplement the agent inside the training framework.
  • Lightweight by design: Agent Lightning v1.0 delivers a complete agent RL control plane in roughly 3,500 lines of code.
  • Native Kubernetes support: agents run as standard Kubernetes jobs on self-managed clusters, cloud Kubernetes, or local infrastructure, with no dependency on paid commercial sandbox services.
  • Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14.6 percentage point gain, using only about 6,000 training samples based on open sourced dataset.
  • Harness 驱动的 Agentic RL:微软亚洲研究院引入了一种训练范式,其中部署时使用的同一 Agent Harness 直接参与强化学习,从而无需在训练框架内重新实现 Agent。
  • 轻量级设计:Agent Lightning v1.0 仅用约 3,500 行代码即提供了一套完整的 Agent RL 控制平面。
  • 原生 Kubernetes 支持:Agent 作为标准 Kubernetes Job 运行于自管集群、云 Kubernetes 或本地基础设施上,无需依赖付费的商业沙箱服务。
  • 数据高效的训练配方:端到端编码 Agent 流水线使 Qwen3.5-9B 在 SWE-bench Verified 上的 Pass@1 指标从 41.8% 提升至 56.4%,提升了 14.6 个百分点,且仅使用了基于开源数据集的约 6,000 个训练样本。

AI agents have evolved from single models to complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness that coordinates them from outside the model. Reinforcement learning (RL) is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. RL can make those agents better, but most agent RL systems require developers to reimplement the agent inside the training framework. That is costly, and it means the agent being trained is not quite the agent that gets deployed.

AI Agent 已从单一模型演变为由模型、工具和执行环境构建的复杂全栈系统。其能力越来越依赖于在模型外部协调它们的 Agent Harness。强化学习(RL)是一种 AI 系统通过试错进行学习的方法,其行动受奖励和惩罚的引导。RL 可以提升这些 Agent 的性能,但大多数 Agent RL 系统要求开发者在训练框架内重新实现 Agent。这不仅成本高昂,还意味着被训练的 Agent 与最终部署的 Agent 并不完全一致。

To address this, researchers at Microsoft Research Asia have introduced the Harnessed Agentic RL training paradigm and open-sourced a fully rebuilt Agent Lightning v1.0 (opens in new tab). Compared with the original, Agent Lightning, v1.0 puts more emphasis on staying lightweight, on integrating with real harnesses, and on a complete, reproducible agent RL training pipeline.

为解决这一问题,微软亚洲研究院的研究人员引入了 Harness 驱动的 Agentic RL 训练范式,并开源了全面重构的 Agent Lightning v1.0(在新标签页中打开)。与原版相比,Agent Lightning v1.0 更加强调保持轻量级、集成真实的 Harness 以及提供完整可复现的 Agent RL 训练流水线。

Agent Lightning v1.0 was rebuilt around Harnessed Agentic RL, with key improvements:

Agent Lightning v1.0 围绕 Harness 驱动的 Agentic RL 进行了重构,主要改进包括:

  • Lightweight: the entire framework is about 3,500 lines of code. Agent Lightning v1.0 implements a complete Harnessed Agentic RL system in a codebase that is small and clear enough to understand, modify, and extend.
  • Training on a real agent harness: agents reach the model through the large language model (LLM) proxy in Agent Lightning v1.0, leaving existing harness code unchanged.
  • Native Kubernetes support: agents run directly as Kubernetes jobs, without external commercial sandbox services. Self-managed clusters and local infrastructure alike can support rollouts at scale.
  • A complete coding agent training example: an end-to-end pipeline built on Qwen3.5-9B raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%, an absolute gain of 14.6 percentage points, using only about 6,000 training samples.
  • 轻量级:整个框架仅约 3,500 行代码。Agent Lightning v1.0 在一个规模小、结构清晰、易于理解、修改和扩展的代码库中实现了完整的 Harness 驱动 Agentic RL 系统。
  • 在真实 Agent Harness 上进行训练:在 Agent Lightning v1.0 中,Agent 通过大语言模型(LLM)代理访问模型,现有的 Harness 代码保持不变。
  • 原生 Kubernetes 支持:Agent 直接作为 Kubernetes Job 运行,无需外部商业沙箱服务。自管集群和本地基础设施均能支持大规模部署。
  • 一个完整的编码智能体训练示例:基于 Qwen3.5-9B 构建的端到端流水线,仅使用约 6,000 个训练样本,便将 SWE-bench Verified 上的 Pass@1 从 41.8% 提升至 56.4%,绝对增益达 14.6 个百分点。

The limits of traditional agentic RL

传统智能体强化学习的局限

Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action, so the whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built this way, which meant training an agent required rebuilding its loop inside the RL framework.

传统智能体强化学习假设训练框架拥有与环境的交互循环。在 ReAct 风格的循环中,模型生成动作,环境返回观察结果,观察结果被追加到上下文中,然后模型生成下一个动作,因此整个 rollout 映射为一条连续的 token 轨迹。早期的 RL 系统如 verl、AReaL 和 slime 都是以此方式构建的,这意味着训练智能体需要在 RL 框架内重建其循环。

Real harnesses have outgrown that assumption. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies, as do general-purpose agent systems. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave in the same way as the deployed agent.

实际的 harness(工具/框架)已超出该假设的适用范围。mini-SWE-agent、OpenHands、OpenCode、Claude Code 和 Codex 等编码智能体各自带来其特有的上下文管理、工具协议、执行逻辑和依赖项,通用智能体系统也是如此。为训练目的重建其中一个成本高昂,且重建后的智能体可能不再表现出与部署时相同的行为了。

Agent Lightning takes a different route. It places an LLM proxy between the agent and the model. The agent continues to run as before: simply point the endpoint that previously called the model API at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, the researchers go further and formally define this paradigm as Harnessed Agentic RL: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training (Figure 1).

Agent Lightning 采取了不同的路径。它在智能体和模型之间放置了一个 LLM 代理。智能体继续按原有方式运行:只需将此前调用模型 API 的端点指向 Agent Lightning,训练框架即可观察并记录其模型调用。在 v1.0 版本中,研究人员更进一步,正式将此范式定义为 Harnessed Agentic RL(受控智能体强化学习):部署时使用的任何智能体 harness,在训练期间都直接参与强化学习(图 1)。

Figure 1. Traditional agentic RL compared with Harnessed Agentic RL. In traditional agentic RL, the training framework manages the environment and the agent loop. In Harnessed Agentic RL, the harness manages both.

图 1. 传统智能体强化学习与 Harnessed Agentic RL 的对比。在传统智能体强化学习中,训练框架管理环境和智能体循环。在 Harnessed Agentic RL 中,harness 同时管理两者。

Four challenges in training with real harnesses

使用真实 harness 进行训练的四大挑战

A core difference between Harnessed Agentic RL and traditional agentic RL is that the environment interaction loop is handled by the agent harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. This brings four key challenges:

Harnessed Agentic RL 与传统智能体强化学习的核心区别在于,环境交互循环由智能体 harness 而非训练框架处理。训练系统只能观察到一系列 LLM 请求和响应对,因此单个 rollout 可能被拆分为数量可变的训练样本。这带来了四个关键挑战:

  • Retokenization and sample merging: Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.
  • Advantage calculation: Retokenization, subagents and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.
  • Loss normalization: Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must also avoid being distorted by it.
  • Training backend scheduling: Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.
  • 重新分词与样本合并:Harness 将上下文保留为文本形式,但强化学习训练需要 rollout 期间采样的 token ID。再次通过聊天模板和分词器处理文本可能会改变 token 边界,因此相邻调用不能总是合并为一个样本。
  • 优势计算:重新分词、子代理和上下文摘要可能将一个 rollout 拆分为多个样本。直接在样本层面计算基线和优势会导致产生更多样本的 rollout 被重复计数,从而改变 rollout 层面的原始统计关系。
  • 损失归一化:按样本数量平均损失会使产生更多样本的 rollout 获得更大的权重。由于样本数量往往只是 harness 行为的产物,损失归一化也必须避免受其扭曲。
  • 训练后端调度:样本数量和长度只有在 harness 完成后才已知,而 GPU 数量和数据/张量并行配置通常是固定的。后端必须将可变的工作负载映射到固定资源中。

PODCAST SERIES

播客系列

The AI Revolution in Medicine, Revisited

医学领域的 AI 革命,再探

Join Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.

加入微软的 Peter Lee,一同探索人工智能如何影响医疗保健以及这对医学未来意味着什么。

Listen now

立即收听

Opens in a new tab

在新标签页中打开

Building a complete agent RL control plane in 3,500 lines of code

用 3,500 行代码构建完整的智能体强化学习控制平面

In system design, Agent Lightning v1.0 treats simplicity as its first principle. The entire framework is about 3,500 lines of code, with three core components: the API Gateway, the Rollout Controller, and the Customized Trainer (Figure 2).

在系统设计中,Agent Lightning v1.0 将简洁性作为其首要原则。整个框架约 3,500 行代码,包含三个核心组件:API 网关、Rollout 控制器和定制训练器(图 2)。

The API gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The rollout controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The customized trainer, built on verl, creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. As a result, for an existing agent harness, simply pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.

API 网关存储 rollouts、模型和事件,并充当兼容 OpenAI 的 LLM 代理。它将来自 harness 的每次模型调用与其对应的 rollout 关联起来,并记录训练所需的提示词、响应和对数概率。Rollout 控制器启动和管理智能体执行,可以是本地进程或标准的 Kubernetes 作业,使智能体执行与训练器分离。基于 verl 构建的定制训练器创建 rollouts,等待它们完成,收集样本,并通过样本适配器组装最终的训练样本。因此,对于现有的智能体 harness,只需将模型端点指向 Agent Lightning 代理,通常就足以快速连接到强化学习训练。

Figure 2. The Agent Lightning v1.0 system architecture showing the API gateway, rollout controller, and customized trainer.

图 2. Agent Lightning v1.0 系统架构,展示了 API 网关、rollout 控制器和定制训练器。

Collocated async RL

同地异步强化学习

Rollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle, while fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. In response, Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.

不同智能体的 rollout 时间差异很大。同步强化学习需要等待批次中最慢的智能体,导致 GPU 空闲;而完全异步强化学习虽然提高了利用率,但需要为 rollout 和训练分别设置独立的 GPU 池。为此,Agent Lightning v1.0 引入了协同位置异步强化学习(Collocated Async RL),使 rollout 和模型更新能够共享同一组 GPU。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件