跳到主内容
@wquguru
精选86r/LocalLLaMA(Reddit)模型发布/更新

GAIR发布801-30B-A3B:首次将递归自改进应用于前沿预训练

[Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon

原文
发到 X
推荐理由

这是RSI从后训练向预训练延伸的首个重磅实证,直接挑战大模型构建的核心范式,对追求极致效率的从业者极具参考价值。

Pre-training is where every foundation model begins; bringing recursive self-improvement (RSI) there would let AI improve the process that creates AI. Yet public RSI work has so far targeted only post-training, which starts from an existing base model, runs trials costing GPU-hours, and returns verifiable scores. Pre-training breaks all three conditions at orders-of-magnitude higher cost: a run consumes hundreds of thousands of GPU-hours over months and cannot be repeated; its objective, post-training headroom, cannot be read off any single score; and what must improve is not a model but the whole production system, from data pipelines and training infrastructure to the teams that run them. To our knowledge, we are the first to apply RSI to an entire frontier pre-training run from scratch and to report it in detail. RSI does not stop at frontier pre-training, but it changes form: since no end-to-end loop can close over a single unrepeatable run, we decompose it into smaller loops that can, each repairing a step the full run breaks. The result is 801-30B-A3B-Base, a 30B-A3B MoE trained from scratch on 11.8T tokens in three months by a newly formed team. We release it as a base model; the lightweight SFT reported here (6B tokens, no RL) only probes post-training headroom. As a base model, 801-30B-A3B outperforms Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, trailing only the larger Qwen3.5-35B-A3B, and keeps this position under identical SFT; with lightweight SFT alone, it achieves the highest average on four agentic benchmarks against fully post-trained models, including Qwen3.6-35B-A3B. Since post-training RSI presupposes a strong base model, bringing RSI into pre-training acts on the foundation every later stage builds on. We open-source 801-30B-A3B-Base and the lightweight SFT data at this https URL .

预训练是每一个基础模型的起点;在此阶段引入递归自我改进(RSI)将使 AI 能够改进创造 AI 的过程。然而,迄今为止公开的 RSI 工作仅针对后训练阶段,该阶段从现有的基础模型出发,运行消耗 GPU 小时的试验,并返回可验证的分数。预训练在数量级上更高的成本下打破了这三个条件:一次运行会消耗数十万计的 GPU 小时,历时数月且无法重复;其目标——后训练的潜力空间——无法从任何单一分数中读取;而需要改进的不是模型本身,而是整个生产系统,包括数据管道、训练基础设施以及运营它们的团队。据我们所知,我们是首个将 RSI 应用于从头开始的整个前沿预训练过程并详细报告的人。RSI 并不止步于前沿预训练,但其形式发生了变化:由于无法在一个不可重复的运行中闭合端到端的循环,我们将其分解为更小的循环,每个循环修复完整运行所破坏的一个步骤。结果是 801-30B-A3B-Base,这是一个 30B-A3B MoE 模型,由一个新成立的团队在三个月内使用 11.8T tokens 从头训练而成。我们将其作为基础模型发布;此处报告的轻量级 SFT(6B tokens,无 RL)仅用于探测后训练潜力。作为基础模型,801-30B-A3B 优于 Qwen3-30B-A3B 和 Nemotron-3.5-Lightning-30B-A3B,仅落后于更大的 Qwen3.5-35B-A3B,并在相同的 SFT 下保持这一地位;仅通过轻量级 SFT,它在四个智能体基准测试中取得了最高的平均分,击败了完全经过后训练的模型,包括 Qwen3.6-35B-A3B。由于后训练 RSI 预设了一个强大的基础模型,将 RSI 引入预训练作用于后续每个阶段所建立的基础之上。我们在 https URL 处开源了 801-30B-A3B-Base 和轻量级 SFT 数据。

  • Paper : https://arxiv.org/abs/2611.10719
  • PDF : https://arxiv.org/pdf/2611.10719
  • GitHub : https://github.com/gair-nlp/801-30b
  • Datasets : https://huggingface.co/datasets/SII-GAIR-NLP/801-30B-A3B-tiny-sft-dataset
  • Models : Coming Soon
  • 论文:https://arxiv.org/abs/2611.10719
  • PDF:https://arxiv.org/pdf/2611.10719
  • GitHub:https://github.com/gair-nlp/801-30b
  • 数据集:https://huggingface.co/datasets/SII-GAIR-NLP/801-30B-A3B-tiny-sft-dataset
  • 模型:即将推出

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件