跳到主内容
@wquguru
精选88Hacker News Best(web_list)论文研究

Dust:无需反向传播预训练Transformer的零阶优化方法

Dust: Pretraining Transformers Without Backpropagation

原文
发到 X
推荐理由

挑战深度学习基石反向传播的新范式,提供可复现的零阶优化方案,对探索高算力下的模型训练路径极具参考价值。

TL;DR

TL;DR(太长不看)

  • We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models. Dust perturbs activations (node perturbation) independently at every token, so each token is a virtual population member and one forward pass evaluates them all in parallel.
  • Dust approximates backprop closely at large population (i.e. substantially more compute) and in multiple settings even exceeds it. This hints that in a compute-rich regime we might be able to surpass backprop.
  • Dust is orders of magnitude more efficient than weight-space ES. From 1M tokens up, Dust is on the order of $10^3$ to $10^4$ times more efficient than a transformer implementation of EGGROLL, a state-of-the-art ES method, based on our extrapolations.
  • Zeroth-order methods are widely believed not to scale to large networks. Strikingly, we find larger models are more population-efficient, not less: a 243M-parameter model outperforms a $120\times$ smaller model at most population sizes.
  • Dust’s gradient estimates align better with backprop’s as population grows, and stay well aligned at every scale we test, up to 1B tokens, which is encouraging for scaling.
  • 我们提出了第一种在预训练 Transformer 语言模型时能与反向传播相竞争的一阶为零方法。Dust 在每个 token 处独立地对激活值进行扰动(节点扰动),因此每个 token 都是一个虚拟种群成员,一次前向传播即可并行评估它们。
  • 在大种群规模下(即显著更多的计算量),Dust 能紧密逼近反向传播的效果;并且在多种设置下甚至超越它。这暗示在计算资源丰富的情况下,我们或许能够超越反向传播。
  • Dust 的效率比基于权重的进化策略(ES)高出几个数量级。根据我们的外推,从 100 万 token 开始,Dust 的效率比 EGGROLL(一种最先进的 ES 方法)的 Transformer 实现高出 $10^3$ 到 $10^4$ 倍。
  • 人们普遍认为一阶为零方法无法扩展到大型网络。令人惊讶的是,我们发现更大的模型具有更高的种群效率,而非更低:在大多数种群规模下,拥有 2.43 亿参数的模型优于参数量小 $120\times$ 的模型。
  • 随着种群规模的增大,Dust 的梯度估计与反向传播更加一致,并且在我们测试的所有尺度上(高达 10 亿 token)都保持良好的对齐,这对扩展性而言是一个积极的信号。

1 Introduction

1 引言

Deep learning has been built around backprop, the only credit assignment algorithm capable of training modern neural nets, including transformer-based language models. Backprop requires differentiability and produces first-order gradients, and deep learning’s architectures, optimizers, and hardware have co-evolved around this constraint.

深度学习一直是围绕反向传播构建的,它是唯一能够训练现代神经网络(包括基于 Transformer 的语言模型)的信用分配算法。反向传播要求可微分并产生一阶梯度,深度学习的架构、优化器和硬件都是围绕这一约束共同演化的。

However, as the amount of compute available in the world increases, we might prefer more generic and brute-force learning algorithms based on search over inductive biases like differentiability, backprop, and approximations of higher-order gradients. The bitter lesson (Sutton, 2019Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019. Blog post.) is that general methods that scale with compute eventually win, and AlphaGo Zero (Silver et al., 2017David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550 (7676): 354–359, 2017. doi: 10.1038/nature24270.) is the obvious example. Bootstrapping AlphaGo on human data helped the network learn faster initially, but with a lot of computation the purely self-play network overtook it. Similarly, differentiability and backprop might be good inductive biases in the low-compute regime, where they make learning efficient, but in the high-compute regime they limit the space of architectures that work. Even within an architecture, gradient-based methods fail to explore the loss landscape optimally (Liu et al., 2020Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. In Advances in Neural Information Processing Systems, volume 33, 2020.). This might also explain why current neural nets require massive amounts of data to generalize. A more flexible credit assignment algorithm based on search is likely an important step towards much better generalization.

然而,随着全球可用计算量的增加,我们可能更倾向于使用基于搜索的更通用和蛮力的学习算法,而不是依赖如可微分性、反向传播和高阶梯度近似等归纳偏置。苦涩教训(Sutton, 2019Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019. Blog post.)指出,随计算量扩展的通用方法最终会获胜,而 AlphaGo Zero (Silver et al., 2017David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis.无需人类知识即可掌握围棋游戏。《自然》,550 (7676): 354–359, 2017. doi: 10.1038/nature24270.) 是一个显而易见的例子。在人类数据上对 AlphaGo 进行自举(Bootstrapping)有助于网络初期更快地学习,但经过大量计算后,纯自我对弈的网络超越了它。同样地,可微性和反向传播在低算力环境下可能是良好的归纳偏置,因为它们使学习高效;但在高算力环境下,它们限制了有效架构的空间。即使在单一架构内,基于梯度的方法也无法最优地探索损失景观(Liu et al., 2020Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. In Advances in Neural Information Processing Systems, volume 33, 2020.)。这也可能解释了为什么当前的神经网络需要海量数据才能泛化。一种基于搜索的更灵活的信用分配算法可能是实现更好泛化的重要一步。

In this paper, we aim to replace backprop with a learning algorithm based much more on brute-force computation and much less on analytic structure. We call it Dust. Dust is a zeroth-order optimization algorithm that perturbs activations, rewards each perturbation by how much it lowers the loss, and averages the reward-weighted perturbations over a population to estimate the gradient. Traditional ES methods that perturb weights (Salimans et al., 2017Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.), like EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.), scale with population, but scaling the population is costly because each member must be materialized and evaluated. We remove both costs with the concept of virtual population, where we avoid materializing every member by bypassing weight space entirely and instead perturb activations, as in node perturbation (Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. In Advances in Neural Information Processing Systems, volume 16, 2003.; Widrow and Lehr, 1990Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE, 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323.). We do so independently at every token, so each token is a member and one forward pass evaluates them all in parallel.

在本文中,我们旨在用一种主要基于暴力计算、较少依赖解析结构的学习算法来替代反向传播。我们将其命名为 Dust。Dust 是一种零阶优化算法,它扰动激活值,根据扰动降低损失的程度给予奖励,并在种群中平均这些加权扰动的奖励以估计梯度。传统扰动权重的 ES 方法(Salimans et al., 2017Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever.进化策略作为强化学习的可扩展替代方案。arXiv 预印本 arXiv:1703.03864, 2017.), 如 EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster.超大规模下的进化策略。arXiv 预印本 arXiv:2511.16652, 2025.), 随种群规模扩展,但扩展种群成本高昂,因为每个成员都必须被实例化并评估。我们通过“虚拟种群”的概念消除了这两项成本,在该概念中,我们通过完全绕过权重空间并改为扰动激活值(如同节点扰动 (Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H.)来避免实例化每个成员。Sebastian Seung。线性前馈网络中随机梯度下降的学习曲线。在 Advances in Neural Information Processing Systems,第16卷,2003年;Widrow和Lehr,1990 Bernard Widrow 和 Michael A. Lehr。自适应神经网络的30年:感知机、Madaline 和反向传播。IEEE Proceedings,78 (9): 1415–1442,1990。doi: 10.1109/5.58323)。我们在每个 token 上独立执行此操作,因此每个 token 都是一个成员,一次前向传递并行评估它们所有。

Activations are a more interesting space to search over than weights. Mechanistic interpretability has shown that reasoning, whether verbalizable or not, lives in the activations (Gurnee et al., 2026Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a global workspace in language models. arXiv preprint arXiv:2607.15495, 2026.; Lindsey et al., 2025Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al. On the biology of a large language model. Transformer Circuits Thread, 2025.), which means this approach could turn training into a search over latent reasoning (Vegesna and Dahal, 2025Akshay Vegesna and Samip Dahal. Decoupling search and learning in neural net training. arXiv preprint arXiv:2509.10973, 2025.). We then pair the activation-space perturbation with a very generic credit assignment rule that assigns different token-level rewards to different layer types in a transformer block. Those two biases, along with a few implementation details and efficiency measures, like avoiding interference between perturbed modules, are the whole algorithm.

激活空间比权重空间更有趣,值得搜索。机械可解释性研究表明,推理(无论是否可言语化)存在于激活中(Gurnee 等人,2026 Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey。可言语化的表示在语言模型中形成全局工作区。arXiv预印本 arXiv:2607.15495,2026;Lindsey 等人,2025 Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al。大型语言模型的生物学。Transformer Circuits Thread,2025),这意味着这种方法可以将训练转化为对潜在推理的搜索(Vegesna 和 Dahal,2025 Akshay Vegesna 和 Samip Dahal。解耦神经网络训练中的搜索和学习。arXiv预印本 arXiv:2509.10973,2025)。然后我们将激活空间扰动与一个非常通用的信用分配规则配对,该规则为 Transformer 块中的不同层类型分配不同的 token 级奖励。这两种偏差,加上一些实现细节和效率措施(例如避免扰动模块之间的干扰),构成了整个算法。

We make the following contributions.

我们做出了以下贡献。

  • We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models. At large populations Dust exceeds backprop in multiple settings, which suggests that in a compute-rich regime we might be able to surpass backprop.
  • Dust is orders of magnitude more efficient than weight-space ES. From 1M tokens up, Dust is on the order of $10^3$ to $10^4$ times more efficient than a transformer implementation of EGGROLL, based on our extrapolations.
  • Contrary to conventional wisdom, larger models are often more population-efficient, not less, and can make use of larger populations. This gives a new view of overparameterization as a larger search space with potentially better geometry.
  • Dust’s gradient estimates align better with backprop’s as the population grows, and the alignment holds up at every scale we test, up to 1B tokens, which is encouraging for scaling.
  • 我们提出了第一种在预训练 Transformer 语言模型方面与反向传播具有竞争力的零阶方法。在大规模种群中,Dust 在多种设置下优于反向传播,这表明在计算丰富的情况下,我们可能能够超越反向传播。
  • Dust 的效率比基于权重的 ES 高出几个数量级。从 100 万 token 开始,根据我们的外推,Dust 的效率比基于 Transformer 实现的 EGGROLL 高出 $10^3$ 到 $10^4$ 倍。
  • 与传统观点相反,更大的模型通常具有更高的种群效率(population-efficient),而非更低,并且能够利用更大的种群。这为过参数化(overparameterization)提供了新的视角:将其视为一个具有潜在更优几何结构的更大搜索空间。
  • 随着种群规模的扩大,Dust 的梯度估计与反向传播(backprop)的梯度估计更加一致,且这种一致性在我们测试的所有规模下均保持稳定,直至 1B tokens,这对扩展性而言是一个令人鼓舞的信号。

更进一步:量化金融体系

看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力

进入量化体系 →

相似阅读

关联信息,但可能不是同一事件