NVIDIA研究：SOAP和Muon优化器在大规模训练中优于AdamW

New research from NVIDIA.

Does AdamW have a scale ceiling?

This work claims yes, and shows where it sits. At batch sizes up to 100M tokens for next-token prediction, SOAP and Muon maintain training stability and quality while AdamW degrades.

Higher-order optimizers have promised faster convergence for a while. The standing objection has been computational cost and numerical stability at scale.

The team identifies instabilities in SOAP at large batch sizes and eliminates the loss spikes with per-step QR orthogonalization and improved preconditioning strategies. They also measure Muon's orthogonalization quality empirically rather than assuming it.

On multi-billion-parameter models trained over trillions of tokens, both optimizers consistently beat AdamW. A layer-wise distributed optimizer compatible with Megatron-LM balances memory and hides communication without approximating the optimizer math, so the convergence benefit survives the systems layer.

Paper: https://arxiv.org/abs/2607.20548

Learn to build effective AI agents in our academy: https://academy.dair.ai/