Templar提出流水线并行训练中的阶段跳过容错机制
Simulating fault tolerance with stage skipping in pipeline-parallel training [R]
针对大规模分布式训练稳定性痛点，提供了具体的流水线并行容错工程方案与模拟数据，对从事底层训练框架优化的同学极具参考价值。
Our most recent work at Templar explores fault tolerance in Crucible, our distributed pre-training platform. The goal is to keep healthy workers training when another pipeline stage goes offline.
我们在 Templar 的最新工作探索了 Crucible（我们的分布式预训练平台）中的容错能力。目标是在另一个流水线阶段下线时，让健康的节点继续训练。
Crucible combines data-parallel replicas with pipeline parallelism. Each replica holds a copy of the model, split into stages on separate workers. SparseLoCo exchanges compressed updates between replicas, while pipeline compression reduces the communication across stage boundaries.
Crucible 将数据并行副本与流水线并行相结合。每个副本持有模型的副本，该模型被分割成多个阶段并分布在不同的节点上。SparseLoCo 在副本之间交换压缩后的更新，而流水线压缩则减少了跨阶段边界的通信量。
We combine those methods with stage skipping. When an inner stage goes offline, activations and gradients bypass it for multiple steps. Healthy stages keep processing tokens instead of waiting for recovery. The bypass omits the unavailable stage’s computation.
我们将这些方法与阶段跳过相结合。当某个内部阶段下线时，激活值和梯度会在多个步骤中绕过它。健康的阶段会继续处理标记，而不是等待恢复。这种绕过机制省略了不可用阶段的计算。
The simulations use a 178M model, eight replicas and four stages per replica. At a 1% per-replica failure probability per global step, validation loss stayed close to the no-failure baseline, even though each simulated outage removed a stage for six global steps. Each configuration is compared with its own no-failure run.
模拟使用了一个拥有 1.78 亿参数的模型，每个副本包含八个副本和四个阶段。在全局步长层面，每个副本的故障概率为 1% 时，验证损失保持在无故障基线附近，尽管每次模拟中断都会导致一个阶段离线六个全局步长。每种配置都与其自身的无故障运行进行比较。
Fixed projections shared across layers improve robustness further when using pipeline compression. This suggests that shared projectors align representations across stage boundaries, making bypasses less disruptive. The alignment explanation remains a hypothesis.
当使用流水线压缩时，跨层共享的投影进一步提高了鲁棒性。这表明共享投影器使跨阶段边界的表示保持一致，从而使绕过操作的影响更小。关于对齐的解释仍是一种假设。
These results point toward training on a broader pool of compute, including unreliable workers and spot instances. This is a simulation of the learning effects of stage failures, rather than a measurement of physical worker replacement or production cost savings.
这些结果指向在一个更广泛的计算资源池上进行训练，包括不可靠的节点和竞价实例。这是对阶段故障学习效果的模拟，而非对物理节点更换或生产成本节省的测量。
The article includes the setup, comparisons and figures:
文章包含了设置、比较和图表：
https://www.tplr.ai/publications/blog/skipping-stages-with-fixed-projections
https://www.tplr.ai/publications/blog/skipping-stages-with-fixed-projections
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力