架构小选择影响长上下文扩展：26个7B模型对比研究
Some architecture choices that make transformers cheaper or more stable also mak…
Some architecture choices that make transformers cheaper or more stable also make them harder to extend to long context.
某些使变换器更廉价或更稳定的架构选择，同时也让它们难以扩展到长上下文。
A model can look completely healthy during short-context pretraining and still be a bad candidate for long context later.
一个模型在短上下文预训练期间可能看起来完全健康，但后来却可能成为长上下文的糟糕候选者。
This paper tests 26 comparable 7B models while keeping the data, tokenizer, and context-extension recipe fixed.
本文在保持数据、分词器和上下文扩展配方不变的情况下，测试了26个可比的7B模型。
The only differences are 4 architecture choices already used across Llama, Qwen, and Olmo: QK normalization, grouped-query attention, sliding-window attention, and shorter pretraining context.
唯一的区别是Llama、Qwen和Olmo中已使用的4种架构选择：QK归一化、分组查询注意力、滑动窗口注意力和较短的预训练上下文。
Individually, most of them hurt only a little. Stack several together and the damage compounds.
单独来看，它们中的大多数只造成轻微损害。但若叠加几个，损害就会累积。
On HELMET at 32K, scores ranged from 29.9 to 56.4, even though standard short-context evaluations barely separated the models.
在HELMET的32K上下文中，分数范围从29.9到56.4，尽管标准的短上下文评估几乎无法区分这些模型。
One example is especially clear: adding sliding-window attention cost just 1.1 points without grouped-query attention, but about 9 points on average when grouped-query attention was already present.
一个例子尤为明显：在没有分组查询注意力时，添加滑动窗口注意力仅损失1.1分，但当分组查询注意力已存在时，平均损失约9分。
More long-context training did not erase the gap either: the worst architecture after 50B extension tokens still trailed the Llama-like architecture after only 1B.
更多的长上下文训练也未能消除差距：在50B扩展token后，最差的架构仍落后于仅用1B token的类Llama架构。
So long-context capability is partly decided before long-context training even begins. Architecture needs an early long-context stress test, not just good short-context loss.
因此，长上下文能力在长上下文训练开始前就已部分决定。架构需要早期长上下文压力测试，而不仅仅是良好的短上下文损失。
– arxiv. org/abs/2608.10296
– arxiv.org/abs/2608.10296
Title: "Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension"
标题：“基础中的裂缝：看似微小的架构选择影响长上下文扩展”
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力