Meta FAIR与NYU研究:多模态视觉编码中预训练非必需但专用参数不可省
[Meta FAIR + NYU] Unbundling Multimodal Vision Encoding: training 200+ MLLMs to find encoder pretraining is optional but vision-only parameters are not [R]
这篇论文用200+模型的严谨实验拆解了多模态架构设计的核心争议,结论直接指导工程取舍,做MLLM的同学值得细读。
X threads (with videos!):
X(推特)上的相关讨论(含视频!):
- https://x.com/choang333/status/2121335484236386574
- https://x.com/_ellisbrown/status/2121335477244428419
Paper: https://www.alphaxiv.org/abs/2611.17334
论文:https://www.alphaxiv.org/abs/2611.17334
Blog and FAQ: https://unbundling.github.io
博客与常见问题解答:https://unbundling.github.io
Labs disagree on vision encoders right now. Gemma 4 12B dropped the encoder, Kimi K3 trains its encoder from scratch, and Inkling 2 Small brought an encoder back. Published models can't settle this, because they differ in data and compute and rarely account for encoder pretraining. A standard encoder also bundles three choices: separate pretraining, parameters that only vision tokens use, and placement before the LLM. We tested each choice separately.
目前各实验室对视觉编码器存在分歧。Gemma 4 12B 移除了编码器,Kimi K3 从头训练其编码器,而 Inkling 2 Small 则重新引入了编码器。已发布的模型无法就此达成共识,因为它们的数据和算力不同,且很少考虑编码器的预训练因素。标准的编码器捆绑了三个选择:单独的预训练、仅供视觉令牌使用的参数,以及放置在 LLM 之前的位置。我们分别测试了每个选择。
Setup:
设置:
- 200+ models, all trained from scratch (LLM included) on the same data.
- Lifetime compute: encoder pretraining counts against the same budget as multimodal training.
- Main comparisons use Qwen3-1.7B and 4B configurations with 0.4B parameters of vision capacity. A scaling ladder covers 0.6B to 6.4B backbone sizes.
- Evaluations are on 16 VQA benchmarks from Cambrian at 448 px. and 12 language benchmarks testing commonsense, QA, and broad knowledge.
- 200 多个模型,均在相同数据上从头训练(包括 LLM)。
- 总计算量:编码器预训练计入与多模态训练相同的预算中。
- 主要比较使用 Qwen3-1.7B 和 4B 配置,其中包含 0.4B 参数的视觉容量。扩展阶梯覆盖 0.6B 到 6.4B 的主干大小。
- 评估基于 Cambrian 的 16 个 VQA 基准(448 px)和 12 个语言基准(测试常识、问答和广泛知识)。
Findings:
发现:
- Separate pretraining is optional. Encoders pretrained with four objectives (contrastive, self-supervised, two captioning) at best tie an encoder trained from scratch with the LLM at matched compute. Even free pretraining doesn't help. Pretraining MetaCLIP 6.5x longer raised ImageNet linear-probe accuracy (80.4% → 84.4%) but not VQA.
- Vision-only parameters are required. Encoder-free models trail at every size from 0.6B to 6.4B and need about 3x the compute to catch up. Giving the LLM the encoder's parameters instead improves text, not VQA.
- Placement before the LLM is optional. "In-fusion" adds zero-initialized vision-only weights (attention projections and FFNs) to the LLM's first 8 layers, and text tokens never use them. It's on par with the encoder from 1.7B to 6.4B, with about 13% lower time to first token. At 1.7B, vision-only branches in layers 1–8 reach 45.0 VQA (encoder 44.4), versus 42.3 when text shares them and 35.1 in the last 8 layers.
- The same holds with a pretrained LLM (vision added to a pretrained 1.7B LLM in midtraining).
- 单独预训练是可选的。使用四种目标(对比、自监督、两种图像描述)预训练的编码器,在匹配算力的情况下,最多只能与从头开始与 LLM 一起训练的编码器持平。即使是免费的预训练也无济于事。将 MetaCLIP 的预训练时间延长 6.5 倍提高了 ImageNet 线性探测准确率(80.4% → 84.4%),但并未提升 VQA 表现。
- 需要仅用于视觉的参数。从 0.6B 到 6.4B 的所有尺寸下,无编码器模型的表现均落后,且需要约 3 倍的算力才能赶上。如果将编码器的参数赋予 LLM,则会提升文本能力而非 VQA。
- 放置在 LLM 之前是可选的。“融合式”(In-fusion)方法将零初始化的仅视觉权重(注意力投影和前馈网络 FFN)添加到 LLM 的前 8 层,文本令牌从不使用它们。在 1.7B 到 6.4B 的尺寸下,其表现与编码器相当,且首个令牌生成时间降低约 13%。在 1.7B 规模下,第 1–8 层中的仅视觉分支达到 45.0 的 VQA 分数(编码器为 44.4),而当文本共享这些权重时为 42.3,在最后 8 层中则为 35.1。
- 在使用预训练 LLM 的情况下同样适用(在中期训练中向预训练的 1.7B LLM 添加视觉模块)。
So vision needs its own parameters, but not its own pretraining or its own encoder.
因此,视觉需要其自己的参数,但不需要其自己的预训练或独立的编码器。
Disclosure: I'm one of the authors. Happy to answer questions!
披露:我是作者之一。欢迎提问!
更进一步:量化金融体系
看懂新闻只是起点——沿量化金融路径,把它变成能交付的工程能力