Soofi S 30B-A3B 预训练报告：基于 NVIDIA Nemotron 架构，27T 词元训练

Their newly released pretraining report reveals that Soofi S 30B-A3B is based on NVIDIA’s open Nemotron 3 Nano reference architecture, the same hybrid Mamba + Transformer MoE design with ~3B active parameters and nearly identical architectural choices

What’s actually new isn’t the architecture, but the training:

-~27 trillion training tokens -German deliberately up-weighted -End-to-end training on Deutsche Telekom’s Industrial AI Cloud -Full training recipe, hyperparameters and evaluation methodology released openly

Ngl, excited to see europe kinda trains their own models.