MICA模型拆分为Ember与Flame，测试平衡训练混合效果
MICA update: the project has split into Ember and Flame, and I’m testing a balanced training mix
非Transformer架构的极致压缩模型探索，提供了完整的训练数据、比特率和生成问题的详细复盘，对研究小模型和替代架构的开发者极具参考价值。
In my previous post, I described MICA as a roughly 4.5 MB experimental language model built from learned integer rewrite rules. Since then, that R1 research line has developed into two models:
在我之前的文章中，我将 MICA 描述为一个大约 4.5 MB 的实验性语言模型，它由学习到的整数重写规则构建而成。此后，这条 R1 研究路线已发展为两个模型：
MICA Ember stays close to the original scaled model’s size, at roughly 4.5 MB.
MICA Ember 保持接近原始缩放模型的大小，约为 4.5 MB。
MICA Flame is a larger version, at about 17 MB.
MICA Flame 是更大的版本，约为 17 MB。
They share the same basic idea. MICA reads text byte by byte into a cellular tape. Learned integer scores select rules that update its state, and an integer readout predicts the next byte. Training uses a soft fitting process, but the exported models run through the integer-rule engine—without a Transformer, GRU, or external language model at inference time.
它们共享相同的基本理念。MICA 将文本逐字节读入细胞磁带（cellular tape）。学习到的整数分数选择规则来更新其状态，而整数读数则预测下一个字节。训练使用软拟合过程，但导出的模型通过整数规则引擎运行——在推理时不需要 Transformer、GRU 或外部语言模型。
The larger Flame model reached 1.786 bits/target on a clean 1,000-record chat set, compared with 1.847 for the previous chat-trained MICA. But Flame isn’t stronger everywhere: it scored 1.966 on everyday text, while our best earlier everyday-trained MICA scored 1.742. Lower is better.
较大的 Flame 模型在干净的 1,000 条记录聊天数据集上达到了 1.786 bits/target，而之前经过聊天训练的 MICA 为 1.847。但 Flame 并非在所有方面都更强：它在日常文本上的得分为 1.966，而我们之前最好的日常训练 MICA 得分为 1.742。得分越低越好。
I’ve also finished evaluating Ember with balanced training mix A using the exact integer engine. It scored 1.861 on chat and 1.876 on everyday text. That gives it a more even result across the two domains, but Flame still leads on chat and the earlier everyday model still leads on everyday text.
我也已完成使用精确整数引擎对 Ember 进行评估，使用的是平衡训练混合 A。它在聊天数据上的得分为 1.861，在日常文本上为 1.876。这使它在两个领域之间的表现更加均衡，但 Flame 在聊天任务上仍领先，而早期的日常文本模型在日常文本任务上仍领先。
The bigger problem remains generation. Mix A still reuses phrases when producing sentences. I’m measuring generated text separately from prediction loss, because a lower bits/target score hasn’t translated into coherent new sentences yet.
更大的问题仍然存在生成方面。混合 A 在生成句子时仍然会重复短语。我单独测量生成文本与预测损失，因为更低的 bits/target 分数尚未转化为连贯的新句子。
Balanced mix B is still training, so I haven’t chosen a balanced version. I’ll compare its exact integer scores and generation samples when it’s ready.
平衡混合 B 仍在训练中，因此我尚未选择平衡版本。待其完成后，我会比较其精确整数分数和生成样本。
I’d appreciate ideas from people who’ve worked on small sequence models:
我非常希望得到那些从事过小型序列模型工作的人的建议：
How would you tell whether repeated phrases are mainly a data-mix problem, a limited-context problem, or a decoding problem?
你会如何判断重复的短语主要是数据混合问题、上下文限制问题还是解码问题？
What would you measure alongside bits/target to track genuinely new, coherent sentences?
除了 bits/target 之外，你会测量哪些指标来追踪真正连贯的新句子？
What would be a fair baseline at Ember’s roughly 4.5 MB size and Flame’s roughly 17 MB size?
在 Ember 大约 4.5 MB 大小和 Flame 大约 17 MB 大小的情况下，什么是公平的基线？
The goals are still coherent new sentences and, eventually, around 1 bit/target. We haven’t reached either. I’ll keep posting updates as the experiments finish, including the ones that don’t work.
目标仍然是连贯的新句子，并最终达到约 1 bit/target。我们尚未实现任一目标。随着实验完成，我会继续发布更新，包括那些未成功的实验。
For transparency: I direct the design and experiments, Claude has done much of the implementation and experiment work, and I used AI to help draft this post.
为了透明起见：我负责设计和实验，Claude 完成了大部分实现和实验工作，我使用 AI 帮助起草了这篇文章。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力