开发者在8GB显存笔记本上训练动态增长MoE模型
mini-AGI - dynamically grown (530M params currently and growing) continual learning model trained from scratch on 8GB VRAM laptop from batch-1 stream of data.
在极低成本硬件上实现从头训练且支持动态扩展的MoE方案，提供了极具参考价值的工程实践路径，适合关注端侧部署与高效训练的开发者深入研读。
Sorry for the pretentious name, I know, I know.. It just contains all the pieces I would like to see a AGI model to have, and I can't stand the temptation. Before throwing rocks at me, please take a glance at the Readme, and I hope it will cover your mood a little bit.
抱歉用了个故作高深的名字，我知道，我知道……它只是汇集了我希望看到一个 AGI 模型所具备的所有要素，而我实在无法抵挡这种诱惑。在向我扔石头之前，请先浏览一下 Readme，我希望它能稍微缓解你的情绪。
So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/heads/main/runs/samples.txt
首先，它是有效的，你可以在这里看到整个训练过程的样本：https://raw.githubusercontent.com/volotat/mini-AGI/refs/heads/main/runs/samples.txt
Here is the scaling law graph I have so far, and it looks very promising:
以下是我目前得到的缩放定律图表，看起来非常有前景：
https://github.com/volotat/mini-AGI/blob/main/assets/scaling.png
https://github.com/volotat/mini-AGI/blob/main/assets/scaling.png
The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.
该模型是在我极度不满的背景下构建的，因此我们无法在消费级硬件上真正训练即使是中等规模的模型（1B+ 参数规模）。我们当然可以进行推理和微调，但我希望对模型在训练过程中所见的内容拥有完全的控制权，使其完全符合我的利益，而非某些公司的利益。
I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.
我思考了一段时间，提出了两个我认为值得深入探索的有趣想法：一是使用大量专家的混合专家模型（MoE），在训练过程中动态添加和剪枝专家，使得在任何特定时刻只有少量专家子集处于实际使用状态；二是在单个连续数据流上进行 batch size 为 1 的训练。
First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.
第一种方法使我们在参数量、加载和卸载方面仅受限于磁盘空间，仅在需要时加载和卸载专家。第二种方法（如果研究透彻且被证明可行）使我们能够摆脱对小显存容量的依赖，因为我们不需要存储大型随机批次及其各自的梯度。
I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.
我开始与 Claude 进行头脑风暴，经过一段时间后，我们发现了一种看似有前景的方法，不出所料，几周后你就能亲眼看到结果了。
Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.
显然，我在制作这个项目的过程中使用了 AI，我非常确信如果没有它，我完全不可能做到类似的事情，所以我认为这完全是正当的。
The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.
该模型仍在运行我选定的第一个包含 78 亿字符的训练语料库，因此权重尚未发布，按照当前的读取速度，还需要等待大约两周才能完成训练。是的，模型只是从数据集中读取连续的交错段落，每段长度为 32K 字符，作为一个单一的数据流。就像你或我会做的那样。
The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.
设置似乎非常简单，你可以 git clone 该项目，运行它并亲自观察一切。
Thanks for your attention.
感谢你的关注。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力