利用CPT与RAG训练本地LLM及评估实战复盘
Training a local LLM using CPT and RAG (with evals)
提供了一套完整的本地模型垂直领域微调工作流，特别是CPT与RAG结合的评估思路非常硬核，做私有化部署的同学可以直接照搬这套配置和测试方法。
I have gone through a series of experiments related to an interesting project where I try to teach a local llm a new domain through continued pretraining (CPT). The different experiments are spread across the four phases below:
我进行了一系列与一个有趣项目相关的实验，该项目旨在通过持续预训练（CPT）教本地大语言模型学习一个新领域。不同的实验分布在以下四个阶段中：
- Phase1 talks about how to teach an llm a new domain through CPT and picking a training set that will generalize well to unseen questions
- Phase 2 does a comparison between the performance of reasoning across internalized knowledge (CPT) vs. RAG injected content
- Phase 3 takes a more practical approach where the CPT trained knowledge is enriched by combining it with RAG instead of viewing the two approaches as competing solutions
- The final part shows the comprehensive eval strategy used to measure performance during the project. Among other things, this involved SFT fine tuning of the CPT trained model to teach it to output responses based on a strict schema instead of English sentences. The schema approach is used to simplify strict eval checks.
- 第一阶段探讨了如何通过 CPT 教大语言模型学习新领域，并选择一个能很好地泛化到未见问题的训练集。
- 第二阶段比较了基于内化知识（CPT）的推理性能与注入 RAG 内容的性能。
- 第三阶段采取了一种更实用的方法，即通过将 CPT 训练的知识与 RAG 结合来丰富知识，而不是将这两种方法视为相互竞争的方案。
- 最后一部分展示了项目中用于衡量性能的综合评估策略。其中包括对经过 CPT 训练的模型进行 SFT 微调，以教会它根据严格的模式输出响应，而不是使用英语句子。采用模式方法是为了简化严格的评估检查。
The local model used for this project is qwen 3.5 4B. Unsloth was used for both CPT and SFT LORA training.
本项目使用的本地模型是 qwen 3.5 4B。Unsloth 被用于 CPT 和 SFT LORA 训练。
I have provided a summary of my findings here in case someone is interested in reading more about it: https://www.teachmecoolstuff.com/viewarticle/domain-specific-training-and-fine-tuning-of-an-llm
我在这里提供了我的发现摘要，以防有人有兴趣阅读更多内容：https://www.teachmecoolstuff.com/viewarticle/domain-specific-training-and-fine-tuning-of-an-llm
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力