RAG助手构建复盘：162篇论文检索管线与踩坑细节
Built RAG Assistant. T'was a Nightmare. Here's Pipeline + Everything That Broke
做RAG落地的同学必看，这篇提供了从数据清洗到评估的完整工程细节和避坑指南，特别是关于幻觉控制和评估纪律的部分极具复用价值。
Built a RAG research assistant over 162 academic papers that turns them into plain-language advice for practitioners, with a concrete action plan in every answer. Haiku pre-filters the corpus, hybrid search inside the shortlist, a second pass that hunts counter-evidence, citations on every claim, and evals that check whether the model invents disagreements (it did, a lot).
基于 162 篇学术论文构建了一个 RAG（检索增强生成）研究助手，将其转化为面向从业者的通俗建议，每个回答都附带具体的行动计划。Haiku 模型先对语料库进行预筛选，在候选列表中进行混合搜索，再进行一次反向证据搜寻，每个主张都提供引用，并通过评估检查模型是否编造了分歧（结果发现它确实经常这样做）。
The point of the whole thing, since the pipeline won't make sense without it: this isn't for academics. The library is research from my field and the reader is a practitioner. The goal is professional advice grounded in actual evidence, in plain language, ending in a concrete action plan. The bet: that beats the generic, confidently-wrong stuff you get from a model answering off vibes.
整个项目的核心所在，因为脱离这一点管道设计将毫无意义：这不是为学术界设计的。该库来自我的研究领域，读者是从业者。目标是提供基于实际证据、用通俗语言表达的专业建议，并以具体的行动计划结尾。我的假设是：这比那些仅凭感觉回答、自信满满却充满错误内容的通用模型输出要好得多。
Was supposed to be a small little value add to my app and ended up being a beast to get right. So thought I'd share what I've built and found out along the way.
原本只是打算给我的应用增加一个小价值点，结果发现要把这件事做好简直是个庞然大物。所以我想分享一下我在此过程中构建的内容和发现的教训。
The library (built once, painful the whole way)
文献库（一次性构建，过程痛苦不堪）
- Extraction. Papers become text, scanned pages through a vision model. That step silently dropped the back pages of 48 papers and I didn't notice for weeks. Re-extracted everything. Sample the output of every ingestion stage, not just the first.
- Summary cards. An LLM writes a short per-paper summary, index-card style. Pays off later: the pre-filter reads cards, not whole papers.
- Chunking into passage-sized pieces.
- Embeddings. Cohere embed-v5, one vector per chunk.
- Topic clustering. k-means gives a rough map of the library. Also told me something uncomfortable: one generic theory cluster holds 41% of it, and it dominates answers more than I'd like.
- 提取。论文被转换为文本，通过视觉模型扫描页面。这一步悄无声息地丢弃了 48 篇论文的末页，而我数周后才察觉。重新提取了所有内容。务必对每个摄入阶段的输出进行抽样检查，而不仅仅是第一阶段。
- 摘要卡片。LLM 为每篇论文撰写简短的摘要，采用索引卡片风格。后续收益显著：预筛选阶段阅读的是卡片而非整篇论文。
- 将内容切分为段落大小的片段。
- 嵌入向量。使用 Cohere embed-v5，每个片段对应一个向量。
- 主题聚类。k-means 算法提供了文献库的大致地图。同时也揭示了一个令人不安的事实：一个通用理论集群占据了 41% 的内容，并在回答中占据的主导地位比我预期的要高。
When someone asks a question
当有人提出问题时的处理流程
- Query expansion. The question gets reworded a few ways so retrieval doesn't hinge on one phrasing.
- LLM pre-filter. Haiku reads every paper's title plus the first 280 characters of its summary (~160 entries) and picks the 9-17 most relevant. If the picks all land in one cluster, the list widens to span at least two. This is where the bias concentrates: 41% of corpus becomes ~50% of picks.
- Hybrid search inside the shortlist. Semantic similarity and exact keywords, run in parallel, merged.
- Counter-evidence pass. Claim-shaped question ("does X improve Y?") triggers a second search for evidence AGAINST, even outside the shortlist. Without it, answers only ever heard one side.
- Rerank. A Cohere reranker scores each passage against the question, top few become the sources. I checked whether this causes the cluster bias: it doesn't. 56% before rerank, 59% after. The bias already exists.
- Theming. Passages get grouped into themes, and a labeller flags themes where papers genuinely disagree, so disagreement never becomes fake consensus.
- 查询扩展。将问题以多种方式重新表述，以便检索不依赖于单一的措辞。
- LLM 预筛选。Haiku 读取每篇论文的标题及其摘要的前 280 个字符（约 160 条条目），并选出最相关的 9-17 篇。如果选出的所有论文都集中在同一个集群，列表会扩大以至少涵盖两个集群。这是偏见集中的地方：语料库中的 41% 变成了约 50% 的选择结果。
- 在候选列表内进行混合搜索。语义相似度和精确关键词并行运行，然后合并结果。
- 反向证据搜寻。针对主张型问题（例如“X 是否能改善 Y？”），触发第二次搜索以寻找反对证据，即使这些证据不在候选列表内。如果没有这一步，回答永远只能听到一面之词。
- 重排序。Cohere 重排序器针对每个段落与问题进行打分，得分最高的几个段落成为来源。我检查了这是否会导致聚类偏差：结果是没有。重排序前为 56%，重排序后为 59%。偏差已经存在。
- 主题化。段落被归类到各个主题中，标注员会标记出论文确实存在分歧的主题，从而确保分歧不会变成虚假的共识。
Writing the answer
撰写答案
- The writer (Sonnet or Opus based on user selection) answers from retrieved passages only, in plain language, every claim carrying a numbered citation back to the paper.
- "Where the evidence splits." When two papers report opposite findings, the answer lays out both sides with citations. The hard part is the opposite failure: it must NOT appear when the research agrees. First version fired on 7 of 8 consensus answers. Inventing academic controversy. Tightened the instruction, got to 0, A/B running now with byte-identical sources to confirm it holds.
- Every answer ends with next steps. Concrete things to go do, capped per mode so the plan stays a plan.
- Long stable instruction prefix is prompt-cached.
- 写手（根据用户选择 Sonnet 或 Opus）仅从检索到的段落中作答，使用通俗语言，每一项主张都带有指向论文的编号引用。
- “当证据出现分歧时。”当两篇论文报告相反的研究结果时，答案会列出双方的观点并附带引用。难点在于避免反面情况：当研究结论一致时，绝不能出现这种表述。第一版在 8 个达成共识的答案中有 7 个触发了该逻辑，凭空制造学术争议。收紧指令后降至 0，目前正在运行 A/B 测试，使用字节级完全相同的来源以确认其稳定性。
- 每个答案都以下一步行动结尾。列出具体的行动建议，每种模式下数量上限固定，以确保计划保持为计划。
- 长而稳定的指令前缀已进行提示缓存。
Evals (build these first next time)
评估（下次构建时优先做这些）
I kept breaking things and finding out from users instead of tests. Bad enough normally, worse when they're asking research questions.
我一直在破坏功能并通过用户反馈而非测试来发现问题。这在正常情况下就很糟糕，而在用户询问研究问题时则更为严重。
- Retrieval: 33 golden questions with known-correct papers. nDCG@5, recall@5, MRR, plus filter-recall so I know WHICH stage broke.
- Honesty: does the answer keep the papers' "it depends" hedges? Invent a disagreement that isn't there? Show both sides when a real split exists?
- Topic balance: whole library, or just the generic cluster?
- A/B discipline: every change judged with/without. The latest re-run exists because an earlier comparison accidentally gave the two versions different sources, quietly invalidating the whole thing.
- 检索：使用 33 个已知正确答案的黄金问题集。指标包括 nDCG@5、recall@5、MRR，以及过滤召回率，以便我知道具体是哪个阶段出了问题。
- 诚实性：答案是否保留了论文中的“视情况而定”等谨慎表述？是否凭空捏造了不存在的分歧？当真实存在分歧时是否展示了双方观点？
- 话题平衡：是针对整个文献库，还是仅针对通用聚类？
- A/B 测试纪律：每次变更都要在有/无对比的情况下进行评估。最近的重跑之所以存在，是因为早期的比较无意中给两个版本分配了不同的来源，悄悄使整个比较失效。
Output modes
输出模式
Same retrieval pipeline for all four; only the writing instructions and token caps differ.
四种模式使用相同的检索管道；区别仅在于写作指令和令牌上限。
- Brief: ~150 words, hard stop 175. Max 3 evidence bullets, 2 next steps. Drop the less important points rather than squash everything in. Hardest mode, honestly: brevity sometimes cuts the losing side of a real evidence split. Still fails my evals. Known issue.
- Standard: 300-500 words, the default. Key evidence plus what it means in practice.
- Detailed: 500+ words. Deeper reasoning, worked examples.
- Ideas: doesn't summarize. Combines findings across papers to come up with new ideas/solutions. Responds in a fixed four-part format, every statement labelled in the UI by confidence:
- Finding (a paper says this),
- Synthesis (combining papers),
- Extrapolation (going beyond them),
- Hypothesis (testable idea). Evidence-splits deliberately off here; it would break the format.
- 简要：约 150 词，硬性上限 175 词。最多 3 条证据要点，2 个下一步行动。与其强行塞入所有内容，不如舍弃次要观点。老实说这是最难的模式：有时简洁会砍掉真实证据分歧中处于劣势的一方。目前仍未通过我的评估。已知问题。
- 标准：300-500 词，默认模式。包含关键证据及其实际意义。
- 详细：500+ 词。更深入的推理和工作示例。
- 创意：不进行总结。跨论文整合发现以提出新想法/解决方案。以固定的四部分格式响应，UI 中每条声明均标注置信度：
- 发现（某论文指出这一点），
- 综合（整合论文）、
- 外推（超越现有内容）、
- 假设（可检验的观点）。此处故意不拆分证据；否则会破坏格式。
Two things on top:
顶层有两件事：
- a classifier decides if you're asking to understand or act. A wrong guess costs a little focus but never deletes the next-steps section, because one misfire did exactly that and a user asked where they went.
- the idea-spinner proposes cross-paper "yes-and" combos before you even ask, then a verifier reads the actual passages (not the summaries) and drops unsupported ideas.
- 一个分类器决定你是要求理解还是行动。猜错只会损失少量注意力，但绝不会删除下一步部分，因为之前有一次误判导致该部分完全消失，用户因此询问它们去了哪里。
- 想法生成器在你提问之前就提议跨论文的“是且加”组合，随后验证器会阅读实际段落（而非摘要）并剔除缺乏支持的设想。
Where I want feedback
我希望获得反馈的地方
Next experiment: the Haiku pre-filter is cheap but still an LLM call reading ~160 titles and summaries per question, it's where the cluster bias concentrates, and it occasionally times out into no filtering at all.
下一个实验：Haiku 预过滤器成本较低，但仍是一次 LLM 调用，每次问题读取约 160 个标题和摘要。这是集群偏差集中的地方，并且偶尔会因超时而完全无法过滤。
About to A/B it against plain cosine similarity over stored title+summary embeddings at matched budgets, same golden questions, same filter-recall metric. My prior: LLM wins recall, embeddings win cost and latency, and the interesting question is whether the gap justifies the LLM. Anyone run this?
即将在相同预算、相同黄金测试题、相同过滤召回率指标下，将其与基于存储的标题+摘要嵌入的普通余弦相似度进行 A/B 测试。我的先验判断是：LLM 在召回率上占优，嵌入模型在成本和延迟上占优，而有趣的问题在于这种差距是否足以证明使用 LLM 的合理性。有人做过这个测试吗？
Also least sure about:
另外，我最不确定的有：
- Whether hard topic clusters are worth it at all. Every representation I've measured has silhouette under 0.04. The corpus might be one field with no crisp subtopics, which would make the clustering and diversity widening theater.
- Whether the counter-evidence pass pulls its weight. It fixed one-sided answers but it's another LLM call on the critical path.
- Whether 33 golden questions is a real eval or I'm fooling myself with small numbers.
- 硬主题聚类是否真的有价值。我测量的所有表示方法的轮廓系数都低于 0.04。语料库可能属于单一领域，没有清晰的主题子集，这会使聚类和多样性扩展沦为形式主义。
- 反证环节是否能发挥其作用。它修正了片面的答案，但它在关键路径上增加了另一次 LLM 调用。
- 33 道黄金测试题是否构成真正的评估，还是我只是被小样本数量所蒙蔽。
Tear it apart and any questions will do my best to answer them.
请尽情批评，任何疑问我都会尽力解答。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力