RAG评估新视角：区分检索失败与知识缺失
A RAG system can say “I don't know” for the wrong reason
做 RAG 的同学必看，这篇指出了当前评估中“拒答”指标的盲区，给出了区分检索失败与知识缺失的五种具体场景，直接优化你的评测集构建逻辑。
I'm no longer convinced that adding a few unanswerable questions is enough to test whether a RAG system knows when to stop.Tae a made-up support example. The documentation explains how to upload a CSV manually. Someone asks, “Can I schedule this to sync every night?” A sensible answer would say the supplied material doesn't establish that, rather than inventing a scheduler. Now change one thing: the scheduling instructions do exist, but retrieval misses them. The model gives exactly the same cautious answer. From the answer aloone, both runs look responsible. From the user's side, one is a useful boundary and the other is a failure. This is where I want to push my FastGPT experiments beyond “did the final answer look right?” Its documented app evaluation takes questions and expected answers, but I think the expected result also needs to describe what evidence the system should have had. Otherwise I could improve a refusal score just by making the system less willing to answer.
我不再确信，仅添加一些无法回答的问题就足以测试 RAG 系统是否知道何时停止。举一个虚构的支持场景为例：文档解释了如何手动上传 CSV 文件。有人问：“我可以安排它每晚同步吗？”一个合理的回答应指出提供的材料并未确立这一点，而不是凭空捏造一个调度器。现在改变一个条件：调度说明确实存在，但检索未能找到它们。模型给出了完全相同的谨慎回答。仅从回答来看，两次运行都显得负责任。但从用户角度看，一个是有用的边界，另一个则是失败。我希望借此推动我的 FastGPT 实验超越“最终答案看起来是否正确”这一层面。其文档化的应用评估需要问题和预期答案，但我认为预期结果也需要描述系统本应具备的证据。否则，我仅仅通过让系统更不愿意回答就能提高拒绝得分。
I'd separate at least these cases:
我至少会区分以下情况：
- The answer is in the allowed documents and reaches the model. Answer it and support the relevant claim.
- The answer is in the allowed documents but never reaches the model. Abstaining may be correct for the answer generator; the retrieval pipeline still failed.
- The documents don't establish the answer. State that limit without turning “not documented” into “not supported.”
- The question is missing a necessary detail. Ask for that detail, rather than giving a generic refusal.
- Two sources disagree and neither clearly supersedes the other. Surface the conflict instead of quietly picking the convenient one.
- 答案存在于允许的文档中并传递到了模型。回答它并支持相关主张。
- 答案存在于允许的文档中，但从未到达模型。对于答案生成器来说，保持沉默可能是正确的；但检索管道仍然失败了。
- 文档无法确立该答案。声明此限制，而不要将“未记录”转化为“不支持”。
- 问题缺少必要的细节。询问该细节，而不是给出通用的拒绝回答。
- 两个来源存在分歧，且没有一个明显优于另一个。揭示冲突，而不是悄悄选择方便的那个。
The score I care about would show both unsupported answers and missed answerable questions. A bot that refuses everything can avoid making things up and still be useless. I also wouldn't grade on whether it says the exact words “I don't know.” In the CSV example, “These instructions only cover manual uploads; I can't establish whether scheduled sync exists” is more useful than an apology. The difficult bit is assigning the expected evidence. For a small test set, I'd rather annotate a handful of close calls carefully than add hundreds of obviously unrelated questions that are easy to reject. If you score abstention, do you mark a missed retrieval separately from a correct refusal? I'd like to see what that distinction looks like in an actual eval report...
我在意的评分指标应同时显示无依据的回答和遗漏的可回答问题。一个拒绝所有问题的机器人可以避免胡编乱造，但仍然毫无用处。我也不打算根据它是否说出确切的话“我不知道”来打分。在 CSV 示例中，“这些说明仅涵盖手动上传；我无法确定是否存在计划同步”比道歉更有用。难点在于分配预期的证据。对于小规模测试集，我更愿意仔细标注少数几个模棱两可的案例，而不是添加数百个显而易见不相关、容易拒绝的问题。如果你对标 abstention（拒绝回答）进行评分，你会将漏检的检索与正确的拒绝分开标记吗？我想看看这种区别在实际的评估报告中是什么样的……
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力