RAG幻觉评测：Gemini在知识缺失问题上全数回答，OpenAI表现最佳
We gave 3 commercial RAG systems questions whose answers were verifiably absent from their knowledge base. Gemini answered 42 out of 42 (wait - what?)
这篇论文用可复现的评测方法揭示了RAG系统在“知识缺失”场景下的幻觉差异，评分机制对优化Agent可靠性有直接参考价值，建议相关开发者阅读并参考其评估思路。
Disclosure: I'm an author of the paper -- "Why RAGs Hallucinate: Penalty-Aware Evaluation of Retrieval-Augmented Generation Systems with Knowledge-Gap Canaries" -- and one of the three systems is my company's. We came third of three.
披露：我是论文《为什么 RAG 会产生幻觉：使用知识缺口金丝雀对检索增强生成系统进行惩罚感知评估》的作者之一，其中三个系统之一是我家公司的产品。我们在三者中排名第三。
Three commercial RAG systems over a 1,000-document knowledge base, three repeats, blind-graded by a three-model LLM-As-A-Judge panel.
针对包含 1,000 份文档的知识库，测试了三个商业 RAG 系统，每个系统重复三次，由一个包含三个模型的 LLM-As-A-Judge（大语言模型即评委）小组进行盲评。
Answering accuracy: 97-98% for all of them - so pretty much indistinguishable from each other.
回答准确率：所有系统的准确率均在 97-98% 之间——彼此之间几乎无法区分。
Then we tested questions whose answers are verifiably absent from the corpus. Answer one of those and you're fabricating from parametric memory.
随后我们测试了那些答案在语料库中可证实不存在的问题。如果回答了这些问题中的任何一个，就意味着你在从参数化记忆中捏造内容。
One sysm (Gemini) answered 42 of 42. Another, 2 of 14.
其中一个系统（Gemini）答对了 42 题中的 42 题。另一个系统仅答对了 14 题中的 2 题。
When accuracy is measure on volume (like in most RAG benchmarks) you can't see this, because "I don't know" and a confident guess both score zero. Guessing is free. Using a strategy from the OpenAI paper "Why LLMs Hallucinate", we scored correct (+1), abstain (0), wrong (MINUS 4) and the true ranking of quality of RAG shows up.
当在体量上衡量准确率时（如大多数 RAG 基准测试所做的那样），你无法发现这一问题，因为“我不知道”和自信的猜测都会得零分。猜测是免费的。我们采用 OpenAI 论文《为什么 LLM 会产生幻觉》中的策略，对正确回答（+1）、放弃回答（0）和错误回答（减 4 分）进行评分，从而显现出 RAG 质量的真实排名。
Quality scores: OpenAI RAG +0.862, Gemini RAG +0.793, CustomGPT .ai RAG +0.767.
质量得分：OpenAI RAG +0.862，Gemini RAG +0.793，CustomGPT .ai RAG +0.767。
Paper: https://arxiv.org/abs/2608.26385
论文：https://arxiv.org/abs/2608.26385
Code, logs, every judge vote: https://github.com/adorosario/why-rags-hallucinate
代码、日志、每位评委的投票：https://github.com/adorosario/why-rags-hallucinate
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力