TypeSafe Jev与IBM STAIR论文RAG幻觉实验复盘

Jev (TypeSafe, released Sep 15) cannot generate text. It takes a passage plus typed questions and returns calibrated choices, scores and yes/no probabilities in about 200ms at $0.042 per million input tokens. I got early access and pointed it at hallucination.

Paired with IBM's STAIR paper (arXiv 2609.03874), which uses a document's table of contents as the retrieval unit instead of fixed-size chunks, reporting 82.6% Recall@1 vs 59.5% for BM25.

Plan: STAIR-style structure finds the section, Jev makes the decisions. Which section, is the answer present, does the quote support the answer.

Experiment 1: SQuAD 2.0, 150 questions, 50 unanswerable. Stack is BM25 plus bge-small embeddings fused, MiniLM cross-encoder, gemini-3.5-flash-lite answering with a verbatim quote that gets string-checked against the source. Failed check becomes NOT_FOUND.

What I found:

• The Jev checkers removed zero hallucinations. On all 8 remaining errors, Jev agreed with the wrong answer at 0.82 to 1.0 confidence. SQuAD 2.0's unanswerables sit adversarially close to the text, and the checker reads them the way the generator does.
• Quote-forcing plus a plain string check is what held hallucination at 5.3%. No model needed.
• Jev pairwise re-ranking was the only gain: +2.7 points correct, 98% section recall.
• ToC routing came last: 69% section recall vs 96% for hybrid search. But Wikipedia is a two-level hierarchy, so this is not a fair test of STAIR. Experiment 2 runs it on textbooks with real tables of contents and real fine-tuned STAIR predictions.

Caveats: 150 questions, one generator, one judge, so a 3-point difference is about 5 questions.

Repo with reports, per-question data and raw API responses: https://github.com/aryanchauhanoffical/no-hallucination

Longer write-up: https://www.linkedin.com/posts/aryanfrontend_rag-llm-aiengineering-share-7507345521741553664-CCd8/

Experiment 2 is where it got weird. Combining retrieval sources made retrieval better and answers worse. Posting that next.Jev (TypeSafe, released Sep 15) cannot generate text. It takes a passage plus typed questions and returns calibrated choices, scores and yes/no probabilities in about 200ms at $0.042 per million input tokens. I got early access and pointed it at hallucination.Paired with IBM's STAIR paper (arXiv 2609.03874), which uses a document's table of contents as the retrieval unit instead of fixed-size chunks, reporting 82.6% Recall@1 vs 59.5% for BM25.Plan: STAIR-style structure finds the section, Jev makes the decisions. Which section, is the answer present, does the quote support the answer.Experiment 1: SQuAD 2.0, 150 questions, 50 unanswerable. Stack is BM25 plus bge-small embeddings fused, MiniLM cross-encoder, gemini-3.5-flash-lite answering with a verbatim quote that gets string-checked against the source. Failed check becomes NOT_FOUND.setup correct hallucinated wrongly refused

baseline, no Jev 87.3% 5.3% 7.3%

plus Jev "is the answer here?" gate (0.7) 85.3% 5.3% 9.3%

plus Jev citation check (their cookbook, 0.8) 86.7% 5.3% 8.0%

plus Jev pairwise re-ranking (their cookbook) 90.0% 6.0% 4.0%

Jev ToC routing, greedy 72.7% 3.3% 24.0%What I found:The Jev checkers removed zero hallucinations. On all 8 remaining errors, Jev agreed with the wrong answer at 0.82 to 1.0 confidence. SQuAD 2.0's unanswerables sit adversarially close to the text, and the checker reads them the way the generator does.

Quote-forcing plus a plain string check is what held hallucination at 5.3%. No model needed.

Jev pairwise re-ranking was the only gain: +2.7 points correct, 98% section recall.

ToC routing came last: 69% section recall vs 96% for hybrid search. But Wikipedia is a two-level hierarchy, so this is not a fair test of STAIR. Experiment 2 runs it on textbooks with real tables of contents and real fine-tuned STAIR predictions.Caveats: 150 questions, one generator, one judge, so a 3-point difference is about 5 questions.Repo with reports, per-question data and raw API responses: https://github.com/aryanchauhanoffical/no-hallucinationLonger write-up: https://www.linkedin.com/posts/aryanfrontend\_rag-llm-aiengineering-share-7507345521741553664-CCd8/Experiment 2 is where it got weird. Combining retrieval sources made retrieval better and answers worse. Posting that next.