Keenable AI开源NEEDLE：每小时更新的实时搜索评测基准
Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour
How do you benchmark a web search API when the thing being tested can read the answer key? A search agent has a fetch tool. If the gold labels sit in a public dataset, the agent can download them mid-evaluation and skip retrieval entirely. A similar problem arises when the answers are already encoded in the model’s parametric memory: a correct response no longer demonstrates that web search worked. Keenable’s answer is NEEDLE, a live open-source benchmark that rebuilds its query set from fresh public sources rather than freezing one. News queries are regenerated hourly from RSS feeds and Google Trends; finance, scholar, legal, and rare-entity queries are regenerated daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs run against the same query text under one protocol, and every score is read against ultimate, a pooled oracle engine that marks what the whole field managed to find.
当被测对象能够读取答案时，如何对网络搜索 API 进行基准测试？搜索智能体拥有抓取工具。如果黄金标签位于公开数据集中，智能体可以在评估过程中下载它们并完全跳过检索步骤。当答案已经编码在模型的参数记忆中时，也会出现类似的问题：正确的回答不再证明网络搜索发挥了作用。Keenable 的解决方案是 NEEDLE，这是一个实时的开源基准测试，它从新的公共来源重建查询集，而不是冻结一个固定的集合。新闻查询每小时从 RSS 源和 Google Trends 重新生成；金融、学者、法律和罕见实体查询每天分别从 SEC XBRL、arXiv、Europe PMC、CourtListener 和公共智能体日志中重新生成。15 个搜索 API 在同一个协议下针对相同的查询文本运行，每个分数都与 ultimate（一个聚合的预言机引擎）进行对比，该引擎标记了整个领域成功找到的内容。
Is it reproducible?
这是否可复现？
Yes, as an open source evaluation harness rather than a product. needle is a Python CLI installed with uv sync and driven by two subcommands per benchmark, generate and run. It needs an OpenRouter key for judging and one API key per engine tested, and runs on a laptop or in CI. It allows recreated all query streams that are being used in addition to the ranking quality judgements.
是的，作为一个开源评估框架而非产品。needle 是一个 Python CLI，通过 uv sync 安装，并由每个基准测试的两个子命令 generate 和 run 驱动。它需要一个 OpenRouter 密钥用于评判，以及每个被测引擎一个 API 密钥，可以在笔记本电脑或 CI 环境中运行。它允许重新创建正在使用的所有查询流，除了排名质量判断之外。
What NEEDLE measures
NEEDLE 衡量什么
NEEDLE stands for News, Everyday, Expert, Deep-tail, and Legal Evaluation. Each vertical models a different agent intent. News projects the newest item from ~124 curated RSS feeds and Google Trends into a keyword query. Finance asks registry facts from Wikidata and GLEIF plus single-quarter 10-Q figures from SEC XBRL. Scholar turns one paper into four query styles: a degraded title, a full-text-only detail, a natural-language clue, and a hedged tip-of-the-tongue description. Deep-tail samples rare-word queries from public agent-trajectory releases including DeepResearchGym, OpenResearcher and LRAT. Legal pulls recent CourtListener opinions across 14 federal courts and eCFR sections.
NEEDLE 代表 News（新闻）、Everyday（日常）、Expert（专家）、Deep-tail（长尾深处）和 Legal Evaluation（法律评估）。每个垂直领域模拟不同的智能体意图。新闻将来自约 124 个精选 RSS 源和 Google Trends 的最新项目转化为关键词查询。金融从 Wikidata 和 GLEIF 获取注册事实，并从 SEC XBRL 获取单季度 10-Q 财报数据。学者将一篇论文转化为四种查询风格：退化的标题、仅全文细节、自然语言线索以及带有保留语气的舌尖提示描述。Deep-tail 从包括 DeepResearchGym、OpenResearcher 和 LRAT 在内的公共智能体轨迹发布中采样罕见词查询。法律从 14 个联邦法院和 eCFR 章节中提取最近的 CourtListener 意见。
Scoring splits along the same line. News and deep-tail have no single correct result, so an LLM judge rates each result 0 to 4 and the harness reports nDCG@5 with a duplicate-URL penalty. Finance reports answer-recall@5: does the fact reach the agent inside a top-5 snippet. Scholar and legal are known-item tasks scored by identifier match.
评分划分遵循相同的原则。新闻和 deep-tail 没有单一的正确结果，因此 LLM 评判员对每个结果进行 0 到 4 的评分，框架报告 nDCG@5 并带有重复 URL 惩罚。金融报告 answer-recall@5：事实是否在顶部 5 个片段内到达智能体。学者和法律是已知项目任务，通过标识符匹配进行评分。
The ceiling is the interesting part
上限才是有趣的部分
Every engine receives the same query text. The runner issues one call at a time, so latency percentiles are comparable and no engine takes concurrent load. Judging happens on the engine’s own ranking, titles and snippets. Pages are never fetched and results are never re-ranked. Evidence is clipped to 2,000 characters for everyone, and the judge does not see the engine name.
每个引擎接收相同的查询文本。运行器一次只发出一个调用，因此延迟百分位数具有可比性，且没有引擎承受并发负载。评判基于引擎自身的排名、标题和摘要。页面永远不会被获取，结果也不会重新排序。证据对所有引擎都截断为 2,000 个字符，且评判者看不到引擎名称。
The more interesting number is the ultimate ceiling. For each query, NEEDLE pools the results returned by every engine into a synthetic oracle engine, then orders that combined set by relevance. That creates an empirical ceiling based on what the entire field was able to retrieve.
更有趣的数字是最终上限。对于每个查询，NEEDLE 将每个引擎返回的结果合并到一个合成的 Oracle 引擎中，然后根据相关性对该组合集进行排序。这基于整个领域能够检索到的内容创建了一个经验性的上限。
The gap to ultimate is therefore an upper bound on agentic search quality as it stands today. A large gap means better results existed but every engine failed to surface or rank them well. A weak ultimate score means something different: even after pooling every provider, the benchmark found little strong evidence. In other words, NEEDLE can distinguish a ranking problem from a retrieval problem shared by the whole market.
因此，与最终上限的差距构成了当前代理搜索质量的上限。较大的差距意味着存在更好的结果，但每个引擎都未能将其展示或良好排名。较弱的最终得分则意味着不同的情况：即使合并了所有提供商，基准测试也发现很少有力的证据。换句话说，NEEDLE 可以区分排名问题与市场整体共享的检索问题。
Where the field actually stands
该领域的实际现状
Numbers below are published 7-day means for the window ending 2026-08-28.
以下数字截至 2026-08-28 窗口期的 7 天平均值。
Finance is close to solved: Exa 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, against an ultimate of 0.965. Scholar spreads out, Keenable 0.774 to Tavily 0.310 against a 0.869 ceiling, because title queries are answerable from metadata and body queries are not. Deep-tail is hardest and closest to real agent traffic: Exa leads at 0.557 of ultimate, Keenable follows at 0.470, Bing sits at 0.199. The gap between delivered and achievable quality widens as queries approach how agents actually search.
金融领域接近解决：Exa 0.910，Keenable 0.872，Perplexity 0.871，Google 0.847，而最终上限为 0.965。学术领域分布较广，Keenable 0.774 至 Tavily 0.310，相对于 0.869 的上限，因为标题查询可通过元数据回答，而正文查询则不能。深尾部分最难且最接近真实的代理流量：Exa 以 0.557 的最终上限领先，Keenable 紧随其后为 0.470，Bing 为 0.199。随着查询方式接近代理实际搜索的方式，已交付的质量与可实现质量之间的差距扩大。
Latency is another important metric here, because agents call search dozens of times per task. Same window: Keenable-realtime 193 ms p50 / 284 ms p95, Exa 1,876 / 2,955, Bing 2,767 / 9,381.
延迟是这里的另一个重要指标，因为代理在每个任务中会调用搜索数十次。同一窗口期：Keenable-realtime p50 193 毫秒 / p95 284 毫秒，Exa 1,876 / 2,955，Bing 2,767 / 9,381。
Key Takeaways
关键要点
- NEEDLE regenerates queries hourly for news and daily for the other four verticals, so there is no fixed set to overfit.
- Five verticals, 15 search APIs, one protocol: same query text, same 2,000-character evidence cap, one request at a time.
- Every leaderboard is read against ultimate, a pooled oracle engine marking the ceiling the whole field reached.
- On rare-entity queries from real agent logs the top engine reaches 0.557 of that ceiling; on finance most engines cluster between 0.77 and 0.91.
- Code is MIT, runs execute in public GitHub Actions, and per-run artifacts ship to a Hugging Face dataset.
- NEEDLE 每小时为新闻重新生成查询，每天为其他四个垂直领域重新生成查询，因此不存在固定的过拟合数据集。
- 五个垂直领域，15 个搜索 API，一个协议：相同的查询文本，相同的 2,000 字符证据上限，一次一个请求。
- 每个排行榜都是针对最终上限进行读取的，这是一个合并的 Oracle 引擎，标志着整个领域达到的上限。
- 在来自真实代理日志的稀有实体查询上，顶级引擎达到了该上限的 0.557；在金融领域，大多数引擎聚集在 0.77 到 0.91 之间。
- 代码采用 MIT 许可证，在公共 GitHub Actions 中运行执行，每次运行的工件都会发布到 Hugging Face 数据集。
Check out the live dashboard, the GitHub repo, the technical write-up, and the archived artifacts. All credit goes to the researchers of this project.
查看实时仪表盘、GitHub 仓库、技术文章以及归档产物。所有功劳归于该项目的研究人员。
Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
此外，欢迎在 Twitter 上关注我们，别忘了加入我们拥有超过 15 万成员的 ML SubReddit 并订阅我们的新闻通讯。等等！你在 Telegram 上吗？现在你也可以加入我们的 Telegram 群组。
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
需要与我们合作推广你的 GitHub 仓库、Hugging Face 页面、产品发布或网络研讨会等？请联系我们
The post Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour appeared first on MarkTechPost.
本文 Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour 首发于 MarkTechPost。
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力