开发者构建28款爬虫路由Frankensurf，盲测覆盖率达96%
I built a router across 28 scraping tools and benchmarked it blind, it can access 96% of the sites
Agent开发者的硬核踩坑复盘，给出了可复用的多工具路由架构与内容校验逻辑，直接解决网页抓取中的幻觉痛点，建议收藏参考。
I've been building agents that read the web, and what kept burning me was pages that came back as a normal 200 with nothing useful on them. Usually a bot check, or a search page whose results hadn't loaded yet. The agent would read it anyway and tell me something wrong with total confidence or no useful info at all.
我一直在构建能够浏览网页的代理，而最让我头疼的是那些返回正常 200 状态码但页面上没有任何有用内容的页面。通常是一些机器人验证页面，或者是搜索结果尚未加载的搜索页。代理仍然会读取这些页面，并以十足的自信告诉我一些错误信息，或者根本没有任何有用信息。
Every scraping tool I tried got through some sites and fell over on others, so I stopped picking one and glued 28 of them together. Frankensurf tries the cheapest tool first, checks the page is real (a search page needs results, a product page needs a price), and climbs to something heavier when it isn't.
我尝试过的每个抓取工具都能通过某些网站，却在其他网站上失败，于是我不再只选一个工具，而是将 28 个工具组合在一起。Frankensurf 优先使用最便宜的工具，检查页面是否真实（搜索页需要有结果，商品页需要有价格），如果检查不通过，则切换到更强大的工具。
Reading pages: 150 sites picked before I read any of them, every tool on its own, same day, same pass/fail check
页面读取：在我读取任何页面之前，先筛选出 150 个网站；每种工具单独运行，在同一天、同一轮测试中进行相同的通过/失败检查
| Tool | Sites read right |
|---|---|
| Frankensurf (free + paid tools) | 96.0% |
| Frankensurf (free tools only, no keys) | 88.0% |
| Zyte | 86.0% |
| Firecrawl | 84.7% |
| Jina Reader | 73.3% |
| Plain HTTP | 48.7% |
| 工具 | 成功读取的网站数 |
|---|---|
| Frankensurf（免费+付费工具） | 96.0% |
| Frankensurf（仅免费工具，无需密钥） | 88.0% |
| Zyte | 86.0% |
| Firecrawl | 84.7% |
| Jina Reader | 73.3% |
| 纯 HTTP | 48.7% |
Search: two new sets of 120 queries, graded blind (results pooled and shuffled, tool hidden)
搜索测试：两组新的各含 120 个查询，采用盲评方式（结果合并并打乱，隐藏所用工具）
| Comparison | Result |
|---|---|
| Frankensurf free vs Firecrawl search | better on 35 queries, worse on 5 |
| Frankensurf with paid sources vs Exa | 4.67 vs 4.66 useful results out of 5 |
| 对比 | 结果 |
|---|---|
| Frankensurf 免费版 vs Firecrawl 搜索 | 在 35 个查询中表现更好，在 5 个查询中表现更差 |
| Frankensurf 结合付费源 vs Exa | 5 个查询中分别获得 4.67 和 4.66 个有用结果 |
What I took away
我的心得
- Check what’s on the page, not the status code. The failures that hurt my agents all came back as a normal 200. A bot check, a login redirect, a search page whose results hadn’t loaded yet. My agent would read it anyway and tell me something wrong with total confidence. If your agent reads the web, “the request worked” means almost nothing until you check the page actually has what you asked for.
- No one tool reads the whole web, so I stopped picking one. Every tool I tried was great on some sites and useless on others. Chaining them, cheapest first and moving up only when the page check fails, read 96% of 150 sites I’d never seen. The best single tool got 86%. It also cost about a quarter as much, because most sites never need the expensive stuff.
- A dumb rule beat an LLM judge. I was sure a model checking every page would beat my boring rules like “a search page needs results”. It scored 88.8% against the rules’ 89.5% and took twice as long. So the rules stayed, and the model only steps in when the agent can tell from its actual task that a page is wrong and asks to try harder.
- My benchmarks lied to me until I stopped letting them peek. My first one scored 98.7% on sites I’d been building against. I just solved the internet! Then I ran it on sites it had never seen and got 71%, losing to Firecrawl. Great. My first search eval was worse. It said I was level with Exa, but the judge was checking the same rules my search used to filter results, so of course it agreed. It was marking my homework with my own answer key. Graded blind, with results shuffled and the tool hidden, Exa was miles ahead, 4.8 useful results out of 5 against my 2.8. Now I pick test sites before I look at them, keep them away from anything I tune on, and grade blind.
- 检查页面上的实际内容，而不是看状态码。导致我的代理失败的案例全部返回了正常的 200 状态码。例如：机器人验证页面、登录重定向、或搜索结果尚未加载的搜索页。代理仍会读取这些页面，并以十足的自信给出错误信息。如果你的代理需要浏览网页，“请求成功”几乎毫无意义，除非你确认页面确实包含你所要求的内容。
- 没有哪个单一工具能读取整个互联网，因此我不再依赖某一个工具。我尝试过的每个工具在某些网站上表现优异，在其他网站上却毫无用处。将它们串联起来，优先使用最便宜的，仅在页面检查失败时才升级到更昂贵的工具，最终成功读取了我从未见过的 150 个网站中的 96%。最好的单一工具只能达到 86%。而且成本还低得多，因为大多数网站根本不需要使用昂贵的工具。
- 一条简单的规则胜过 LLM 裁判。我曾确信，让模型逐页检查会比我的简单规则（如“搜索页必须有结果”）更有效。但模型得分仅为 88.8%，低于规则的 89.5%，且耗时是规则的两倍。因此我保留了规则，模型仅在代理能从其实际任务中判断页面有误，并请求重试时才会介入。
- 我的基准测试曾欺骗了我，直到我停止让它们窥探。我的第一个测试在我一直用来构建对比的网站上得分 98.7%。我以为自己刚刚征服了互联网！然后我在它从未见过的网站上运行该测试，结果只有 71%，输给了 Firecrawl。太棒了。我的第一次搜索评估更糟。它声称我与 Exa 持平，但裁判检查的是我的搜索用于过滤结果的相同规则，所以当然它会同意。这就像是用我自己的答案卷批改我的作业。在盲评模式下，结果经过打乱且工具被隐藏，Exa 遥遥领先：5 个结果中有 4.8 个有用，而我的只有 2.8 个。现在，我在查看测试网站之前先选定它们，将它们与我调优的任何内容隔离开来，并进行盲评。
It's open source and very much an alpha. The benchmark script, site lists and every raw row are in the repo if you want to check my work: https://github.com/yail259/frankensurf
它是开源的，并且仍处于非常早期的 alpha 阶段。如果你想核查我的工作，仓库里包含了基准测试脚本、网站列表以及每一行原始数据：https://github.com/yail259/frankensurf
更进一步：量化金融体系
看懂新闻只是起点——沿量化金融路径，把它变成能交付的工程能力