实测6款代码索引工具：Token节省宣称未兑现

Scroll to bottom for tldr

In July, JetBrains reran the headline claims of two token-saving tools on real agent workloads.

Caveman claimed 65% and measured 8.5%. RTK claimed 60–90% and ended up slightly more expensive than using nothing. How the tools fool you??

It felt like every tool out there was overclaiming, so I benchmarked 5 token saving tools with conditions closer to how agents actually use them

my setup was :

• 48 Django questions drawn from SWE-bench
• Five question types, selected before running anything
• Same agent, prompt, repository commit and tool access
• Fresh index for every tool
• One no-tools baseline
• 261 runs on this harness
• 43 of the 48 questions finished in all six arms. The run hit an API usage cap near the end and lost the last five from every arm equally, so the paired comparisons are over those 43.

Since the first version of this post I've added cocoindex and codebase-memory-mcp, a third harness on a local model, and a call graph correctness benchmark graded by a compiler.

Disclosure: I work on repowise. Token saving was never its main goal; it's a side effect of the index it builds for agents (call graph, git history, code health, architectural decisions, searchable docs). The harness, raw data, preregistration files and invalidated runs are all public

Nobody saved 60%.

The best result was about 32% of the agent's output tokens, and once you correct for testing five tools at once, three of those five reductions hold up and two are marginal. CodeGraph is a real second at 24.4%, so the correct reading is that more than one tool here works.

Serena was very weird. It writes less than the bare agent while calling tools 42% more often, which makes it busier rather than leaner.

Indexing is also part of the tradeoff. Repowise saved the most tokens here but took the longest to index, because it builds several additional intelligence layers in the same pass. For a straightforward call graph, CodeGraph is 22x faster. And 366.8s is the version with prose generation switched off; a default init on this repo is 1,058s.

Why there is no Claude Code table here

I ran the same questions, servers and indexes under Claude Code with Sonnet 5, and then again with Opus. Those tables are on the benchmarks page, because under Claude Code most of these tools were barely called at all. code-review-graph was never called once across 15 questions. Graphify was called three times, Serena four. Nothing was different about the servers, the questions or the indexes between the two harnesses, and Codex called every tool on every question.

The likely explanation is harness behavior. Claude Code loads MCP schemas on demand, so the agent has to go looking before it can call anything, and frequently never does. Codex mounts them up front

Update: this turned out to be unstable for every tool, mine included. repowise went from 15 of 15 in the first run to 4 and then 3 of 15 in later reruns. Opus landed at 7 of 15, which I had defined before the run as inconclusive.

Third harness: local qwen3:8b

Same 15 questions under Ollama via opencode, and the agent called the tool on every question. repowise cut output tokens by 40.8% and wall clock time by 27.5%. With only the local tools enabled (no hosted model at all), the cuts were 47.9% on tokens and 41.5% on time. It used fewer tokens on 15 of 15 questions, p = 0.00006. I only ran repowise on this harness, so this row has no competitor comparison.

Quality

There was no meaningful quality winner, including repowise.

A blind judge scored every tool in the field, mine included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10 point scale. None of those gaps is distinguishable from zero, and all of them are smaller than the 0.69 points the same benchmark moves when it is rerun unchanged.

Deterministic retrieval benchmark

Token counts still depend on an LLM deciding what to write, so I also ran a deterministic benchmark using ContextBench

Each task has a known list of files touched by the real fix. The score simply measures whether a tool retrieves those files. So there was no LLM judge.

Coverage on its rewards whoever serves highest files, which is why precision and files served sit next to it. get_answer finds the most and hands back about 19 files to do it. code-review-graph finds the least and is the most precise thing in the table, 0.240 from 5.4 files, so if you are paying per token that row reads better than its coverage suggests. Graphify serves 34.5 files for 0.546

The 42 instances were a sealed half, held out and never touched until the final run. cocoindex was added later; one of its instances never returned an answer, so its n is 41.

This required 748 index builds and roughly 78 hours of indexing for 1,129 graded instance/tool pairs. Every tool indexed every repository independently at the task's original base commit, with no shared cache.

Is the call graph actually correct

A call graph can be wrong in two ways: it can miss real calls (recall), or it can invent calls that don't exist (precision). Here the answer key comes from the compilers, not from me: Go's own callgraph/rta and TypeScript's tsc. That's 37,853 edges across 5 tools and 7 repo cells.

• No tool found as much of the call graph as repowise while also getting more of it right, and that held in all 7 cells.
• I lose recall in all five Go cells. codebase-memory-mcp and CodeGraph find a bigger graph, but on syft more than a third of the leader's edges don't exist according to the Go compiler.
• repowise is the most precise tool outright in only 1 of 7 cells. code-review-graph hits 0.997 precision on cobra, but its graph covers 17% of the calls there.

Graph-only build across 35 repos: repowise had the lowest memory on all 35 (median 75 MB against 757 MB for CodeGraph). Speed was a tie, with CodeGraph fastest on 16 repos and repowise on 14.

Two mistakes I nearly published

I nearly published a cost table showing code-review-graph as 43% cheaper than the baseline, in the same run where Claude Code never called it once.

The reason was prompt-cache warming. Whichever arm ran first paid the full price, and later arms reused the cache

That is why the tables report output tokens rather than API cost.

There is a larger version of this mistake, which is measuring one retrieved payload instead of the complete agent session. Loading one commit's context through repowise takes 393 tokens against 13,984 for reading the changed files, so 35.6x, and that is the easy number this whole category tends to publish. The full-session figure is 31.6% on Codex and 15.9% on Claude Code. Agents re-read, backtrack and re-plan, so a compression that looks enormous on a single payload nets out much smaller across a session.

Full methodology, raw data and reproducibility files:

I'm happy to add other tools if there are ones people think are worth testing. The harness is public, so you can also rerun or challenge the results directly.