AI成本下降但账单反升：Token爆炸悖论解析

Hi all,

Last week, we explored what tokenmaxxing means for CFOs and how firms can buffer unexpected AI costs.

We go further today to show you why AI bills are hard to forecast today and what will happen as we crack that problem.

Subscribe now

👾 Play our one‑off, AI‑themed word quiz created just for this edition – and win a prize.

The token explosion paradox

We estimate that the number of tokens processed per quarter has grown by around 17,000x over four years.1

Token prices have collapsed during this time. Demand for machine intelligence is highly elastic, meaning that as prices fall, consumption increases by more than the decline in price.

One reason is that cheaper tokens have made agents economically viable. At the same time, agents use tokens at rates that are orders of magnitude higher than those of chatbots for single-turn queries. That shows up as the total tokens processed per output token—advanced models do a lot of processing below the surface that a user doesn’t see.

A lot of this growth is driven by China’s domestic demand and its model providers, especially ByteDance and Alibaba.

Subscribe now

The cost of the ghost token

When you use an AI agent, the final result you get is really just a summary of all the work the agent has undertaken. There may be dozens of tool calls to browse the web or load up a file to check and validate the work it has done. All of these are steps consume tokens: they become hidden multipliers.

The first of these is token amplification.2 A coding agent that operates over 10 turns might need to re-read its full context every turn. That repetitive reading of context could use as many as 55x more tokens than a single-turn query for the same task.

Actual active inference is probably only 15-20% of the total token consumption. The rest is invisible work that you, as a user, and possibly the company paying for it all, haven’t modelled.

The long tail of tool calls

Agents make anywhere between five and twenty-five tool calls per task. And each call adds more context, tokens and API costs. It also increases the likelihood that the model will need to retry the task to get it right.

The price of safety

Governance and safety costs3 are between 20% and 40% of total spend. Amazon Bedrock charges $0.15 per 1,000 text units for content filters, and Azure charges $0.38 per 1,000 text units. When you add those charges on top of model inference, you could be looking at 15-30% on top of the existing costs for every call.

For firms that use a frontier model as a judge to evaluate another LLM’s output, the evaluation step will cost almost as much as running the main model again. One study found this approach to be 18x more expensive than using smaller, purpose-built guardrail models.

Subscribe now

Forecasting is off

What makes this particularly hard to manage is that the tools themselves cannot yet forecast their own costs. A new paper found that frontier models fail to accurately predict their token use.4

Runs on the same task can vary by as much as 30x in total tokens. So even those of us who have developed a sense for token consumption run into surprises. In the study, 6.7% of tasks lasting less than 15 minutes cost more than the average 1-hour task.

Subscribe now

The cost management ladder

So, what does this mean for companies? We will see firms climb the maturity ladder as they figure out how to track token spend and likely evolve into more sophisticated observation and monitoring of model calls.

Firms will benchmark their internal processes against their previous experience and, when the data is available, against those of other companies. The ambition will be to optimize price-to-outcome ratios. This will all help clarify the unit economics for specific commercial outcomes. The value of observability and monitoring will increase, as evidenced by the share price rise of Datadog, a leading observability platform.

Long-term, attributing costs to specific teams, workflows, and customers will help finance teams budget for AI spending and decide how to route and tune model use.

Taken together, the firms with the most mature approach to their AI deployments should start to understand the true unit economics of their outputs in the coming quarters.

1

This is the total volume of tokens (input + output) processed by LLMs across all providers. Our methodology uses a bottom-up, per-provider approach built up across 50+ providers. Cross-checks: revenue÷price, GPU utilisation, API call volume.

2

Token amplification is when agents do the things that inflate the number of tokens used per human interaction.

3

Content and policy filters.

4

The correlation between predicted and actual token consumption is weak to moderate, with a maximum of r = 0.39.