从工时到Token：衡量AI投入回报的陷阱

Everyone is talking about “return on tokens.” Vendors love it (as long as the news is good). Companies that had long since given up on any sort of value architecture or on understanding the ROI of product work are suddenly chomping at the bit. Which makes sense: they haven’t been able to lay off enough people to cover their token budgets while still hitting their promises to the board.

It is hard to know what sits below the surface: a genuine effort to understand how AI might augment human capabilities, or some version of “how many people can we safely fire?” Or both?

It is as if the whole world suddenly shifted from a buffet model to buying each dumpling one at a time, and now everyone is freaking out about the ROI of each dumpling. But we’ve been here before. We’ve been here with hours. We’ve been here with capacity. And the problems have always been roughly the same:

• Spending more time on the “I” side than the “R” side of ROI.
• Myopically choosing shorter-term, easier-to-attribute use cases for the “R” side.
• Gravitating toward whatever is easiest to measure. And tokens are very easy to measure.

This post takes a trip through hours and flow metrics, detours into an alternative way of thinking about ROI, and eventually gets to TOKENS—everyone’s favorite dumpling.

My goal is to show that the underlying measurement problems are remarkably universal. AI introduces some genuinely new twists, but in the end, it still hinges on a theory of value.

With these comparisons, ask yourself, “How might this relate to the current return on tokens question?”

Leave a Coffee Tip!

Hours

Hours, as commonly used to understand investment, represent a major construct validity problem. There’s nothing inherently wrong with measuring “time we spent on things” (hours) to understand better where effort is going, provided you make it clear that:

• Time is not a fungible thing that can be infinitely allocated/re-allocated. This holds across skill sets, team context, and even across a “normal day”. For example, the “magic hour” of uninterrupted morning productivity can be vastly more productive than trying to wrap your head around something complex between 4 and 5 PM. The developer deep in the onboarding space, an expert in onboarding analytics and the user journey, working primarily with what customers see/interact with, can’t be immediately swapped into a deep, legacy backend refactoring effort.
• Where you allocate time tells you nothing about the “quality”, efficiency, or efficacy of that expenditure of time. The highly skilled developer reactively called in to debug an area of the code they have no experience with will not have the same leverage as someone more junior, but more contextually situated. Ten people spending ten hours each is not equal to one person spending one hundred hours. Sometimes the entire system is constrained by a single specialist, a single decision, a single dependency, or a single review. Spending time elsewhere has negative leverage.
• The context switching tax is real. Most efforts to understand the allocation of time, whether explicit or implicit, leave out large time investments. Imagine alternating between two tasks every ten minutes for a six-hour day. You’ll spend three hours (at least) on context switching and calibrating/orienting around the new task. Coordination is multiplicative, while time accounting is additive.
• The relationship between time and outcomes is non-linear. 2x-ing where you spend time doesn’t 2x the outcomes. You might bang your head against a problem for eight hours and get nowhere, and the ninth hour “unlocks” all the value. The metric treats the input as continuous even though the production function contains thresholds and fixed setup costs.
• Where we invest focus can have a long-term effect. Imagine spending 5% longer on every enhancement to pay off a bit of debt and “garden” the codebase. Three years later, a version of the team—some people have stayed, but some have left—is absolutely flying. Where do you “book” that time? And how do you connect that investment to the long-evolving, game-changing “return”? Some hours multiply future options. Some hours limit future options.
• Time allocation schemes often force allocations into individual buckets, but there are typically causal relationships between those buckets. They aren’t mutually exclusive. One impacts the other.

Lean/Flow Metrics (and “story points”)

Helpful, when focused.

Lean/flow metrics include cycle time, lead time, practical WIP limits, and throughput (e.g., N stories per week). Most of these metrics have a storied history in manufacturing, which is both a positive—there’s actual math, theory, and practice behind them—and a negative when misapplied to software development.

Note the following:

• In manufacturing, it matters what you are producing. Things that have sufficiently different work processes, value profiles, etc. are always divided out. This is sometimes referred to as “classes of service,” and it is a critical component of using these metrics effectively. Without that segmentation, changes in throughput, cost, or efficiency may simply reflect changes in the mix of what was produced rather than actual changes in performance.
• A lot of product work is more akin to R&D, experimentation, and initial “design” than to manufacturing and mass production. In this setting, “waste” isn’t an inherently bad thing. If you try five things and one works, then you’ve purchased the knowledge for that fifth thing.
• What constitutes a work unit is also mushy. Say you have a more open-ended stream of experiments meant to move a metric; what do you measure? Experiments per week? The lead time of the whole stream? Both? Where do you put the opportunities that get discovery work, but ultimately don’t get greenlit?
• When we talk about “capacity” in manufacturing, you are talking about the sustainable ability to meet a particular type (or types) of demand. “This factory can produce N of this variety of shoes every month.” You invest in a factory that can meet this forecasted demand, with optionality to scale up the factory based on new demand. So, going back to some of the issues with hours, capacity is not (always) fungible and is not something that is “spent” at the time of use. It is an emergent quality of the factory’s design.
• The boundaries of the system matter enormously. “Lead time” from when? Customer request? Commitment? Ticket creation? First code change? And when is something done: merged, deployed, adopted, or producing value? Moving the boundary can dramatically change the metric without changing the work.
• You have to be ready for the uncomfortable reality that “work” spends a majority of its time not being “worked on”, as well as facing all of your bad habits when it comes to keeping people busy.
• Many of these metrics describe a relatively stable flow system that remains stable over a sufficiently long observation period. If the system is rapidly changing and has a lot of variability in work types, you can be lulled into a sense of apparent precision.
• Don’t get me started on story points. Any effort to use story points for anything beyond a disposable tool to spark discussion, right-size, or reality-check a near-term cycle commitment is a dereliction of duty. We have decades of evidence to support this.

The main thing with manufacturing is that it is essentially a demand-based pull system. The things being produced have a price, and there is a clear distinction between “sitting on the lot” and “in the customer’s hands.” That makes the economics an order of magnitude easier to reason about. Theoretically, we could imagine features going unused as inventory, unfinished work sitting in queues as WIP, or capabilities built ahead of actual demand as overproduction.