Dan Luu：让AI自动执行代码毫无意义，人类必须持续监督

In early 2025, I started seeing people turn off their brain as they use LLMs1. They would have an LLM take an action (summarize text, write some code, etc.), and just assume that it worked2. This generally didn't work in early 2025 and the result was often quite silly.

As LLMs have gotten better, I've seen more of this. Sometimes, people will try to get the LLM to write some code for them and basically just assume that it works3. Sometimes there's a human in the loop and, if the thing doesn't work, they'll ask the LLM to figure out the problem and solve it. Niklas Gruhn calls some variants of doing this being a meat proxy.4

Being a for loop meat proxy works better than it did in early 2025 and the software I've tried that's developed like this sometimes actually sort of works. Not well enough that I'd want to use it or that it's successful, but I'm impressed at how effective being a meat proxy is in September 2026. You could imagine LLMs improving enough that brain-off meat-proxy development produces average quality software in the foreseeable future, or even that LLMs improve enough that they produce great software without a human in the loop.

Let's say that happens. What reason is there for the company to employ the meat proxy? The company can just run the LLM in a loop and lay off the employee. There's no point at which this methodology will work for the employee5.

Thanks to Max Bittker, Yossi Kreinin, Luke Burton, Thomas Dullien, Dennis Snell, Peter Geoghegan, and Jamie Brandon for comments/corrections/discussion.

• I've been having this thought for about a year and a half now. I have it more frequently now as LLMs get better and I see people spend more time turning their brain off when interacting with LLMs. [return]
• Luke Burton had this comment:
• I think being able to do this says more about the type of work being done than people think. I will only walk away from work like this if the task is quite low value, if it can afford to fail.
• For high value tasks, the probability of an LLM one-shotting them is much lower. I have to assume the role of QA, engineering manager, and architect. The while loop often feels like a crunch time. I feel the nagging suspicion I've missed something and that a badly specified prompt could result in an architectural choice that needs to be undone.
• Another observation is that the high throughput causes me to raise my own bar for what I ship. Whereas before I might have shipped an MVP and iterated, now I have agents polish and explore edge cases well beyond my norm, which they invariably fail to do unless prompted.
• Maybe it raises some uncomfortable thoughts for people, but my question for the meat proxies out there if the agents are nailing it so easily: 1) is it possible you've been coasting a bit already? 2) why aren't you pushing agents well beyond tasks they can tackle so easily?
• We've been doing something you'd think is extremely amenable to "hands off" automation, which is converting [redacted] to build with Bazel. It has taken us months even with agents. There's a lot of intangible, hard-to-specify requirements buried inside this task and having agents walk that line means constant supervision. Giving them a prompt like "convert this to Bazel" and walking away is at minimum many months in the future, maybe years, and maybe not ever? There are too many decision points, and too many unknown unknowns involved.
• Like how often does this scenario come up: you encounter some code and it's not clear why it functions this way, but knowing that materially changes what course of action you should take. Maybe it changes the dev experience, maybe you don't know if some customer has started using it, so on and so forth. How exactly do you meat proxy your way through that?
• Conversely you review what you've done with some stakeholder and they say "oh that? that part of it wasn't needed, we aren't even using that any more". What kind of decisions got made around the false assumption that a certain element needed to be preserved?
• [End of Luke's comment, comment from me]. A place where it's more obvious you need to make decisions is when the agent runs into something that's out of distribution. A minor version of this was when we compared how well agents use different programming languages and agents were much worse at obscure languages, which they're trained on, just not as much as with mainstream languages. A more out of distirbution example is if you try to play a board game (especially a modern game and not one of the classical games like chess or go). In general, for a game like Lost Cities or Dominion, a SOTA model and harness is worse than a human who's reasonable at board games but has never played the game before. If you ask the agent about the game, it knows a lot about the game and can say things that sound like they make sense to someone who doesn't understand the game, but are obviously wrong to anyone who does understand the game. I recently played some Dominion with a new player who thought that using ChatGPT to help them understand the game would help them learn and play the game. I was quite skeptical of this and suggested that it will probably make them worse (which, AFAICT, it did). After playing a few games, I looked at what ChatGPT was telling them, and it was maybe half right and half wrong, but the half wrong parts were steering them to a worse place than someone who generally plays games well and uses general game playing heurisitics would do. BTW, there's enough public information out there that I think that someone who'd never played before, but decided to spend, say, five hours reading about the game and seeing what information is out there, could easily be 99%-ile or above at the game if they did some pre-reading (maybe 30 minutes if using references while playing is allowed). I think that would be un-fun and I wouldn't recommend that anyone do it, but given that agents can do searches, query APIs, etc., it shows you the gap between a human and an agent today when approaching an out of distribution problem. For all I know, the next big model release will flip this around, but the gap is still fairly large today.
• Anyway, my point here is that, even when doing coding tasks, you often run into out of distribution questions where the agent behaves very poorly compared to a reasonable human being. If you want a good overall result today, you need to notice these cases and deal with them.
• [return]
• Some examples of what goes wrong when someone just assumes things will work are this case, where agents (sometimes) heavily overfit to tests or this case where agents heavily overfit to a metric. I've heard a theory that agents do more cheating on eval-shaped problems. I'm not sure that's true, but even assuming it's true and that, in my work and personal projects, I tend to create more eval-shaped instructions than most people even when not running evals, I've seen other people who don't create very eval-shaped things run into the same problem (I think actually more severely) when they write some instructions and let agents go wild without supervision (I've had luck doing that with minimal supervision, but only by fencing the agents in quite a bit, which makes the thing more eval-shaped than what most people seem to do).
• When I try software from people who've outsourced thinking to the LLM, the software has serious issues. I've had people tell me this kind of thing works, but the software is often at a level where I would say that it doesn't work according to the standard discussed here.