Meta 论文：强化学习优化代码为何常失败及解法

New Meta Paper.

Code optimization looks like an easy extension of reinforcement learning: reward correct programs, then give extra credit to the faster ones.

This paper shows why that recipe usually fails.

Says that code optimization requires co-designing tests, timing infrastructure, rewards, and GRPO.

Runtime is a noisy, sparse signal, and small flaws in the tests, sandbox, reward, or GRPO update can make the model barely faster while also hurting correctness.

The authors rebuild the entire feedback path: larger optimization tests, a calibrated remote execution service, problem-relative ranking against human solutions, correctness-gated binary rewards, and GRPO changes for noisy zero-advantage batches.

They stabilize GRPO by sampling more solutions per prompt, enlarging batches, dropping group variance normalization, and discarding stale timing data.

Qwen 2.5 7B moved from 18.0% to 31.3% at the top-50% speed threshold only after the full timing pipeline was rebuilt.

– arxiv. org/abs/2607.25970

Title: "Reinforcement Learning for Code Optimization"