本周 AI 论文精选：LLM 评测、技能提升与上下文管理

1. Judges as a Lifecycle

Most teams validate an LLM judge once, ship it, and never look at it again. Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members on mobile, and this writeup describes what it takes to keep one honest at that volume.

• Four phases instead of one artifact: Birth defines multiple evaluation criteria and builds curated benchmarks with human labels and rationales. Training refines the rubric. Deployment puts the judge to work. Monitoring watches for drift and triggers re-tuning behind a review gate.
• Rubric tuning carries the learning signal: Reasoning-Aligned Rubric Tuning uses a meta-judge over the judge’s reasoning output as the learning signal, so mismatches between judge and human get traced back to specific rubric language rather than patched with more prompt text.
• One judge, two roles: The same judge gates quality and drives reflective generation, appending its rationale to the generator prompt so failed explanations get revised instead of dropped.
• Why it matters: A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions against a no-explanation control, with no quality-related takedowns. This is the rare LLM-judge writeup with production consequences attached.

Paper | Tweet

2. Skill Lift

Enterprise teams reviewing shared skill libraries almost always gate on a scanner that checks structure, style, and security. NVIDIA measured whether that gate predicts anything about how a skill actually performs, and the answer is close to no.

• The review gate is nearly uncorrelated with quality: Across 145 real skills from internal and public catalogs, structural scan scores correlate with LLM-judge quality at a Spearman rho of 0.14. Passing the scanner tells you the skill is well formatted, nothing more.
• Measure the delta, not the document: ACES proposes Skill Lift. Run the same task twice under the same model, sandbox, workspace, and scorer, once with the skill loaded and once without, then measure the difference in what the agent completed.
• Results compare across harnesses: 947 paired cases from 58 production skills were scored across four harnesses, with trajectories normalized into a shared Agent Trajectory Interchange Format so a skill’s lift in Claude Code can be read against its lift in Cursor.
• Why it matters: The largest process-metric gains show up in skill execution, behavior check, and skill efficiency, which points at what skills are actually for. If you run a review process today, this gives you the paired-run design to replace it with something that measures outcomes.

Paper | Tweet

3. Context Management as Code

Every memory system asks you to design a schema up front, then rewrite it when the agent starts doing something you did not anticipate. Scroll, from Alibaba, removes the schema entirely and hands context construction to the model as a programming problem.

• State lives outside the prompt: Each session is backed by an append-only event log and a sandboxed, persistent Python kernel. Tool outputs, retrieved history, and derived state bind to typed variables across model calls instead of being serialized into the prompt every turn.
• Only printed projections cross the boundary: Model-written code searches and transforms that state, and just the explicitly printed output enters the working view. The event log keeps lossless ground truth, so nothing gets committed to a compressed form before you know what will matter.
• Eviction stays recoverable: When the working view nears its budget, stale spans are evicted but remain retrievable. An eviction index keeps compact landmarks tied to exact event-log addresses, so the agent navigates back to a region instead of searching the whole log.
• Why it matters: With Qwen3.8-Max it reaches 94.8% on LongMemEval_S, 73.1% on BEAM_10M (5.1 points over the best published memory system), and 86.7% on LOCA_256K. Because context management runs as code, it inherits every future improvement in model coding ability.

Paper | Tweet

4. JIT-Agent

Harnesses are hand-built and then frozen, which means one design has to serve deep research, product generation, and long-horizon coding equally well. JIT-Agent is a model whose output is a harness, synthesized per task.

• A fixed protocol, a variable harness: The harness is formalized as a composable artifact under a four-module protocol covering memory, planning, action protocol, and tool orchestration. JIT-Agent instantiates those modules for the task at hand rather than picking from a menu of presets.
• Repair happens mid-run: Harnesses get patched during execution, and the system self-evolves by distilling performance signals from an expanding archive of prior configurations, so recurring task shapes converge on better starting designs. Nothing about the backbone changes, only the scaffolding wrapped around it.
• Backbones move a long way: With JIT-Agent attached, DeepSeek-V4-Flash surpasses GPT-5.6 on DeepSearchQA (+9.1) and OdysseyBench (+4.3), and GLM-5.2 gains up to +20.2 points. The generated harnesses are performance-competitive with mature runtimes like OpenCode and Claude Code.
• Why it matters: Generated harnesses are still an underexplored direction, and the appendix is worth the read on its own for the named designs that emerge (Palimpsest, Trapdoor, Origami, Gearbox). Those are legible patterns you can steal by hand even if you never run the generator.

Paper | Tweet

Message from the Editor

We just released Introduction to Exo, a hands-on DAIR Academy lab on the open-source agent harness built for recursive self-improvement. Across 6 labs, you drive the real exo CLI in a live terminal, give an agent a shell, read its raw event log, and fork a conversation to travel back in time. Free for a limited time.

Get Started

5. Prime Agent

Prime Intellect released an open-source harness built for long-horizon work, and what persists between runs sets it apart. Most harnesses reset everything except the files on disk, which caps how much a system can compound.

• The model programs its own context: A persistent IPython REPL lets the model process its context programmatically instead of reading a flat transcript, so filtering, aggregating, and re-deriving state become code the model writes rather than tokens it re-reads.
• A Continual Harness carries the rest: Histories, memories, skills, prompts, and subagent specifications persist across trajectories. Improvements accumulate across runs instead of being rebuilt from scratch each time the agent starts.
• The jump on ARC-AGI-3 is large: Holding the model class fixed, RHAE Best@1 moves from 30% to 95.5%. It also matches or beats native harnesses on long-context coding, GPU kernel generation, and autonomous nanoGPT speedruns.
• Why it matters: This is a working reference implementation of the compounding-harness idea rather than a paper describing one, and it is open source. If you have been reading about self-improving harnesses and wanted something to run, start here. The state hierarchy diagram alone is a useful map of what belongs in the prompt and what belongs in managed storage.

Paper | Tweet

6. What Compaction Destroys

If you keep safety rules or coding standards in an AGENTS.md or a CLAUDE.md, this one is worth your time. Researchers measured what context compaction actually destroys across 20 production agent configurations, and safety rules are among the first casualties.

• Everything gets summarized at the same rate: A safety rule and an episodic log compete for the same tokens, and when the budget overflows both get compressed equally. Only the rule needs exact wording to stay enforceable, and nothing in the pipeline knows that.