DAIR.AI周报：HySparse2长上下文优化与SIFT代码Agent自改进

In agent workloads, a short tool call can return a long search result or execution trace that has to be prefilled before decoding resumes, and the context keeps growing across turns. Xiaomi’s MiMo team built HySparse2, the attention architecture behind the upcoming MiMo-V3, to lower prefill cost and KV-cache size while improving long-context retrieval.

• Two levels of KV sharing: Following YOCO, the model is split into a self-decoder and a cross-decoder. KV Bridging builds the cross-decoder’s full-attention KV caches from the hidden states of the self-decoder’s full-attention layers, and KV Reuse lets each sparse layer reuse the KV cache and selection indices of the full-attention layer before it.
• Prefill exits early: Because every cross-decoder KV cache comes from the self-decoder, prefill can stop once the self-decoder finishes. Token-level selection replaces block-level selection, and a forced window of recent tokens replaces the separate sliding-window branch, so local and global tokens share one cache.
• Cheaper at 1M tokens: On 80B-A3B MoE models trained on the same data, HySparse2 cuts prefill FLOPs by 5.02x against the Hybrid SWA design used in the MiMo-V2 series and by 2.92x against HySparse. Its KV cache takes 2.69 GB, against 12.09 GB for Hybrid SWA.
• Why it matters: After light post-training, HySparse2 scores 11.30 points higher than HySparse on MRCR-v2 and 19.81 points higher on RULER-v2, with lower AgentPPL and LongPPL than both baselines up to 256k tokens. For long-running agents that prefill and store a long observation every turn, it lowers both costs and improves retrieval accuracy.

Paper

Message from our Sponsor

Step 5 Preview is StepFun’s new flagship model for agentic work, a 600B-total, 27B-active MoE with a 1M-token context and vision input. Open weights are scheduled for October 15.

I tested it early as a coding agent, in a minimal harness against GLM 5.3 on two real tasks in the same repo. Both models solved both tasks with no regressions. Step 5 Preview checked its work and declared itself done both times, while GLM 5.3 kept going until the step limit ended the run. In a separate long-context test, it found all five clues hidden in about 368K tokens of incident tickets and connected them to the root cause in about 90 seconds.

It sits on the Pareto frontier for cost against capability, so it is worth trying in your coding agent.

Try Step 5 Preview

Coding agents that rewrite their own implementation can improve on benchmarks, but prior methods such as the Darwin Gödel Machine (DGM) are expensive to run. Researchers from MIT and Sakana AI trace most of that cost to one step and make it cheaper.

• Evaluation is the bottleneck: Earlier approaches score every candidate self-modification by re-running benchmark tasks with the modified agent. That evaluation dominates runtime, so SIFT adds a cheaper signal beforehand.
• Judge first, benchmark later: An LLM judge compares candidate patches pairwise, a regularized Bradley-Terry model turns the win-loss record into strength scores, and those scores drive parent sampling in a disaggregated tree search. Full benchmark runs are reserved for the most promising nodes.
• A tenth of the compute: With o3-mini, SIFT reaches 35.1% on Polyglot after 30 expansions, against 30.7% for DGM after 80 nodes, in under 50 CPU hours and under 5 hours of wall clock. The Qwen3-30B configuration runs its full search at 224 CPU hours and $34 of API spend, about a tenth of the DGM baseline.
• Why it matters: Judge quality decides the outcome. On TerminalBench, gpt-5.4-high as the judge finds a 36.7% agent from a 29.2% start, while gpt-5 finds 34.5% and ranks a weaker candidate first. If you run self-improvement loops, the judge deserves as much attention as the search.

Paper | Tweet

LLM plans for long-horizon robot tasks often break embodiment constraints, fail to recover from mistakes, or lose track of objects they cannot see. GAVEL adds an explicit graph world model around the LLM and more than doubles the success rate of a small model without changing its weights.

• A graph that tracks the world: The graph holds object relations, action preconditions and effects, and probabilistic beliefs about where unobserved objects are. Before the robot executes an LLM-generated action, the graph predicts what the action would do.
• Repair without calling the model: It catches violations before execution. When the fix follows directly from the world model, GAVEL repairs it on its own, and only errors that need semantic reasoning go back to the LLM for replanning.
• Large gains on BEHAVIOR-1K: With Qwen3-8B, single-task success rises from 41.2% to 91.8% across 100 long-horizon tasks, and multi-task success rises from 19.9% to 92.6% across 500 instructions. Reasoning over the distribution of possible object locations also reorders subtasks and cuts travel distance by about 5.4%.
• Why it matters: Many long-horizon agent failures come from losing track of state rather than from weak reasoning. A symbolic model outside the LLM catches those failures cheaply, and the same idea applies to software agents that need to track the state of files, tickets, or accounts.

Paper | Tweet

More agents now store long-term memory as an LLM Wiki, a folder of markdown pages linked to each other. Each page holds dense text, and the links provide structure; WFM is a Wiki Foundation Model trained to use both when retrieving.

• Wiki as a graph: WFM formalizes a Wiki Graph schema in which entity relations and passage nodes live in one graph. The explicit links stay intact while each page keeps its full text, which is harder to capture with sparse graph embeddings.
• Query-conditioned retrieval: Retrieval runs message passing over the graph, conditioned on the query, with attentive aggregation and a regularizer on attention variance. Page text and link structure shape the result together.
• Faster distributed training: The team built a GPU-to-GPU exchange protocol over NCCL that avoids CPU serialization and memory copies when the graph is split across GPUs. Training runs 10.5x faster on distributed clusters, targeting the overhead that makes graph encoders hard to deploy at scale.
• Why it matters: WFM reports strong results on five long-term agent memory and multi-hop reasoning benchmarks. If your agent’s memory is already a folder of linked markdown files, this retrieval model is designed for that format instead of one adapted from plain RAG or sparse knowledge graphs.

Paper | Tweet

Running a frontier LLM as the judge on every eval gets expensive at scale. This paper tests a cheaper setup: a decision-only judge handles most calls, and only uncertain ones go to a frontier model.

• A judge with no reasoning text: JEV, TypeSafe AI’s decision-only model, returns a verdict and label probabilities. It costs $0.044 per 1,000 judgments at a median latency of 0.152 seconds, against $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper.
• Close on ordinary judgments: Compared with sixteen generative and reward-model judges under blinded human adjudication, JEV stays within 3 points of GPT-6 on preference and evidence-grounded factuality, with 92.2% against 93.5% on RewardBench and 87.5% against 86.7% on HaluEval.
• Weaker on hard checks: The gap grows to 9 to 20 points when a judgment requires checking a derivation or rejecting an elaborately written wrong answer, such as 78.6% against 93.1% on JudgeBench. On several benchmarks, those errors cluster in JEV’s low-confidence decisions.
• Why it matters: Because the errors cluster there, a cascade that accepts confident verdicts and escalates the rest to GPT-6 Astra kept 99% of GPT-6’s accuracy at about 57% of its fee on 510 held-out preference pairs. The escalation threshold did not transfer across every fallback model, so the authors recommend setting it on your own data.

Paper | Tweet