Contrastive-LM发布CLM-8B：比Jev快9倍的开源Agent评

Contrastive-LM has released CLM-8B, the first open model in a new class called Contrastive Language Models (CLMs). CLM does not generate text. It scores a set of candidate actions against the current state and returns probabilities. Their main baseline is Jev, the proprietary System One model from TypeSafe AI.

Is it deployable? Yes. The Apache-2.0 head weighs 75 MB. It runs on 1 NVIDIA GPU under Linux, with vLLM serving the Qwen3-8B encoder.

What a System One Model Does

Jev entered limited early access on 15 September 2026. It returns typed values with probabilities instead of text. CLM targets the same interface. The CLM GitHub repo serves CLM-8B behind a TypeSafe-compatible API. It exposes 3 question types:

• Noul: returns the probability that a statement is true.
• Choice: picks one option from a declared set, with probabilities.
• Score: returns an expected level on an ordered rubric.

A request written for TypeSafe’s API can be replayed through CLM’s Python client.

How CLM Works

CLM trains a state encoder and an action encoder with a bidirectional InfoNCE loss. Each encoder is a frozen Qwen3-8B backbone plus a 20M-parameter trainable projection head. Training pulls each state toward the action actually taken and pushes it away from the others.

At inference, CLM scores each candidate by the dot product of the state and action embeddings. A softmax over those scores becomes the answer distribution. The same primitive ranks best-of-N solutions, routes tools and answers typed decisions.

This design disaggregates states and actions. In an agent loop, the state changes every step while the action set stays mostly fixed. clm-serve reserves a slab of GPU memory, similar to vLLM’s KV cache, and reuses cached vectors. On 1 RTX 4090 with 3 actions, revisited states drop from 1.7 ms to 0.6 ms. The model card reports CLM running 13× faster than Jev with about 1,000 candidates.

A 3-Stage Training Recipe

• Pre-training on ~60M Nemotron DQA question-answer pairs.
• Mid-training on ~30M synthetic hard negatives generated by Gemini 2.5 Flash-Lite.
• Post-training on ~1M agent trajectories from Agent Data Protocol, Endless-Terminals and LiteCoder-Terminal-SFT.

On ~100K held-out questions, pre-training alone reaches 52.1% top-1 accuracy. Mid-training lifts it to 69.2%. Training on hard negatives from the start peaks at 62.4%, then overfits.

Zero-Shot Results Against Jev

The 9× figure comes from the T-Rex game, where actions repeat across states. CLM matches Jev on T-Rex and Super Mario. It trails on tool calling and WikiRacing while running faster on every task.

CLM as a Verifier for Coding Agents

Here a generator samples several candidate solutions and the verifier picks one. Opus 5 produced DeepSWE candidates (best-of-4). Fable 5 produced Terminal-Bench 2.1 candidates (best-of-5). The team evaluated 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks. Latency was measured on an H100.

The research team reports these as new SOTA verifier results. Jev scores below pass@1 on both benchmarks, so selecting with Jev is worse than taking 1 sample. CLM runs 4.1× to 5.7× faster. These numbers use lightweight fine-tuned heads, not the zero-shot checkpoint. They are held-out subset results, not full leaderboard submissions.

Interactive Explainer

Key Takeaways

• CLM-8B scores candidate actions instead of generating text.
• Up to 9× lower latency than Jev in zero-shot tests.
• Fine-tuned heads reach 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1 subsets.
• Cached state and action vectors cut agent-loop latency.
• Apache-2.0 head, self-hosted on 1 NVIDIA GPU.

Check out the Blog, Code and Data & Models. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.