Contrastive-LM Releases CLM-8B: An Open System One Model That Scores Agent Actions Up to 9× Faster Than Jev
Contrastive-LM released CLM-8B, an open model that scores agent actions up to 9× faster than Jev.
Contrastive-LM released CLM-8B, an Apache-2.0 model that scores candidate actions rather than generating text, using a frozen Qwen3-8B encoder and a 75 MB head served with vLLM. Against TypeSafe AI’s proprietary Jev model, zero-shot latency was as much as about 9 times lower, including 16.5 ms versus 149.8 ms on a T-Rex task. Fine-tuned verifier heads reached 81.6% on a 38-task DeepSWE subset and 87.6% on 30 Terminal-Bench 2.1 tasks. Training used roughly 60 million Nemotron DQA pairs, 30 million Gemini 2.5 Flash-Lite hard negatives, and 1 million agent trajectories.
- CLM-8B scores candidate actions with probabilities instead of generating text.
- It uses frozen Qwen3-8B encoders and a 75 MB Apache-2.0 head.
- Zero-shot T-Rex latency is 16.5 ms versus 149.8 ms for Jev.
- Fine-tuned heads reach 81.6% on DeepSWE and 87.6% on Terminal-Bench subsets.
- Training used about 60M DQA pairs, 30M hard negatives, and 1M trajectories.
Full article755 words · extracted from marktechpost.com · click to collapse
Contrastive-LM has released CLM-8B, the first open model in a new class called Contrastive Language Models (CLMs). CLM does not generate text. It scores a set of candidate actions against the current state and returns probabilities. Their main baseline is Jev, the proprietary System One model from TypeSafe AI.
Is it deployable? Yes. The Apache-2.0 head weighs 75 MB. It runs on 1 NVIDIA GPU under Linux, with vLLM serving the Qwen3-8B encoder.
What a System One Model Does
Jev entered limited early access on 15 September 2026. It returns typed values with probabilities instead of text. CLM targets the same interface. The CLM GitHub repo serves CLM-8B behind a TypeSafe-compatible API. It exposes 3 question types:
- Noul: returns the probability that a statement is true.
- Choice: picks one option from a declared set, with probabilities.
- Score: returns an expected level on an ordered rubric.
A request written for TypeSafe’s API can be replayed through CLM’s Python client.
How CLM Works
CLM trains a state encoder and an action encoder with a bidirectional InfoNCE loss. Each encoder is a frozen Qwen3-8B backbone plus a 20M-parameter trainable projection head. Training pulls each state toward the action actually taken and pushes it away from the others.
At inference, CLM scores each candidate by the dot product of the state and action embeddings. A softmax over those scores becomes the answer distribution. The same primitive ranks best-of-N solutions, routes tools and answers typed decisions.
This design disaggregates states and actions. In an agent loop, the state changes every step while the action set stays mostly fixed. clm-serve reserves a slab of GPU memory, similar to vLLM’s KV cache, and reuses cached vectors. On 1 RTX 4090 with 3 actions, revisited states drop from 1.7 ms to 0.6 ms. The model card reports CLM running 13× faster than Jev with about 1,000 candidates.
A 3-Stage Training Recipe
- Pre-training on ~60M Nemotron DQA question-answer pairs.
- Mid-training on ~30M synthetic hard negatives generated by Gemini 2.5 Flash-Lite.
- Post-training on ~1M agent trajectories from Agent Data Protocol, Endless-Terminals and LiteCoder-Terminal-SFT.
On ~100K held-out questions, pre-training alone reaches 52.1% top-1 accuracy. Mid-training lifts it to 69.2%. Training on hard negatives from the start peaks at 62.4%, then overfits.
Zero-Shot Results Against Jev
| Task | CLM-8B latency | Jev latency | CLM-8B success | Jev success |
|---|---|---|---|---|
| T-Rex game | 16.5 ms | 149.8 ms | 5/5 | 5/5 |
| Tool calling (BFCL v4) | 76.8 ms | 125.5 ms | 95.2% | 99.2% |
| WikiRacing | 79.8 ms | 225 ms | 26/30 | 30/30 |
| Super Mario | 33.5 ms | 132.6 ms | 5/5 | 5/5 |
The 9× figure comes from the T-Rex game, where actions repeat across states. CLM matches Jev on T-Rex and Super Mario. It trails on tool calling and WikiRacing while running faster on every task.
CLM as a Verifier for Coding Agents
Here a generator samples several candidate solutions and the verifier picks one. Opus 5 produced DeepSWE candidates (best-of-4). Fable 5 produced Terminal-Bench 2.1 candidates (best-of-5). The team evaluated 38 held-out DeepSWE tasks and 30 held-out Terminal-Bench 2.1 tasks. Latency was measured on an H100.
| Benchmark | Pass@1 | CLM (fine-tuned) | Jev | CLM latency | Jev latency |
|---|---|---|---|---|---|
| DeepSWE | 73.7% | 81.6% | 71.1% | 79 ms | 449 ms |
| Terminal-Bench 2.1 | 84.0% | 87.6% | 83.1% | 32 ms | 131 ms |
The research team reports these as new SOTA verifier results. Jev scores below pass@1 on both benchmarks, so selecting with Jev is worse than taking 1 sample. CLM runs 4.1× to 5.7× faster. These numbers use lightweight fine-tuned heads, not the zero-shot checkpoint. They are held-out subset results, not full leaderboard submissions.
Interactive Explainer
Key Takeaways
- CLM-8B scores candidate actions instead of generating text.
- Up to 9× lower latency than Jev in zero-shot tests.
- Fine-tuned heads reach 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1 subsets.
- Cached state and action vectors cut agent-loop latency.
- Apache-2.0 head, self-hosted on 1 NVIDIA GPU.
Check out the Blog, Code and Data & Models. All credit goes to the researcher of this project. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Michal Sutter
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.