autotrust/JEV-27B-VL — new model trending #30 on Hugging Face

| computer use: numbered boxes only | 10% | 37% | 15% |

| time per robot-arm decision | 239 ms | 163 ms | 61 ms |

* Use System 1 for simple visual questions inside a control loop. Asked to pick one of 8 motor commands directly, it

completed 0 of 10 scenes.

* For computer use, give it the element text (as an accessibility tree would). With the numbered boxes alone it often

declares the task complete too early.

Demo code: [JEV-9B `vl/demos/`](https://huggingface.co/autotrust/JEV-9B/tree/main/vl/demos) (set `JEV_URL` to this

server). Per-episode results: [`reports/demos/`](reports/demos).

## New (1 October 2026)

* **System 1 over plain HTTP.** `serve_decide.py` adds `POST /v1/decide` to the vLLM server: send

`{kind, state, question, options}` and get a calibrated probability for every option, for text or images. See

[Quick start](#quick-start).

* **Up to 256 options per choice question**, with no retraining. Zero-shot on CLINC150 with all 150 intents as options:

89.5% with intent names alone, 93.8% with a one-line description per option. See

[Choice questions with up to 256 options](#choice-questions-with-up-to-256-options).

* **How to write prompts, measured.** One-line "use when" descriptions for similar options help; question wording, JSON

vs plain text and extra instructions make no measurable difference. See [Writing System 1 prompts](#writing-system-1-prompts).

* **256K-token prompts.** Decisions that hinge on one sentence hidden at a random depth in up to 250K tokens of text:

20 of 20 correct at every length tested. See [Context length](#context-length).

## Highlight: zero-shot short-video recommendation (for TikTok-style feeds)

A short-video feed has to decide, for every user and every new clip, whether to show it. JEV-27B-VL System 1 can make that

call from the pictures alone: it looks at the covers of the last 5 videos a user watched plus one candidate cover, and returns

**P(this user watches it)** in one forward pass. It needs no interaction logs, no item embeddings and no training, so a

brand-new video can be ranked the moment it is uploaded, before anyone has watched it.

Offline evaluation on [MicroLens-100k](https://github.com/westlake-repl/MicroLens), a public dataset from a real short-video

platform with the original covers: for 200 users, the video each one actually watched next is hidden among 19 videos other

users watched within ±3 days (what the feed would have been showing), and every method ranks the 20 candidates.


| method | uses interaction logs? | AUC | HR@5 | NDCG@10 |

|---|---|---:|---:|---:|

| random order | no | 0.489 | 0.205 | 0.219 |

| title similarity (TF-IDF) | no | 0.602 | 0.385 | 0.367 |

| JEV-27B-VL System 1, titles only | **no, zero-shot** | 0.649 | 0.455 | 0.399 |

| **JEV-27B-VL System 1, covers only** | **no, zero-shot** | **0.727** | **0.590** | 0.498 |

| item-based collaborative filtering | yes, 59,045 other users | 0.728 | 0.490 | 0.503 |

* **Matches collaborative filtering with zero behaviour data:** the same AUC as item-based CF learned from 59,045 users'

watch histories (difference 0.000, 95% interval −0.041 to +0.040), and a higher top-5 hit rate (59% vs 49%).

* **Solves the cold-start problem:** collaborative filtering needs co-watch history; JEV only needs the cover, so new videos

and new creators can be recommended from the first second.

* **Pictures beat words:** covers alone score +0.078 AUC over titles (95% interval +0.031 to +0.126).

* **Fast enough for a ranking stage:** 20 candidate covers scored in about 2.6 s on one GPU in the live demo.

Code, data pipeline and a live web demo: [JEV-27B-DEMO / 07-video-recommendation](https://github.com/yuhai-china/JEV-27B-DEMO/tree/master/07-video-recommendation).

## Watch it decide

Every move below is one System 1 decision: a probability for each possible action, read off in a single forward pass. The

panels show the action probabilities and the time each decision took.

**Super Mario Bros.** Each step is a movement choice plus a jump decision, about 150 ms per decision (video at 5× speed).

<video src="https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/videos/vl1.mp4" controls autoplay loop muted playsinline width="100%"></video>

**Rubik's Cube.** Solving a 25-turn scramble, choosing each move from 11 standard formulas, about 46 ms per decision

(10× replay).

<video src="https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/videos/vl2.mp4" controls autoplay loop muted playsinline width="100%"></video>

**Tetris.** Choosing the column and orientation of each piece, about 122 ms per decision (5× speed).

<video src="https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/videos/vl3.mp4" controls autoplay loop muted playsinline width="100%"></video>

**Quick, Draw!** Guessing what a player is sketching, stroke by stroke, among 16 answers. On 320 real players' sketches

from the Quick, Draw! dataset it names the drawing **88%** of the time once it is finished, and **62%** of the time when only

60% of the strokes are down (random guessing: 6%). About 270 ms per guess.

<video src="https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/videos/vl4.mp4" controls autoplay loop muted playsinline width="100%"></video>

**Snake.** Looking at the board and a list of positions, choosing a direction at every step: 18 pieces of food in 131 steps,

about 300 ms per decision (4× playback).

<video src="https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/videos/vl5.mp4" controls autoplay loop muted playsinline width="100%"></video>

## Highlight: agent judge

**Plan-RewardBench (ACL 2026).** Which of two complete trajectories of a tool-using agent is better? On 1,171 pairs (planning,

error recovery, safe refusal, tool irrelevance), System 1 reaches **73.2%** macro-average accuracy, the top of the paper's

table (Table 4): Qwen-Plus 70.0, DeepSeek-V3.2 69.6, Inf-ORM-Llama3.1-70B 69.2, Gemini-3-Flash 69.1, GPT-5 68.5.


**AgentRewardBench (2025, with screenshots).**

Did the web agent really finish the task? Given the user's goal, the agent's actions and final message, and the final

screenshot, System 1 returns P(task completed) in one forward pass, zero-shot. On

[AgentRewardBench](https://agent-reward-bench.github.io) (McGill, 2025; 1,302 expert-labelled trajectories from GPT-4o, Claude

3.7, Llama 3.3 and Qwen2.5-VL agents) its precision is **higher than every judge on the official leaderboard at that judge's own

recall**:

| judge | its precision | its recall | **JEV-27B-VL at the same recall** |

|---|---:|---:|---:|

| Rule-based | 83.8 | 55.9 | **86.2** |

| WebJudge (o4-mini) | 82.0 | 47.8 | **87.7** |

| WebJudge-7B (trained judge) | 75.7 | 58.0 | **86.2** |

| World-State-Model-7B (trained judge) | 71.2 | 72.2 | **77.7** |

| GPT-4o (accessibility tree) | 69.8 | 83.1 | **72.9** |

| Claude 3.7 Sonnet (screenshot) | 69.4 | 76.3 | **75.6** |

| Qwen2.5-VL-72B (screenshot) | 64.5 | 86.1 | **67.9** |

At threshold 0.5: precision 78.4, recall 70.2; AUROC 0.91. All 1,302 judgements in about 4 minutes on one GPU. Code:

[JEV-27B-DEMO / 10-agent-judge](https://github.com/yuhai-china/JEV-27B-DEMO/tree/master/10-agent-judge).


## Highlight: multimodal judge

Given an image, a question and two answers, System 1 says which answer is better, zero-shot and in one forward pass per order.

On [VL-RewardBench](https://vl-rewardbench.github.io) (CVPR 2025, 1,247 human-verified pairs) it reaches **78.3% overall

accuracy**, above every model on its official leaderboard (26 models, updated May 2025):

| judge | general | hallucination | reasoning | **overall** | macro |

|---|---:|---:|---:|---:|---:|

| **JEV-27B-VL System 1** | 58.0 | **83.2** | **78.2** | **78.3** | **73.1** |

| Skywork-VL-Reward-7B | **65.6** | 80.2 | 61.3 | 73.3 | 69.0 |

| Gemini 2.0 Flash | 50.8 | 72.6 | 70.1 | 68.8 | 64.5 |

| Gemini 1.5 Pro | 50.8 | 72.5 | 64.2 | 67.2 | 62.5 |

| GPT-4o | 49.1 | 67.6 | 70.5 | 65.8 | 62.4 |

| Claude 3.5 Sonnet | 43.4 | 55.0 | 62.3 | 55.3 | 53.6 |

Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/autotrust/JEV-27B-VL