JevOut: Natural Context Can Flip Decision Models
Natural context additions flip Jev and peer decision models onto high-confidence wrong choices.
JevOut studies decision models that map language to probabilities over finite choices used to route requests, select tools, and trigger actions. Short, natural-looking context additions, optimized while keeping the source, question, choices, and gold answer fixed, redirect Jev on 312 of 508 initially correct decisions (61.4%); in 229 cases the wrong option receives probability at least 0.7. Across seven datasets, three other decision systems show targeted flip rates of 64.9% to 73.2% on items they initially answer correctly. The authors argue this fragility makes raw probability outputs unreliable decision interfaces.
- Contexts redirect Jev on 312 of 508 initially correct decisions, or 61.4%.
- In 229 cases Jev assigns at least 0.7 probability to the fixed wrong option.
- Three other systems flip 64.9% to 73.2% of initially correct decisions.
- Authors warn these probability outputs are unreliable interfaces for routing and actions.
From the original article
1 min read at arxiv.orgDedicated decision models such as Jev map unstructured language to probability distributions over finite choices, allowing their outputs to directly route requests, select tools, and trigger actions. Yet real-world inputs rarely arrive in isolation: they come with background details and surrounding context.
By Zixiang Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.