Labels Override Definitions in Jev-Style Typed Decision Models
Open typed decision models mostly follow option labels, not definitions, because of prompt rendering.
Typed decision models score caller-defined options that each carry a short label and a written definition, an interface used for routing, moderation, and triage. Across four open-weight models, three Qwen2.5 readout methods, eleven tasks, and PolicyBench, answers mostly follow labels rather than definitions. Deleting every definition leaves laya-td accuracy essentially unchanged at 0.8559 versus 0.8487, while renaming options to A and B raises accuracy by 0.1511. The gap tracks prompt rendering: laya writes the label with the definition and von writes only the definition, and swapping that expression removes or creates the bias without changing weights.
- Removing all definitions leaves laya-td accuracy at 0.8559 versus 0.8487.
- Renaming options to A and B improves accuracy by 0.1511.
- Definitions alone still score 0.7971, so they contain usable signal.
- laya includes the label in the prompt; von writes only the definition.
- Changing only that expression flips the bias in both code bases.
From the original article
1 min read at huggingface.co
A typed decision model answers a fixed question about an input by returning a probability for each of several caller-defined options. Each option carries a short label and a written definition, which is where a developer states the rule the model should apply.
By Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud Pedram · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.