Paper tests which rule and input changes drive decision-model answers
Original title: Which Differences Matter? Controlled Tests of Rule Transfer, Joint Reliability, and Information Use in Decision Models
Controlled arithmetic-rule tests separate transfer from joint reliability and show wording and table edits change answers.
An arXiv paper argues that aggregate accuracy does not show which rule and information changes guide discrete answers from decision models. The authors use executable arithmetic rules, semantic rule separation, and crossed rule-and-state tests to measure transfer and joint success separately. Equivalent rewrites can have opposing task-level effects, and a decision service changes answers more across equivalent wordings than identical-text repeats, usually following targets implied by altered tables.
- Adaptation gains rise with nearest-source answer agreement.
- Fully conflicting quartets yield small pooled gains and mixed signs.
- Joint-score departures depend on which associations the reference preserves.
- Equivalent rewrites link transfer gains to how rules are presented.
From the original article
1 min read at arxiv.orgDecision models map stated rules and observed states to discrete answers. Aggregate accuracy leaves unresolved which rule and information changes guide those answers. We connect controlled input changes to transfer, joint reliability, and directed information use through executable arithmetic rules.
By Tianyou Lai · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.