Logged bandit feedback trains calibrated decision heads on Qwen3
Original title: Learning a calibrated decision head from logged bandit feedback
CAPS beats confirmed-answer fine-tuning by 2.6 accuracy points on Banking77 and cuts ECE by 0.053.
The paper adapts a one-pass decision head with LoRA on Qwen3-0.6B-Base and 1.7B-Base using only logged bandit feedback. It finds Brier or log scores of the sampled action are unusable REINFORCE rewards and can collapse accuracy, while pathwise scoring and rewards r and r minus p avoid that. Chosen-Action Proper Scoring (CAPS) beats fine-tuning on confirmed answers only, and at 5,000 samples stays close to full-label training. Code, adapters, logs and per-record results are public.
Unconfirmed: Several reported gains, including emotion data and noisy-label rates, use one seed.
- One run cut ordered-scale accuracy from 0.636 to 0.111 when the reward collapsed.
- On one emotion-data seed with an often-wrong logger, CAPS gained 21.7 points.
- At N 5000, CAPS trailed full labels by 1.5 points at 0.6B and 0.6 at 1.7B.
- Propensity weights added nothing on this log; early stopping blocked noisy memorization in three runs.
From the original article
1 min read at arxiv.orgA decision head reads a state and a list of options and returns a probability per option in one forward pass. Once deployed, it usually observes only whether the option it chose was right: logged bandit feedback. We adapt such a head, LoRA on Qwen3-0.
By Ye Yuan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.