Bayesian Learning to Select Fair Classifiers
Researchers use Thompson Sampling to pick classifiers that balance accuracy and subgroup fairness online.
The paper studies online selection of the classifier, model, or human labeler that best balances predictive accuracy and subgroup fairness when labels may arrive late. It formulates the problem as a Bayesian multi-armed bandit solved with Thompson Sampling. Experiments on real tabular, large-language-model, and human-labeling tasks report substantial regret reductions versus baselines, alongside theoretical guarantees covering both accuracy and fairness.
- Online policy selects agents balancing accuracy and subgroup fairness.
- Cast as a Bayesian bandit using Thompson Sampling with delayed labels.
- Tested on tabular classifiers, large language models, and human labelers.
- Reports lower regret than baselines, with accuracy and fairness guarantees.
From the original article
1 min read at arxiv.orgEvaluations of classifiers (AI or human or agents) often assume a batch setting in which every agent produces labels on every example in a test set, which can be expensive or impractical in many contexts. This motivates online policies that adaptively select agents to query as examples arrive and update policies as outcomes are observed.
By Saatvik Kher · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.