Selective auditing bounds unaudited AI errors while cutting expert reviews
Original title: Reliable AI-Assisted Decisions via Selective Auditing
A family of algorithms includes one that asymptotically attains the best auditing rate for adversarial and i.i.d. streams.
The paper treats online monitoring of AI decisions that arrive with confidence scores, which may come from a separate verifier. At each round the system either accepts the decision or sends it to an expert who accepts or corrects it. The goal is to minimize expert audits while keeping the fraction of unaudited AI errors under a user-defined threshold. Experiments on harmful-request screening, qualitative text annotation, and data labeling report valid error control and fewer human audits than baselines.
- Guarantees require no assumptions on the data stream.
- The proposer and the confidence scorer need not be the same model.
- The optimal method is asymptotically efficient for both adversarial and i.i.d. streams.
- Reported tasks include screening harmful requests to large language models.
From the original article
1 min read at arxiv.orgWe consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score estimating the trustworthiness of the decision.
By Isaac Gibbs · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.