AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
Researchers release AMLworld-Compact, cutting AML LLM evaluation targets by up to 611x with weighted precision and F1 estimates.
AMLworld-Compact keeps every illicit AMLworld test transaction and samples benign ones from supervised-score strata so LLM evaluation is cheaper. The HT-Coreset protocol reduces targets 271x on HI-Small and 611x on LI-Small, with exact recall and Horvitz-Thompson estimates of precision and F1. At released sizes, F1 RMSE is 0.00–2.58 percentage points versus 4.44–7.41 for uniform sampling. Seven open-weight LLMs stay near a predict-all-illicit precision baseline, and lexical checks pass for 84.5% of correct and 56.2% of incorrect verdicts in a 1,000-output audit.
- HT-Coreset cuts HI-Small targets 271x and LI-Small targets 611x.
- Recall is exact; precision and F1 are Horvitz-Thompson ratio estimates.
- F1 RMSE is 0.00–2.58 points versus 4.44–7.41 for uniform sampling.
- Seven open-weight LLMs score near a predict-all-illicit precision baseline.
- Lexical checks do not reliably separate correct and incorrect verdicts.
- Organizations
- typhoon-ai
From the original article
1 min read at huggingface.co
Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata.
By Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.