Paper trains LLMs to amortize sequential experimental design
Original title: Amortizing Sequential Experimental Design in Reasoning LLMs
Information-gain policies beat training- and inference-time BED baselines on Twenty Questions and transfer to Medi-Q.
Researchers train LLM reasoning that chooses experiments using information-gain rewards, while specifying the generative model independently of the model's own beliefs. They compare generative-model constructions, reward estimation, hypothesis spaces, multi-step objectives, and compute cost with Bayesian experimental design baselines. On Twenty Questions, the learned policies acquire more information, improve task performance, cost less compute, and generalize to unseen categories, with positive transfer to the Medi-Q clinical dialogue benchmark.
- Expected information gain is optimized under a chosen generative model, not the LLM's beliefs.
- Policies outperform inference-time and training-time BED baselines at lower computational cost.
- Twenty Questions policies generalize to categories not seen during training.
- Training on Twenty Questions shows positive transfer to Medi-Q.
From the original article
1 min read at huggingface.co
Despite their extensive pretraining knowledge and strong reasoning capabilities, large language models (LLMs) struggle to adaptively gather information in experimental design tasks. We study how to amortize sequential Bayesian experimental design (BED) into an LLM by explicitly training the reasoning that guides action selection using information-gain rewards.
By Anita Kriz, Antonin Berthon, Nicolás Astorga · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.