Improving LLM Collaboration via Multi-Agent Preference Learning
MAPL improves multi-agent LLM collaboration by learning from preferences instead of hard-to-specify joint rewards.
The paper formulates preference-based multi-agent LLM systems from decentralized and centralized views and introduces MAPL, a general multi-agent preference learning framework. MAPL is instantiated as MARL from human feedback with a learned reward model and as multi-agent direct preference optimization. On collaborative writing, coding, tool use, and travel planning, MAPL improves quality and efficiency and approaches MARL with fixed rewards; MARLHF usually beats MADPO but is sensitive to data coverage and model choice.
- MAPL learns multi-agent LLM collaboration from comparative feedback.
- It covers decentralized and centralized collaboration settings.
- Instantiations are MARLHF with a learned reward and multi-agent DPO.
- Gains appear on writing, coding, tool use, and travel planning.
From the original article
1 min read at arxiv.orgSeveral works have explored multi-agent reinforcement learning (MARL) in LLM collaboration. However, constructing reliable rewards is difficult in practice, as complete and accurate metrics are often unavailable and hard to aggregate. Preference learning provides an alternative by learning from comparative human or AI feedback.
By Shuo Liu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.