Many Right Answers, One Training Example: Removing Responder Noise from GRPO for LLM Recommenders
ReMaRk rewards LLM recommenders against all valid replies jointly, matching Rank-GRPO in fewer steps.
Rank-GRPO trains LLM recommenders against one reply per example, so about 85% of advantage variance reflects which reply was drawn rather than list quality. ReMaRk instead scores every generated list once against the union of all replies' titles and samples each request equally. On seven backbones and three datasets, it reaches the validation recall of 2,000 per-reply RL steps in 25–53% of the steps. Trained until neither improves, it is never significantly behind Rank-GRPO and matches its best with fewer GPU-hours in four of seven pairs; the same joint reward helps on ProtoQA.
- Rank-GRPO's advantage variance is about 85% from reply sampling.
- ReMaRk rewards each list against the union of all reply titles.
- It matches 2,000-step recall in 25–53% of the steps.
- Never significantly behind Rank-GRPO; fewer GPU-hours in four of seven pairs.
- The joint reward also helps on ProtoQA's many human answers.
From the original article
1 min read at huggingface.co
A recommendation request on Reddit draws several replies, each naming different titles. When an LLM recommender is trained on such threads with reinforcement learning (RL), which replies should a generated list be rewarded against? The state-of-the-art recipe, Rank-GRPO, picks one reply per training example, so the advantage mostly records which reply was drawn (85% of its variance) rather than how good the list is.
By Xin Li, Hao Jiang, Jinghao Guo · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.