Mosaic: Multi-Session Agentic Reinforcement Learning
Mosaic improves multi-session agentic RL by crediting session contributions instead of broadcasting trajectory advantages.
Most agentic reinforcement learning assumes one continuous session, but context compaction and subagent forking produce multi-session trajectories. Mosaic applies critic-free hierarchical optimization, adding session-usage signals to trajectory advantages and shaping each session by estimated marginal contribution. On BrowseComp with Qwen3.5-4B, it outperforms advantage broadcasting for both compaction and forking while using fewer sessions on average.
- Extends agentic RL to trajectories with any number of sessions.
- Session-usage signals inform trajectory-level advantage estimates.
- Each session advantage reflects estimated marginal contribution.
- On BrowseComp with Qwen3.5-4B, Mosaic uses fewer sessions than broadcasting.
Full article188 words · extracted from huggingface.co · click to collapse
Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.22174