Theory says REINFORCE can collapse coverage in LM post-training
Resetting the KL reference in Approximate Policy Mirror Descent is shown to restore global convergence for stylized PPO and GRPO.
Stories per day · 30 days
2 RSS
Relatedreinforcement-learning3actionopd1agent-distillation1causal-inference1confounding1domain-shift1grpo1language-models1offline-reinforcement-learning1offline-rl1olmo1on-policy-distillation1post-training1ppo1qwen31rlvr1self-distillation1tool-use1
18 stories, newest first · one card per story
Stories tagged #policy-optimization
Offline policy optimization under all-policy linear realizability
The method uses one convex regression per layer and has sample dependence of order epsilon to the minus two.
Offline RL method handles domain shift and unmeasured confounding
CAT-CCB and CAT-CMDP use causal transportability and pessimistic optimization, with finite-sample suboptimality bounds.
Function-space policy optimization improves RLVR training for language models
Authors report about 4 percentage points higher average accuracy than the strongest baseline, with peak memory falling from about 70 GB to 56 GB on Olmo3-7B-Instruct.
ActionOPD: Distilling Decisions, Not Tokens
ActionOPD distills black-box teacher decisions into Qwen3 agents, gaining 3.3-6.9 points out of distribution.
RSI-Anything: Unlocking Recursive Self-Improvement with World Models
RSI-Anything uses predictive world models to recursively improve policies across code, GUI, lab, and chemistry tasks.
Rethinking Relative Policy Differences Transfer of On-Policy Distillation
Relative policy differences, not expert strength alone, determine useful on-policy distillation transfer.
ViSkill: Reinforcing VLM Agents with Evolving Visual-Native Skills
ViSkill stores visual skill cards for VLM agents, reaching 0.89 success and 0.91 with cold-start on grid tasks.
Rationale-Guided Policy Optimization: Learning to Reason with Adaptive Rationale Scaffolding
RGPO uses temporary rationale scaffolds to cut reward sparsity in reasoning RL for text and multimodal models.
On the Convergence of Success Conditioning for Policy Optimization
Paper proves success conditioning converges to optimal policies on broad MDP classes, with O(1/ε^p) iterations for discounted MDPs and O(log(1/ε)) for single-period settings.
Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
DepGPO improves RL for terminal agents by building command dependency graphs from execution traces to assign credit to reads and writes that actually affect task outcomes.
Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation
Neighborhood OPSD lifts Qwen3 math scores by routing student prefixes to nearby expert teachers.
PolyOCR-Venus: Unified OCR Foundation Models for Text-Centric Visual Intelligence
PolyOCR introduces multi-scale OCR foundation models and a corrected OCRBench, reporting competitive document and scene-text results.
ReSPO: Reshaped Sequence Policy Optimization for Gradient Starvation in Off-Policy Learning
ReSPO replaces clipping in off-policy RL to fix gradient starvation on reused rollouts for Qwen3.
EvolveTrade: Experience-Driven Policy Refinement for Self-Evolving LLM Trading Agents
EvolveTrade lets LLM trading agents self-refine their tool-use policy from realized portfolio feedback, improving Sharpe ratios.
Bellman Policy Optimization
Bellman Policy Optimization, a critic-free RLVR method derived from Policy Mirror Descent, improves LLM mathematical reasoning without intermediate state-value estimation.
Bellman Policy Optimization
Bellman Policy Optimization is a critic-free RLVR method that reformulates Policy Mirror Descent as a trajectory-level objective for LLM reasoning training.
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
Researchers propose ERPO, enabling test-time reinforcement learning for code generation via probe-executed consensus rewards, rank masking, and entropy regularization.