Search: “policy-optimization”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

Expert-Space Exploration in MoE Reinforcement Learning

ESRL explores MoE expert-routing space during RL post-training, improving Qwen3-30B-A3B Pass@1 by 3.2 points over GRPO without extra compute.

The paper shows perturbing expert routing increases rollout diversity similarly to higher decoding temperature, but naive perturbation degrades quality. ESRL anchors high-confidence experts, restricts stochastic routing to a plausible candidate pool, adapts perturbation strength via router entropy, and replays recorded expert paths during policy optimization. It achieves the best results across top-K, top-1, and shared-expert MoE backbones on math, science, and code tasks; on Qwen3-30B-A3B it improves average Pass@1 and Pass@8 over GRPO by 3.2 and 4.5 percentage points.

Hugging Face daily papers·updated Sep 11, 16:58 UTC · 4d ago·first Sep 10, 20:00 UTC · 5d agoAI research 2 sources

45

Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation

Researchers propose ERPO, enabling test-time reinforcement learning for code generation via probe-executed consensus rewards, rank masking, and entropy regularization.

The paper introduces probe-driven test-time reinforcement learning (TTRL) for code generation, where output-free probe inputs are constructed from problem statements and candidate programs are executed on them to compute a Probe Consensus Reward (PCR). Because PCR can be gamed through spurious consensus, the authors propose Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which turns low-PCR outcomes into conservative negative updates via rank masking and constrains policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:54 UTC · 7d agoAI research1

38

Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport

OptiFlow learns one-step multimodal flow policies for offline RL via state-wise entropic optimal transport, avoiding critic overestimation and mode collapse.

The paper introduces OptiFlow, a framework that frames one-step flow policy learning as a structured sample-allocation problem in offline reinforcement learning. It jointly trains a value-aware reference flow policy and a one-step policy, coupling action samples through state-wise entropic optimal transport where critic values set distillation priority and action-distance cost ensures geometrically compatible pairings. By avoiding direct critic maximization, it anchors the policy to high-value dataset-supported modes without out-of-distribution divergence. Code is released on GitHub and the method performs strongly across diverse offline RL benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:10 UTC · 1d agoAI research

14

Bellman Policy Optimization

Bellman Policy Optimization, a critic-free RLVR method derived from Policy Mirror Descent, improves LLM mathematical reasoning without intermediate state-value estimation.

The paper introduces Bellman Policy Optimization (BPO), a critic-free reinforcement learning method for LLMs with verifiable rewards, derived from Policy Mirror Descent. BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective for autoregressive generation with terminal rewards, avoiding state-value estimation at intermediate states. The authors prove BPO shares the same unique optimal solution as PMD and validate it on mathematical reasoning benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:59 UTC · 1d agoAI research

38

MInTRL: Off-policy Intervention can boost On-policy RL

MInTRL injects sparse judge corrections into on-policy RL rollouts, expanding exploration beyond on-policy sampling while preserving learnability on math and code benchmarks.

Minimal Intervention Reinforcement Learning periodically has a judge-intervention policy replace erroneous suffixes of the current policy's output with short corrections, then returns control, keeping trajectories largely on-policy. Training uses a sequence-level advantage-regression objective that removes the need for importance sampling. Across math and code benchmarks it consistently beats standard on-policy and off-policy baselines, remains effective with self-intervention, and performs best at moderate intervention intensity.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

40

Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation

OPRD distillation enables weak-to-strong generalization by amplifying verifier-supported policy updates, outperforming existing RL and distillation methods with fewer student updates.

On-Policy Reverse Distillation (OPRD) evaluates a weak teacher's policy shift relative to its reference policy on student rollouts and amplifies the verifier-supported component of the student's policy gradient. This rescaling preserves the stationary points of policy optimization while letting the student learn beyond the teacher's capacity ceiling. In successive model transfer and multi-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches, and response-style analysis shows students remain closer to verifier-RL-trained models than to their weak teachers.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

33

Near-Optimal Reinforcement Learning with Multi-Step Transition Lookahead

Theorists prove multi-step lookahead RL planning is NP-hard for every fixed rational discount factor yet give a randomized polynomial-time approximation scheme.

The paper resolves open questions about reinforcement learning with multi-step transition lookahead. It shows exact planning remains NP-hard for every fixed rational discount factor in (0,1), not just discounts arbitrarily close to one, and introduces a randomized polynomial-time approximation scheme for every fixed lookahead depth. Extending to unknown transitions and stochastic rewards via optimism and variance-adaptive confidence bounds, the algorithm achieves cumulative regret matching classical tabular discounted RL up to logarithmic factors.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 16:58 UTC · 5d agoAI research1

22

Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning

Review connects control theory, optimal transport, probabilistic inference, thermodynamics, and machine learning via free-energy optimization under constraints.

The review unifies five fields: control theory, optimal transport, probabilistic inference, non-equilibrium thermodynamics, and machine learning. The common conceptual thread is optimization of free-energy-like functionals under dynamical or statistical constraints. Selected applications are presented in reinforcement learning, variational inference, and generative modeling. The tutorial-style text assumes no prior familiarity and begins from physics principles.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:16 UTC · 1d agoAI research

15

Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models

Drift-Constrained Optimization reformulates fine-tuning as update-direction selection, letting Qwen3 models improve target tasks within a behavioral drift budget.

The paper specifies a behavioral drift budget before optimization and shows that update direction is the remaining degree of freedom, reformulating fine-tuning as a direction-selection problem. In a stringent QA-only setting where instruct models must still generate multi-step reasoning at inference, a coarse layer-selective probe reverses the failure of QA-only fine-tuning. Across Qwen3-8B and Qwen3-14B, these directions substantially improve scientific reasoning and multilingual translation, matching or outperforming dedicated translation systems over 100+ languages and giving stronger initialization for reinforcement learning.

Hugging Face daily papers·Sep 11, 20:00 UTC · 4d agoAI research

48

Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization

Researchers release OR-Clarify, a benchmark testing whether LLM agents ask clarifying questions before formulating optimization models from incomplete requests.

OR-Clarify evaluates pre-formulation clarification in operations research: each task gives a partial problem description, withholds structured hidden slots, and scores agents via bounded interaction with a simulated user, measuring slot recovery, stopping behavior, silent assumptions, and interaction cost. The authors also propose InterOPT, a two-stage framework that identifies formulation-critical gaps to decide when to ask or stop. In choice-based experiments InterOPT substantially outperforms all baselines in exact slot recovery and remains competitive in the open-ended setting.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

35

Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails

Research shows on-policy expert correction, not imitation fine-tuning, lets weaker agent models catch up under evolved harnesses.

Researchers study how to combine automated agent-harness evolution with lightweight fine-tuning across seven enterprise agent tasks. Naively training weaker models (Qwen3-Coder, Gemma 4) on expert trajectories under an evolved harness regressed performance by 4 to 30 points on all tasks. They propose an on-policy correction pipeline, automated by a meta-level MLE agent, where an expert rewrites only the failing turn of the weaker model's rollout, preserving model-harness fit.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:53 UTC · 7d agoAI research

35

Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails

Research shows imitation of expert trajectories breaks weaker models' harness fit, while on-policy expert correction preserves gains across seven enterprise agent tasks.

The paper studies combining automated agent-harness evolution with lightweight fine-tuning across seven enterprise agent tasks using Qwen3-Coder and Gemma 4. Training weaker models on complete expert trajectories under an evolved harness regressed performance by 4-30 points on all tasks, disrupting model-harness fit. The authors propose an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that rewrites only failing turns and preserves the model's planning style.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

48

PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization

PLC-DPO corrects noisy preference labels during DPO by routing each pair as clean, flip, or tie using calibrated policy-reference margins.

PLC-DPO addresses reversed, weak, or ambiguous preference labels in Direct Preference Optimization by using the calibrated policy-reference margin as online evidence to route each pair's training signal as clean, flip, or tie. This actively corrects supervision direction and strength rather than only filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO achieves the best mean win rate of 60.5 versus 55.5 for the next-best method. Injected-noise and tie stress tests plus human disagreement analysis show the routing stays stable and distinguishes flipped from weakly directional pairs.

Hugging Face daily papers·Aug 30, 20:00 UTC · 16d agoAI research

28

AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems

AgentGrad introduces intervention-guided prompt optimization for LLM multi-agent systems, achieving state-of-the-art results with 2.5x faster optimization.

AgentGrad is a prompt optimization framework for LLM-based multi-agent systems that addresses limitations in textual gradient extraction and aggregation. It uses sequential intervention to identify the agent whose prompt modification resolves a given failure, then applies agent-level supervision and semantic gradient clustering to build generalized gradients. Experiments report state-of-the-art performance across five MAS benchmarks and a 2.5x average reduction in wall-clock optimization time versus the next-fastest baseline.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

32

MasterControl Seventeen Every Time

Governed enterprise analytics study shows deterministic policy execution matched 110/110 answer-and-evidence contracts while runtime agent planning matched none.

The paper studies a governed approach where a language model interprets the question while deterministic policy selects and runs a pre-approved analytical program returning results and evidence. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B only interpreted intent and policy executed the approved program. None of 330 runtime-planning episodes satisfied the full answer-and-evidence contract, whereas the policy-executed analyzer matched 110 of 110. The authors note this is configuration-specific and expressiveness is preserved via relational operations, aggregation, comparison, windows, ranking, and similarity with replayable results.

Hugging Face daily papers·Sep 1, 20:00 UTC · 14d agoAI research

24

Robust Policy Optimization via Adversarial Importance Sampling

Adversarial Importance Sampling estimates worst-case RL returns without extra interactions; authors also release the advrl PyTorch library.

The paper introduces Advis, which uses importance sampling over trajectories from standard training to estimate and optimize verifiable worst-case returns, requiring no additional environment interactions or auxiliary networks. It also releases advrl, a modular PyTorch library of single-file robustness methods and adversarial attacks for reproducible evaluation. The authors show adversarial hyperparameters do not transfer across agents, so they evaluate with 6-14x more attacker configurations than prior work. Effectiveness is demonstrated on continuous control environments.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:40 UTC · 4d agoAI research1

20

MCRL2: Multi-resource Cross-attention-based Representation Learning-augmented Reinforcement Learning for Cloud Microservice Scheduling

MCRL2 augments reinforcement learning with multi-resource cross-attention representations to improve cloud microservice scheduling and load balancing.

MCRL2 combines a multi-resource cross-attention representation learning module (MCRL) with an actor-critic architecture and maximum entropy objective for microservice scheduling. The approach captures interdependencies among nodes, resources, and microservices in data centers. Experiments on real production cluster traces show improvements in load balancing, scheduling success rate, and average completion time versus baselines.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:44 UTC · 4d agoAI research1

15

Quantile-based Loss Filtering for Outlier-Robust Stochastic Gradient Descent

Quantile-k-Loss SGD filters corrupted component losses by quantile sampling, proving linear convergence while outperforming standard and min-k-loss SGD.

The paper proposes Quantile-k-Loss SGD (Q(k)L-SGD), a loss-filtering framework for finite-sum optimization with corrupted components that samples k losses per iteration and updates using an index from the lower empirical q-quantile. The authors prove linear convergence under standard convexity, requiring sample size to scale with the number of corruptions, plus a complementary small-sample probabilistic analysis. Experiments on polynomial regression, regularized logistic regression, and hinge loss show intermediate quantiles often outperform both standard SGD and min-k-loss SGD.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:38 UTC · 4d agoAI research

18

Safe Meta-Reinforcement Learning via Information Space Reachability

Safe meta-RL framework reasons about safety in information space, learning a safety value function used for safety filtering and constrained policy optimization.

The paper proposes safe meta-RL that reasons about safety in information space, capturing both physical state and the agent's belief over the underlying task. A safety value function measures the probability of avoiding unsafe regions indefinitely and satisfies a self-consistency condition and Bellman equation, making it learnable via meta-RL. The resulting algorithm uses the learned function for safety filtering and constrained policy optimization, with effectiveness demonstrated on meta-RL benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:27 UTC · 1d agoAI research

25

A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning

Paper unifies regularization-based robust RL methods via new performance-gap upper bounds and jointly learned Lagrange multipliers.

The authors derive new upper bounds on the gap between nominal and worst-case deep RL policies, each expressible as an existing regularization objective plus a KL-divergence penalty. Robust training is reformulated as constrained optimization, where prior methods correspond to a fixed Lagrange multiplier. The multiplier is instead updated jointly with the policy, auto-tuning the regularization weight. Adversarial evaluations across several continuous control tasks validate the theory.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:45 UTC · 4d agoAI research1

20

Silver Rate Is (Almost) Optimal for Gradient Descent Acceleration

New optimization theory paper proves near-optimal lower bounds for gradient descent with predetermined stepsizes, confirming silver-schedule optimality.

The paper studies the limits of accelerating gradient descent using predetermined nonnegative stepsizes in smooth convex optimization, with the key constant p_sil = log2(1 + sqrt(2)). It proves a non-anytime lower bound of Omega(n^(-p_sil - O(sqrt(log log n / log n)))) on the error achievable by any such stepsize schedule. In the anytime setting, it shows every infinite nonnegative schedule must incur error Omega(n^(-2*p_sil/(1+p_sil) - O(sqrt(log log n / log n)))) at infinitely many horizons. Combined with the silver-schedule upper bound of Altschuler and Parrilo (2025) and the anytime upper bound of Zhang et al. (2025), these results determine the optimal polynomial convergence exponents in both settings.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:59 UTC · 7d agoAI research

28

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

FlowBalance is a verifier-grounded self-improvement method that beats FlowRL on Qwen3-4B and Qwen3-8B math reasoning while improving training stability.

FlowBalance calibrates dense self-guidance scores with verifier-derived group advantages: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when rollout groups show no outcome preference. The method exponentially reweights a reference policy via trajectory balance, with guarantees including within-group contrast preservation and a minimum-change reverse-KL characterization. On mathematical reasoning it outperforms FlowRL on Qwen3-4B and Qwen3-8B, trains faster and more stably, avoids direct OPSD's response-length collapse, and shows higher correct-strategy diversity on AIME24.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

32

Learning-Guided Planning in Large Dynamic Action Spaces: Budgeted Tree Search for One-to-Many Mobile Charging

LP-BTS uses graph proposal policies, learned critics, and budgeted PUCT search to plan mobile charging across dynamic action spaces up to 2,813 stops.

LP-BTS is a learning-guided planning architecture for one-to-many mobile charging, where N=250 sensors induce roughly 1,125 initial candidate charging stops. A graph proposal policy concentrates candidate support, a learned value critic evaluates leaves, and edge-budgeted PUCT compares simulated futures, letting a single frozen checkpoint cover action universes from 736 to 2,813 stops. On a sealed 30-scenario confirmatory bank it attains the highest observed survival (0.4545) and alive-AUC (0.8031), though its +0.0066 survival edge over the strongest engineered comparator is statistically unresolved.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 16:42 UTC · 18h agoAI research

13

Multi-Agent Reinforcement Learning for Autonomous UAV Exploration in Wildfire Response

Researchers trained multi-agent deep reinforcement learning UAV agents for autonomous wildfire monitoring, with converging policies tracking fire boundaries in simulation.

The study develops a deep reinforcement learning framework for training UAV agents to navigate and monitor simulated wildfire environments. Agents showed increasingly stable and effective behavior over time, evidenced by converging loss trends, improved rewards, and consistent navigation patterns such as fire-boundary tracking. The findings highlight DRL-based UAV potential for autonomous wildfire monitoring and show that environmental structure and reward design influence policy effectiveness.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:45 UTC · 6d agoAI research1

18

One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

A review paper frames on-policy self-distillation collapse as governed by three levers: token weighting, privileged information, and guidance decay.

The paper critically reviews On-Policy Self-Distillation (OPSD), where a language model trains on its own generations scored token-by-token by a teacher conditioned on privileged information such as reference solutions or environment feedback. It identifies collapse, the progressive narrowing of producible reasoning paths, as the dominant failure mode and analyzes it through three levers: signal weighting, the nature of privileged information, and teacher dynamics. The review is restricted to mathematical reasoning, reports no new experiments, and offers a shared vocabulary separating settled findings from disputed ones.

Hugging Face daily papers·Aug 25, 20:00 UTC · 21d agoAI research

45

ScienceBuddy: Recursive-in-Recursive Self-Improvement for Interactive Scientific Agents

ScienceBuddy couples harness evolution with model reinforcement learning so scientific agents continually self-improve from researcher feedback in an interactive workspace.

The authors release ScienceBuddy, an interactive scientific research workspace that transforms researcher requests, feedback, and execution evidence into tasks and evaluation rubrics for continual learning. Its recursive-in-recursive self-improvement paradigm couples harness evolution (inner recursion, model fixed) with model reinforcement learning under the improved harness (outer recursion). Case studies span four scientific task families, and the system is released to the scientific community as a research product.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:55 UTC · 17h agoAI research 2 sources

35

Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks

Researchers propose Feedback-Enriched Environments (FEEs) that reduce reward sparsity and improve RL training of Qwen3-based agents on SciWorld and BFCL.

The paper proposes shifting from agent-side warmup (SFT) to environment-side adaptation via Feedback-Enriched Environments to address severe reward sparsity in RL training of long-horizon LLM agents. A pilot study defines a feedback strategy that transitions from action guidance to observation enrichment in later training stages. Large-scale experiments on SciWorld and BFCL across Qwen3 model scales and GRPO, GSPO, and DAPO show consistent gains, plus stabilized training dynamics and proactive exploration.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

25

When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay

Researchers derive an exact law linking learning-rate schedules and weight decay in normalized networks, pinpointing when scale-invariant optimization destabilizes.

The paper shows that normalization makes large parts of neural networks scale-invariant, creating a hidden feedback loop where learning-rate schedules and weight decay interact through the parameter norm to control the effective optimizer step. An exact discrete-time law with a single scalar quantity separates contraction- and expansion-dominated effective learning-rate regimes, and the balance point is intrinsically unstable, so constant learning rate with weight decay produces recurrent behavior instead of a stable equilibrium. A unified homogeneous-optimizer framework explains why adaptive methods stabilize more weakly under normalization. The law is validated with high precision on MLPs, CNNs, and GPT-2 across MNIST, CIFAR, WikiText, and OpenWebText, with code released on GitHub.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:46 UTC · 7d agoAI research

30

Agent as Policy for Robotic Manipulation

Agent as Policy lets a general-purpose agent drive a physical robot via runtime reasoning and program generation, reaching 100% success on manipulation tasks.

The paper introduces Agent as Policy (AGP), which puts task planning and execution for a physical robot under a general-purpose agent's control with no task-specific or environment-specific training. The agent interprets visual evidence, writes executable programs, issues motion commands, and revises actions based on physical outcomes. AGP was evaluated on real-world manipulation tasks including assembly from human videos, block construction from goal images, die reorientation, targeted throwing, and bimanual towel folding. It achieved success rates of 100%, 100%, and 80% on three block construction configurations.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

40