Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents
DepGPO improves RL for terminal agents by building command dependency graphs from execution traces to assign credit to reads and writes that actually affect task outcomes.
Dependency-Aware Group Policy Optimization (DepGPO) constructs a command dependency graph from terminal execution traces and traces backward from resources inspected by the task verifier. Credit is assigned to relevant writes and supporting reads along these paths, then used to redistribute trajectory advantages across steps. Experiments and ablations show improved task performance and training stability on complex terminal tasks like coding and debugging.
- Builds command dependency graphs from terminal execution traces
- Traces backward from verifier-inspected resources to find outcome-relevant operations
- Redistributes trajectory advantages across steps using dependency-based credit
- Improves performance and training stability on complex terminal tasks
- Organizations
- DepGPO
From the original article
1 min read at arxiv.orgTerminal-using agents benefit from reinforcement learning (RL) in coding, debugging, and other multi-step terminal tasks. In these tasks, later commands often depend on information or intermediate results produced by earlier commands. However, existing trajectory-level and step-level credit assignment methods do not explicitly trace the read-write dependencies through which commands affect the final outcome.
By Yu Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.