H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning
H-JEPA trains hierarchical action-conditioned world models, raising Visual AntMaze planning success from 18% to 73%.
H-JEPA is an end-to-end recipe for training a hierarchy of action-conditioned JEPA world models, with each level predicting farther ahead in its own latent space. Planning proceeds top-down: the top level optimizes progress toward the goal, and its predictions become subgoals for the planner below. Across four simulated navigation and manipulation environments, hierarchical planning beats a flat JEPA; on Visual AntMaze a three-level hierarchy raises success from 18% to 73% with less planner compute. With inverse-dynamics supervision, the hierarchy also improves offline planning fidelity on real-robot videos from DROID.
- Each JEPA level predicts farther ahead in its own learned latent space.
- Planning is top-down, with higher predictions becoming lower-level subgoals.
- A three-level hierarchy raises Visual AntMaze success from 18% to 73%.
- Inverse-dynamics supervision extends the method to real DROID robot videos.
From the original article
1 min read at arxiv.orgLong-horizon planning with latent world models requires reasoning across timescales and levels of abstraction. Existing task-agnostic JEPA world models predict and plan at a single timescale or with multiple horizons in one shared latent space. We introduce H-JEPA, an end-to-end recipe for training a hierarchy of action-conditioned JEPAs in which each level predicts farther ahead in its own learned latent space.
By Wancong Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.