JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shifts
JEPA-TTT adapts a pretrained latent world model at test time when environment dynamics shift.
JEPA-TTT continually updates only the latent dynamics predictor of a pretrained action-conditioned JEPA world model during test time, using self-supervised dense replay while freezing the visual encoder and reward head. Planning requires neither a goal image nor online environment reward. Across eight dynamics shifts in four continuous-control environments, it improves planning on every shift. After 500 test-time episodes, autoregressive latent prediction error falls 83% on average and planning performance rises 153% versus the frozen JEPA model.
- Updates only the latent dynamics predictor at test time
- Visual encoder and reward head remain frozen
- Dense replay builds prediction windows at every time offset
- After 500 episodes, prediction error falls 83% and planning rises 153%
Full article165 words · extracted from huggingface.co · click to collapse
World models enable agents to plan by predicting future states of the environment, but their predictions can become unreliable when test-time dynamics differ from those seen during training. We present JEPA-TTT, which adapts the latent dynamics predictor of a pretrained action-conditioned Joint-Embedding Predictive Architecture world model throughout test time. Self-supervised updates accumulate across episodes, while the visual encoder and reward head remain fixed, preserving the pretrained representation and task objective. Planning requires neither a goal image nor online environment reward. JEPA-TTT uses dense replay, which forms prediction windows at every temporal offset, retains them in a growing buffer, and samples minibatches from that buffer for predictor updates. Across eight dynamics shifts in four continuous-control environments, JEPA-TTT improves planning on every shift. After 500 test-time episodes, it reduces autoregressive latent prediction error by 83% on average and improves planning performance by 153% over the frozen JEPA world model. These results show that persistent self-supervised test-time training can adapt a pretrained latent world model under changed dynamics.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2610.00722