MIST finds video world models fail to execute changes, not only memory
Original title: Execution Before Memory: Diagnosing Memory Failures in Video World Models
M3-Agent more than triples the MIST reveal score by revising instructions so a frozen model executes the change.
The paper argues that video world-model memory tests conflate scene setup, whether an instructed change was carried out, and whether the last frame shows the result. MIST separates the hidden change from later recall and finds the bottleneck is execution: the model sets the scene but does not carry out the change. M3-Agent recursively revises instruction details so the frozen model performs the requested move. Relative to the baseline, it more than triples the MIST reveal score.
- The change stays hidden until after the scene is set.
- The same execution gap appears in general memory evaluations.
- WRBench score T rises by 26.1% and MBench-T score U by 52.7%.
- The method is presented as a stricter baseline for frozen models.
From the original article
1 min read at huggingface.co
A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory.
By Weiyang Jin, Jiwen Yu, Sihan Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.