Skip to content
NewA calmer, faster ZeroHour

SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation

Hugging Face daily papersby Mikhail Dereviannykh, Vikram Voleti, Simon Donne ()1 min read3
AI summary · grok-4.7Summaries by Vani AI

SemanTok aligns video tokens with frozen DINO features so short prefixes carry clip semantics.

SemanTok is a flexible video tokenizer that feeds frozen DINO features into its encoder and reconstructs them from each retained token prefix with lightweight heads. A 201M-parameter SemanTok autoregressive model matches or beats a VideoFlexTok model 3.4 times larger, and larger SemanTok models further improve fidelity. Short prefixes are cheaper to predict and produce better generation quality, leaving pixel detail to later tokens. Semantic alignment holds on out-of-distribution classes and at every decoder noise level, including pure noise.

Key facts
  • SemanTok reconstructs frozen DINO features from each retained token prefix.
  • A 201M SemanTok model matches a VideoFlexTok model 3.4 times larger.
  • Short prefixes cost less to predict and improve generation fidelity.
  • Alignment holds for out-of-distribution classes and at pure noise.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics.

Hugging Face daily papers

By Mikhail Dereviannykh, Vikram Voleti, Simon Donne · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.