Hugging Face trending modelspublished Aug 31, 20:14 UTC (15d ago)ingested Sep 10, 09:36 UTCViggle

Viggle/Viggle-Animate — new model trending #28 on Hugging Face

AI summary · glm-5.3-flash

Viggle released Viggle-Animate, a 33.1B MiniMax-H3 finetune replacing video characters from one repainted frame, rendering 124 frames in 26 seconds on one GPU.

Viggle-Animate replaces the character in a video using only a driving video and one of its own repainted frames, with no pose estimator, segmentation mask, face tracker, or text encoder. It is a 33.1B full finetune of MiniMax-H3's ref2va transformer, jointly distilled with DMD across two teachers split by noise level, so rendering takes three forward passes per clip. On a B200 GPU it renders 124 frames in 26 seconds, 6.1x faster per clip than Wan2.2-Animate-14B in matched comparisons. The method assumes no person-specific representation, so it generalizes beyond humans; a demo, research write-up, and ComfyUI nodes are available.

Two inputs only: driving video plus one repainted frame; no pose, segmentation, or text encoders.
Joint DMD distillation across two teachers cuts rendering to three forward passes per clip.
6.1x faster per clip than Wan2.2-Animate-14B in matched B200 comparisons.
Method generalizes beyond humans since it assumes no person-specific representation.
Demo, research write-up, and ComfyUI nodes published alongside the weights.

Read the original

VendorsViggle

ProductsViggle-Animate ComfyUI

OrganizationsViggle

AI modelsViggle-Animate MiniMax-H3 Wan2.2-Animate-14B

Tagsviggle-animate video-generation character-replacement minimax-h3 distillation dmd comfyui

Full article2,628 words · extracted from huggingface.co · click to collapse

# Viggle-Animate

### Character Replacement in Video from a Single Repainted Frame

**[Try the demo](https://huggingface.co/spaces/Viggle/viggle-animate)**  · 

**[Research write-up](https://viggle.ai/research/viggle-animate-character-replacement-from-a-repainted-frame?utm_source=huggingface&utm_medium=social&utm_campaign=viggle-animate&utm_content=viggle/viggle-animate)**  · 

**[ComfyUI nodes](https://github.com/Saganaki22/ComfyUI-Viggle-Animate-H3)**  · 

**[viggle.ai/h3](https://viggle.ai/h3)**  · 

Built on **[MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)**

</a>

<sub>Made with Viggle-Animate by <a href="https://x.com/cocktailpeanut/status/2097332291844399514">@cocktailpeanut</a>, not by us. Click through to watch it on X.</sub>

**Viggle-Animate replaces the character in a video with whatever you paint into one of its own

frames** — motion, camera and timing untouched. You prepare that one frame in an image editor;

from there the video stage runs no pose estimator, segmenter, face tracker or text encoder. Two

inputs, three forward passes, 26 seconds a shot on one GPU.

**It is strongest where replacement is hardest: fast motion, and pose transfer accurate enough to

follow it.** Whipping heads, full kicks, jumps — tracked frame for frame, not smeared through.

## Abstract

Controlled character replacement is usually built on intermediate representations — pose skeletons,

segmentation masks, background plates, face crops. Each needs its own extractor, and each extractor

is another model to run and another place to lose information. Recent work drops the skeleton but

keeps a mask channel. **Viggle-Animate uses neither.** Its two inputs are a driving video and one of

that video's own frames with the character repainted, and its only task is to propagate that edit

across the shot.

Because the reference is a frame of the clip, its pose, camera, framing and lighting already agree

with the footage, and nothing downstream has to align them again. The model is never told what the

new character is: no class, no identity encoder, and no user-provided text prompt.

**Viggle-Animate is a 33.1 B full finetune of MiniMax-H3's `ref2va` transformer, jointly distilled

with DMD to three forward passes.** In a matched comparison on the same machine and B200 GPU, using

the same source videos, output resolution and frame count, it renders 124 frames in 26 s, 6.1×

faster per clip than Wan2.2-Animate-14B.

## Method

Character replacement asks two questions at once: *what does the new character look like*, and *how

does it move through this shot*. Systems that condition on a standalone character photograph must

answer both, and reconciling a photograph with footage it was never part of is what the scaffolding

exists for.

State-of-the-art image models have finished that job. Give `gpt-image` a frame and an instruction

and it replaces the character while following the prompt exactly — transferring the pose, matching

the lighting, preserving the background. The hard reconciliation is already solved, once, on one

image. This model is the second half of that pipeline, not the whole of it.

<img src="https://huggingface.co/Viggle/Viggle-Animate/resolve/main/examples/media/pipeline.svg"

alt="Two inputs — a driving video and one of its own frames, repainted in any image editor — enter Viggle-Animate. No pose skeleton, segmentation mask, face crop, background plate, depth map or user-provided text prompt enters the video model." width="100%">

Appearance enters only through the repainted frame; geometry enters only through the driving video.

**The text encoder is never loaded.** Conditioning is one frozen embedding shipped with the weights

([`assets/fixed_prompt.txt`](assets/fixed_prompt.txt)), identical for every render.

Left panel is the driving video, right panel is this model:

**It is fast twice over.** Once the repainted frame exists there is nothing else to run — no pose

estimator, no segmenter, no face tracker, no text encoder. And the sampler is distilled, so a

finished clip is three forward passes rather than thirty. The two compound: one model, one GPU,

and no orchestration to get wrong.

The distillation is **joint, across two teachers split by noise level.** Our finetune supervises the

high-noise end of the schedule, where the replacement itself is decided — it is the model that gets

the swap right. The original MiniMax-H3 supervises the low-noise end, where detail and texture are

decided — it is the model with the better image quality. Distilling each end against the teacher that

owns it keeps both properties in one student, instead of inheriting the finetune's visual

regressions along with its replacement ability.

**It generalizes past humans**, because nothing in the loop assumes one. A pose skeleton has a neck

and two arms; a mask has a person-shaped hole. We have neither, so the model holds no representation

that a character must be a person. What it can animate is bounded by what you can paint.

## Efficiency

<b>per render</b><br>

<small style="opacity:.7">124 frames at 24 fps, 480×832, a single B200</small>

</div>

<b>forward passes</b><br>

<small style="opacity:.7"><code>--steps 4</code> names four sigma boundaries, so three passes</small>

</div>

<b>inputs</b><br>

<small style="opacity:.7">a clip, and one of its own frames repainted</small>

</div>

<b>other models</b><br>

<small style="opacity:.7">in the video stage — no pose estimator, segmenter, face tracker or text encoder</small>

</div>

One B200, 480×832, 124 frames at 24 fps, bf16, no compile, no offload. Wan ran its documented

replacement recipe — 20 steps, `sample_shift 5.0`, `--refert_num 1 --replace_flag

--use_relighting_lora`, `--w_len 1 --h_len 1` — after its own preprocessing pass.

| | Viggle-Animate | Wan2.2-Animate-14B |

|---|---|---|

| Inputs | driving video + one repainted frame | driving video + character image, then a **preprocessing pass** producing pose, face, mask and background tracks |

| Render, after weights load | **26 s** | 160 s |

| — of which sampling | **13.6 s** | 140 s |

| Forward passes | **3** | 40 (20 steps × 2 chunks) |

| Parameters | 33.1 B | 17.3 B |

**6.1× faster per render, 10.3× on sampling alone.** Wan's preprocessing pass is not counted in

its 160 s.

### Qualitative comparison

Four panels each: **painted reference · driving video · this model · Wan2.2-Animate-14B**, the last

at its documented replacement settings. The gap is widest under fast motion: where the comparison

smears, this model stays sharp and lands the pose on the right frame.

## Generalization

The model is never told what it is animating, so how far the character can get from a person is an

empirical question rather than a list of supported categories. Three panels each: **painted

reference · driving video · this model.** Every clip below is one paint and one render at the

shipped defaults, `--seed 42` — no best-of-N.

**Animals.** Ears, eye patches and flippers move on limbs the driving clip does not have — the paint

places them, the render animates them as if they had always been arms and a head.

**Not humanoid.** The airliner is the hardest case we have: the paint binds wings to arms and landing

gear to legs, and the model's job is to keep that binding for 124 frames. The robot has to relight

specular metal as it turns.

Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/Viggle/Viggle-Animate