From 650 s to 72 s: reference-to-video on a single RTX 4090
Disclosure: we wrote the open-source node described here, and we sell optimisation work to teams. The offer is at the end, clearly marked. The technical part stands without it.
A 5-second, 24 fps reference-to-video clip with MiniMax H3 took about 650 s on our RTX 4090 with the native dense pipeline at 50 steps. The same clip now takes about 72 s warm (95 s on the first run), and peak VRAM went from about 22.5 GB to about 13.5 GB.
This post explains what we changed, how we measured it, what we did not measure, and what you can reuse even if you never touch this model. Code, workflow and videos: github.com/Kablex/ComfyUI-Ref2VA-VSA.
Measurement setup
Every number below comes from the "Comprehensive Performance Comparison" table in the repo README. The README states that the compared videos were generated on the same NVIDIA GeForce RTX 4090 (24 GB), with the same prompt, the same character reference image, the same seed (981445682258077) and the same resolution: 1344×768, 5.16 s at 24 fps (124 frames).
| Pipeline | Attention | Steps / sampler | 1st run | Warm | Peak VRAM |
|---|---|---|---|---|---|
| MiniMax H3 native dense | Dense global softmax | 50 / Euler | ~650 s | ~650 s | ~22.5 GB |
| H3 Turbo LoRA (dense) | Dense global softmax | 4 / Euler | ~210 s | ~190 s | ~21.0 GB |
| Video Delta Net (VDN-H3) | Windowed + linear delta | 8 / er_sde | 213.0 s | ~135 s | ~18.5 GB |
| Ref2VA VSA (this repo) | 75 % video sparsity + dense prefix | 4 / Euler | 95.2 s | ~72 s | ~13.5 GB |
We report the first run (95 s) and a later warm run (~72 s), as reported in the README. The README reports one configuration on one GPU, so read these as single-setup measurements, not as a distribution.
The image and the videos
Here are the reference image and the two videos from the table, as published in the README. The time on each clip is the first-run time.
Powered by MiniMax H3 (MiniMax). MiniMax H3 is licensed under the MiniMax H3 Community License Agreement. Not affiliated with MiniMax.
Everything in this section is AI-generated. The person in the reference is synthetic, generated by us with SDXL.
Where the time went
Two changes stack:
- Fewer steps. A 4-step Turbo LoRA (loaded at strength 1.0) replaces the 50-step schedule. In the table, that takes dense attention from ~650 s to ~190 s.
- Sparse attention on the generated video. On top of the 4-step setup, attention over the generated-video tokens is pruned to the top 25 % of tiles (75 % sparsity). That takes ~190 s to ~72 s.
Dividing the table's numbers: 650 → 190 is about 3.4×, 190 → 72 is about 2.6×, and 650 → 72 is about 9×. The bases differ, so check which one you are quoting. The 9× against native H3 is warm (650 / 72). The README's 2.24× against VDN-H3 is first run (213.0 / 95.2). Warm against warm, VDN-H3 vs VSA is 135 / 72, about 1.9×.
One thing in the table is worth stopping on. Going from 50 steps to 4 is 12.5× fewer steps, but wall time only dropped about 3.4×. The README has no breakdown of that, so we do not explain it here. It is the kind of gap you find by profiling.
The method
The idea is Visual Sparse Attention (VSA), introduced for text-to-video by FastVideo. Video tokens are grouped into 3D spatial-temporal tiles of 4×4×4 (64 tokens). A learned gating projection (to_gate_compress) decides which tiles each query attends to, and 75–90 % of tiles are pruned. Attention cost falls because most key tiles are never visited.
Reference-to-video breaks the obvious way of doing this. The model prepends conditioning segments (reference image latents, reference audio latents and text tokens) in front of the generated video. If you tile the whole sequence naively, tokens from different modalities land in the same tile. That corrupts the conditioning masks and hurts identity fidelity, which is the whole point of a reference model.
The patch (Ref2VAVSAGatePatch) uses a two-tier layout:
- Dense-exempt prefix. Text, reference image and reference audio tokens are packed into segment-pure tiles. Those tiles are exempt from top-k pruning, so reference tokens are always attended densely.
- Sparse video only. Top-k pruning applies only to key tiles of the generated video.
- Gate transplant. The 50 learned
to_gate_compressmatrices are copied onto the Ref2VA base model (minimax_h3_ref2va_pruned_int8_convrot.safetensors). The gate file is distributed separately, andtools/extract_vsa_gate.pycan extract it from a FastH3 VSA checkpoint.
The benchmark workflow wires it as: UNETLoader (Ref2VA INT8) → Turbo 4-step LoRA → Ref2VAVSAGatePatch (sparsity 0.75) → sigma shift (video 12, audio 3) → 4-step simple scheduler with the euler sampler.
A small design choice we liked: the CUDA path requires BF16 and head dim 128, and the code raises an error otherwise instead of silently falling back to the quadratic eager attention. A silent fallback would be slower (we have not measured by how much), and you would blame the wrong thing.
Limits and trade-offs
- One GPU, one setup. All benchmark numbers are from an RTX 4090. The README lists 24 GB cards such as the RTX 3090, A5000 and L40S under requirements, but it publishes no timings for them. We do not know their speed.
- The node is marked experimental. Its display name says so. The README's own wiring uses sparsity 0.75, while the node's default is 0.70 and its description suggests starting at 0.50–0.70 and comparing against the same dense seed. Do that.
- Quality is not scored. The README shows output videos for the VSA run and the VDN-H3 run, and you can watch them above, in "The image and the videos". It does not publish a quality metric, and it does not include the native 50-step output for comparison. We therefore do not claim "same quality". Quality is not measured in the README, so check any fidelity cost on your own cases.
- VRAM attribution. The README reports peak VRAM per pipeline (22.5 → 13.5 GB) but not how much of the drop comes from each change. Do not read it as "sparsity saves 9 GB".
- Dependencies. It needs a recent ComfyUI with native MiniMax-H3 support, comfy-kitchen built with CUDA
sol_attn, and the checkpoints listed in the README. The node's own error messages also point to a ComfyUI pull request (#15958) for the gate attribute. - The model has its own licence. The node code is Apache 2.0. The README says nothing about the licence of the MiniMax H3 weights, which is separate and may restrict some uses or territories. Read it before you use the model.
What transfers to other pipelines
None of this is specific to video. The habits are:
- Profile before you optimise. The step-count result above is a reminder that the thing you assume dominates the runtime may not. Split time per stage first.
- Fix the test. The README's two videos (VSA and VDN-H3) share the GPU, prompt, reference, seed and resolution. Otherwise you are comparing noise.
- Report cold and warm separately. A 95 s first run and a 72 s warm run are both true. Your users may live in either.
- Measure memory, not just time. Peak VRAM decides whether a job fits on the card at all.
- Change one thing per row. The table moves one lever at a time (steps, then attention), so you can see what each bought.
- Fail loudly. Guard the fast path so a misconfiguration cannot silently run the slow one.
- Keep the quality check next to the speed number. Same seed, side by side, on your real cases.
We do this for teams
Commercial section. Grawwww, the studio that wrote this node, sells this service.
Pipeline diagnosis, €300 excl. VAT, one week, deducted if we continue → https://grawwww.xyz/en/teams/
You send us one real case from your own pipeline and models. In a week you get a written report: what we measured, where time, money or quality is lost, how much can be gained, and a closed-price quote to do it. If there is no gain worth chasing, the report says so with numbers.
This is optimisation and diagnosis of your own pipeline, not a hosted generation service.
Another article: Agents don't review themselves →