Abstract
Comment: 40 pages
Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers
- Authors
- Chongjian Ge, Hanwen Jiang, Tianyu Wang...
- Categories
- cs.CV
Brief
Chimera is a hybrid visual diffusion transformer that combines O(N) Kimi Delta Attention, Multi-head Latent Attention, short spatiotemporal convolutions, and Sparse MoE, operating on a raster-ordered token stream without positional embeddings. Using a new HeteroP module-wise scaling recipe tuned to Chinchilla-style compute laws, the authors train an 11B-parameter (2B activated) model that achieves 1.7×–7.3× compute efficiency vs a matched full-attention baseline and generalizes zero-shot from 5s to 30s video with only 6.5% FID loss. Summary based on the paper abstract (full text not provided here).
Why it matters
Chimera is a hybrid visual diffusion backbone combining Kimi Delta Attention (KDA, O(N) long-context state tracking), interleaved Multi-head Latent Attention (MLA), modality-aware short convolutions, and Sparse Mixture-of-Experts (MoE); it processes text/image/video tokens in one raster-ordered stream without positional embeddings and uses HeteroP, a module-wise scaling recipe.
Key details
- The authors trained an 11B-parameter Chimera with 2B activated parameters and report compute-efficiency gains: the dense Chimera backbone is 1.7× more compute-efficient (by pretraining diffusion loss) than a matched full-attention Wan-2.1 2B baseline, while the complete Chimera system reaches a 7.3× efficiency improvement.
- Chimera zero-shot extrapolates from 5-second training clips to 30-second videos with only a 6.5% FID degradation in the last five seconds; fitted Chinchilla-style compute-optimal laws show image pretraining splits compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets.