ArXiv

Token Radius Attention for Efficient Video Generation

2026-08-03 · 17:05 UTC ·Jiayu Chen, Zhikun Jiang, Maoliang Li... ·1 min read
Authors
Jiayu Chen, Zhikun Jiang, Maoliang Li...
Categories
cs.CV
arXiv
https://arxiv.org/abs/2608.02504v1
PDF
https://arxiv.org/pdf/2608.02504v1

Brief

Token Radius Attention (TRA) targets the quadratic cost of dense 3D self-attention in Video Diffusion Transformers by using per-query attention entropy to compute an analytic token budget and a temporally decayed, query-centered radius, avoiding key ranking. Training-free TRA plus fused entropy extraction, warm-up reuse, and block-sparse masks reduces interactions to 9–19% and speeds models 1.56–2.05× on Wan2.1/2.2 and HunyuanVideo T2V/I2V, while preserving generation quality (based on the abstract; full text not reviewed).

Why it matters

Observation: retained attention density varies per-query and correlates log-linearly with attention entropy; dominant interactions form query-centered neighborhoods with token-dependent radii, motivating a token-specific sparsification strategy.

Key details

  • Method & results: Token Radius Attention (TRA) is training-free and maps query entropy to an analytic token budget and temporally decayed radius (no explicit key ranking). Across seven Wan2.1/Wan2.2/HunyuanVideo T2V/I2V configurations, TRA keeps only 9–19% of interactions and yields 1.56×–2.05× speedups with competitive generation quality (arXiv 2026-08-03).
Source evidence

Abstract

Video Diffusion Transformers (VDiTs) enable high-fidelity generation but incur quadratic cost from dense 3D self-attention. Existing head- and block-level sparse methods share computation budgets across queries, overlooking token-specific attention demand. We observe that retained density varies across queries yet correlates log-linearly with attention entropy, while dominant interactions form query-centered neighborhoods with token-dependent radii. Based on these findings, we propose Token Radius Attention (TRA), a training-free framework that maps query entropy to an analytic token budget and converts it into a temporally decayed radius without explicit key ranking. Fused entropy extraction, warm-up reuse, and block-sparse mask construction further reduce overhead. Across seven Wan2.1, Wan2.2, and HunyuanVideo T2V/I2V configurations, TRA retains only 9-19% of attention interactions and achieves 1.56x-2.05x speedup with competitive generation quality. Code is available at https://github.com/IF-LAB-PKU/Token-Radius-Attention.