ArXiv

Keep The Essentials: Efficient Reference Conditioned Generation via Token Dropping

2026-06-22 · 17:59 UTC ·Rishubh Parihar, Ayush Raina, R. Venkatesh Babu... ·1 min read
Authors
Rishubh Parihar, Ayush Raina, R. Venkatesh Babu...
Categories
cs.CV
arXiv
https://arxiv.org/abs/2606.23682v1
PDF
https://arxiv.org/pdf/2606.23682v1

Brief

Reference-conditioned diffusion models face high runtime cost as reference count grows. Parihar et al. introduce Sparse Context, which trains models with randomized reference-token dropping and at inference selects task-prioritized tokens, keeping only the most informative regions. The approach yields up to 4× speedups for multi-reference and 2× for single-reference generation without degrading visual quality.

Why it matters

Parihar et al. (published 2026-06-22) propose Sparse Context: a method that constructs sparse reference representations by retaining a reduced subset of reference tokens and fine-tuning diffusion models with random token dropping to make them robust and decoupled from any specific token selection rule.

Key details

  • Using task-aware token selection at inference, Sparse Context achieves up to 4× inference speedup for multi-reference generation and 2× for single-reference generation while preserving visual quality on spatially-aligned editing and subject-driven generation (per the paper's abstract).
Source evidence

Abstract

Reference-based diffusion models enable highly controllable image generation by leveraging elements from input images to guide prompt-driven synthesis. However, these models are computationally expensive in runtime, and their cost scales severely with the number of input references. While the efficiency of diffusion models has been extensively studied in the context of prompt-driven generation, it remains largely under-explored in the realm of reference-based models. This setting presents unique challenges not addressed by methods focusing solely on generation. In particular, the wasteful representation of references as dense token grids offers significant opportunities for improvement. In this work, we present Sparse Context, a method for constructing sparse reference representations by retaining only a reduced subset of reference tokens. We observe that even without modifying the model, dropping a significant portion of reference tokens at inference time largely preserves its generation capabilities. To fully realize this potential, we fine-tune the model with random token dropping at varying ratios, encouraging robustness to partial reference representations. Crucially, this training strategy decouples the model from any specific token selection rule, allowing flexible control at inference time. At inference time, instead of random dropping, we apply task-aware token selection strategies that prioritize the most informative regions of the reference images, adapting the token budget to the input and task requirements. Extensive experiments show our method achieves a 4x increase in inference speed for multi-reference generation and an 2x for single reference generation. Importantly, this efficiency is achieved without compromising visual quality across both spatially-aligned editing and subject-driven generation.

Comment: Project Page: https://sparsecontext.github.io