DnA: Denoising Attention for Visual Tasks

2026-06-25 · 17:59 UTC ·Ron Campos, Subhajit Maity, Xin Li... ·1 min read

Brief

DnA (Denoising Attention) modifies multihead attention by adding positive and negative queries and projecting their interactions into two subspaces with larger principal angles to separate relevant from correlated-but-irrelevant features. Evaluated with a ViT‑B backbone, DnA improves ImageNet‑1K by 0.8% and yields gains on video tasks (+1.8% video transformers, +0.5% video LLMs); extensive experiments are reported to support design choices.

Why it matters

DnA (Denoising Attention) uses a positive query to select class-relevant image features and a negative query to select closely associated but irrelevant features, then projects their interactions into two distinct subspaces with larger principal angles to promote subspace separation and improved discriminability.

Key details

With a ViT-B backbone DnA yields an absolute +0.8% top-1 on ImageNet‑1K; it also improves video understanding by +1.8% for video transformers and +0.5% for video LLMs.

The authors report extensive empirical analyses that justify the two interacting-subspace design and the claimed denoising effect versus standard softmax attention, which they identify as producing noisy attention patterns.

Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.