ArXiv

Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

2026-06-22 · 17:58 UTC ·Dingzhi Yu, Hongyi Tao, Yuanyu Wan... ·1 min read
Authors
Dingzhi Yu, Hongyi Tao, Yuanyu Wan...
Categories
cs.LG, cs.AI, math.OC, stat.ML
arXiv
https://arxiv.org/abs/2606.23676v1
PDF
https://arxiv.org/pdf/2606.23676v1

Brief

The paper examines AdamW under heavy-tailed stochastic-gradient noise observed in LLM pretraining and asks whether its second-moment accumulator prevents convergence. Under this open problem (Yu et al., 2026-06-22), the authors provide a positive weighted-metric benchmark and a corridor-style lower bound that demonstrates how denominator memory can mask large gradients. (Abstract only; full text not assessed.)

Why it matters

The paper (Yu et al., arXiv 2026-06-22) poses the open problem whether AdamW converges under heavy-tailed stochastic gradient noise typical in LLM pretraining, noting existing theory mostly assumes finite-variance noise.

Key details

  • Authors prove a positive weighted-metric benchmark for AdamW and present a corridor lower-bound mechanism showing the optimizer's second-moment (denominator) memory can hide large gradients.
  • By contrast, recent work shows sign-based optimizers (Lion, Muon) attain sharp heavy-tailed rates and AdaGrad can converge under heavy tails; no prior rigorous heavy-tailed convergence result existed for AdamW.
Source evidence

Abstract

AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.