ArXiv

Sharp Stability Threshold and Certification for Designing Stable Residual Architectures

2026-07-16 · 05:13 UTC ·Hyemin Gu, Michael Tyrrell, Tuhin Sahai... ·1 min read
Authors
Hyemin Gu, Michael Tyrrell, Tuhin Sahai...
Categories
cs.LG, stat.ML
arXiv
https://arxiv.org/abs/2607.14576v1
PDF
https://arxiv.org/pdf/2607.14576v1

Brief

The paper introduces the sublinear-growth principle for deep residual architectures, characterizing block velocities by ‖v(x,t)‖ ≤ c‖x‖^q + b and proving q ≤ 1 is the sharp stability threshold via ODE existence results and an HJB optimal-control selection argument. It supplies an algebra of input-magnitude exponents for five primitive operations to certify architectural stability, gives a parameter-free fix that lowers Mamba's q from 5 to 1, and reports experiments on Mamba and PatchTST showing q ≤ 1 yields stable training. Summary based on the abstract (full text not available here).

Why it matters

Proposes the sublinear-growth principle and proves a sharp stability threshold q = 1 for residual-block velocity fields obeying ‖v(x,t)‖ ≤ c‖x‖^q + b (q ∈ [0,1]); classical ODE theory gives global forward flow for q ≤ 1 and divergent velocity fields for any q > 1.

Key details

  • An optimal-control / HJB analysis shows the training optimum is bang–bang on the admissible-class boundary: optima with q > 1 blow up while q ≤ 1 are safe, giving a necessary and sufficient condition for stable training and enabling an arithmetic of input-magnitude exponents across five architectural operations to certify stability at the primitive level.
  • Provides a parameter-free modification that reduces a supercritical Mamba block from q = 5 to q = 1 without layer normalization; experiments on Mamba and PatchTST confirm that q ≤ 1 variants train stably, indicating stability depends on the input-magnitude exponent rather than the mere presence of normalization.
Source evidence

Abstract

We propose \emph{the sublinear-growth principle} for deep residual architectures -- a sharp stability threshold on the input-magnitude exponent of every residual block's velocity field: $$\|v(x, t)\| \leq c\,\|x\|^q + b, \qquad q \in [0, 1].$$ The threshold $q = 1$ is established via two independent arguments. Classical ODE theory gives a global forward flow on $[0, T]$ at $q \le 1$ and exhibits divergent velocity fields at any $q > 1$. The optimal-control analysis, via the Hamilton-Jacobi-Bellman equation, sharpens this to a selection statement: the training optimum is bang-bang on the boundary of the admissible class, so the optimum at $q > 1$ blows up while the optimum at $q \le 1$ is safe by construction. The exponent criterion $q \le 1$ is thereby a necessary and sufficient condition for stable training. It clarifies architectural placements that ensure the stability of training and inference, explaining, for instance, the stabilizing role of layer normalization. The sublinear-growth velocity fields form \emph{the right function space} on which forward dynamics, adjoint sensitivity, and architectural composition are all well-controlled. An arithmetic of input-magnitude exponents under the five operations that build residual blocks enables efficient certification of $q_k \le 1$ at the level of architectural primitives, in place of ad hoc trial and error in the search for stable neural architectural designs. A parameter-free modification reduces the supercritical Mamba block from $q = 5$ to $q = 1$ without layer normalization, demonstrating this point. Experiments on Mamba and PatchTST confirm that the $q \le 1$ variants train stably: the criterion is the input-magnitude exponent, not the presence of a normalization layer.