ArXiv

Feature Bagging Provides Stability

2026-07-29 · 14:26 UTC ·Yuheng Ma, Qiang Sun ·1 min read
Authors
Yuheng Ma, Qiang Sun
Categories
stat.ML, cs.LG, math.ST
arXiv
https://arxiv.org/abs/2607.26964v1
PDF
https://arxiv.org/pdf/2607.26964v1

Brief

Feature bagging is studied through algorithmic stability by introducing feature instability (FI), analogous to instance instability (II). The authors analyze bagging in a linear parametric model and a model-free, recursive feature-subsampling setting, proving that feature bagging reduces instability (more so with aggressive subsampling) and that few bagging rounds suffice to approach infinite-bagging stability. Summary based on the paper abstract; full text was not available here.

Why it matters

Yuheng Ma and Qiang Sun (arXiv:2607.26964v1, published 2026-07-29) introduce feature instability (FI), the feature-axis analogue of instance instability (II), and report experiments showing FI captures generalization-relevant information complementary to II.

Key details

  • They analyze feature bagging (aggregating base learners trained on random feature subsamples) in a parametric linear model and a model-free, random-forest-inspired setting, proving formal guarantees that bagging improves stability versus non-bagged learners, with larger gains under more aggressive subsampling and rapid convergence to the infinite-bagging stability limit with a modest number of rounds.
Source evidence

Abstract

We study feature bagging through the lens of algorithmic stability. Feature bagging is an ensemble strategy that aggregates base learners trained on randomly subsampled feature subsets, possibly in a data-dependent manner. We introduce feature instability (FI), the feature-axis analogue of instance instability (II), which measures sensitivity to removing a single feature. Smaller values of II or FI correspond to stronger stability, and our experiments show that FI captures generalization-relevant information complementary to II. Within this framework, we analyze feature bagging in both a parametric linear model and a model-free setting inspired by recursive feature subsampling in random forests. In both settings, we establish formal guarantees showing that feature bagging improves the relevant stability relative to its non-bagged counterpart, with larger improvements under more aggressive subsampling. We further show that a modest number of bagging rounds is sufficient to approach the infinite-bagging stability level.