ArXiv

Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

2026-06-24 · 17:27 UTC ·Juliana Li, Diya Sreedhar ·1 min read
Authors
Juliana Li, Diya Sreedhar
Categories
cs.LG, cond-mat.dis-nn, cs.AI, cs.CL
arXiv
https://arxiv.org/abs/2606.26050v1
PDF
https://arxiv.org/pdf/2606.26050v1

Brief

Natural ungrokking introduces within-run reversals where a pretraining corpus determines which learned rules persist. Li and Sreedhar run controlled experiments (2 corpora, 3 budgets, 3 seeds) and analyze Pythia checkpoints to show support frequency predicts rule fate, collapse is driven by a competing surface pattern (margin flips within ~100 steps), and destroying but not restoring rules is possible (recovery fails even at 450× support).

Why it matters

A small language model learned a pronoun-gender rule (e.g., 'Sue cried because' → 'she'), reaching 0.94 generalization accuracy by training step 925, then fell to near zero by step 3,500 despite the rule remaining evidenced in the corpus.

Key details

  • Across un-intervened runs (2 corpora, 3 budgets, 3 seeds), a rule's survival is predicted by its support frequency in the training stream; data-to-parameter ratio only modulates collapse depth. Public Pythia checkpoints show the same emerge-then-collapse dynamics, with collapse depth ordered by model scale.
  • Forgetting is a displacement by a competing surface pattern (log-probability margin crosses zero within ~100 steps of behavioral collapse). Control is asymmetric: flipping support to counter-evidence reliably kills rules (monotone dose-response for two rules), but re-injecting support—even up to 450× natural levels—fails to restore them.
Source evidence

Abstract

Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ("Sue cried because"), it resolves the next pronoun to she, generalizing to held-out probes (0.94 by step 925). By step 3,500 the same model scores near zero on the same probes, although the rule's evidence is still in the training data. We call this within-run reversal natural ungrokking: the corpus decides, with no trace in the loss curve, which learned rules a model keeps. Which rules survive is predictable from one corpus statistic: how often the training stream shows the rule winning. Across un-intervened runs (two corpora, three budgets, three seeds), support frequency decides a rule's fate; the data-to-parameter ratio only modulates how deeply a doomed rule falls. The same emerge-then-collapse dynamics appear in public Pythia checkpoints, collapse depth ordered by model scale as predicted. The forgetting is a displacement: a competing surface pattern out-competes the rule, and the log-probability margin between them crosses zero within 100 training steps of the behavioral collapse. Control over this fate is asymmetric: the same edit that destroys a rule on demand cannot restore it. Flipping support to counter-evidence in place kills the rule with monotone dose-response in two unrelated rules; but injecting support back, even to 450 times the level that naturally sustains it, buys no recovery. Every confirmatory threshold and prediction was pre-registered before the data it governed was read.

Comment: Foundations of Deep Generative Models (FoGen) Workshop at ICML 2026. 23 pages (5-page main text plus appendices), 5 figures. Code: https://github.com/lijuliana/Natural-Ungrokking