Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining
A small language model learned a pronoun-gender rule (e.g., 'Sue cried because' → 'she'), reaching 0.94 generalization accuracy by training step 925, then fell to near zero by step 3,500 despite the rule remaining evidenced in the corpus.
Natural ungrokking introduces within-run reversals where a pretraining corpus determines which learned rules persist. Li and Sreedhar run controlled experiments (2 corpora, 3 budgets, 3 seeds) and analyze Pythia checkpoints to show support frequency predicts rule fate, collapse is driven by a competing surface pattern (margin flips within ~100 steps), and destroying but not restoring rules is possible (recovery fails even at 450× support).
Across un-intervened runs (2 corpora, 3 budgets, 3 seeds), a rule's survival is predicted by its support frequency in the training stream; data-to-parameter ratio only modulates collapse depth. Public Pythia checkpoints show the same emerge-then-collapse dynamics, with collapse depth ordered by model scale.Open reader