Reader · no content
No body text on file.
Open the original to read the full piece.
The next big breakthrough will be AIs learning on the job
Brief
Dwarkesh surveys technical remedies: architecture work (sparse attention, KV compaction) helps but may not be the core bottleneck; the loss function and training recipe matter. He outlines On-Policy Self-Distillation (OPSD) — distilling a long-session “teacher” back into a base model via per-token mismatches — as a promising, sample-efficient way to consolidate session learning without catastrophic overwrites, and contrasts it with naive supervised fine-tuning and RL. He also describes a more speculative “dreaming” or test-time training idea (inspired by EfficientZero) where models generate internal simulators to rehearse vast quantities of experience. He references Dario’s comment about training vs serving context lengths and an impromptu OPSD lecture with Sasha Rush. Finally, he sketches a near-term scenario (2027–2028) where agents co-work for week-long sessions, receive thumbs-up reviews, and distill learned behaviors back into weights — enabling deployed models to improve from economy-wide interactions, a change he calls simultaneously exciting and alarming.
Why it matters
Dwarkesh: Labs are betting on large-scale RLVR — training agents on “millions of verifiable tasks across thousands of diverse RL environments” — hoping this will produce problem-solving agents that can sustain open-ended work and approach AGI.
Key details
- Dwarkesh: Progress on real-world computer use lags because such tasks are hard to make “grindable” (deterministic, replayable simulators); unlike coding (where you can clone containers), web workflows resist parallel rollouts and botting, slowing sample-efficient RL improvements.
- Dwarkesh: Around 30–50% of a lab's compute is spent on inference during deployment and currently doesn't improve model weights; he argues that the most valuable learning signal appears at deployment and is being wasted without continual learning into the weights.
- Dwarkesh: Proposes On-Policy Self-Distillation (OPSD) as a practical continual-learning loss: train the base model to match a veteran teacher's per-token predictions after a long contextual session, giving denser supervision than a single reward signal and avoiding naive transcript memorization from SFT.
- Dwarkesh: Suggests a speculative complementary approach — “dreaming” or test-time training (inspired by EfficientZero/AlphaZero) where models build internal simulators to rehearse strategies — and sketches a 2027–2028 scenario where weekly deployed sessions are thumbs-up/downs and distilled back into weights.