Two of the people most responsible for scaling the transformer are now betting on a next act.
@MillionInt ran the Reasoning 🍓 team at OpenAI. @arohan was a pre-training lead on Gemini after years at Google Brain and Anthropic. They just started @coreautoai to find what comes next.
Their core argument: (1) models are trained in the lab but deployed in the real world and can't keep learning once they leave; (2) AI research is done by humans today but models will be able to explore and uncover new advances more rapidly and systematically (controversial but timely w this week's petition).
The conversation covers:
— why Jerry expected AGI in 2025 and what changed his mind
— the two kinds of learning from experience, and why RL only captures one
— the computational depth problem baked into today's architectures
— why the biggest labs can't afford to look for a transformer replacement
— the kernel competition where humans + $100K of coding agents found a 60x speedup no frontier model comes close to
— a definition of AGI you can actually test: a model that improves itself with no human in the loop
00:00 Introduction
01:46 Appreciating Transformers
02:44 Scaling Hits Limits
04:54 Why Architecture Matters
05:32 RL Reality Check
07:32 Test Time Learning
09:52 Economics Of Scaling
12:47 Why Start A Company
14:24 Rohan On Transformers
19:11 Computational Depth Problem
20:32 When Transformers Top Out
23:22 Beyond Reinforcement Learning
26:41 Optimization And Efficiency
34:24 Building An Automated Lab
39:45 Kernel Automation Roadmap
Video