World Action Models Enable Continual Imitation Learning with Recurrent Generative Replays

2026-06-25 · 17:59 UTC ·Manish Kumar Govind, Dominick Reilly, Smit Patel... ·1 min read

Brief

World Action Models (WAMs) are used to generate future visual observations and underpin Recurrent Generative Replay (REGEN), a continual imitation-learning method that synthesizes pseudo-replay trajectories conditioned on prior task instructions and current observations. Evaluated in simulation and real-world manipulation, REGEN cuts catastrophic forgetting by up to 50% versus sequential fine-tuning and nears privileged experience-replay performance; primary failure modes are long-horizon visual degradation and action–observation inconsistency.

Why it matters

Recurrent Generative Replay (REGEN) leverages World Action Models (WAMs) to synthesize pseudo-replay trajectories by recursively querying a generative world+action model conditioned on prior task instructions and current-task observations; evaluated in both simulation and real-world robot manipulation (Govind et al., arXiv 2026-06-25).

Key details

REGEN reduces catastrophic forgetting by up to 50% relative to sequential fine-tuning and approaches the performance of privileged experience-replay methods that require access to stored real demonstration data.

The authors identify long-horizon visual degradation and action–observation inconsistency as the primary bottlenecks limiting the fidelity and effectiveness of generated replay trajectories.

Abstract

Going beyond predicting robot actions, World Action Models (WAMs) can also generate future visual observations. We build on this generative capability to propose Recurrent Generative Replay (REGEN), a continual imitation learning framework that synthesizes pseudo-replay trajectories, enabling a robot policy to rehearse previously learned tasks without storing their original human demonstrations. During continual adaptation, REGEN recursively queries the WAM to synthesize pseudo-replay trajectories conditioned only on prior task instructions and current-task observations. Experiments in both simulation and real-world manipulation settings show that REGEN reduces catastrophic forgetting by up to $50\%$ relative to sequential fine-tuning, while approaching the performance of privileged experience replay methods that require access to real replay data. Finally, we analyze the factors limiting generated replay, identifying long-horizon visual degradation and action-observation inconsistency as the primary bottlenecks. Our results establish WAMs as a promising foundation for continual robot learning without stored demonstrations.