Most robot datasets store one sentence per episode: the task. Real problems need more: a plan, a memory, room to answer questions and change course.
We just shipped full language support in the LeRobot dataset format. Now lerobot-annotate points a vision-language model at each episode and writes the subtasks, plans, memory, and grounded Q&A itself - roughly $1.20 for 100 episodes.
From there, recipes turn the annotated rows into policy-agnostic training messages, so the same dataset can teach planning, memory, tool calls, or low-level control, no dataset redesign required.
Read how it all works on the blog: huggingface.co/spaces/lerobo…
Video