ArXiv

Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation

2026-06-22 · 17:19 UTC ·Kuancheng Wang, Vaibhav Saxena, Shuo Cheng... ·1 min read
Authors
Kuancheng Wang, Vaibhav Saxena, Shuo Cheng...
Categories
cs.RO
arXiv
https://arxiv.org/abs/2606.23625v1
PDF
https://arxiv.org/pdf/2606.23625v1

Brief

See2Act addresses manipulation under partial observability by conditioning action prediction on actively inferred viewpoints: it couples action denoising with viewpoint refinement and trains policies using camera poses anchored to keyframe actions from offline demos. Results (abstract) show robust viewpoint recovery under severe occlusion, up to 34% improvement on RLBench, and zero-shot sim-to-real pick-and-place transfer after 50 simulated demonstrations using depth inputs. Summary based on the paper abstract.

Why it matters

See2Act couples action denoising with viewpoint refinement: the policy is trained using camera poses anchored to keyframe actions from offline demonstrations so it jointly learns where to look and how to act.

Key details

  • Empirical gains: the method recovers informative viewpoints under severe occlusions (Ravens) and improves RLBench task performance by up to 34% over prior methods.
  • Sim-to-real: after collecting 50 demonstrations in a digital twin and training with depth observations, the policy achieves zero-shot sim-to-real transfer on pick-and-place tasks.
Source evidence

Abstract

Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.

Comment: Project website: see2act.github.io