ArXiv

OctoSense: Self-Supervised Learning for Multimodal Robot Perception

2026-06-25 · 17:30 UTC ·Anthony Bisulco, Jeremy Wang, Kostas Daniilidis... ·1 min read
Authors
Anthony Bisulco, Jeremy Wang, Kostas Daniilidis...
Categories
cs.CV, cs.RO
arXiv
https://arxiv.org/abs/2606.27317v1
PDF
https://arxiv.org/pdf/2606.27317v1

Brief

OctoSense presents an open-source multimodal sensor platform and a 59-hour time-synchronized driving dataset (stereo RGB, event cams, LiDAR, thermal, IMU, RTK GPS, proprioception). They train a late-fusion masked autoencoder with modality-specific tokenizers and cached tokens to handle heterogeneous frequencies/latencies. The model is real-time (6.68 ms / 112 ms) and outperforms image-only baselines on flow, depth, segmentation, and ego-motion, remaining robust in nighttime/degraded conditions. Summary based on the paper abstract (PDF link provided).

Why it matters

OctoSense releases a 59-hour time-synchronized driving dataset (published 2026-06-25) containing stereo RGB + event cameras, LiDAR, thermal camera, IMU, RTK GPS, and proprioception (CAN bus and quadruped joint angles) across varied environments and times of day, including highly degraded-sensor scenarios.

Key details

  • The proposed model is a late-fusion masked autoencoder with modality-specific tokenizers and cached modality tokens at inference; representation computation is fast: 6.68 ms on an NVIDIA 5090 and 112 ms on an Orin NX.
  • Their multimodal self-supervised approach outperforms existing image-only foundation models on downstream tasks (optical flow, depth, semantic segmentation, ego-motion: translation, rotation, steering angle) and yields robust predictions at nighttime or under degraded sensing; dataset and code are open-source (project page link provided).
Source evidence

Abstract

We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors. We demonstrate multi-modal self-supervised learning using such real-world robotics data, where sensors have different representations, frequencies, latencies and noise. Our approach, a "late-fusion" masked autoencoder, (i) uses modality-specific tokenizers to account for different spatiotemporal characteristics of these sensors, and (ii) caches modality-specific tokens at inference time to process new measurements as they come. This architecture (i) is fast (6.68 ms and 112 ms on NVIDIA 5090 and Orin NX respectively, to compute the representation), (ii) performs better than existing image-only foundation models on tasks such as estimation of optical flow, depth, semantic segmentation, and ego-motion (translation, rotation, and steering angle), and (iii) predicts robustly at nighttime or in situations where sensory data is degraded. See our project page for links to the dataset, code, and supplementary videos: https://abisulco.com/octosense/.