Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments

2026-08-06 · 15:36 UTC ·Giorgio Tonetti, Laurent Kneip, Abel Gawel... ·1 min read

Brief

Prior-SG addresses region segmentation for hierarchical 3D scene graphs in arbitrarily structured environments by turning graph generation into a probabilistic alignment problem. It fuses RGB-D into an Instance Graph with multi-scale, open-vocabulary features, then performs MAP inference guided by a Prior Graph produced by an LLM and optimized via an MRF combining visual, geometric, and object experts. Validation on simulated and real open-plan scenes shows state-of-the-art segmentation and zero-shot, task-driven re-partitioning. (Summary based on the abstract; full text not reviewed.)

Why it matters

Prior-SG (Tonetti, Kneip, Gawel, Hutter; published 2026-08-06) casts scene-graph generation as a probabilistic alignment: it aggregates an RGB-D stream into a multi-scale, open-vocabulary Instance Graph and infers high-level semantics via a MAP estimate guided by a Prior Graph synthesized on-the-fly by a Large Language Model.

Key details

The system optimizes a Markov Random Field that fuses heterogeneous experts (visual, geometric, discrete objects) to resolve local ambiguities; validated on diverse simulated residential datasets and large open-plan real-world environments, Prior-SG reports state-of-the-art semantic region segmentation accuracy, robustly delineates functional boundaries without walls, and enables zero-shot ontological flexibility for task-driven re-partitioning.

Abstract

Hierarchical 3D scene graphs are a promising representation for high-level spatial reasoning in autonomous mobile platforms. However, existing extraction frameworks typically rely on purely local visual clustering or strict geometric heuristics, such as wall-separated rooms, which fail in open-plan or arbitrarily-structured environments. We propose Prior-SG, a task- and prior-driven framework that casts scene graph generation fundamentally as a probabilistic alignment problem. As the robot explores, it continuously aggregates an incoming RGB-D sensor stream into a physically grounded Instance Graph utilizing a multi-scale, open-vocabulary feature fusion strategy. The system then infers the high-level functional semantics of this map through a Maximum A Posteriori (MAP) estimate, guided by a Prior Graph-a logical expectation of the environment's structure and task-relevant vocabulary synthesized dynamically by a Large Language Model. By optimizing a Markov Random Field that fuses heterogeneous experts (visual, geometric, and discrete objects) with these topological priors, the system resolves local perceptual ambiguities. We validate this approach across diverse simulated residential datasets and large, open-plan real-world environments. Prior-SG achieves state-of-the-art semantic region segmentation accuracy compared to recent baselines, robustly delineates distant functional boundaries in the absence of physical walls, and uniquely provides zero-shot ontological flexibility, enabling the robot to entirely restructure its spatial partitioning based on a given high-level task.