Explainable and Resource-Efficient Spatial Reasoning in Multimodal LLMs for Decision-Critical Applications

2026-07-29 · 17:20 UTC ·Piyush Jain, Kousik Dasgupta, Rajarshi Roy... ·1 min read

Brief

ByDeWay-V2 tackles MLLM spatial errors by combining monocular depth layering with explicit, auditable spatial predicates derived via an open-vocabulary detector (YOLO-World-L). Without training, it injects pairwise projective and topological relations into prompts, improving hallucination grounding and fine-grained object-to-object reasoning; experiments show major F1 gains (46% relative vs LDP) and CPU-friendly operation under 40-token budgets.

Why it matters

ByDeWay-V2 (Jain, Dasgupta, Roy, Tripathi; arXiv 2026-07-29) augments Layered-Depth-Based Prompting by injecting human-readable pairwise spatial predicates (projective and topological: e.g., "left of", "inside") computed from an open-vocabulary detector (YOLO-World-L) plus monocular depth; the method is training-free and provides auditable spatial evidence.

Key details

Evaluation on VSR and BLINK (hallucination grounding via POPE) shows a 46% relative F1 improvement over LDP on the BLINK spatial subset for Qwen2.5-VL, and recovers BLIP-Base on VSR from near-random to F1 = 0.53; the lightest config runs within a 40-token context budget on CPU.

Abstract

As Multimodal Large Language Models (MLLMs) are increasingly deployed in decision-critical pipelines such as robotics, embodied AI, and safety monitoring, the opacity of their spatial judgments limits operator trust and auditability. MLLMs demonstrate strong reasoning but often struggle with fine-grained spatial understanding and object hallucination. Prior work, ByDeWay, introduced Layered-Depth-Based Prompting (LDP), a training-free framework that mitigates hallucinations by structuring prompts using monocular depth estimation. However, coarse depth layering falls short in resolving object-to-object spatial relationships within the same geometric plane, such as projective ("left of", "above") and topological ("inside", "touching") relations. We propose ByDeWay-V2, which integrates explicit spatial relational context alongside depth cues, expressed as human-readable predicates that serve as auditable evidence for downstream decision support. Using an open-vocabulary object detector (YOLO-World-L), our framework computes pairwise geometric relations between detected objects and injects them as structured spatial predicates into the MLLM prompt, bridging 3D scene depth and 2D spatial semantics without any training. We evaluate ByDeWay-V2 on the Visual Spatial Reasoning (VSR) and BLINK benchmarks across multiple MLLMs, with hallucination grounding assessed via POPE. On the BLINK spatial subset, ByDeWay-V2 achieves a 46 percent relative F1 improvement over LDP for Qwen2.5-VL, and recovers BLIP-Base's spatial reasoning on VSR from near-random performance to a competitive F1 of 0.53. Our lightest configuration operates under a strict 40-token context budget on CPU, showing the framework's suitability for resource-constrained, real-time decision-support settings.

Comment: 14 pages