ArXiv

What Can Latent World Models Know? Physical Parameter Identifiability in Multimodal Predictive Representations

2026-07-29 · 15:15 UTC ·Kaizhen Tan, Xin Xu, Siru Tao... ·1 min read
Authors
Kaizhen Tan, Xin Xu, Siru Tao...
Categories
cs.LG, cs.RO
arXiv
https://arxiv.org/abs/2607.27017v1
PDF
https://arxiv.org/pdf/2607.27017v1

Brief

Latent world models are evaluated for physical-parameter identifiability using a certificate-gated protocol and controlled interventions in POKEWORLD and on RH20T (4,258 episodes). The paper shows two organizing mechanisms—inputs limit what is recoverable, while prediction targets determine what enters the latent—and a frontier: some parameters (stiffness) require forecasting touch, while others (drag) are recoverable but not acquired by common prediction objectives.

Why it matters

Certificate-gated interventions in POKEWORLD show contact stiffness enters the latent only when touch is forecast (R^2 = 0.50), but is not encoded when touch is merely fused into inputs (R^2 = -0.02), demonstrating prediction targets—not raw input fusion—drive retention of some parameters.

Key details

  • Drag has a recoverability certificate of 0.89 yet latent readout under deterministic prediction objectives plateaus near R^2 ≈ 0.13, while a supervised head on the same trunk attains R^2 = 0.45, indicating prediction objectives fail to acquire slow/ratio-type parameters even when they are in principle observable.
  • RH20T robot experiments (4,258 episodes across two robots) replicate these mechanisms: arms missing information or prediction pressure show flat identifiability across a fivefold data range, and only the full multimodal predictive objective forecasts force beyond a persistence baseline, with held-out gains that grow with scale.
Source evidence

Abstract

A central premise of latent world models is that predicting the future forces a representation to internalize the physics of its environment. Which physical quantities does a trained latent actually contain, and what decides this? We answer with controlled interventions in POKEWORLD, an interactive environment whose visually identical objects hide mass, drag, and contact stiffness. A certificate-gated protocol first certifies each parameter as recoverable from raw observations, then measures whether it enters the latent, so a null result can be attributed to the objective rather than to the environment. The resulting identifiability map has two organizing mechanisms and one frontier. Inputs limit what can be known, while prediction targets decide what is retained. Stiffness enters the latent only when touch is forecast ($R^2=0.50$, compared with $-0.02$ when the same signal is merely fused into the input), and under single-step prediction a vision-only latent discards even perfectly visible object state. Drag marks the frontier. It carries a recoverability certificate of 0.89 yet plateaus near 0.13 under every deterministic prediction objective we test, while a supervised head on the same trunk reaches 0.45. Parameters whose readout is slow and ratio-type under the sensed coordinates fall outside what these objectives acquire. On RH20T, an input-target factorial across scaling curves reproduces both mechanisms across two robots and 4,258 episodes. Every arm missing information or prediction pressure stays flat over a fivefold data range, and only the full multimodal objective forecasts force beyond a persistence baseline, with held-out gains that grow with scale. Objective structure determines which physical parameters a latent acquires, and additional data improves only the parameters it already acquires.