RayPE: Ray-Space Positional Encoding for 3D-Aware Video Generation

2026-06-25 · 17:51 UTC ·Minghao Yin, Jiahao Lu, Wenbo Hu... ·1 min read

Brief

RayPE augments video diffusion transformers with 3D-aware positional encoding by injecting per-token 6D Plucker ray coordinates into queries and keys (with a query/key flip) so attention bilinearly captures the Plucker reciprocal product. The additive design yields separable content, geometry, and cross-terms; stability is achieved via direction/magnitude decoupling, log-magnitude gating, and RMSNorm. The zero‑init module adds <0.1% params to a pretrained video DiT and improves camera control and 3D consistency on a four-dataset mixture.

Why it matters

RayPE injects per-token 6D Plucker coordinates additively into queries and keys of self-attention (with a query/key flip) so the symmetric identity matches the Plucker reciprocal product; the resulting attention score cleanly decomposes into a content term, a geometry term, and two cross-terms, each found necessary by experiments.

Key details

To stabilize across heterogeneous camera-translation scales, RayPE decouples ray direction from moment magnitude, gates the encoding by a learned function of the log-magnitude, and applies RMSNorm to align with QKNorm-normalized content; the module is zero-initialized, adds <0.1% parameters to a pretrained video DiT, and improves camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset training mixture.

Abstract

Modern video diffusion transformers position their tokens through RoPE on the (u,v,t) axes -- a description of the camera's sampling grid that says nothing about the 3D structure of the scene. We observe that the geometric relation between two camera rays is captured by the Plucker reciprocal product, which is bilinear in the two rays -- the same algebraic form as the dot product in Transformer attention. Building on this analogy, we propose RayPE, a positional-encoding extension that injects per-token 6D Plucker coordinates additively into the queries and keys of self-attention, with a query/key flip arrangement under which the symmetric identity configuration coincides exactly with the reciprocal product. The injection is additive, the resulting attention score decomposes into a content term, a geometry term, and two content and geometry cross-terms -- all of which our experiments find individually necessary. To make the encoding stable across video data with heterogeneous camera-translation scales (SfM, deep SLAM, metric), we further decouple ray direction from moment magnitude, gate the encoding by a learned function of the log-magnitude, and apply RMSNorm to align it with the QKNorm-normalized content branch. The full module adds less than 0.1% parameters to a pretrained video DiT, is zero-initialized to start from the pretrained weights, and improves camera controllability, cross-frame 3D consistency, and overall video quality on a four-dataset training mixture.