Apple proposes RayRoPE for geometry-aware positional encoding in multi-view transformers

Apple’s Machine Learning Research team proposed RayRoPE, a positional encoding method for multi-view transformers that process tokens from posed input images. The approach encodes patch positions based on associated rays but uses a predicted point along the ray rather than the ray direction to achieve geometry-aware encoding.

To ensure SE(3)-invariant attention, RayRoPE computes query-frame projective coordinates for multi-frequency similarity. Because the predicted 3D point along a ray may not be precise, the method includes a mechanism to analytically compute the expected position encoding under uncertainty.

The team validated RayRoPE on novel-view synthesis and stereo depth estimation, reporting consistent improvements over alternative positional encoding schemes. On the CO3D dataset, RayRoPE achieved a 15% relative improvement on LPIPS compared to alternatives.

RayRoPE also supports seamless integration of RGB-D input, which further increases performance gains over methods that cannot positionally encode this information.