Researchers propose RIG-RoPE, a gated rotary positional encoding method for multimodal LLMs

Researchers propose RIG-RoPE, a modification to rotary positional encoding (RoPE) designed to reduce interference in multimodal transformer models. The method introduces three augmentations per token: a modality indicator, a visual instance identifier, and a scalar information-duration coordinate.

The authors identify two limitations of static multidimensional position assignment in interleaved multimodal contexts. First, height/width rotations may be applied to token pairs whose spatial displacement is not a well-defined geometric object, leading to cross-modal and inter-instance spatial interference. Second, temporal coordinates are often treated as equal-step counters, causing text tokens, image blocks, and video segments to advance temporal phase comparably despite differing information density.

RIG-RoPE enables height/width rotations only for query-key pairs from the same visual instance, marginalizing unknown spatial displacements otherwise. Temporal rotations use interpolated cumulative block durations: text tokens consume unit duration, images use a dimension-aware logarithmic spatial scale, and videos apply a logarithmic temporal extension over effective frames.

The report includes theoretical arguments: a gauge-invariance argument against ordinary cross-instance spatial rotation, an impossibility result for static IDs under shared height/width subspaces, and a duration-consistency argument against equal-step multimodal time.

RIG-RoPE adds no learned parameters and is designed to be implemented inside tiled attention kernels with constant additional metadata per token. The authors describe this as a preliminary technical report that establishes formulation and validation path without claiming empirical superiority.