Runway debuts WorldPrompt for real-time interactive world models with persistent context and timed actions

Runway’s GWM Worlds 2 is a research preview that turns high-fidelity video and audio generation into real-time interactive simulation using an autoregressive diffusion model. The system streams continuous 720p video at 24 frames per second and audio at 48,000 Hz, updating frame-by-frame while the user observes.

A new feature called WorldPrompt acts as a control layer for characters, cameras, and environments, allowing users to fix aspects of a simulated world such as the first frame and then specify a series of timestamped events or actions that can be prompted in real time. Unlike scriptable virtual worlds, WorldPrompt is a prompting mechanism and does not provide scripting or explicit state control.

To achieve real-time performance, Runway fine-tuned its foundational audio-video generation model to the WorldPrompt format, post-trained it to generate autoregressively, and applied distillation methods to reduce latency, including reducing diffusion steps from around 50 to four with some quality trade-offs.

Current limitations include error accumulation from feeding generated frames back into the model, challenges managing infinite generations and context memory to avoid GPU memory blowup, and imperfect long-term memory. Instruction-following reliability varies by action complexity, with movement reported as more reliable than physics or ability constraints.

Runway’s co-founder and co-CEO Anastasis Germanidis described the approach as starting from bidirectional diffusion that generates an entire video at once and then making it autoregressive to emit one frame or a few frames at a time.

The company acknowledges that autoregressive models suffer from error accumulation over time and that ensuring plausible consequences across different user actions remains an open research problem, especially for counterfactual scenarios like simulating failed versus successful outcomes in football.

Beyond gaming, Runway highlights robotics as a potential use case, suggesting simulated environments for testing policies before deployment.

Runway is valued at $5.3 billion following a $315 million funding round in February, and its first release, GWM Worlds, launched in December. The company is one of several working on real-time interactive world models, alongside Google DeepMind’s Genie 3, Odyssey-2 Pro, and World Labs’ RTFM.