NVIDIA releases Cosmos-H-Dreams, a real-time generative simulator for surgical robotics

NVIDIA has released Cosmos-H-Dreams, a real-time, action-conditioned generative simulator designed to support training and evaluation of surgical robotics policies. The system builds on the earlier Cosmos-H-Surgical-Simulator, a world foundation model trained on the Open-H-Embodiment dataset and capable of generating future surgical video conditioned on robot actions. Cosmos-H-Dreams distills this capability into a causal student model optimized for real-time interaction.

The distillation process includes several technical components. A bidirectional teacher model is fine-tuned on the JHU dVRK tabletop mixture, which includes both successful demonstrations and failure cases such as needle drops and missed throws. Training progressively increases the temporal horizon from 12 to 72 frames, with warm-start initialization at each stage. A causal student is then initialized from the teacher and pre-trained on cached denoising trajectories to learn causal attention and streaming key/value caching. Self-forcing distillation further trains the student to operate on its own generated context, matching the deployment conditions and enabling few-step diffusion with as few as two denoising steps per latent frame.

Cosmos-H-Dreams is served through FlashDreams, an accelerated inference library for autoregressive world and video models. FlashDreams applies optimizations such as streaming KV cache, CUDA Graph capturing, and model compilation to achieve interactive operation. On a single NVIDIA RTX PRO 6000 GPU, the system delivers approximately 160 frames per second, a significant improvement over the roughly ten frames per second achieved by the standard Cosmos-H-Surgical-Simulator inference.

The simulator supports multiple interaction modalities. A browser client can send keyboard commands and receive generated frames over WebRTC, while a Meta Quest client can map tracked controller motion into robot actions and display the synthesized scene via WebXR. The same model can also interface with learned surgical policies in a closed loop for end-to-end evaluation.

While a pre-trained checkpoint is provided for da Vinci Research Kit (dVRK) tabletop suturing, the system is designed for extensibility. NVIDIA provides a step-by-step guide for training a real-time student model on custom datasets, enabling researchers to adapt the simulator to their own embodiments and tasks.