NVIDIA releases Cosmos3-H-Dreams, a generative surgical simulator for real-time clinical training

NVIDIA, in collaboration with researchers, released Cosmos3-H-Dreams, a generative simulator designed to bring interactive, real-time surgical simulation to the visual and procedural complexity of clinical surgery. The model advances prior work by training on synchronized video and robot kinematics recorded during clinical procedures on the CMR Surgical Versius platform, rather than controlled tabletop environments. According to the announcement, this is the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control.

Cosmos3-H-Dreams is powered by the Cosmos 3 foundation and accelerated with FlashDreams, NVIDIA’s streaming-inference library, enabling interactive operation on a single NVIDIA RTX PRO 6000 GPU. The model addresses challenges in temporal consistency and responsiveness by specializing a bidirectional teacher on curated cholecystectomy data and distilling it into causal student models optimized for sequential generation with low latency.

The release includes two student checkpoints—four-step and two-step—each representing a trade-off between visual quality and inference speed. The four-step model uses four denoising steps per generated video block, while the two-step model is separately trained to reduce steps for faster generation at the cost of visual fidelity. Both are designed for developers to explore this balance in real-time applications.

The system is trained using 832 x 480 video with actions aligned at 10 Hz across 28 action dimensions, including pose and gripper information for both surgical instruments, energy controls, and controller inputs. The training pipeline uses a teacher-to-student distillation process that extends the generation horizon from 24 frames to 72 frames (2.4 to 7.2 seconds at 10 Hz), followed by self-forcing to mitigate error accumulation during autoregressive rollouts.

Cosmos3-H-Dreams also introduces adaptations for long rollouts, including an adapted FadeMem module that consolidates the growing key/value cache in FlashDreams, supporting sustained streaming generation. The team demonstrates generalization to unseen instrument trajectories, showing coherent tissue deformation and instrument behavior when conditioned on novel action sequences.

The release includes a step-by-step guide for adapting the model to new datasets and controller integrations, provided the action representations and conventions match those used during training. NVIDIA positions this work as part of a broader research agenda to model full surgical procedures and improve the realism of embodied AI systems in clinical contexts.