Researchers propose FluidPD for SLO-aware in-place elasticity in disaggregated LLM serving

Prefill-decode disaggregation is increasingly used in LLM serving to separate two phases with distinct execution patterns and service-level objective (SLO) requirements. Existing systems rely on fixed worker ratios and request routing, but real-world workloads experience short bursts and sustained shifts in the prefill-to-decode demand ratio. This can cause a well-provisioned configuration to quickly become mismatched, resulting in latency SLO violations even when idle capacity exists elsewhere.

FluidPD, a new P/D-disaggregated serving system, introduces two mechanisms to address these imbalances. FluidToken handles transient imbalance by offloading a bounded portion of prefill computation to decode workers when decode-side slack is available. FluidRole addresses sustained imbalance by reassigning running workers between prefill and decode roles in place, avoiding model reloads and engine restarts.

Both mechanisms are guided by lightweight pressure indices that expose prefill and decode-side resource pressure before SLO violations occur. In evaluation on production Azure trace workloads, FluidPD improves overall SLO attainment over a static SGLang baseline by up to 94.6 percentage points, showing that SLO-aware in-place elasticity can enhance service quality without provisioning additional workers.