FineServe dataset released to characterize real-world LLM serving workloads

Researchers from Tiancheng Zhang and colleagues published a paper introducing FineServe, an in-the-wild, multi-model LLM serving workload dataset collected from a global commercial marketplace. The dataset enables fine-grained characterization of real-world serving dynamics across heterogeneous models and tasks.

The authors report that existing studies often rely on proxy traces or coarse-grained characterizations, which fail to capture the heterogeneity of modern multi-model LLM platforms. FineServe addresses this gap by providing detailed traces of arrival dynamics and token behavior.

Using FineServe, the team conducted a comprehensive analysis revealing fundamentally different fluctuation regimes across model architectures, scales, and task intents. They attribute these differences to the distinct operational characteristics of diverse models and workloads.

Building on these insights, the researchers developed the FineServe workload generator, which composes fine-grained, model-aware workloads into configurable mixtures tailored for benchmarking multi-model serving platforms. The generator allows researchers to simulate realistic, mixed workloads for evaluating routing, scheduling, and capacity-planning strategies.

The FineServe dataset and workload generator are made publicly available on GitHub, providing a realistic foundation for systems researchers to evaluate and compare serving strategies under realistic conditions.