Constraint-aware GPU allocator raises utilization by up to 33 percentage points in enterprise AI workloads

A Dharma AI–led team reports building a constraint-aware GPU allocator and benchmarking it against a FIFO scheduler across seven scenarios on identical hardware and identical workloads.

They observed GPU utilization rise by as much as 33 percentage points and priority-weighted output increase in every scenario, by as much as 105%.

The authors note that every reported gain is expressed as improvement over the FIFO result on the same scenario, with utilization in percentage points and value as a percentage increase in priority-weighted output.

The core scheduling challenge arises from two incompatible allocation shapes competing for the same hardware: batch-like workloads (training, batch inference, quantization) that require contiguous blocks until completion, and real-time inference that is elastic and varies with demand.

Under contention, a FIFO scheduler reserves GPUs for each real-time application’s maximum daily demand, leaving many idle outside peak hours and blocking batch jobs from using that capacity.

The allocator addresses this by treating real-time demand as a curve and allocating against demand at each timestep, while placing batch-like jobs by priority across the entire scheduling horizon.

Across five benchmark scenarios built for genuine contention, utilization moved from a 52–85% band under FIFO to a 72–88% band with the allocator, and priority-weighted value rose between 24.6% and 105.1%, averaging 52%.

In a scale test with 30 jobs across 64 GPUs, utilization and throughput were identical between FIFO and the allocator, but the allocator delivered 15.9% more priority-weighted value, illustrating that occupancy alone can mask differences in business-critical output.