AWS SageMaker adds NVIDIA MIG support for GPU partitioning in inference endpoints
SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) to divide a single GPU into hardware-isolated partitions, enabling more models per instance and finer cost control.
1 source · cross-referenced
- SageMaker Inference endpoints can now use NVIDIA MIG to partition a single GPU into isolated units for running multiple models or smaller workloads.
- Partitioning is supported only on endpoints that use SageMaker inference components; single-model, serverless, and heterogeneous instance pool endpoints are excluded.
- AWS measured partitioning efficiency against whole-GPU deployments, showing how granular resource allocation reduces waste and improves utilization.
Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG), a feature that divides a single physical GPU into hardware-isolated partitions. Each partition receives a fixed share of compute units and dedicated memory, enforced at the hardware level.
This capability is available on SageMaker Inference endpoints that use inference components, a construct for deploying multiple models to one endpoint with configurable compute per model. Partitioning is not supported on single-model endpoints, serverless endpoints, or heterogeneous instance pools.
Previously, deploying a model to a GPU endpoint required requesting at least one whole accelerator, even if the model only needed a fraction of its resources. This often led to either underutilization on newer GPUs or insufficient capacity on older ones as workloads grew.
With MIG, users can place a different model in each partition or allocate a smaller slice of a newer accelerator, improving GPU utilization and cost efficiency. AWS describes this as a direct benefit for deploying more models than available GPUs while maintaining isolation between them.
The announcement highlights that customers previously had limited options for fine-grained GPU allocation. Alternatives like running fewer larger endpoints, using multi-model endpoints with shared but unisolated GPUs, or self-managing Kubernetes with MIG were either inefficient or operationally burdensome.
AWS notes that MIG support is integrated into SageMaker’s existing inference components, avoiding the need for customers to manage driver versioning, partition configuration, device plugins, or scheduling themselves.
- Nov 11, 2026 · Hugging Face
Hugging Face proposes token pooling-aware training to improve compressed multi-vector retrieval
Trust79 - Nov 11, 2026 · TechCrunch — AI
Fractile raises $720M Series C to develop AI chips and compiler stack
Trust79 - Nov 11, 2026 · The Verge — AI
Google’s Tabletop AI experiment lets users build custom collaborative apps via waitlist
Trust79