Skip to content
Tools · Nov 7, 2026

AWS SageMaker adds NVIDIA MIG support for GPU partitioning in inference endpoints

SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG) to divide a single GPU into hardware-isolated partitions, enabling more models per instance and finer cost control.

Trust79
HypeLow hype

1 source · cross-referenced

ShareXLinkedInEmail
TL;DR
  • SageMaker Inference endpoints can now use NVIDIA MIG to partition a single GPU into isolated units for running multiple models or smaller workloads.
  • Partitioning is supported only on endpoints that use SageMaker inference components; single-model, serverless, and heterogeneous instance pool endpoints are excluded.
  • AWS measured partitioning efficiency against whole-GPU deployments, showing how granular resource allocation reduces waste and improves utilization.

Amazon SageMaker Inference now supports NVIDIA Multi-Instance GPU (MIG), a feature that divides a single physical GPU into hardware-isolated partitions. Each partition receives a fixed share of compute units and dedicated memory, enforced at the hardware level.

This capability is available on SageMaker Inference endpoints that use inference components, a construct for deploying multiple models to one endpoint with configurable compute per model. Partitioning is not supported on single-model endpoints, serverless endpoints, or heterogeneous instance pools.

Previously, deploying a model to a GPU endpoint required requesting at least one whole accelerator, even if the model only needed a fraction of its resources. This often led to either underutilization on newer GPUs or insufficient capacity on older ones as workloads grew.

With MIG, users can place a different model in each partition or allocate a smaller slice of a newer accelerator, improving GPU utilization and cost efficiency. AWS describes this as a direct benefit for deploying more models than available GPUs while maintaining isolation between them.

The announcement highlights that customers previously had limited options for fine-grained GPU allocation. Alternatives like running fewer larger endpoints, using multi-model endpoints with shared but unisolated GPUs, or self-managing Kubernetes with MIG were either inefficient or operationally burdensome.

AWS notes that MIG support is integrated into SageMaker’s existing inference components, avoiding the need for customers to manage driver versioning, partition configuration, device plugins, or scheduling themselves.

Sources
  1. 01AWS — Machine Learning Blog — SageMaker Inference now supports Multi-Instance GPU to maximize GPU utilization for model endpoints
Also on Tools

Stories may contain errors. Dispatch is assembled with AI assistance and curated by human editors; despite the trust-score filter, mistakes happen. We correct publicly — every article links to its revision history. Nothing here is financial, legal, or medical advice. Verify before relying on any claim.

© 2026 Dispatch. No ads. No sponsorships. No paid placement. Reader-supported via Ko-fi.

Built by a person who cares about honest AI news.