AWS SageMaker AI adds container caching to cut generative AI inference scale-out latency by up to half

Amazon SageMaker AI now caches container images on supported accelerator instance types so that new instances launched during scale-out events do not need to pull the container image from Amazon Elastic Container Registry (Amazon ECR). This removes the container image download step from the scale-out path, which AWS says can reduce end-to-end startup latency by up to 51% in tests with early access customers.

The improvement is most pronounced for large generative AI containers such as SageMaker Large Model Inference (LMI, powered by vLLM), vLLM, and NVIDIA Triton. In one example cited by AWS, a Qwen3-8B (16 GB) model on an ml.g6.2xlarge instance using the LMI container (17.7 GB compressed) saw end-to-end startup latency drop from 525 seconds to 258 seconds after enabling container caching, with the container pull step eliminated and model download latency reduced from 168 seconds to 77 seconds due to reduced network contention.

Container caching works alongside two other auto-scaling optimizations introduced earlier: sub-minute Amazon CloudWatch metrics that detect scale-out needs up to 6x faster than standard 1-minute metrics, and an inference component data caching layer that stores container images and model artifacts on already running instances. Together, the three optimizations target different parts of the scale-out latency chain—detection, reuse of running instances, and new-instance launches.

Early access customers reported P50 latency improvements ranging from 38% to 65% depending on instance type, container image size, and model size. For example, one customer using an ml.g4dn.xlarge instance with a 15.7 GB image and no model size reported P50 latency falling from 381 seconds to 134 seconds (65% improvement).

Security and tenant isolation are preserved: each cache is dedicated to a single customer endpoint and purged automatically when the endpoint is deleted. No code changes are required for supported configurations, and container caching is available in all commercial AWS Regions where SageMaker AI inference is supported.