Salesforce uses SageMaker Inference Components placement to meet Multi-AZ high availability requirements

Salesforce adopted Amazon SageMaker Inference Components to co-host multiple models on shared GPUs, achieving an 8x reduction in infrastructure costs. However, the default placement behavior of SageMaker’s Inference Components did not guarantee even distribution of model copies across Availability Zones (AZs), creating potential single points of failure for high availability (HA) compliance.

To address this, Salesforce used the new Inference Component placement capability, exposed through the SchedulingConfig parameter in the CreateInferenceComponent API. This feature allows explicit control over where Inference Components are deployed, enabling Salesforce to distribute model copies evenly across multiple AZs and meet their Multi-AZ HA requirements.

The challenge Salesforce faced was that the default placement algorithm optimizes each deployment operation independently, which could result in uneven distribution of model copies across AZs. This per-operation view risked concentrating copies of a specific model in a single AZ, undermining resilience goals.

By leveraging the SchedulingConfig parameter, Salesforce ensured that Inference Components were strategically placed to avoid single points of failure, thereby aligning with their compliance and resilience objectives without sacrificing the cost benefits of multi-model co-hosting.