SageMaker Inference Adds Multi-Instance GPU Support

What Happened

AWS SageMaker Inference now supports Multi-Instance GPU (MIG) partitioning on model endpoints. The feature allows a single physical GPU on supported instance types to be subdivided into isolated compute instances, each with dedicated memory and compute slices, and assigned across multiple endpoints or endpoint variants. AWS positions this as a density play: pack more model endpoints onto the same physical accelerator rather than dedicating a full GPU per endpoint.

Why It Matters

The default economics of managed inference have historically forced a floor on cost per endpoint: if a model does not saturate a full GPU, the operator still pays for the whole device. For small-to-mid models — embedding services, classification heads, rerankers, distilled task models, and low-QPS generation endpoints — that floor is the dominant cost driver, not the model itself. MIG support removes the requirement to over-provision hardware to isolate workloads, letting operators right-size the compute envelope per endpoint to actual utilization. The organizations that benefit most are those running portfolios of small models or many tenant-isolated endpoints where consolidation was previously blocked by memory and fault-isolation concerns rather than raw FLOPs. The strategic implication is that GPU memory, not GPU count, becomes the unit of capacity planning for a large class of inference workloads.

Technical Details

MIG partitions a physical GPU into up to seven instances, each with its own memory slice, cache, and compute engines, with hardware-level isolation between partitions so a fault or memory-pressure event in one endpoint does not propagate to neighbors. Support on SageMaker is scoped to specific GPU-backed instance families (A100-class and comparable MIG-capable hardware) and specific instance sizes; operators should confirm the instance type and partition profile against the current documentation before planning migrations. Each endpoint or endpoint variant is bound to a MIG profile rather than an entire device, and the partition's memory ceiling is fixed at provisioning time — a model whose weights plus KV cache exceed the slice will not fit, with no elastic spillover. Because partitioning reduces per-endpoint SM count and memory bandwidth proportionally, latency-sensitive or high-throughput workloads will see different performance envelopes per slice, and autoscaling behavior must be re-baselined against the smaller partition rather than the full device.

Operational Impact

The immediate workflow change is in capacity planning: instead of mapping one endpoint to one GPU, operators map endpoints to partition profiles and co-schedule multiple models on shared hardware, which cuts per-endpoint infrastructure spend for underutilized models. Isolation also simplifies multi-tenant deployments — customer-specific endpoints can share a device without sharing an execution context, which previously required either dedicated GPUs or softer process-level isolation with weaker guarantees. Model packaging and validation need revision: teams must verify that container images, CUDA/cuDNN versions, and serving frameworks (e.g., Triton, TorchServe, DJL) correctly enumerate and target the assigned MIG device, since misconfigured containers that assume full-device visibility will fail or underperform. Autoscaling policies, load tests, and SLO baselines must be re-derived per partition, and cost allocation tooling needs to attribute shared-device spend across the endpoints consuming each slice. Endpoints that were already saturating full GPUs gain nothing and should remain unpartitioned.