Amazon SageMaker Inference Adds NVIDIA Multi-Instance GPU Support for Hardware Partitioning
Frontier AI models · Enterprise AI
Amazon SageMaker Inference now supports GPU partitioning via NVIDIA Multi-Instance GPU (MIG) on endpoints using inference components. This allows a single physical GPU to be divided into up to seven hardware-isolated partitions with dedicated compute units and memory blocks, enabling multi-model hosting on managed endpoints without managing self-hosted Kubernetes clusters.
AWS announced that Amazon SageMaker Inference now supports GPU partitioning using NVIDIA Multi-Instance GPU (MIG) technology on endpoints configured with inference components. Deploying models previously required allocating at least one whole accelerator, limiting instance density and leading to resource underutilization. With this update, SageMaker AI automates GPU configuration, placement, and scaling across supported GPU families, including NVIDIA A100 (P4d), H100 (P5), RTX PRO 4500 Blackwell (G7), and RTX PRO 6000 Blackwell Server Edition (G7e). Single-model endpoints, serverless endpoints, and heterogeneous instance pools do not support partitioning.
Key takeaways
- NVIDIA Multi-Instance GPU divides a physical GPU into up to seven independent partitions with dedicated compute, cache, memory, and memory bandwidth enforced in hardware.
- The capability is available exclusively for endpoints using inference components, while single-model, serverless, and heterogeneous instance pools cannot be partitioned.
- Configuration uses two fields without new API operations: AcceleratorPartitionLayouts sets the device division, and AcceleratorPartitionType maps models to specific partition profiles.
- Hardware isolation ensures predictable latency, eliminates cold starts by keeping models loaded, and confines crashes or memory errors to the affected partition.
Source：AWS Machine Learning Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human.