SageMaker HyperPod: Topology-Aware AI Workload Scheduling

Amazon SageMaker HyperPod task governance introduces topology-aware scheduling, a new capability to optimize AI workload efficiency and reduce network latency. This feature leverages EC2 instance topology information to improve resource allocation and task prioritization within Amazon EKS clusters. Generative AI workloads, which heavily rely on inter-instance communication, benefit significantly from this optimization. By considering the physical placement of instances within a data center’s hierarchical infrastructure (nodes and node sets), the system minimizes network hops and routing, resulting in faster processing times. Data scientists can use this feature to enhance their workflow by confirming topology information, identifying instances on the same network nodes using provided scripts, and then submitting topology-aware training tasks. Two methods are available for task submission: modifying Kubernetes manifest files using annotations (kueue.x-k8s.io/podset-required-topology or kueue.x-k8s.io/podset-preferred-topology) to specify topology requirements, or using the SageMaker HyperPod CLI with the –preferred-topology or –required-topology parameters. Prerequisites include an EKS cluster, a SageMaker HyperPod cluster with topology-enabled instances, the HyperPod task governance add-on (version 1.2.2 or later), and kubectl. The solution offers improved training efficiency, reduced latency, and better resource utilization, making it ideal for organizations working with large-scale generative AI workloads. The system provides tools and scripts to visualize cluster topology and guide users through the process of topology-aware task submission.

(Source: https://aws.amazon.com/blogs/machine-learning/schedule-topology-aware-workloads-using-amazon-sagemaker-hyperpod-task-governance/)