feat(aws/sagemaker): HyperPod — Slurm & EKS clusters, task governance, workload DX

[sam-goodwin]

Adds SageMaker HyperPod: the AWS.SageMaker.Cluster resource (Slurm or EKS orchestrated), task governance (ClusterSchedulerConfig + ComputeQuota), a first-class hyperpod prop on AWS.EKS.Job/Deployment, examples/aws-hyperpod covering every workload tier, and a dedicated docs page at /aws/compute/hyperpod (with EKS, choosing-a-runtime, and the AWS hub updated to point at it).

HyperPod is a persistent fleet; how work lands on it depends on the orchestrator, so the DX is tiered the same way ECS/EKS were in #867:

	Slurm (default)	EKS (orchestrator: { Eks })
Provision	AWS.SageMaker.Cluster	AWS.EKS.Cluster + AWS.SageMaker.Cluster
Low-level workloads	ssm start-session → sbatch	AWS.EKS.Manifest (any K8s object)
High-level workloads	— (no submission API)	AWS.EKS.Job / Deployment with hyperpod: props
Governance	Slurm accounting	ClusterSchedulerConfig + ComputeQuota (Kueue)

Provision (both orchestrators)

// Slurm (default): sbatch over SSM, LifeCycleConfig required
const cluster = yield* AWS.SageMaker.Cluster("TrainingCluster", {
  instanceGroups: {
    controller: {
      InstanceType: "ml.t3.medium",
      InstanceCount: 1,
      ExecutionRole: role.roleArn,
      LifeCycleConfig: { SourceS3Uri: script.sourceS3Uri, OnCreate: "on_create.sh" },
    },
  },
});

// EKS: attach to a control plane; nodes join as ordinary EKS nodes
const hyperpod = yield* AWS.SageMaker.Cluster("HyperPod", {
  orchestrator: { Eks: { ClusterArn: eks.clusterArn } },
  vpcConfig: { SecurityGroupIds: [...], Subnets: network.privateSubnetIds },
  instanceGroups: { workers: { /* same shape — LifeCycleConfig required here too */ } },
});

Instance groups reconcile in place (removing one computes InstanceGroupsToDelete); VPC change replaces; createCluster retries the IAM propagation race on fresh execution roles like Lambda does.

Low level on Slurm — sbatch over SSM

Slurm has no remote submission API; jobs are submitted on the cluster. Every node is an SSM target:

aws sagemaker list-cluster-nodes --cluster-name <clusterName>
aws ssm start-session --target sagemaker-cluster:<cluster-id>_controller-<instance-id>
# on the node:
sbatch --nodes=4 train.sbatch

Low level on EKS — any Kubernetes object, pinned by node labels

const job = yield* AWS.EKS.Manifest("GovernedJob", {
  cluster: eks,
  manifest: {
    apiVersion: "batch/v1",
    kind: "Job",
    metadata: {
      namespace: Output.interpolate`hyperpod-ns-${quota.teamName}`,
      labels: { "kueue.x-k8s.io/queue-name": Output.interpolate`hyperpod-ns-${quota.teamName}-localqueue` },
    },
    spec: { template: { spec: {
      nodeSelector: {
        "sagemaker.amazonaws.com/node-health-status": "Schedulable",
        "sagemaker.amazonaws.com/instance-group-name": "workers",
      },
      containers: [{ name: "train", image: "ghcr.io/acme/train:v3" }],
    } } },
  },
});

High level on EKS — effectful Jobs/Deployments with hyperpod: props

The #867 surfaces run on HyperPod unchanged. The hyperpod prop references resources through the graph: instance groups are declared as a keyed object whose keys carry through to the cluster's attributes as types — hyperpod.instanceGroups.workers is typed per key (a typo'd name is a compile error) — and governance comes from the team's ComputeQuota resource, so the node selector, namespace, Kueue labels, and cluster/quota → workload ordering all derive from the data flow:

const hyperpod = yield* AWS.SageMaker.Cluster("HyperPod", {
  instanceGroups: { workers: { InstanceType: "ml.g5.xlarge", InstanceCount: 4, ... } },
  ...
});

const train = yield* AWS.EKS.Job(
  "Train",
  {
    cluster: eks,
    main: import.meta.url,
    hyperpod: {
      instanceGroup: hyperpod.instanceGroups.workers, // key-typed, Output-connected
      quota: researchQuota,      // → hyperpod-ns-research + Kueue queue label
      priorityClass: "training", // → training-priority
    },
  },
  Effect.gen(function* () {
    const putItem = yield* AWS.DynamoDB.PutItem(results); // IAM → pod identity
    return { run: Effect.gen(function* () { /* ... */ }) };
  }).pipe(Effect.provide(AWS.DynamoDB.PutItemHttp)),
);

Task governance

yield* AWS.EKS.Addon("TaskGovernance", { clusterName, addonName: "amazon-sagemaker-hyperpod-taskgovernance" });

const policy = yield* AWS.SageMaker.ClusterSchedulerConfig("Scheduler", {
  clusterArn: hyperpod.clusterArn,
  schedulerConfig: { PriorityClasses: [{ Name: "training", Weight: 75 }], FairShare: "Enabled" },
});

const quota = yield* AWS.SageMaker.ComputeQuota("ResearchQuota", {
  clusterArn: hyperpod.clusterArn,
  computeQuotaTarget: { TeamName: "research", FairShareWeight: 10 }, // creates hyperpod-ns-research
  computeQuotaConfig: { ComputeQuotaResources: [{ InstanceType: "ml.g5.xlarge", Count: 2 }] },
});

Verified live

• Slurm: full cluster lifecycle (create → InService → destroy, ~5 min on 1× ml.t3.medium) behind AWS_TEST_SAGEMAKER_HYPERPOD=1.
• EKS: the entire eks.run.ts stack deployed green against real AWS — network → EKS 1.34 → HyperPod dependencies chart → HyperPod attach (InService) → amazon-sagemaker-hyperpod-taskgovernance add-on → scheduler policy → team quota → the governed raw-manifest Job in hyperpod-ns-research → the effectful TrainJob via hyperpod: props.
• Governance lifecycles (gated on AWS_TEST_SAGEMAKER_HYPERPOD_EKS_CLUSTER_ARN): ComputeQuota create → in-place update with version bump → destroy, all green live; ClusterSchedulerConfig create/reconcile verified via the stack, and the test asserts the typed one-policy-per-cluster conflict when the cluster is already governed (full lifecycle on a bare cluster). Ungated typed-error probes for all three resources run in every CI pass; full SageMaker suite 18/18.
• Constraints discovered live, now encoded in the example + JSDoc: EKS auth mode must be API (CONFIG_MAP rejected), HyperPod trails the newest EKS version (1.28–1.35), LifeCycleConfig is required for EKS-orchestrated groups (docs claim otherwise), the HyperPod dependencies helm chart is mandatory pre-attach (its vendored legacy mpi-operator CRD fails k8s ≥1.34 strict SSA — disabled via values), and AWS allows one ClusterSchedulerConfig per cluster (typed as ClusterSchedulerConfigAlreadyExists via a distilled patch).
• Drive-by provider fixes found by the live runs: EKS Job/CronJob names are capped at Kubernetes' 63-char limit (long app+stage combos were rejected via the batch.kubernetes.io/job-name label); EKS.Addon waits up to 20 min for node-bound addons and reports status in AddonNotReady; the K8s client retries transport failures and fresh-cluster auth propagation; EKS.Addon.read tolerates rows from crashed pre-resolve runs; EC2.InternetGateway delete retries DependencyViolation for ~5 min (EKS control planes release ENI addresses slowly).

🤖 Generated with Claude Code

[alchemy-version-bot]

Install the packages built from this commit:

alchemy

bun add alchemy@https://pkg.ing/alchemy/429e7ee

@alchemy.run/better-auth

bun add @alchemy.run/better-auth@https://pkg.ing/@alchemy.run/better-auth/429e7ee

@alchemy.run/pr-package

bun add @alchemy.run/pr-package@https://pkg.ing/@alchemy.run/pr-package/429e7ee