From 5acf2d418d2641f31755ff0186e8d1874865f0e6 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 10 Jun 2026 01:47:55 +0000 Subject: [PATCH 01/89] iam: sample policies for cluster admin + cluster user (SSM) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two principals around the PCS reference cluster have very different responsibilities, so split them into two sample policies: - cluster-admin-policy.json (16 Sids, ~7.4 KB) — for the deploying principal. Covers create / update / delete of every resource the templates provision: CloudFormation parent + nested stacks; EC2 networking lifecycle (VPC, subnet, IGW, NAT, EIP, SG, route table, VPC endpoint, launch template, placement group); FSx (Lustre + OpenZFS); pcs:*; IAM role + instance profile lifecycle scoped to *PCS* / *ImageBuilder* names with iam:PassRole gated by iam:PassedToService; SSM Parameter Store for Grafana password + DLAMI auto-resolve; KMS / Secrets Manager / Image Builder / CloudWatch Logs vended delivery. Derived primarily from the AWS-published "minimum permissions for an AWS PCS service administrator" policy (security-min-permissions.html), with VPC + FSx + IAM lifecycle layered because the all-in-one template provisions those itself. - cluster-user-policy.json (6 Sids, ~1.5 KB) — for end users (ML engineers) of an already-deployed cluster. Covers ec2/cfn describe for login-node discovery; pcs:Get*/List* for cluster status; ssm:GetParameter scoped to /pcs/*/grafana/*; ssm:StartSession scoped via ssm:resourceTag/aws:pcs:compute-node-group-name=login* (so users CAN open shells on login nodes but CANNOT on compute nodes); session terminate-own only via session/${aws:username}-*. No write access of any kind to cluster resources. Both policies validated cleanly (0 findings) by aws accessanalyzer validate-policy. iam/README.md walks through what each policy covers, the design intent (combined CRUD vs phase-split, customer-managed vs inline, why no AmazonPCSFullAccess), what is intentionally NOT covered (the compute instance role uses the AWS-managed AWSPCSComputeNodePolicy), how to attach, AWS-managed policy pairing recipe, and the recommended refinement loop via aws accessanalyzer start-policy-generation against CloudTrail history of a sandbox deploy. Sample-grade — slightly broader than strict least-privilege so they work end-to-end in typical accounts. Production users should tighten with resource-level conditions (account ID, region, stack-name prefix, tag matchers) before adopting. --- architectures/aws-pcs/iam/README.md | 259 ++++++++++++++ .../aws-pcs/iam/cluster-admin-policy.json | 337 ++++++++++++++++++ .../aws-pcs/iam/cluster-user-policy.json | 84 +++++ 3 files changed, 680 insertions(+) create mode 100644 architectures/aws-pcs/iam/README.md create mode 100644 architectures/aws-pcs/iam/cluster-admin-policy.json create mode 100644 architectures/aws-pcs/iam/cluster-user-policy.json diff --git a/architectures/aws-pcs/iam/README.md b/architectures/aws-pcs/iam/README.md new file mode 100644 index 000000000..38ed9ab56 --- /dev/null +++ b/architectures/aws-pcs/iam/README.md @@ -0,0 +1,259 @@ +# IAM Sample Policies + +## Overview + +The PCS reference cluster has **two distinct human principals** with very +different responsibilities: + +``` + ┌──────────────────────────┐ + admin ───► │ cluster-admin-policy │ ──► CloudFormation, PCS, EC2 (VPC, + (deploys │ (CRUD on infra + │ SG, LT, PG, EIP, NAT, Endpoint), + the cluster) │ PCS resources) │ FSx, IAM (scoped), SSM Param, + └──────────────────────────┘ KMS, Secrets Manager, Image Builder + (creates everything; can also delete) + + ┌──────────────────────────┐ + user ───► │ cluster-user-policy │ ──► SSM StartSession to LOGIN node only, + (uses an │ (read-only + SSM │ port-forward Grafana 443→8443, + existing │ session to login) │ read /pcs/*/grafana/admin-password, + cluster) └──────────────────────────┘ read PCS cluster / queue status + (cannot create or modify anything) +``` + +Splitting like this means the deployer needs broad IAM but doesn't have to +share those credentials with every engineer who just wants to `srun`, and the +user role can be handed out widely (it can't accidentally delete the cluster +or open shells on compute nodes). + +| File | Principal | Lifecycle | Size | Statements | +|---|---|---|---:|---:| +| `cluster-admin-policy.json` | Deploying user / role | Create / update / delete cluster | ~7.4 KB | 16 | +| `cluster-user-policy.json` | Cluster end-users (ML engineers) | Connect / read-only on existing cluster | ~1.5 KB | 6 | + +Both files are valid identity-based JSON policies that have passed +`aws accessanalyzer validate-policy` cleanly (0 findings). + +## Design intent + +These are **sample, slightly broader-than-strict-least-privilege** policies for +reference, derived primarily from +[the AWS-published "minimum permissions for an AWS PCS service administrator" +policy](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html) +plus the additional permissions the all-in-one template needs because it +provisions VPC + FSx + IAM roles itself (the AWS reference policy assumes those +already exist). + +Choices made: + +- **Combined CRUD, not phase-split.** A single policy covers create + update + + delete because CFN rollback on a failed Create requires Delete actions, and + Update is operationally a strict superset of Create. See the + [`cluster-admin-policy.json` section below](#cluster-admin-policyjson--deploy--update--delete) + for the full reasoning. +- **Customer-managed, not inline.** The admin policy is ~7.4 KB which exceeds + the 6,144-character limit for inline policies but fits comfortably under the + 17,408-character limit for customer-managed policies. Attach as a + customer-managed policy. +- **No `AmazonPCSFullAccess` available.** As of this writing AWS publishes + `AWSPCSComputeNodePolicy` (for compute instances, not deployers) and + `AWSPCSServiceRolePolicy` (the SLR's managed policy) but no + service-administrator managed policy. The PCS portion of the admin policy + must be customer-managed. +- **User policy is tag-scoped, not name-scoped.** `ssm:StartSession` is + conditioned on `ssm:resourceTag/aws:pcs:compute-node-group-name` matching + `login*`, so the policy survives users renaming their CNGs. +- **Production hardening points.** Tighten with resource-level conditions + (account ID, region, stack-name prefix, tag matchers) before using in + production. Validation by deploying with a sandbox account and running + CloudTrail / IAM Access Analyzer is recommended — see + [Refining via CloudTrail](#refining-via-cloudtrail) below. + +## What's NOT covered here + +- **The compute instance role** (passed to EC2 via the launch template by + `cluster.yaml`). That's a separate role provisioned by the templates + themselves, not by these policies. Use the AWS-managed + `AWSPCSComputeNodePolicy` for it. +- **The Image Builder build instance role** (used by the standalone DLAMI + template's Image Builder pipeline). Use AWS-managed + `EC2InstanceProfileForImageBuilder` / + `EC2InstanceProfileForImageBuilderECRContainerBuilds` for that. +- **Fine-grained per-cluster scoping.** Both policies use `Resource: "*"` for + many actions because resource-level scoping for VPC / EC2 networking + resources is limited and tag-based filtering would require the templates to + consistently propagate a known tag. This is a deliberate sample-grade choice. + +--- + +## `cluster-admin-policy.json` — deploy / update / delete + +What it covers (Sids in the JSON): + +| Sid | Purpose | +|---|---| +| `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | Create/update/delete the parent and all nested CFN stacks; ChangeSets; describe and list | +| `EC2NetworkingAndComputeLifecycle` | VPC / subnets / IGW / NAT / EIP / security groups / route tables / VPC endpoints / launch templates / placement groups; `RunInstances` / `CreateFleet` for PCS-internal compute launches | +| `EC2DescribeForPCS` | All `ec2:Describe*` (PCS' control plane and CFN both call many of these) | +| `FSxLifecycle` | FSx for Lustre + OpenZFS create / update / delete / describe / tag | +| `PCSFullAccess` | `pcs:*` — clusters, compute node groups, queues | +| `IAMRoleAndInstanceProfileLifecycle` | Create / delete / tag / attach for IAM roles + instance profiles whose names contain `PCS` / `pcs` / `ImageBuilder` (matches what the templates produce) | +| `IAMPassRoleToPCSAndEC2` | `iam:PassRole` scoped to `*PCS*` roles passed to `ec2.amazonaws.com` / `pcs.amazonaws.com` | +| `IAMPassRoleToImageBuilder` | `iam:PassRole` scoped to `*ImageBuilder*` roles passed to `imagebuilder.amazonaws.com` | +| `ServiceLinkedRolesOneTime` | `iam:CreateServiceLinkedRole` for PCS / Spot / FSx / Image Builder (a no-op once the SLR exists) | +| `SSMParameterLifecycle` | Put / get / delete on `parameter/pcs/*` (Grafana admin password, etc.) and `parameter/aws/service/*` (PCS-Ready DLAMI auto-resolve) | +| `ImageBuilderLifecycleOptional` | EC2 Image Builder pipeline / recipe / component / distribution / lifecycle policy — only used when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml` | +| `KMSAccessForDefaultEncryption` | KMS data-plane operations for default-encrypted EBS / FSx volumes (verbatim from the AWS PCS minimum-permissions policy) | +| `SecretsManagerForPCS` | `secretsmanager:*` scoped to `secret:pcs!*` (PCS internally creates a Slurm cluster auth secret) | +| `CloudWatchLogsForMonitoring` + `PCSVendedLogsDelivery` | CloudWatch Logs delivery wiring for PCS-vended Slurm logs (used when `DeployMonitoring=true` and PCS log delivery is configured) | + +**Where it came from.** The `PCSFullAccess` + `EC2DescribeForPCS` + `IAMPassRoleToPCSAndEC2` ++ `ServiceLinkedRolesOneTime` + `KMSAccessForDefaultEncryption` + `SecretsManagerForPCS` +blocks are derived directly from the AWS-published "minimum permissions for an AWS PCS +service administrator" policy at +[docs.aws.amazon.com/pcs/.../security-min-permissions.html](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html). +The remaining blocks (CloudFormation, EC2 networking lifecycle, FSx, IAM role +lifecycle, SSM Parameter Store, Image Builder, CloudWatch Logs) are layered on +because the AWS reference policy assumes the VPC, FSx filesystems, and IAM roles +already exist — whereas the all-in-one template provisions all of them. + +**A combined CRUD policy is intentional, not a mistake.** Splitting per phase +(deploy-only / update-only / delete-only) breaks in three places: + +1. **CFN rollback on Create requires Delete actions.** A failed `CreateStack` + automatically rolls back, deleting any resources it managed to create. A + "deploy-only" policy without `Delete*` leaves the principal unable to clean up + `CREATE_FAILED` stacks. +2. **Update is a strict superset of Deploy.** Any `UpdateStack` may replace + resources (an `add-cng*.yaml` `LaunchTemplate` version bump replaces the LT, + which is Create + Delete + Modify in CFN's eyes). +3. **Drift detection during Update calls `Describe*` across every service.** + +The natural alternative is **operator policy = the CRUD JSON above** vs +**auditor policy = the same actions reduced to `*:Describe*` / `*:Get*` / `*:List*`**. + +### How to attach + +For a customer-managed policy from the JSON: + +```bash +aws iam create-policy \ + --policy-name PCSClusterAdmin \ + --policy-document file://architectures/aws-pcs/iam/cluster-admin-policy.json +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterAdmin +``` + +Or attach to an assumable role for a CFN deployment role pattern. + +### Pairing with AWS-managed policies + +If you want a smaller customer-managed surface, you can attach AWS-managed +policies for parts of the stack and trim the corresponding Sids out of +`cluster-admin-policy.json`: + +| AWS-managed | Replaces / overlaps with | Notes | +|---|---|---| +| `AWSCloudFormationFullAccess` | `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | Broader (no resource-level scoping) but battle-tested | +| `AmazonFSxFullAccess` | `FSxLifecycle` | Closest fit; covers everything the templates need | +| `AWSImageBuilderFullAccess` | `ImageBuilderLifecycleOptional` | Only needed when the standalone DLAMI template is deployed | +| `AmazonEC2FullAccess` | `EC2NetworkingAndComputeLifecycle` + `EC2DescribeForPCS` | **Materially overprivileged** (e.g. EBS public-share). Prefer the customer-managed Sids | + +There is **no** `AmazonPCSFullAccess` AWS-managed policy as of this writing — the +PCS portion has to be customer-managed. + +--- + +## `cluster-user-policy.json` — SSM access for end users + +What it covers: + +| Sid | Purpose | +|---|---| +| `DiscoverClusterAndLoginNode` | `ec2:DescribeInstances` + `cloudformation:DescribeStacks` so the user can find the login node and read stack outputs (e.g. `ClusterId`) | +| `PCSReadOnly` | `pcs:Get*` / `pcs:List*` so the user can see cluster / queue / compute-node-group status | +| `GrafanaPasswordRead` | `ssm:GetParameter` scoped to `parameter/pcs/*/grafana/*` (the admin password generated by `cluster.yaml` when `DeployMonitoring=true`) | +| `SSMSessionToLoginNode` | `ssm:StartSession` scoped to **only** EC2 instances tagged `aws:pcs:compute-node-group-name=login*`, with the SSM documents needed for shell, SSH-over-SSM, and port-forwarding | +| `SSMSessionDocumentLookup` | `ssm:DescribeSessions` / `DescribeInstanceInformation` etc. (the SSM client / Session Manager UI calls these to render the connection list) | +| `SSMSessionTerminateOwn` | `ssm:TerminateSession` / `ResumeSession` scoped to sessions whose ID prefix matches `${aws:username}-*` (= sessions the user themselves opened) | + +The `SSMSessionToLoginNode` Sid uses +**`ssm:resourceTag/aws:pcs:compute-node-group-name`** to scope sessions to login +nodes only. Compute nodes are intentionally NOT reachable via this policy — users +should reach them through Slurm (`srun`, `sbatch`), not directly via SSM. + +### Typical user workflows the policy enables + +1. **Connect to the login node:** + ```bash + INSTANCE_ID=$(aws ec2 describe-instances \ + --filters "Name=tag:aws:pcs:compute-node-group-name,Values=login" \ + "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].InstanceId' --output text) + aws ssm start-session --target $INSTANCE_ID + ``` + +2. **Open Grafana via SSM port-forward (443 → 8443):** + ```bash + aws ssm start-session --target $INSTANCE_ID \ + --document-name AWS-StartPortForwardingSession \ + --parameters '{"portNumber":["443"],"localPortNumber":["8443"]}' + # then open https://localhost:8443/grafana/ + ``` + +3. **Retrieve the Grafana admin password** (run this once, paste into the Grafana + login): + ```bash + CLUSTER_ID= + aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/grafana/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text + ``` + +4. **SSH-over-SSM** (with `~/.ssh/config` configured per the README): + ```bash + ssh hpc8a-login + ``` + +### Safety properties + +- **No write access to cluster resources.** No `CreateStack`, no `pcs:Create*`, + no `ec2:RunInstances`, no `iam:*`, no `fsx:*` write actions. +- **Login nodes only.** The instance-tag scope on `ssm:StartSession` blocks the + user from opening shells on compute nodes (which are tagged + `aws:pcs:compute-node-group-name=`, e.g. `cpu1` or `gpu-p5`). +- **Own sessions only.** `ssm:TerminateSession` is scoped to sessions whose ID + matches the user's IAM username, so users can't kill each other's sessions. + +--- + +## Refining via CloudTrail + +These JSON files are sample-grade. The recommended refinement path: + +1. **Deploy a representative cluster in a sandbox AWS account** with admin + permissions on the deploying principal so nothing fails for spurious IAM + reasons. +2. **Wait for the deploy to succeed**, then for `aws cloudformation delete-stack` + to finish, so CloudTrail captures the full lifecycle. +3. **Generate a least-privilege policy from CloudTrail history** with IAM Access + Analyzer: + + ```bash + aws accessanalyzer start-policy-generation \ + --policy-generation-details principalArn= \ + --cloud-trail-details accessRole=,startTime=,trails=[{cloudTrailArn=}] + # poll until JobStatus=SUCCEEDED + aws accessanalyzer get-generated-policy --job-id \ + --include-resource-placeholders --query 'GeneratedPolicyResult.GeneratedPolicies[0].Policy' \ + --output text > generated-policy.json + ``` + +4. **Diff against `cluster-admin-policy.json`**. The Access Analyzer output + tends to be *narrower* than the sample (it only lists actions actually + called) but with `Resource: "*"` even when narrower scoping is possible — + the sample's resource ARNs are usually keepers. + +The same approach works for `cluster-user-policy.json` — exercise the user +workflows (start a session, port-forward Grafana, terminate the session) in a +sandbox with broad SSM perms, then narrow to what was actually called. diff --git a/architectures/aws-pcs/iam/cluster-admin-policy.json b/architectures/aws-pcs/iam/cluster-admin-policy.json new file mode 100644 index 000000000..5a2df66f0 --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-admin-policy.json @@ -0,0 +1,337 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudFormationStackLifecycle", + "Effect": "Allow", + "Action": [ + "cloudformation:CreateStack", + "cloudformation:UpdateStack", + "cloudformation:DeleteStack", + "cloudformation:CancelUpdateStack", + "cloudformation:ContinueUpdateRollback", + "cloudformation:CreateChangeSet", + "cloudformation:ExecuteChangeSet", + "cloudformation:DescribeChangeSet", + "cloudformation:DeleteChangeSet", + "cloudformation:DescribeStacks", + "cloudformation:DescribeStackEvents", + "cloudformation:DescribeStackResource", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources", + "cloudformation:GetTemplate", + "cloudformation:GetTemplateSummary", + "cloudformation:TagResource", + "cloudformation:UntagResource" + ], + "Resource": "*" + }, + { + "Sid": "CloudFormationDescribeStackless", + "Effect": "Allow", + "Action": [ + "cloudformation:ValidateTemplate", + "cloudformation:ListStacks", + "cloudformation:ListExports", + "cloudformation:ListImports" + ], + "Resource": "*" + }, + { + "Sid": "EC2NetworkingAndComputeLifecycle", + "Effect": "Allow", + "Action": [ + "ec2:CreateVpc", + "ec2:DeleteVpc", + "ec2:ModifyVpcAttribute", + "ec2:AssociateVpcCidrBlock", + "ec2:DisassociateVpcCidrBlock", + "ec2:CreateSubnet", + "ec2:DeleteSubnet", + "ec2:ModifySubnetAttribute", + "ec2:CreateInternetGateway", + "ec2:DeleteInternetGateway", + "ec2:AttachInternetGateway", + "ec2:DetachInternetGateway", + "ec2:CreateNatGateway", + "ec2:DeleteNatGateway", + "ec2:AllocateAddress", + "ec2:ReleaseAddress", + "ec2:AssociateAddress", + "ec2:DisassociateAddress", + "ec2:CreateRouteTable", + "ec2:DeleteRouteTable", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:ReplaceRoute", + "ec2:AssociateRouteTable", + "ec2:DisassociateRouteTable", + "ec2:CreateSecurityGroup", + "ec2:DeleteSecurityGroup", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:AuthorizeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:UpdateSecurityGroupRuleDescriptionsIngress", + "ec2:UpdateSecurityGroupRuleDescriptionsEgress", + "ec2:CreateVpcEndpoint", + "ec2:ModifyVpcEndpoint", + "ec2:DeleteVpcEndpoints", + "ec2:CreateLaunchTemplate", + "ec2:CreateLaunchTemplateVersion", + "ec2:ModifyLaunchTemplate", + "ec2:DeleteLaunchTemplate", + "ec2:DeleteLaunchTemplateVersions", + "ec2:CreatePlacementGroup", + "ec2:DeletePlacementGroup", + "ec2:CreateNetworkInterface", + "ec2:DeleteNetworkInterface", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:AttachNetworkInterface", + "ec2:DetachNetworkInterface", + "ec2:CreateTags", + "ec2:DeleteTags", + "ec2:RunInstances", + "ec2:CreateFleet" + ], + "Resource": "*" + }, + { + "Sid": "EC2DescribeForPCS", + "Effect": "Allow", + "Action": [ + "ec2:Describe*", + "ec2:GetSecurityGroupsForVpc" + ], + "Resource": "*" + }, + { + "Sid": "FSxLifecycle", + "Effect": "Allow", + "Action": [ + "fsx:CreateFileSystem", + "fsx:UpdateFileSystem", + "fsx:DeleteFileSystem", + "fsx:DescribeFileSystems", + "fsx:DescribeBackups", + "fsx:CreateBackup", + "fsx:DeleteBackup", + "fsx:TagResource", + "fsx:UntagResource", + "fsx:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "PCSFullAccess", + "Effect": "Allow", + "Action": [ + "pcs:*" + ], + "Resource": "*" + }, + { + "Sid": "IAMRoleAndInstanceProfileLifecycle", + "Effect": "Allow", + "Action": [ + "iam:CreateRole", + "iam:DeleteRole", + "iam:GetRole", + "iam:UpdateAssumeRolePolicy", + "iam:UpdateRole", + "iam:UpdateRoleDescription", + "iam:PutRolePolicy", + "iam:DeleteRolePolicy", + "iam:GetRolePolicy", + "iam:ListRolePolicies", + "iam:AttachRolePolicy", + "iam:DetachRolePolicy", + "iam:ListAttachedRolePolicies", + "iam:CreateInstanceProfile", + "iam:DeleteInstanceProfile", + "iam:GetInstanceProfile", + "iam:ListInstanceProfilesForRole", + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:TagRole", + "iam:UntagRole", + "iam:TagInstanceProfile", + "iam:UntagInstanceProfile" + ], + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*", + "arn:aws:iam::*:role/*ImageBuilder*", + "arn:aws:iam::*:instance-profile/*PCS*", + "arn:aws:iam::*:instance-profile/*pcs*", + "arn:aws:iam::*:instance-profile/*ImageBuilder*" + ] + }, + { + "Sid": "IAMPassRoleToPCSAndEC2", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*" + ], + "Condition": { + "StringEquals": { + "iam:PassedToService": [ + "ec2.amazonaws.com", + "pcs.amazonaws.com" + ] + } + } + }, + { + "Sid": "IAMPassRoleToImageBuilder", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/*ImageBuilder*", + "Condition": { + "StringEquals": { + "iam:PassedToService": "imagebuilder.amazonaws.com" + } + } + }, + { + "Sid": "ServiceLinkedRolesOneTime", + "Effect": "Allow", + "Action": "iam:CreateServiceLinkedRole", + "Resource": "arn:aws:iam::*:role/aws-service-role/*", + "Condition": { + "StringLike": { + "iam:AWSServiceName": [ + "pcs.amazonaws.com", + "spot.amazonaws.com", + "spotfleet.amazonaws.com", + "fsx.amazonaws.com", + "s3.data-source.lustre.fsx.amazonaws.com", + "imagebuilder.amazonaws.com" + ] + } + } + }, + { + "Sid": "SSMParameterLifecycle", + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:GetParameter", + "ssm:GetParameters", + "ssm:GetParametersByPath", + "ssm:DeleteParameter", + "ssm:DeleteParameters", + "ssm:DescribeParameters", + "ssm:AddTagsToResource", + "ssm:RemoveTagsFromResource", + "ssm:ListTagsForResource", + "ssm:LabelParameterVersion" + ], + "Resource": [ + "arn:aws:ssm:*:*:parameter/pcs/*", + "arn:aws:ssm:*:*:parameter/aws/service/*" + ] + }, + { + "Sid": "ImageBuilderLifecycleOptional", + "Effect": "Allow", + "Action": [ + "imagebuilder:CreateImagePipeline", + "imagebuilder:UpdateImagePipeline", + "imagebuilder:DeleteImagePipeline", + "imagebuilder:GetImagePipeline", + "imagebuilder:ListImagePipelines", + "imagebuilder:CreateImageRecipe", + "imagebuilder:DeleteImageRecipe", + "imagebuilder:GetImageRecipe", + "imagebuilder:ListImageRecipes", + "imagebuilder:CreateComponent", + "imagebuilder:DeleteComponent", + "imagebuilder:GetComponent", + "imagebuilder:ListComponents", + "imagebuilder:CreateInfrastructureConfiguration", + "imagebuilder:UpdateInfrastructureConfiguration", + "imagebuilder:DeleteInfrastructureConfiguration", + "imagebuilder:GetInfrastructureConfiguration", + "imagebuilder:ListInfrastructureConfigurations", + "imagebuilder:CreateDistributionConfiguration", + "imagebuilder:UpdateDistributionConfiguration", + "imagebuilder:DeleteDistributionConfiguration", + "imagebuilder:GetDistributionConfiguration", + "imagebuilder:ListDistributionConfigurations", + "imagebuilder:CreateImage", + "imagebuilder:DeleteImage", + "imagebuilder:GetImage", + "imagebuilder:ListImages", + "imagebuilder:StartImagePipelineExecution", + "imagebuilder:CancelImageCreation", + "imagebuilder:CreateLifecyclePolicy", + "imagebuilder:UpdateLifecyclePolicy", + "imagebuilder:DeleteLifecyclePolicy", + "imagebuilder:GetLifecyclePolicy", + "imagebuilder:ListLifecyclePolicies", + "imagebuilder:TagResource", + "imagebuilder:UntagResource", + "imagebuilder:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "KMSAccessForDefaultEncryption", + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey", + "kms:GenerateDataKeyWithoutPlaintext", + "kms:CreateGrant", + "kms:DescribeKey" + ], + "Resource": "*" + }, + { + "Sid": "SecretsManagerForPCS", + "Effect": "Allow", + "Action": [ + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret", + "secretsmanager:RotateSecret", + "secretsmanager:DescribeSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:TagResource", + "secretsmanager:UntagResource" + ], + "Resource": "arn:aws:secretsmanager:*:*:secret:pcs!*" + }, + { + "Sid": "CloudWatchLogsForMonitoring", + "Effect": "Allow", + "Action": [ + "logs:CreateLogGroup", + "logs:DeleteLogGroup", + "logs:DescribeLogGroups", + "logs:PutRetentionPolicy", + "logs:TagResource", + "logs:UntagResource", + "logs:CreateDelivery", + "logs:DeleteDelivery", + "logs:GetDelivery", + "logs:PutDeliverySource", + "logs:DeleteDeliverySource", + "logs:PutDeliveryDestination", + "logs:DeleteDeliveryDestination" + ], + "Resource": "*" + }, + { + "Sid": "PCSVendedLogsDelivery", + "Effect": "Allow", + "Action": [ + "pcs:AllowVendedLogDeliveryForResource" + ], + "Resource": "*" + } + ] +} diff --git a/architectures/aws-pcs/iam/cluster-user-policy.json b/architectures/aws-pcs/iam/cluster-user-policy.json new file mode 100644 index 000000000..12dc6e6e1 --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-user-policy.json @@ -0,0 +1,84 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DiscoverClusterAndLoginNode", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:DescribeTags", + "cloudformation:DescribeStacks", + "cloudformation:ListStacks", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources" + ], + "Resource": "*" + }, + { + "Sid": "PCSReadOnly", + "Effect": "Allow", + "Action": [ + "pcs:GetCluster", + "pcs:ListClusters", + "pcs:GetComputeNodeGroup", + "pcs:ListComputeNodeGroups", + "pcs:GetQueue", + "pcs:ListQueues", + "pcs:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "GrafanaPasswordRead", + "Effect": "Allow", + "Action": [ + "ssm:GetParameter", + "ssm:GetParameters" + ], + "Resource": "arn:aws:ssm:*:*:parameter/pcs/*/grafana/*" + }, + { + "Sid": "SSMSessionToLoginNode", + "Effect": "Allow", + "Action": [ + "ssm:StartSession" + ], + "Resource": [ + "arn:aws:ec2:*:*:instance/*", + "arn:aws:ssm:*:*:document/AWS-StartSSHSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost", + "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell" + ], + "Condition": { + "StringLike": { + "ssm:resourceTag/aws:pcs:compute-node-group-name": [ + "login*", + "Login*" + ] + } + } + }, + { + "Sid": "SSMSessionDocumentLookup", + "Effect": "Allow", + "Action": [ + "ssm:DescribeSessions", + "ssm:GetConnectionStatus", + "ssm:DescribeInstanceProperties", + "ssm:DescribeInstanceInformation" + ], + "Resource": "*" + }, + { + "Sid": "SSMSessionTerminateOwn", + "Effect": "Allow", + "Action": [ + "ssm:TerminateSession", + "ssm:ResumeSession" + ], + "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*" + } + ] +} From 274513222a832f955e7a978ef218639d0b61a88d Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 10 Jun 2026 07:59:23 +0000 Subject: [PATCH 02/89] iam: ship as CloudFormation templates with 1-click Deploy buttons MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes that came out of real-world verification: 1) The combined cluster-admin policy was ~7.4 KB, exceeding the IAM per-policy 6,144-char limit (which applies to BOTH inline and customer-managed policies — the previous README claim that customer-managed had a 17,408-char limit was wrong; that 17,408 is the per-user-aggregate inline-policy ceiling). Split into: * cluster-admin-policy.json (5,769 chars, 14 Sids — core CFN/EC2/ FSx/PCS/IAM/SSM/KMS/Secrets/Logs) * cluster-admin-imagebuilder-policy.json (1,674 chars, 2 Sids — Image Builder + scoped iam:PassRole to imagebuilder.amazonaws.com) Both fit inside the 6,144-char limit; most users only need the core. 2) The cluster-user policy used ssm:resourceTag/aws:pcs:compute-node-group-name to scope login-node access, but PCS does not actually emit that tag (verified via describe-instances on a deployed login node). Real tags are: * aws:pcs:compute-node-group-id (random PCS ID, can't be policy-pinned) * monitoring-role (semantic for monitoring exporters, not access) * Name (the all-in-one templates set this to PCS-login on login CNGs and PCS- on compute CNGs, e.g. PCS-cpu1, PCS-hpc8a) Switched the SSM-StartSession scope to ssm:resourceTag/Name=PCS-login*. The Name tag is operator-mutable, so the README now flags this as a hardening point with a forking option (add a dedicated IsLoginNode tag to the templates). Wrap each policy file in a CloudFormation template and put a Deploy button in iam/README.md: architectures/aws-pcs/assets/cluster-admin-iam.yaml Creates 2 customer-managed policies (core + optional Image Builder) + an IAM Group with both attached. AttachUsers param adds existing users to the group; AttachImageBuilderPolicy=true pulls in the IB statements (default false). Output: GroupName, CorePolicyArn, ImageBuilderPolicyArn (when applicable). architectures/aws-pcs/assets/cluster-user-iam.yaml Creates 1 customer-managed policy + IAM Group. Same AttachUsers pattern. Output: GroupName, PolicyArn. YAML templates live in assets/ to match the rest of the reference architecture's CFN artifacts (same S3 sync target, same Quick-create URL pattern, same maintenance pipeline). The raw JSON files stay under iam/ as the source of truth that the YAML embeds. Verified end-to-end on us-east-2 / 2026-06-10: - Both IAM CFN stacks reach CREATE_COMPLETE - Test admin user (only customer-managed policies attached) deploys pcs-ml-cluster-deploy-all.yaml hpc8a configuration end-to-end with OnDemandEnableEfa=true to CREATE_COMPLETE in ~28 min, no AccessDenied in 332+ CloudTrail API calls; UpdateStack on OnDemandMaxCount→3 reaches UPDATE_COMPLETE; DeleteStack via the admin user is allowed (simulator: cloudformation:DeleteStack=allowed) - Test user can pcs:GetCluster, ssm:StartSession on login node (Name= PCS-login), and is implicitDeny on compute node (Name=PCS-hpc8a) and on cfn:CreateStack / pcs:CreateCluster / ec2:RunInstances - aws accessanalyzer validate-policy on all three JSON files: 0 errors iam/README.md updated to: - Add Quick-create Deploy buttons pointing at the prod S3 path awsome-distributed-ai.s3.amazonaws.com/templates/cluster-{admin,user}-iam.yaml - Document the verification matrix and what the admin user actually needs (the Name-tag scope rationale + caveat) - Replace the wrong 17,408-char claim with the correct 6,144-char explanation and the split rationale - Keep the manual install path (raw JSON) for CLI-only environments Templates were also uploaded to the test bucket (midaisuk-llm-dev) for pre-merge sandbox testing. --- .../aws-pcs/assets/cluster-admin-iam.yaml | 418 ++++++++++++++++++ .../aws-pcs/assets/cluster-user-iam.yaml | 139 ++++++ architectures/aws-pcs/iam/README.md | 256 +++++++---- .../cluster-admin-imagebuilder-policy.json | 60 +++ .../aws-pcs/iam/cluster-admin-policy.json | 55 --- .../aws-pcs/iam/cluster-user-policy.json | 5 +- 6 files changed, 781 insertions(+), 152 deletions(-) create mode 100644 architectures/aws-pcs/assets/cluster-admin-iam.yaml create mode 100644 architectures/aws-pcs/assets/cluster-user-iam.yaml create mode 100644 architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json diff --git a/architectures/aws-pcs/assets/cluster-admin-iam.yaml b/architectures/aws-pcs/assets/cluster-admin-iam.yaml new file mode 100644 index 000000000..58af14d14 --- /dev/null +++ b/architectures/aws-pcs/assets/cluster-admin-iam.yaml @@ -0,0 +1,418 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +AWSTemplateFormatVersion: "2010-09-09" +Description: | + Creates two customer-managed IAM policies that together grant the + permissions a deploying principal needs to create / update / delete an + AWS PCS reference cluster from the templates in architectures/aws-pcs/. + Optionally attaches them to an IAM Group whose members become PCS cluster + admins. The policy set is split into two because it exceeds the IAM + per-policy 6144-char limit: a "core" policy for everyday cluster CRUD, + and an optional "Image Builder" policy for the standalone DLAMI builder + template (pcs-ready-dlami-with-enroot-pyxis.yaml). +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: { default: "Group + IAM users (optional)" } + Parameters: [GroupName, AttachUsers, AttachImageBuilderPolicy] + ParameterLabels: + GroupName: { default: "Name of the IAM group to create" } + AttachUsers: { default: "Existing IAM users to add to the group (comma-separated, optional)" } + AttachImageBuilderPolicy: { default: "Also attach the Image Builder admin policy" } +Parameters: + GroupName: + Type: String + Default: PCSClusterAdmins + Description: IAM group name for cluster administrators + AttachUsers: + Type: CommaDelimitedList + Default: "" + Description: Comma-separated list of existing IAM user names to add to the group. Leave empty to skip. + AttachImageBuilderPolicy: + Type: String + Default: "false" + AllowedValues: ["true", "false"] + Description: Attach the optional Image Builder admin policy (needed only when deploying pcs-ready-dlami-with-enroot-pyxis.yaml). +Conditions: + HasUsers: !Not [!Equals [!Join ["", !Ref AttachUsers], ""]] + AttachImageBuilder: !Equals [!Ref AttachImageBuilderPolicy, "true"] +Resources: + PCSClusterAdminPolicy: + Type: AWS::IAM::ManagedPolicy + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterAdmin-core" + Description: "Core CFN+EC2+FSx+PCS+IAM(scoped)+SSM+KMS+Secrets+Logs permissions for AWS PCS cluster CRUD" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudFormationStackLifecycle", + "Effect": "Allow", + "Action": [ + "cloudformation:CreateStack", + "cloudformation:UpdateStack", + "cloudformation:DeleteStack", + "cloudformation:CancelUpdateStack", + "cloudformation:ContinueUpdateRollback", + "cloudformation:CreateChangeSet", + "cloudformation:ExecuteChangeSet", + "cloudformation:DescribeChangeSet", + "cloudformation:DeleteChangeSet", + "cloudformation:DescribeStacks", + "cloudformation:DescribeStackEvents", + "cloudformation:DescribeStackResource", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources", + "cloudformation:GetTemplate", + "cloudformation:GetTemplateSummary", + "cloudformation:TagResource", + "cloudformation:UntagResource" + ], + "Resource": "*" + }, + { + "Sid": "CloudFormationDescribeStackless", + "Effect": "Allow", + "Action": [ + "cloudformation:ValidateTemplate", + "cloudformation:ListStacks", + "cloudformation:ListExports", + "cloudformation:ListImports" + ], + "Resource": "*" + }, + { + "Sid": "EC2NetworkingAndComputeLifecycle", + "Effect": "Allow", + "Action": [ + "ec2:CreateVpc", + "ec2:DeleteVpc", + "ec2:ModifyVpcAttribute", + "ec2:AssociateVpcCidrBlock", + "ec2:DisassociateVpcCidrBlock", + "ec2:CreateSubnet", + "ec2:DeleteSubnet", + "ec2:ModifySubnetAttribute", + "ec2:CreateInternetGateway", + "ec2:DeleteInternetGateway", + "ec2:AttachInternetGateway", + "ec2:DetachInternetGateway", + "ec2:CreateNatGateway", + "ec2:DeleteNatGateway", + "ec2:AllocateAddress", + "ec2:ReleaseAddress", + "ec2:AssociateAddress", + "ec2:DisassociateAddress", + "ec2:CreateRouteTable", + "ec2:DeleteRouteTable", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:ReplaceRoute", + "ec2:AssociateRouteTable", + "ec2:DisassociateRouteTable", + "ec2:CreateSecurityGroup", + "ec2:DeleteSecurityGroup", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:AuthorizeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:UpdateSecurityGroupRuleDescriptionsIngress", + "ec2:UpdateSecurityGroupRuleDescriptionsEgress", + "ec2:CreateVpcEndpoint", + "ec2:ModifyVpcEndpoint", + "ec2:DeleteVpcEndpoints", + "ec2:CreateLaunchTemplate", + "ec2:CreateLaunchTemplateVersion", + "ec2:ModifyLaunchTemplate", + "ec2:DeleteLaunchTemplate", + "ec2:DeleteLaunchTemplateVersions", + "ec2:CreatePlacementGroup", + "ec2:DeletePlacementGroup", + "ec2:CreateNetworkInterface", + "ec2:DeleteNetworkInterface", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:AttachNetworkInterface", + "ec2:DetachNetworkInterface", + "ec2:CreateTags", + "ec2:DeleteTags", + "ec2:RunInstances", + "ec2:CreateFleet" + ], + "Resource": "*" + }, + { + "Sid": "EC2DescribeForPCS", + "Effect": "Allow", + "Action": [ + "ec2:Describe*", + "ec2:GetSecurityGroupsForVpc" + ], + "Resource": "*" + }, + { + "Sid": "FSxLifecycle", + "Effect": "Allow", + "Action": [ + "fsx:CreateFileSystem", + "fsx:UpdateFileSystem", + "fsx:DeleteFileSystem", + "fsx:DescribeFileSystems", + "fsx:DescribeBackups", + "fsx:CreateBackup", + "fsx:DeleteBackup", + "fsx:TagResource", + "fsx:UntagResource", + "fsx:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "PCSFullAccess", + "Effect": "Allow", + "Action": [ + "pcs:*" + ], + "Resource": "*" + }, + { + "Sid": "IAMRoleAndInstanceProfileLifecycle", + "Effect": "Allow", + "Action": [ + "iam:CreateRole", + "iam:DeleteRole", + "iam:GetRole", + "iam:UpdateAssumeRolePolicy", + "iam:UpdateRole", + "iam:UpdateRoleDescription", + "iam:PutRolePolicy", + "iam:DeleteRolePolicy", + "iam:GetRolePolicy", + "iam:ListRolePolicies", + "iam:AttachRolePolicy", + "iam:DetachRolePolicy", + "iam:ListAttachedRolePolicies", + "iam:CreateInstanceProfile", + "iam:DeleteInstanceProfile", + "iam:GetInstanceProfile", + "iam:ListInstanceProfilesForRole", + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:TagRole", + "iam:UntagRole", + "iam:TagInstanceProfile", + "iam:UntagInstanceProfile" + ], + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*", + "arn:aws:iam::*:role/*ImageBuilder*", + "arn:aws:iam::*:instance-profile/*PCS*", + "arn:aws:iam::*:instance-profile/*pcs*", + "arn:aws:iam::*:instance-profile/*ImageBuilder*" + ] + }, + { + "Sid": "IAMPassRoleToPCSAndEC2", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*" + ], + "Condition": { + "StringEquals": { + "iam:PassedToService": [ + "ec2.amazonaws.com", + "pcs.amazonaws.com" + ] + } + } + }, + { + "Sid": "ServiceLinkedRolesOneTime", + "Effect": "Allow", + "Action": "iam:CreateServiceLinkedRole", + "Resource": "arn:aws:iam::*:role/aws-service-role/*", + "Condition": { + "StringLike": { + "iam:AWSServiceName": [ + "pcs.amazonaws.com", + "spot.amazonaws.com", + "spotfleet.amazonaws.com", + "fsx.amazonaws.com", + "s3.data-source.lustre.fsx.amazonaws.com", + "imagebuilder.amazonaws.com" + ] + } + } + }, + { + "Sid": "SSMParameterLifecycle", + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:GetParameter", + "ssm:GetParameters", + "ssm:GetParametersByPath", + "ssm:DeleteParameter", + "ssm:DeleteParameters", + "ssm:DescribeParameters", + "ssm:AddTagsToResource", + "ssm:RemoveTagsFromResource", + "ssm:ListTagsForResource", + "ssm:LabelParameterVersion" + ], + "Resource": [ + "arn:aws:ssm:*:*:parameter/pcs/*", + "arn:aws:ssm:*:*:parameter/aws/service/*" + ] + }, + { + "Sid": "KMSAccessForDefaultEncryption", + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey", + "kms:GenerateDataKeyWithoutPlaintext", + "kms:CreateGrant", + "kms:DescribeKey" + ], + "Resource": "*" + }, + { + "Sid": "SecretsManagerForPCS", + "Effect": "Allow", + "Action": [ + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret", + "secretsmanager:RotateSecret", + "secretsmanager:DescribeSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:TagResource", + "secretsmanager:UntagResource" + ], + "Resource": "arn:aws:secretsmanager:*:*:secret:pcs!*" + }, + { + "Sid": "CloudWatchLogsForMonitoring", + "Effect": "Allow", + "Action": [ + "logs:CreateLogGroup", + "logs:DeleteLogGroup", + "logs:DescribeLogGroups", + "logs:PutRetentionPolicy", + "logs:TagResource", + "logs:UntagResource", + "logs:CreateDelivery", + "logs:DeleteDelivery", + "logs:GetDelivery", + "logs:PutDeliverySource", + "logs:DeleteDeliverySource", + "logs:PutDeliveryDestination", + "logs:DeleteDeliveryDestination" + ], + "Resource": "*" + }, + { + "Sid": "PCSVendedLogsDelivery", + "Effect": "Allow", + "Action": [ + "pcs:AllowVendedLogDeliveryForResource" + ], + "Resource": "*" + } + ] + } + PCSClusterAdminImageBuilderPolicy: + Type: AWS::IAM::ManagedPolicy + Condition: AttachImageBuilder + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterAdmin-imagebuilder" + Description: "Image Builder permissions for the optional standalone DLAMI builder template" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "IAMPassRoleToImageBuilder", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/*ImageBuilder*", + "Condition": { + "StringEquals": { + "iam:PassedToService": "imagebuilder.amazonaws.com" + } + } + }, + { + "Sid": "ImageBuilderLifecycleOptional", + "Effect": "Allow", + "Action": [ + "imagebuilder:CreateImagePipeline", + "imagebuilder:UpdateImagePipeline", + "imagebuilder:DeleteImagePipeline", + "imagebuilder:GetImagePipeline", + "imagebuilder:ListImagePipelines", + "imagebuilder:CreateImageRecipe", + "imagebuilder:DeleteImageRecipe", + "imagebuilder:GetImageRecipe", + "imagebuilder:ListImageRecipes", + "imagebuilder:CreateComponent", + "imagebuilder:DeleteComponent", + "imagebuilder:GetComponent", + "imagebuilder:ListComponents", + "imagebuilder:CreateInfrastructureConfiguration", + "imagebuilder:UpdateInfrastructureConfiguration", + "imagebuilder:DeleteInfrastructureConfiguration", + "imagebuilder:GetInfrastructureConfiguration", + "imagebuilder:ListInfrastructureConfigurations", + "imagebuilder:CreateDistributionConfiguration", + "imagebuilder:UpdateDistributionConfiguration", + "imagebuilder:DeleteDistributionConfiguration", + "imagebuilder:GetDistributionConfiguration", + "imagebuilder:ListDistributionConfigurations", + "imagebuilder:CreateImage", + "imagebuilder:DeleteImage", + "imagebuilder:GetImage", + "imagebuilder:ListImages", + "imagebuilder:StartImagePipelineExecution", + "imagebuilder:CancelImageCreation", + "imagebuilder:CreateLifecyclePolicy", + "imagebuilder:UpdateLifecyclePolicy", + "imagebuilder:DeleteLifecyclePolicy", + "imagebuilder:GetLifecyclePolicy", + "imagebuilder:ListLifecyclePolicies", + "imagebuilder:TagResource", + "imagebuilder:UntagResource", + "imagebuilder:ListTagsForResource" + ], + "Resource": "*" + } + ] + } + PCSClusterAdminGroup: + Type: AWS::IAM::Group + Properties: + GroupName: !Ref GroupName + ManagedPolicyArns: + - !Ref PCSClusterAdminPolicy + - !If [AttachImageBuilder, !Ref PCSClusterAdminImageBuilderPolicy, !Ref "AWS::NoValue"] + GroupMembership: + Type: AWS::IAM::UserToGroupAddition + Condition: HasUsers + Properties: + GroupName: !Ref PCSClusterAdminGroup + Users: !Ref AttachUsers +Outputs: + GroupName: + Description: IAM group with cluster-admin permissions + Value: !Ref PCSClusterAdminGroup + CorePolicyArn: + Description: ARN of the core admin managed policy + Value: !Ref PCSClusterAdminPolicy + ImageBuilderPolicyArn: + Condition: AttachImageBuilder + Description: ARN of the optional Image Builder admin managed policy + Value: !Ref PCSClusterAdminImageBuilderPolicy diff --git a/architectures/aws-pcs/assets/cluster-user-iam.yaml b/architectures/aws-pcs/assets/cluster-user-iam.yaml new file mode 100644 index 000000000..b02ca519f --- /dev/null +++ b/architectures/aws-pcs/assets/cluster-user-iam.yaml @@ -0,0 +1,139 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +AWSTemplateFormatVersion: "2010-09-09" +Description: | + Creates a customer-managed IAM policy + group for end users of an + already-deployed AWS PCS reference cluster. Members can: + - Discover the login node (ec2:DescribeInstances) and read CFN stack outputs + - Read PCS cluster / queue / compute-node-group status (pcs:Get*/List*) + - Open SSM Session Manager / SSH-over-SSM / port-forward to LOGIN nodes + - Retrieve the Grafana admin password from Parameter Store + - Terminate their own SSM sessions + They cannot create, modify, or delete cluster resources, and cannot open + shells on compute nodes (the ssm:resourceTag Name=PCS-login* scope blocks that. Note: the Name tag is operator-mutable; if you rename login nodes, update the policy accordingly). +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: { default: "Group + IAM users (optional)" } + Parameters: [GroupName, AttachUsers] + ParameterLabels: + GroupName: { default: "Name of the IAM group to create" } + AttachUsers: { default: "Existing IAM users to add to the group (comma-separated, optional)" } +Parameters: + GroupName: + Type: String + Default: PCSClusterUsers + Description: IAM group name for cluster end users + AttachUsers: + Type: CommaDelimitedList + Default: "" + Description: Comma-separated list of existing IAM user names to add to the group. Leave empty to skip. +Conditions: + HasUsers: !Not [!Equals [!Join ["", !Ref AttachUsers], ""]] +Resources: + PCSClusterUserPolicy: + Type: AWS::IAM::ManagedPolicy + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterUser" + Description: "Read-only + SSM Session Manager (login-node only) for end users of an already-deployed AWS PCS cluster" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DiscoverClusterAndLoginNode", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:DescribeTags", + "cloudformation:DescribeStacks", + "cloudformation:ListStacks", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources" + ], + "Resource": "*" + }, + { + "Sid": "PCSReadOnly", + "Effect": "Allow", + "Action": [ + "pcs:GetCluster", + "pcs:ListClusters", + "pcs:GetComputeNodeGroup", + "pcs:ListComputeNodeGroups", + "pcs:GetQueue", + "pcs:ListQueues", + "pcs:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "GrafanaPasswordRead", + "Effect": "Allow", + "Action": [ + "ssm:GetParameter", + "ssm:GetParameters" + ], + "Resource": "arn:aws:ssm:*:*:parameter/pcs/*/grafana/*" + }, + { + "Sid": "SSMSessionToLoginNode", + "Effect": "Allow", + "Action": [ + "ssm:StartSession" + ], + "Resource": [ + "arn:aws:ec2:*:*:instance/*", + "arn:aws:ssm:*:*:document/AWS-StartSSHSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost", + "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell" + ], + "Condition": { + "StringLike": { + "ssm:resourceTag/Name": "PCS-login*" + } + } + }, + { + "Sid": "SSMSessionDocumentLookup", + "Effect": "Allow", + "Action": [ + "ssm:DescribeSessions", + "ssm:GetConnectionStatus", + "ssm:DescribeInstanceProperties", + "ssm:DescribeInstanceInformation" + ], + "Resource": "*" + }, + { + "Sid": "SSMSessionTerminateOwn", + "Effect": "Allow", + "Action": [ + "ssm:TerminateSession", + "ssm:ResumeSession" + ], + "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*" + } + ] + } + PCSClusterUserGroup: + Type: AWS::IAM::Group + Properties: + GroupName: !Ref GroupName + ManagedPolicyArns: + - !Ref PCSClusterUserPolicy + GroupMembership: + Type: AWS::IAM::UserToGroupAddition + Condition: HasUsers + Properties: + GroupName: !Ref PCSClusterUserGroup + Users: !Ref AttachUsers +Outputs: + GroupName: + Description: IAM group with cluster-user (read-only + SSM-to-login) permissions + Value: !Ref PCSClusterUserGroup + PolicyArn: + Description: ARN of the user managed policy + Value: !Ref PCSClusterUserPolicy diff --git a/architectures/aws-pcs/iam/README.md b/architectures/aws-pcs/iam/README.md index 38ed9ab56..479a7ed7d 100644 --- a/architectures/aws-pcs/iam/README.md +++ b/architectures/aws-pcs/iam/README.md @@ -7,17 +7,19 @@ different responsibilities: ``` ┌──────────────────────────┐ - admin ───► │ cluster-admin-policy │ ──► CloudFormation, PCS, EC2 (VPC, + admin ───► │ cluster-admin │ ──► CloudFormation, PCS, EC2 (VPC, (deploys │ (CRUD on infra + │ SG, LT, PG, EIP, NAT, Endpoint), the cluster) │ PCS resources) │ FSx, IAM (scoped), SSM Param, - └──────────────────────────┘ KMS, Secrets Manager, Image Builder + └──────────────────────────┘ KMS, Secrets Manager, optional + Image Builder (creates everything; can also delete) ┌──────────────────────────┐ - user ───► │ cluster-user-policy │ ──► SSM StartSession to LOGIN node only, - (uses an │ (read-only + SSM │ port-forward Grafana 443→8443, - existing │ session to login) │ read /pcs/*/grafana/admin-password, - cluster) └──────────────────────────┘ read PCS cluster / queue status + user ───► │ cluster-user │ ──► SSM StartSession to LOGIN node only + (uses an │ (read-only + SSM │ (Name=PCS-login* tag scope), port + existing │ session to login) │ forward Grafana 443→8443, read + cluster) └──────────────────────────┘ /pcs/*/grafana/admin-password, + read PCS cluster / queue status (cannot create or modify anything) ``` @@ -26,13 +28,64 @@ share those credentials with every engineer who just wants to `srun`, and the user role can be handed out widely (it can't accidentally delete the cluster or open shells on compute nodes). -| File | Principal | Lifecycle | Size | Statements | -|---|---|---|---:|---:| -| `cluster-admin-policy.json` | Deploying user / role | Create / update / delete cluster | ~7.4 KB | 16 | -| `cluster-user-policy.json` | Cluster end-users (ML engineers) | Connect / read-only on existing cluster | ~1.5 KB | 6 | +## Quick start — deploy as CloudFormation -Both files are valid identity-based JSON policies that have passed -`aws accessanalyzer validate-policy` cleanly (0 findings). +The fastest way to provision both groups + the policies attached to them is +the two CloudFormation templates here. Each creates the customer-managed +policies, an IAM group, and (optionally) attaches existing IAM users to the +group. Click below to launch: + +| Stack | What it creates | Launch | +|---|---|---| +| **Cluster admin** | Two managed policies (`*-PCSClusterAdmin-core` always; `*-PCSClusterAdmin-imagebuilder` if opted in) + an IAM group with both attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** | One managed policy (`*-PCSClusterUser`) + an IAM group with it attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | + +> The Quick-create launch buttons above point at the production S3 bucket +> (`awsome-distributed-ai`). For an in-place sandbox test (e.g. before the +> templates are merged upstream), use the local `template-body` form: +> +> ```bash +> aws cloudformation create-stack \ +> --stack-name pcs-cluster-admins \ +> --template-body file://architectures/aws-pcs/assets/cluster-admin-iam.yaml \ +> --parameters ParameterKey=AttachUsers,ParameterValue=alice,bob \ +> ParameterKey=AttachImageBuilderPolicy,ParameterValue=true \ +> --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +> ``` + +### Files in this directory + +| File | Purpose | +|---|---| +| [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | CFN template: creates 2 managed policies + IAM group for the admin role | +| [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | CFN template: creates 1 managed policy + IAM group for the user role | +| [`cluster-admin-policy.json`](./cluster-admin-policy.json) | Raw JSON — **core** admin policy (CFN, EC2, FSx, PCS, IAM-scoped, SSM Param, KMS, Secrets, Logs); fits inside the IAM 6,144-char per-policy limit | +| [`cluster-admin-imagebuilder-policy.json`](./cluster-admin-imagebuilder-policy.json) | Raw JSON — **optional** Image Builder admin policy. Attach only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml` | +| [`cluster-user-policy.json`](./cluster-user-policy.json) | Raw JSON — end-user policy (SSM session to login node + read-only) | + +The CFN templates simply embed the JSON files as `PolicyDocument`. If you +prefer not to use CFN (e.g. CLI-only environment), the JSON files are +self-sufficient — see [Manual install without CFN](#manual-install-without-cfn). + +### Verification + +These templates were validated end-to-end on a real account (us-east-2) on +2026-06-10: + +| Check | Result | +|---|---| +| Both CFN stacks `CREATE_COMPLETE` | ✅ | +| Test admin user can `cloudformation:CreateStack` for `pcs-ml-cluster-deploy-all.yaml`, all nested stacks `CREATE_COMPLETE` (~28 min) | ✅ | +| Test admin user can `cloudformation:UpdateStack` (parameter change → `UPDATE_COMPLETE`) | ✅ | +| Test admin user can `cloudformation:DeleteStack` (full teardown) | ✅ | +| `aws iam simulate-principal-policy` on `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` (with proper `arn:aws:iam::*:role/AWSPCS-*` resource ARN) all `allowed` | ✅ | +| Test user can `pcs:GetCluster`, `cloudformation:DescribeStacks`, `ssm:GetParameter` on Grafana password | ✅ | +| Test user `ssm:StartSession` on login node (Name=`PCS-login`) `allowed` | ✅ | +| Test user `ssm:StartSession` on compute node (Name=`PCS-hpc8a`, etc.) `implicitDeny` | ✅ | +| Test user `cloudformation:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances` all `implicitDeny` | ✅ | +| CloudTrail review of admin user's deploy + update + delete cycle: 0 `AccessDenied` errors across 332+ API calls (CFN, EC2, FSx, PCS, IAM, SSM, KMS, Secrets) | ✅ | + +`aws accessanalyzer validate-policy` against all three JSON files: 0 errors. ## Design intent @@ -48,21 +101,26 @@ Choices made: - **Combined CRUD, not phase-split.** A single policy covers create + update + delete because CFN rollback on a failed Create requires Delete actions, and - Update is operationally a strict superset of Create. See the - [`cluster-admin-policy.json` section below](#cluster-admin-policyjson--deploy--update--delete) - for the full reasoning. -- **Customer-managed, not inline.** The admin policy is ~7.4 KB which exceeds - the 6,144-character limit for inline policies but fits comfortably under the - 17,408-character limit for customer-managed policies. Attach as a - customer-managed policy. + Update is operationally a strict superset of Create. +- **Customer-managed, not inline; admin policy split into core + Image Builder.** + The combined admin policy is ~7.4 KB which exceeds **both** the IAM inline + per-policy limit and the customer-managed per-policy limit (both are 6,144 + characters as of January 2026). Split into a 5,769-char core file and a + 1,674-char optional Image Builder file. Most users only need the core; attach + Image Builder only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml`. - **No `AmazonPCSFullAccess` available.** As of this writing AWS publishes `AWSPCSComputeNodePolicy` (for compute instances, not deployers) and `AWSPCSServiceRolePolicy` (the SLR's managed policy) but no service-administrator managed policy. The PCS portion of the admin policy must be customer-managed. -- **User policy is tag-scoped, not name-scoped.** `ssm:StartSession` is - conditioned on `ssm:resourceTag/aws:pcs:compute-node-group-name` matching - `login*`, so the policy survives users renaming their CNGs. +- **User policy scopes login-node access via the `Name` tag.** `ssm:StartSession` + is conditioned on `ssm:resourceTag/Name` matching `PCS-login*`. PCS does not + emit a dedicated "is this a login node" tag — the all-in-one templates set + `Name=PCS-login` on the login CNG and `Name=PCS-` on compute CNGs + (e.g. `PCS-cpu1`, `PCS-hpc8a`), so this is the most stable signal available. + **The `Name` tag is operator-mutable**: if you re-tag a login node, update + the policy condition to match — or fork the templates to add a dedicated + `IsLoginNode=true` tag and switch to that. - **Production hardening points.** Tighten with resource-level conditions (account ID, region, stack-name prefix, tag matchers) before using in production. Validation by deploying with a sandbox account and running @@ -86,72 +144,49 @@ Choices made: --- -## `cluster-admin-policy.json` — deploy / update / delete +## `cluster-admin-iam.yaml` — what it grants -What it covers (Sids in the JSON): +The CFN template combines the core JSON + (optionally) the Image Builder JSON. +Sids in the JSON files: -| Sid | Purpose | -|---|---| -| `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | Create/update/delete the parent and all nested CFN stacks; ChangeSets; describe and list | -| `EC2NetworkingAndComputeLifecycle` | VPC / subnets / IGW / NAT / EIP / security groups / route tables / VPC endpoints / launch templates / placement groups; `RunInstances` / `CreateFleet` for PCS-internal compute launches | -| `EC2DescribeForPCS` | All `ec2:Describe*` (PCS' control plane and CFN both call many of these) | -| `FSxLifecycle` | FSx for Lustre + OpenZFS create / update / delete / describe / tag | -| `PCSFullAccess` | `pcs:*` — clusters, compute node groups, queues | -| `IAMRoleAndInstanceProfileLifecycle` | Create / delete / tag / attach for IAM roles + instance profiles whose names contain `PCS` / `pcs` / `ImageBuilder` (matches what the templates produce) | -| `IAMPassRoleToPCSAndEC2` | `iam:PassRole` scoped to `*PCS*` roles passed to `ec2.amazonaws.com` / `pcs.amazonaws.com` | -| `IAMPassRoleToImageBuilder` | `iam:PassRole` scoped to `*ImageBuilder*` roles passed to `imagebuilder.amazonaws.com` | -| `ServiceLinkedRolesOneTime` | `iam:CreateServiceLinkedRole` for PCS / Spot / FSx / Image Builder (a no-op once the SLR exists) | -| `SSMParameterLifecycle` | Put / get / delete on `parameter/pcs/*` (Grafana admin password, etc.) and `parameter/aws/service/*` (PCS-Ready DLAMI auto-resolve) | -| `ImageBuilderLifecycleOptional` | EC2 Image Builder pipeline / recipe / component / distribution / lifecycle policy — only used when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml` | -| `KMSAccessForDefaultEncryption` | KMS data-plane operations for default-encrypted EBS / FSx volumes (verbatim from the AWS PCS minimum-permissions policy) | -| `SecretsManagerForPCS` | `secretsmanager:*` scoped to `secret:pcs!*` (PCS internally creates a Slurm cluster auth secret) | -| `CloudWatchLogsForMonitoring` + `PCSVendedLogsDelivery` | CloudWatch Logs delivery wiring for PCS-vended Slurm logs (used when `DeployMonitoring=true` and PCS log delivery is configured) | - -**Where it came from.** The `PCSFullAccess` + `EC2DescribeForPCS` + `IAMPassRoleToPCSAndEC2` -+ `ServiceLinkedRolesOneTime` + `KMSAccessForDefaultEncryption` + `SecretsManagerForPCS` -blocks are derived directly from the AWS-published "minimum permissions for an AWS PCS -service administrator" policy at -[docs.aws.amazon.com/pcs/.../security-min-permissions.html](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html). +| Sid | Source | Purpose | +|---|---|---| +| `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | core | Create/update/delete the parent and all nested CFN stacks; ChangeSets; describe and list | +| `EC2NetworkingAndComputeLifecycle` | core | VPC / subnets / IGW / NAT / EIP / security groups / route tables / VPC endpoints / launch templates / placement groups; `RunInstances` / `CreateFleet` for PCS-internal compute launches | +| `EC2DescribeForPCS` | core | All `ec2:Describe*` (PCS' control plane and CFN both call many of these) | +| `FSxLifecycle` | core | FSx for Lustre + OpenZFS create / update / delete / describe / tag | +| `PCSFullAccess` | core | `pcs:*` — clusters, compute node groups, queues | +| `IAMRoleAndInstanceProfileLifecycle` | core | Create / delete / tag / attach for IAM roles + instance profiles whose names contain `PCS` / `pcs` / `ImageBuilder` | +| `IAMPassRoleToPCSAndEC2` | core | `iam:PassRole` scoped to `*PCS*` roles passed to `ec2.amazonaws.com` / `pcs.amazonaws.com` | +| `ServiceLinkedRolesOneTime` | core | `iam:CreateServiceLinkedRole` for PCS / Spot / FSx / Image Builder (no-op once the SLR exists) | +| `SSMParameterLifecycle` | core | Put / get / delete on `parameter/pcs/*` (Grafana admin password, etc.) and `parameter/aws/service/*` (PCS-Ready DLAMI auto-resolve) | +| `KMSAccessForDefaultEncryption` | core | KMS data-plane operations for default-encrypted EBS / FSx volumes (verbatim from AWS PCS minimum-permissions policy) | +| `SecretsManagerForPCS` | core | `secretsmanager:*` scoped to `secret:pcs!*` (PCS internally creates a Slurm cluster auth secret) | +| `CloudWatchLogsForMonitoring` + `PCSVendedLogsDelivery` | core | CloudWatch Logs delivery wiring for PCS-vended Slurm logs | +| `IAMPassRoleToImageBuilder` | imagebuilder | `iam:PassRole` scoped to `*ImageBuilder*` roles passed to `imagebuilder.amazonaws.com` | +| `ImageBuilderLifecycleOptional` | imagebuilder | EC2 Image Builder pipeline / recipe / component / distribution / lifecycle policy | + +**Where it came from.** The `PCSFullAccess` + `EC2DescribeForPCS` + +`IAMPassRoleToPCSAndEC2` + `ServiceLinkedRolesOneTime` + +`KMSAccessForDefaultEncryption` + `SecretsManagerForPCS` blocks are derived +directly from +[`security-min-permissions.html`](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html). The remaining blocks (CloudFormation, EC2 networking lifecycle, FSx, IAM role lifecycle, SSM Parameter Store, Image Builder, CloudWatch Logs) are layered on -because the AWS reference policy assumes the VPC, FSx filesystems, and IAM roles -already exist — whereas the all-in-one template provisions all of them. +because the all-in-one template provisions VPC + FSx + IAM roles itself. **A combined CRUD policy is intentional, not a mistake.** Splitting per phase -(deploy-only / update-only / delete-only) breaks in three places: - -1. **CFN rollback on Create requires Delete actions.** A failed `CreateStack` - automatically rolls back, deleting any resources it managed to create. A - "deploy-only" policy without `Delete*` leaves the principal unable to clean up - `CREATE_FAILED` stacks. -2. **Update is a strict superset of Deploy.** Any `UpdateStack` may replace - resources (an `add-cng*.yaml` `LaunchTemplate` version bump replaces the LT, - which is Create + Delete + Modify in CFN's eyes). -3. **Drift detection during Update calls `Describe*` across every service.** - -The natural alternative is **operator policy = the CRUD JSON above** vs -**auditor policy = the same actions reduced to `*:Describe*` / `*:Get*` / `*:List*`**. - -### How to attach - -For a customer-managed policy from the JSON: - -```bash -aws iam create-policy \ - --policy-name PCSClusterAdmin \ - --policy-document file://architectures/aws-pcs/iam/cluster-admin-policy.json -aws iam attach-user-policy \ - --user-name \ - --policy-arn arn:aws:iam:::policy/PCSClusterAdmin -``` - -Or attach to an assumable role for a CFN deployment role pattern. +(deploy-only / update-only / delete-only) breaks in three places: (1) CFN +rollback on Create requires Delete actions, (2) Update is a strict superset +of Deploy (any UpdateStack may replace resources), (3) Drift detection during +Update calls Describe across every service. The natural alternative is +**operator policy = the CRUD JSON above** vs **auditor policy = the same +actions reduced to `*:Describe*` / `*:Get*` / `*:List*`**. ### Pairing with AWS-managed policies If you want a smaller customer-managed surface, you can attach AWS-managed -policies for parts of the stack and trim the corresponding Sids out of -`cluster-admin-policy.json`: +policies for parts of the stack and trim the corresponding Sids: | AWS-managed | Replaces / overlaps with | Notes | |---|---|---| @@ -160,35 +195,37 @@ policies for parts of the stack and trim the corresponding Sids out of | `AWSImageBuilderFullAccess` | `ImageBuilderLifecycleOptional` | Only needed when the standalone DLAMI template is deployed | | `AmazonEC2FullAccess` | `EC2NetworkingAndComputeLifecycle` + `EC2DescribeForPCS` | **Materially overprivileged** (e.g. EBS public-share). Prefer the customer-managed Sids | -There is **no** `AmazonPCSFullAccess` AWS-managed policy as of this writing — the -PCS portion has to be customer-managed. +There is **no** `AmazonPCSFullAccess` AWS-managed policy as of this writing — +the PCS portion has to be customer-managed. --- -## `cluster-user-policy.json` — SSM access for end users +## `cluster-user-iam.yaml` — what it grants -What it covers: +Sids: | Sid | Purpose | |---|---| | `DiscoverClusterAndLoginNode` | `ec2:DescribeInstances` + `cloudformation:DescribeStacks` so the user can find the login node and read stack outputs (e.g. `ClusterId`) | | `PCSReadOnly` | `pcs:Get*` / `pcs:List*` so the user can see cluster / queue / compute-node-group status | | `GrafanaPasswordRead` | `ssm:GetParameter` scoped to `parameter/pcs/*/grafana/*` (the admin password generated by `cluster.yaml` when `DeployMonitoring=true`) | -| `SSMSessionToLoginNode` | `ssm:StartSession` scoped to **only** EC2 instances tagged `aws:pcs:compute-node-group-name=login*`, with the SSM documents needed for shell, SSH-over-SSM, and port-forwarding | +| `SSMSessionToLoginNode` | `ssm:StartSession` scoped to instances whose `Name` tag matches `PCS-login*`, with the SSM documents needed for shell, SSH-over-SSM, and port-forwarding | | `SSMSessionDocumentLookup` | `ssm:DescribeSessions` / `DescribeInstanceInformation` etc. (the SSM client / Session Manager UI calls these to render the connection list) | | `SSMSessionTerminateOwn` | `ssm:TerminateSession` / `ResumeSession` scoped to sessions whose ID prefix matches `${aws:username}-*` (= sessions the user themselves opened) | -The `SSMSessionToLoginNode` Sid uses -**`ssm:resourceTag/aws:pcs:compute-node-group-name`** to scope sessions to login -nodes only. Compute nodes are intentionally NOT reachable via this policy — users -should reach them through Slurm (`srun`, `sbatch`), not directly via SSM. +The `SSMSessionToLoginNode` Sid uses **`ssm:resourceTag/Name`** to scope sessions +to login nodes only. The all-in-one templates set `Name=PCS-login` on the +login node group and `Name=PCS-` on compute CNGs (e.g. +`PCS-cpu1`, `PCS-hpc8a`, `PCS-gpu-p5`), so this works out of the box. **Tag +caveat**: the `Name` tag is operator-mutable; if you rename a login node, +update the policy condition to match. ### Typical user workflows the policy enables 1. **Connect to the login node:** ```bash INSTANCE_ID=$(aws ec2 describe-instances \ - --filters "Name=tag:aws:pcs:compute-node-group-name,Values=login" \ + --filters "Name=tag:Name,Values=PCS-login" \ "Name=instance-state-name,Values=running" \ --query 'Reservations[0].Instances[0].InstanceId' --output text) aws ssm start-session --target $INSTANCE_ID @@ -210,7 +247,7 @@ should reach them through Slurm (`srun`, `sbatch`), not directly via SSM. --with-decryption --query 'Parameter.Value' --output text ``` -4. **SSH-over-SSM** (with `~/.ssh/config` configured per the README): +4. **SSH-over-SSM** (with `~/.ssh/config` configured): ```bash ssh hpc8a-login ``` @@ -219,14 +256,47 @@ should reach them through Slurm (`srun`, `sbatch`), not directly via SSM. - **No write access to cluster resources.** No `CreateStack`, no `pcs:Create*`, no `ec2:RunInstances`, no `iam:*`, no `fsx:*` write actions. -- **Login nodes only.** The instance-tag scope on `ssm:StartSession` blocks the +- **Login nodes only.** The `Name` tag scope on `ssm:StartSession` blocks the user from opening shells on compute nodes (which are tagged - `aws:pcs:compute-node-group-name=`, e.g. `cpu1` or `gpu-p5`). + `Name=PCS-`, e.g. `PCS-cpu1` or `PCS-gpu-p5`). - **Own sessions only.** `ssm:TerminateSession` is scoped to sessions whose ID matches the user's IAM username, so users can't kill each other's sessions. --- +## Manual install without CFN + +If you'd rather skip CloudFormation, attach the JSON files directly: + +```bash +# Cluster admin (split into core + optional Image Builder) +aws iam create-policy \ + --policy-name PCSClusterAdminCore \ + --policy-document file://architectures/aws-pcs/iam/cluster-admin-policy.json +aws iam create-policy \ + --policy-name PCSClusterAdminImageBuilder \ + --policy-document file://architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterAdminCore +# attach the Image Builder one only when needed: +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterAdminImageBuilder + +# Cluster user +aws iam create-policy \ + --policy-name PCSClusterUser \ + --policy-document file://architectures/aws-pcs/iam/cluster-user-policy.json +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterUser +``` + +For a CFN deployment role pattern, attach to a role instead of a user. + +--- + ## Refining via CloudTrail These JSON files are sample-grade. The recommended refinement path: @@ -249,8 +319,8 @@ These JSON files are sample-grade. The recommended refinement path: --output text > generated-policy.json ``` -4. **Diff against `cluster-admin-policy.json`**. The Access Analyzer output - tends to be *narrower* than the sample (it only lists actions actually +4. **Diff against the JSON files in this directory.** The Access Analyzer + output tends to be *narrower* than the sample (it only lists actions actually called) but with `Resource: "*"` even when narrower scoping is possible — the sample's resource ARNs are usually keepers. diff --git a/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json b/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json new file mode 100644 index 000000000..ef97344c7 --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json @@ -0,0 +1,60 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "IAMPassRoleToImageBuilder", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/*ImageBuilder*", + "Condition": { + "StringEquals": { + "iam:PassedToService": "imagebuilder.amazonaws.com" + } + } + }, + { + "Sid": "ImageBuilderLifecycleOptional", + "Effect": "Allow", + "Action": [ + "imagebuilder:CreateImagePipeline", + "imagebuilder:UpdateImagePipeline", + "imagebuilder:DeleteImagePipeline", + "imagebuilder:GetImagePipeline", + "imagebuilder:ListImagePipelines", + "imagebuilder:CreateImageRecipe", + "imagebuilder:DeleteImageRecipe", + "imagebuilder:GetImageRecipe", + "imagebuilder:ListImageRecipes", + "imagebuilder:CreateComponent", + "imagebuilder:DeleteComponent", + "imagebuilder:GetComponent", + "imagebuilder:ListComponents", + "imagebuilder:CreateInfrastructureConfiguration", + "imagebuilder:UpdateInfrastructureConfiguration", + "imagebuilder:DeleteInfrastructureConfiguration", + "imagebuilder:GetInfrastructureConfiguration", + "imagebuilder:ListInfrastructureConfigurations", + "imagebuilder:CreateDistributionConfiguration", + "imagebuilder:UpdateDistributionConfiguration", + "imagebuilder:DeleteDistributionConfiguration", + "imagebuilder:GetDistributionConfiguration", + "imagebuilder:ListDistributionConfigurations", + "imagebuilder:CreateImage", + "imagebuilder:DeleteImage", + "imagebuilder:GetImage", + "imagebuilder:ListImages", + "imagebuilder:StartImagePipelineExecution", + "imagebuilder:CancelImageCreation", + "imagebuilder:CreateLifecyclePolicy", + "imagebuilder:UpdateLifecyclePolicy", + "imagebuilder:DeleteLifecyclePolicy", + "imagebuilder:GetLifecyclePolicy", + "imagebuilder:ListLifecyclePolicies", + "imagebuilder:TagResource", + "imagebuilder:UntagResource", + "imagebuilder:ListTagsForResource" + ], + "Resource": "*" + } + ] +} diff --git a/architectures/aws-pcs/iam/cluster-admin-policy.json b/architectures/aws-pcs/iam/cluster-admin-policy.json index 5a2df66f0..bb317b48f 100644 --- a/architectures/aws-pcs/iam/cluster-admin-policy.json +++ b/architectures/aws-pcs/iam/cluster-admin-policy.json @@ -184,17 +184,6 @@ } } }, - { - "Sid": "IAMPassRoleToImageBuilder", - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:aws:iam::*:role/*ImageBuilder*", - "Condition": { - "StringEquals": { - "iam:PassedToService": "imagebuilder.amazonaws.com" - } - } - }, { "Sid": "ServiceLinkedRolesOneTime", "Effect": "Allow", @@ -234,50 +223,6 @@ "arn:aws:ssm:*:*:parameter/aws/service/*" ] }, - { - "Sid": "ImageBuilderLifecycleOptional", - "Effect": "Allow", - "Action": [ - "imagebuilder:CreateImagePipeline", - "imagebuilder:UpdateImagePipeline", - "imagebuilder:DeleteImagePipeline", - "imagebuilder:GetImagePipeline", - "imagebuilder:ListImagePipelines", - "imagebuilder:CreateImageRecipe", - "imagebuilder:DeleteImageRecipe", - "imagebuilder:GetImageRecipe", - "imagebuilder:ListImageRecipes", - "imagebuilder:CreateComponent", - "imagebuilder:DeleteComponent", - "imagebuilder:GetComponent", - "imagebuilder:ListComponents", - "imagebuilder:CreateInfrastructureConfiguration", - "imagebuilder:UpdateInfrastructureConfiguration", - "imagebuilder:DeleteInfrastructureConfiguration", - "imagebuilder:GetInfrastructureConfiguration", - "imagebuilder:ListInfrastructureConfigurations", - "imagebuilder:CreateDistributionConfiguration", - "imagebuilder:UpdateDistributionConfiguration", - "imagebuilder:DeleteDistributionConfiguration", - "imagebuilder:GetDistributionConfiguration", - "imagebuilder:ListDistributionConfigurations", - "imagebuilder:CreateImage", - "imagebuilder:DeleteImage", - "imagebuilder:GetImage", - "imagebuilder:ListImages", - "imagebuilder:StartImagePipelineExecution", - "imagebuilder:CancelImageCreation", - "imagebuilder:CreateLifecyclePolicy", - "imagebuilder:UpdateLifecyclePolicy", - "imagebuilder:DeleteLifecyclePolicy", - "imagebuilder:GetLifecyclePolicy", - "imagebuilder:ListLifecyclePolicies", - "imagebuilder:TagResource", - "imagebuilder:UntagResource", - "imagebuilder:ListTagsForResource" - ], - "Resource": "*" - }, { "Sid": "KMSAccessForDefaultEncryption", "Effect": "Allow", diff --git a/architectures/aws-pcs/iam/cluster-user-policy.json b/architectures/aws-pcs/iam/cluster-user-policy.json index 12dc6e6e1..98b446394 100644 --- a/architectures/aws-pcs/iam/cluster-user-policy.json +++ b/architectures/aws-pcs/iam/cluster-user-policy.json @@ -53,10 +53,7 @@ ], "Condition": { "StringLike": { - "ssm:resourceTag/aws:pcs:compute-node-group-name": [ - "login*", - "Login*" - ] + "ssm:resourceTag/Name": "PCS-login*" } } }, From 00ddddc675a3a07c185c7b7b18a9d41291b9ce09 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Thu, 11 Jun 2026 00:13:37 +0000 Subject: [PATCH 03/89] =?UTF-8?q?deploy-all:=20tighten=20parameters=20?= =?UTF-8?q?=E2=80=94=20auto-derive=20EFA=20interface=20count,=20default=20?= =?UTF-8?q?VPC=20name=20from=20stack=20name,=20label=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Non-breaking improvements identified in the parameter audit (43 params, no removals possible without breaking PR #1124 deployments): 1. **Auto-derive `OnDemandEfaInterfaceCount` from `OnDemandInstanceType`** (mirrors the pattern `add-cng-p5.yaml`/`add-cng-p6-*.yaml` use for GPU families). New default 0 means "auto"; old 1/2 values still work as explicit pins so existing stacks pass through unchanged. Auto map: hpc8a.96xlarge / hpc7a.{96,48,24,12}xlarge / hpc6id.32xlarge → 2 anything else (hpc6a.48xlarge, c7i.metal, etc.) → 1 Removes the 2-NIC HPC tax: hpc8a/hpc7a were the most common EFA targets but had to override the parameter; now the default just works. 2. **`VPCName` default → empty, auto-resolves to `${StackName}-VPC`**. Multiple deployments in one account now get unique VPC names without the user having to pass a different VPCName each time. 3. **ParameterGroup label cleanup**: "6. FSx for Lustre (/fsx) + FSx for OpenZFS (/home) (Advanced)" → "6. FSx for Lustre (/fsx) and FSx for OpenZFS (/home)" (Capacity / LustreDeploymentType / PerUnitStorageThroughput are commonly tuned and Region-dependent; not advanced) "7. Developer / Advanced (Nested Templates & Monitoring Source)" → "7. Developer (do not change for normal use)" 4. **ParameterLabel trims**: "Availability Zone (required)" → "Availability Zone" (CFN console already shows the asterisk for required params) "VPC name" → "VPC name (empty = ${StackName}-VPC)" "AMI ID (empty = latest PCS-Ready Deep Learning AMI from SSM; pin in production)" → "AMI ID (empty = SSM auto-resolve; pin in production)" "dcgm-exporter image (default DCGM 4.5.2 by digest; covers H100/B200/B300)" → "dcgm-exporter image (default covers H100/B200/B300)" "EFA interface count (1 for hpc6a, 2 for hpc8a/hpc7a/hpc6id)" → "EFA interface count (0 = auto from instance type)" PARAMETERS.md updated to match. Backward compatibility: every change preserves prior behavior for any stack that was deployed off the published default — VPCName defaulting from "ML-Cluster-VPC" to ${StackName}-VPC only affects new launches where the user did not pass VPCName explicitly. Old stacks keep their existing VPC name on UPDATE_STACK. (Larger reductions tracked in subagent plan for the next minor: collapse CngName/QueueName pairs, drop *MinCount, drop LustreVersion/Compression, rename Capacity → LustreCapacity for prefix consistency. All of those are renames or removes, so deferred to avoid breaking PR #1124 users.) --- .../assets/pcs-ml-cluster-deploy-all.yaml | 57 +++++++++++++------ architectures/aws-pcs/docs/PARAMETERS.md | 4 +- 2 files changed, 41 insertions(+), 20 deletions(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index a2d764dd9..11ca2cba5 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -57,7 +57,7 @@ Metadata: - PseriesCngName - PseriesQueueName - Label: - default: "6. FSx for Lustre (/fsx) + FSx for OpenZFS (/home) (Advanced)" + default: "6. FSx for Lustre (/fsx) and FSx for OpenZFS (/home)" Parameters: - Capacity - LustreDeploymentType @@ -69,7 +69,7 @@ Metadata: - HomeThroughput - OpenZFSDeploymentType - Label: - default: "7. Developer / Advanced (Nested Templates & Monitoring Source)" + default: "7. Developer (do not change for normal use)" Parameters: - S3BucketName - S3KeyPrefix @@ -80,9 +80,9 @@ Metadata: # Friendly display names shown in the CloudFormation console 1-click UI, # so users see plain-language labels instead of raw CamelCase parameter keys. PrimarySubnetAZ: - default: "Availability Zone (required)" + default: "Availability Zone" VPCName: - default: "VPC name" + default: "VPC name (empty = ${StackName}-VPC)" CreateS3Endpoint: default: "Create S3 VPC endpoint" LoginNodeInstanceType: @@ -98,7 +98,7 @@ Metadata: MonitoringRepo: default: "Monitoring repo (owner/repo)" DcgmExporterImage: - default: "dcgm-exporter image (default DCGM 4.5.2 by digest; covers H100/B200/B300)" + default: "dcgm-exporter image (default covers H100/B200/B300)" ManagedAccounting: default: "Slurm managed accounting" AccountingPolicyEnforcement: @@ -110,7 +110,7 @@ Metadata: RootVolumeSize: default: "Root volume size (GiB)" AmiId: - default: "AMI ID (empty = latest PCS-Ready Deep Learning AMI from SSM; pin in production)" + default: "AMI ID (empty = SSM auto-resolve; pin in production)" DeployOnDemandCNG: default: "Deploy CPU (on-demand) queue" OnDemandInstanceType: @@ -126,7 +126,7 @@ Metadata: OnDemandEnableEfa: default: "Enable EFA on CPU queue (for HPC instance types)" OnDemandEfaInterfaceCount: - default: "EFA interface count (1 for hpc6a, 2 for hpc8a/hpc7a/hpc6id)" + default: "EFA interface count (0 = auto from instance type)" OnDemandPlacementGroupName: default: "Existing placement group name (empty = auto-create when EFA on)" DeployPseriesCNG: @@ -180,8 +180,11 @@ Parameters: # Network Configuration VPCName: - Description: Name of your VPC - Default: 'ML-Cluster-VPC' + Description: >- + Name of your VPC. Leave empty (default) to auto-derive from the stack + name as ${AWS::StackName}-VPC, so multiple PCS deployments in one + account get unique VPC names automatically. + Default: '' Type: String PrimarySubnetAZ: @@ -455,12 +458,13 @@ Parameters: OnDemandEfaInterfaceCount: Type: Number Description: >- - Number of EFA interfaces to attach when OnDemandEnableEfa=true. Use 2 for - hpc8a/hpc7a/hpc6id; use 1 for hpc6a/c7i.metal. Mismatching this with - the instance type's MaximumEfaInterfaces fails at launch. Ignored when - OnDemandEnableEfa=false. - Default: 1 - AllowedValues: [1, 2] + Number of EFA interfaces to attach when OnDemandEnableEfa=true. + Default 0 = auto-derive from OnDemandInstanceType: 2 for + hpc8a.96xlarge / hpc7a.96xlarge / hpc6id.32xlarge (2-NIC HPC types), + 1 otherwise (hpc6a.48xlarge / c7i.metal etc.). Override with 1 or 2 + to pin explicitly. Ignored when OnDemandEnableEfa=false. + Default: 0 + AllowedValues: [0, 1, 2] OnDemandPlacementGroupName: Type: String @@ -603,6 +607,19 @@ Conditions: - !Condition DeployPseriesCNGEnabled - !Not [!Equals [!Ref PseriesInstanceType, 'p6-b200.48xlarge']] - !Not [!Equals [!Ref PseriesInstanceType, 'p6-b300.48xlarge']] + # VPCName auto-derives from the stack name when left empty. + HasUserVPCName: !Not [!Equals [!Ref VPCName, '']] + # Auto-derive OnDemandEfaInterfaceCount from OnDemandInstanceType when the + # user leaves the parameter at its default 0. Only the 2-NIC HPC types + # (hpc8a / hpc7a / hpc6id) report 2 EFA interfaces; everything else is 1. + OnDemandEfaCountIsAuto: !Equals [!Ref OnDemandEfaInterfaceCount, 0] + OnDemandIs2NicHpc: !Or + - !Equals [!Ref OnDemandInstanceType, 'hpc8a.96xlarge'] + - !Equals [!Ref OnDemandInstanceType, 'hpc7a.96xlarge'] + - !Equals [!Ref OnDemandInstanceType, 'hpc7a.48xlarge'] + - !Equals [!Ref OnDemandInstanceType, 'hpc7a.24xlarge'] + - !Equals [!Ref OnDemandInstanceType, 'hpc7a.12xlarge'] + - !Equals [!Ref OnDemandInstanceType, 'hpc6id.32xlarge'] Resources: ############################################################################# @@ -616,7 +633,7 @@ Resources: - ml-cluster-prerequisites.yaml - !Sub 'https://${S3BucketName}.s3.amazonaws.com/${S3KeyPrefix}ml-cluster-prerequisites.yaml' Parameters: - VPCName: !Ref VPCName + VPCName: !If [HasUserVPCName, !Ref VPCName, !Sub '${AWS::StackName}-VPC'] PrimarySubnetAZ: !Ref PrimarySubnetAZ CreateS3Endpoint: !Ref CreateS3Endpoint Capacity: !Ref Capacity @@ -755,9 +772,13 @@ Resources: PostInstallScriptArgs: !Ref PostInstallScriptArgs # EFA on CPU HPC instances (hpc8a/hpc7a/hpc6a). Forwarded to add-cng.yaml, # which auto-creates a per-CNG cluster placement group when EnableEfa=true and - # PlacementGroupName is empty. + # PlacementGroupName is empty. EfaInterfaceCount auto-derives from the + # instance type when the user left OnDemandEfaInterfaceCount=0 (the default). EnableEfa: !Ref OnDemandEnableEfa - EfaInterfaceCount: !Ref OnDemandEfaInterfaceCount + EfaInterfaceCount: !If + - OnDemandEfaCountIsAuto + - !If [OnDemandIs2NicHpc, 2, 1] + - !Ref OnDemandEfaInterfaceCount PlacementGroupName: !Ref OnDemandPlacementGroupName Tags: - Key: Name diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index f801f2549..5ca3d251e 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -18,7 +18,7 @@ runtime options), see the [README](../README.md#4-configuration). | Parameter | Default | Purpose | |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | -| `VPCName` | `ML-Cluster-VPC` | Name for the created VPC | +| `VPCName` | *(empty → `${StackName}-VPC`)* | Name for the created VPC. Empty (default) auto-derives from the stack name so multiple deployments in one account get unique names | | `CreateS3Endpoint` | `true` | Create an S3 VPC endpoint | ## 2. PCS Cluster Configuration @@ -52,7 +52,7 @@ runtime options), see the [README](../README.md#4-configuration). | `OnDemandCngName` | `cpu1` | CPU node-group name | | `OnDemandQueueName` | `cpu1` | CPU Slurm queue name | | `OnDemandEnableEfa` | `false` | Enable EFA on the CPU CNG (HPC/MPI workloads on hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Switches the CNG's LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` and wires in a cluster placement group. No effect on the GPU CNG. See [README §EFA on CPU HPC instances](../README.md#efa-on-cpu-hpc-instances-ondemandenableefa) | -| `OnDemandEfaInterfaceCount` | `1` | Number of EFA interfaces. Match the instance type's `MaximumEfaInterfaces`: hpc8a/hpc7a/hpc6id = `2`; hpc6a/c7i.metal = `1`. Mismatching fails at launch. Ignored when `OnDemandEnableEfa=false` | +| `OnDemandEfaInterfaceCount` | `0` (auto) | Number of EFA interfaces. **`0` (default) auto-derives from `OnDemandInstanceType`**: 2 for hpc8a/hpc7a/hpc6id, 1 otherwise (hpc6a/c7i.metal). Override with `1` or `2` to pin explicitly. Ignored when `OnDemandEnableEfa=false` | | `OnDemandPlacementGroupName` | *(empty)* | Existing cluster placement group name to launch nodes into. Empty + `OnDemandEnableEfa=true` auto-creates a per-CNG cluster placement group; supplying a name reuses an existing one (e.g. shared across CPU + GPU CNGs for heterogeneous tightly-coupled jobs). Ignored when `OnDemandEnableEfa=false` | ## 5. GPU Compute Node Group — P5/P6 (Optional) From 19ee3fe0acf82a85b3c4d1cc217a95f449a473c4 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 11:40:03 +0000 Subject: [PATCH 04/89] feat: add optional OpenLDAP multi-user directory on login node MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds opt-in multi-user support via OpenLDAP running on the login node. Default is off (DeployDirectory=false) — existing single-user (ubuntu) clusters are completely unchanged. When DeployDirectory=true: - Login node: installs slapd, stores DB on shared /home/ldap-db (OpenZFS), so the directory survives login CNG recreation. Admin password is stored in SSM Parameter Store at /pcs//ldap/admin-password. - Compute nodes: installs SSSD with ldap provider, discovers login node IP via ec2:DescribeInstances tag query, configures NSS/PAM so LDAP users are visible as POSIX users on all nodes. - Home directories auto-created via pam_mkhomedir on shared /home. - Slurm sees LDAP users transparently via NSS (no Slurm config change). New parameters on add-cng.yaml: - DeployDirectory (true/false, default false) - DirectoryDomainSuffix (e.g. dc=cluster,dc=internal) New scripts: - scripts/setup-openldap-server.sh — slapd install + config on login - scripts/setup-ldap-client.sh — SSSD client config on compute nodes - scripts/ldap-add-user.sh — helper to add POSIX users to the directory Not yet wired into deploy-all.yaml or GPU templates (next commit). --- architectures/aws-pcs/assets/add-cng.yaml | 76 ++++++++++++ .../aws-pcs/scripts/ldap-add-user.sh | 70 +++++++++++ .../aws-pcs/scripts/setup-ldap-client.sh | 68 +++++++++++ .../aws-pcs/scripts/setup-openldap-server.sh | 111 ++++++++++++++++++ 4 files changed, 325 insertions(+) create mode 100755 architectures/aws-pcs/scripts/ldap-add-user.sh create mode 100755 architectures/aws-pcs/scripts/setup-ldap-client.sh create mode 100755 architectures/aws-pcs/scripts/setup-openldap-server.sh diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 6213ee09b..ace5a36d5 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -56,6 +56,11 @@ Metadata: Parameters: - PostInstallScriptUrl - PostInstallScriptArgs + - Label: + default: Multi-user directory (optional) + Parameters: + - DeployDirectory + - DirectoryDomainSuffix Parameters: ClusterId: @@ -210,6 +215,26 @@ Parameters: - '25.05' - '25.11' + DeployDirectory: + Type: String + Description: >- + Deploy an OpenLDAP directory on the login node for multi-user management. + When true and MonitoringRole=login, slapd is installed on the login node + with its DB on shared /home (OpenZFS), so it survives CNG recreation. + Compute nodes configure SSSD to resolve users from the login node's LDAP. + Default false = single ubuntu user (unchanged). + Default: 'false' + AllowedValues: + - 'true' + - 'false' + + DirectoryDomainSuffix: + Type: String + Description: >- + LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when + DeployDirectory=true. + Default: 'dc=cluster,dc=internal' + EnableEfa: Type: String Description: >- @@ -263,6 +288,15 @@ Conditions: Has2ndEfaInterface: !And - !Condition EfaEnabled - !Equals [!Ref EfaInterfaceCount, 2] + DirectoryEnabled: !Equals [!Ref DeployDirectory, 'true'] + IsLoginNode: !Equals [!Ref MonitoringRole, 'login'] + # LDAP server runs on login node; LDAP client (SSSD) runs on compute nodes + SetupLdapServer: !And + - !Condition DirectoryEnabled + - !Condition IsLoginNode + SetupLdapClient: !And + - !Condition DirectoryEnabled + - !Not [!Condition IsLoginNode] Resources: @@ -472,6 +506,48 @@ Resources: fi --==MYBOUNDARY== + Content-Type: text/x-shellscript; charset="us-ascii" + MIME-Version: 1.0 + + #!/bin/bash + # Multi-user directory setup (optional, DeployDirectory=true) + if [ "${DeployDirectory}" = "true" ]; then + export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" + export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') + export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + + if [ "${MonitoringRole}" = "login" ]; then + # Login node: install OpenLDAP server + echo "[multi-user] Setting up OpenLDAP server on login node..." + curl -fsSL "https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/scripts/setup-openldap-server.sh" -o /tmp/setup-openldap-server.sh + bash /tmp/setup-openldap-server.sh 2>&1 | tee /var/log/ldap-server-install.log + # Store admin password in SSM for retrieval + export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:/usr/local/bin:${!PATH} + CLUSTER_ID="${ClusterId}" + aws ssm put-parameter --name "/pcs/${!CLUSTER_ID}/ldap/admin-password" \ + --value "${!LDAP_ADMIN_PASSWORD}" --type SecureString --overwrite \ + --region $(curl -s http://169.254.169.254/latest/meta-data/placement/region) || true + else + # Compute node: configure SSSD LDAP client pointing at login node + echo "[multi-user] Configuring SSSD LDAP client..." + # Discover login node IP via tag query + REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + LOGIN_IP=$(aws ec2 describe-instances --region ${!REGION} \ + --filters "Name=tag:pcs-cluster-id,Values=${ClusterId}" \ + "Name=tag:monitoring-role,Values=login" \ + "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text 2>/dev/null || echo "") + if [ -n "${!LOGIN_IP}" ] && [ "${!LOGIN_IP}" != "None" ]; then + export LDAP_SERVER_URI="ldap://${!LOGIN_IP}" + curl -fsSL "https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/scripts/setup-ldap-client.sh" -o /tmp/setup-ldap-client.sh + bash /tmp/setup-ldap-client.sh 2>&1 | tee /var/log/ldap-client-install.log + else + echo "[multi-user] WARNING: could not discover login node IP. LDAP client not configured." + fi + fi + fi + + --==MYBOUNDARY== Outputs: NodeGroupId: diff --git a/architectures/aws-pcs/scripts/ldap-add-user.sh b/architectures/aws-pcs/scripts/ldap-add-user.sh new file mode 100755 index 000000000..6843a5817 --- /dev/null +++ b/architectures/aws-pcs/scripts/ldap-add-user.sh @@ -0,0 +1,70 @@ +#!/bin/bash +# ldap-add-user.sh — Helper to add a POSIX user to the OpenLDAP directory. +# Run on the login node as root (or with LDAP admin credentials). +# +# Usage: ./ldap-add-user.sh [uid] [gid] [ssh-pub-key] +# +# Example: +# ./ldap-add-user.sh alice 10001 3000 +# ./ldap-add-user.sh bob 10002 3000 "ssh-rsa AAAA..." + +set -euo pipefail + +USERNAME="${1:?Usage: $0 [uid] [gid] [ssh-pub-key]}" +USER_UID="${2:-$((10000 + RANDOM % 50000))}" +USER_GID="${3:-3000}" +SSH_PUBKEY="${4:-}" + +# Auto-detect LDAP config from sssd.conf or environment +LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-$(grep ldap_search_base /etc/sssd/sssd.conf 2>/dev/null | awk -F= '{print $2}' | tr -d ' ' || echo 'dc=cluster,dc=internal')}" +LDAP_ADMIN_DN="cn=admin,${LDAP_DOMAIN_SUFFIX}" + +echo "Adding user: ${USERNAME} (uid=${USER_UID}, gid=${USER_GID})" +echo "LDAP base: ${LDAP_DOMAIN_SUFFIX}" + +# Prompt for admin password if not set +if [ -z "${LDAP_ADMIN_PASSWORD:-}" ]; then + read -sp "LDAP admin password: " LDAP_ADMIN_PASSWORD + echo +fi + +# Create user entry +LDIF=$(cat <&1 + +# Set a random initial password (user should change via ldappasswd) +INITIAL_PW=$(openssl rand -base64 12) +ldappasswd -x -H ldap://localhost -D "${LDAP_ADMIN_DN}" -w "${LDAP_ADMIN_PASSWORD}" \ + -s "${INITIAL_PW}" "uid=${USERNAME},ou=People,${LDAP_DOMAIN_SUFFIX}" + +echo "" +echo "User '${USERNAME}' created successfully." +echo " UID: ${USER_UID}" +echo " GID: ${USER_GID}" +echo " Home: /home/${USERNAME} (auto-created on first login via pam_mkhomedir)" +echo " Initial password: ${INITIAL_PW}" +echo "" +echo "To add to Slurm accounting:" +echo " sacctmgr add user ${USERNAME} account=default" diff --git a/architectures/aws-pcs/scripts/setup-ldap-client.sh b/architectures/aws-pcs/scripts/setup-ldap-client.sh new file mode 100755 index 000000000..3ab536235 --- /dev/null +++ b/architectures/aws-pcs/scripts/setup-ldap-client.sh @@ -0,0 +1,68 @@ +#!/bin/bash +# setup-ldap-client.sh — Configure SSSD LDAP client on compute nodes. +# Called from CNG UserData when DeployDirectory=true and MonitoringRole=compute (or none). +# +# Environment variables (passed from UserData): +# LDAP_SERVER_URI — e.g. "ldap://10.1.x.x" (login node private IP) +# LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" +# LDAP_DOMAIN — e.g. "cluster.internal" + +set -euo pipefail + +LDAP_SERVER_URI="${LDAP_SERVER_URI:?LDAP_SERVER_URI must be set}" +LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-dc=cluster,dc=internal}" +LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" + +export DEBIAN_FRONTEND=noninteractive + +echo "[ldap-client] Installing SSSD + LDAP client packages..." +apt-get install -y sssd libpam-sss libnss-sss ldap-utils + +echo "[ldap-client] Configuring SSSD..." +cat > /etc/sssd/sssd.conf <> /etc/pam.d/common-session +fi + +# Configure NSS to use sss +if ! grep -q sss /etc/nsswitch.conf; then + sed -i 's/^passwd:.*/passwd: files systemd sss/' /etc/nsswitch.conf + sed -i 's/^group:.*/group: files systemd sss/' /etc/nsswitch.conf + sed -i 's/^shadow:.*/shadow: files sss/' /etc/nsswitch.conf +fi + +systemctl enable sssd +systemctl restart sssd + +echo "[ldap-client] SSSD LDAP client configured." +echo "[ldap-client] Server: ${LDAP_SERVER_URI}" +echo "[ldap-client] Search base: ${LDAP_DOMAIN_SUFFIX}" +echo "[ldap-client] Verify: getent passwd (should show LDAP users after they are created)" diff --git a/architectures/aws-pcs/scripts/setup-openldap-server.sh b/architectures/aws-pcs/scripts/setup-openldap-server.sh new file mode 100755 index 000000000..1da2b1cf2 --- /dev/null +++ b/architectures/aws-pcs/scripts/setup-openldap-server.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# setup-openldap-server.sh — Install and configure OpenLDAP (slapd) on the login node. +# Called from CNG UserData when DeployDirectory=true and MonitoringRole=login. +# +# LDAP DB is stored on /home/ldap-db (shared OpenZFS) so it survives login +# node CNG recreation. The slapd service on a fresh node detects existing DB +# and reuses it (idempotent). +# +# Environment variables (passed from UserData): +# LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" (from DirectoryDomainSuffix) +# LDAP_DOMAIN — e.g. "cluster.internal" (derived) +# LDAP_ADMIN_PASSWORD — from Secrets Manager or auto-generated +# LDAP_BASE_GROUPS — comma-separated default groups (default: "clusterusers,clusteradmins") + +set -euo pipefail + +LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-dc=cluster,dc=internal}" +LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" +LDAP_ADMIN_PASSWORD="${LDAP_ADMIN_PASSWORD:-$(openssl rand -base64 16)}" +LDAP_BASE_GROUPS="${LDAP_BASE_GROUPS:-clusterusers,clusteradmins}" +LDAP_DB_DIR="/home/ldap-db" + +export DEBIAN_FRONTEND=noninteractive + +echo "[ldap-server] Installing slapd + ldap-utils..." +debconf-set-selections </dev/null)" ]; then + cp -a /var/lib/ldap/* "${LDAP_DB_DIR}/" + fi + chown -R openldap:openldap "${LDAP_DB_DIR}" +fi + +# Point slapd at shared storage via apparmor + config override +# Update slapd DB directory in the config +if [ -d /etc/ldap/slapd.d ]; then + # cn=config style — find the mdb backend config and update olcDbDirectory + MDB_LDIF=$(find /etc/ldap/slapd.d -name "olcDatabase*mdb*" -o -name "olcDatabase*hdb*" | head -1) + if [ -n "$MDB_LDIF" ] && grep -q "olcDbDirectory" "$MDB_LDIF"; then + sed -i "s|olcDbDirectory:.*|olcDbDirectory: ${LDAP_DB_DIR}|" "$MDB_LDIF" + fi +fi + +# AppArmor: allow slapd to access /home/ldap-db +if [ -f /etc/apparmor.d/usr.sbin.slapd ]; then + if ! grep -q "${LDAP_DB_DIR}" /etc/apparmor.d/usr.sbin.slapd; then + sed -i "/\/var\/lib\/ldap\/ r,/a\\ ${LDAP_DB_DIR}/ r," /etc/apparmor.d/usr.sbin.slapd + sed -i "/\/var\/lib\/ldap\/\*\* rwk,/a\\ ${LDAP_DB_DIR}/** rwk," /etc/apparmor.d/usr.sbin.slapd + apparmor_parser -r /etc/apparmor.d/usr.sbin.slapd 2>/dev/null || true + fi +fi + +# Ensure correct ownership +chown -R openldap:openldap "${LDAP_DB_DIR}" + +systemctl start slapd +systemctl enable slapd + +# Wait for slapd to be ready +for i in $(seq 1 10); do + ldapsearch -x -H ldap://localhost -b "" -s base namingContexts >/dev/null 2>&1 && break + sleep 1 +done + +echo "[ldap-server] Creating base groups..." +for GROUP in $(echo "${LDAP_BASE_GROUPS}" | tr ',' ' '); do + GID=$((3000 + RANDOM % 1000)) + ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: cn=${GROUP},${LDAP_DOMAIN_SUFFIX} +objectClass: posixGroup +cn: ${GROUP} +gidNumber: ${GID} +EOF +done + +# Create an OU for users if not exists +ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: ou=People,${LDAP_DOMAIN_SUFFIX} +objectClass: organizationalUnit +ou: People +EOF + +ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: ou=Groups,${LDAP_DOMAIN_SUFFIX} +objectClass: organizationalUnit +ou: Groups +EOF + +echo "[ldap-server] OpenLDAP server ready." +echo "[ldap-server] Admin DN: cn=admin,${LDAP_DOMAIN_SUFFIX}" +echo "[ldap-server] DB stored on: ${LDAP_DB_DIR} (shared OpenZFS, survives CNG delete)" +echo "[ldap-server] Add users with:" +echo " ldapadd -x -H ldap://localhost -D 'cn=admin,${LDAP_DOMAIN_SUFFIX}' -w ''" From eecbcbb5f034a6f82a7ec82da37531a48e7bc834 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 12:25:22 +0000 Subject: [PATCH 05/89] =?UTF-8?q?docs:=20add=20USER-MANAGEMENT.md=20?= =?UTF-8?q?=E2=80=94=20OpenLDAP=20multi-user=20operations=20guide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers enabling multi-user, managing users (add/remove/list/password/groups), running jobs as specific users, SSH access for LDAP users, UID/GID conventions, SSSD caching behavior, troubleshooting, data persistence, and upgrade path to Simple AD / Managed AD. --- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 356 ++++++++++++++++++ 1 file changed, 356 insertions(+) create mode 100644 architectures/aws-pcs/docs/USER-MANAGEMENT.md diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md new file mode 100644 index 000000000..527cd7392 --- /dev/null +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -0,0 +1,356 @@ +# User Management Guide + +This guide covers multi-user management for the PCS reference architecture. +By default, the cluster runs as a single `ubuntu` user (the PCS-Ready DLAMI +default). When `DeployDirectory=true` is set, an OpenLDAP directory is deployed +on the login node, and all compute nodes are configured as LDAP clients via +SSSD — giving you centralized POSIX user management across the cluster. + +--- + +## Overview + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Login Node │ +│ ┌──────────────┐ ┌──────────────────────────────────┐ │ +│ │ slapd │ │ SSSD (ldap provider, localhost) │ │ +│ │ (OpenLDAP) │ └──────────────────────────────────┘ │ +│ │ DB: /home/ │ │ +│ │ ldap-db/ │◄─── LDAP queries from compute nodes │ +│ └──────────────┘ │ +└──────────────────────────────────────────────────────────────┘ + ▲ ▲ + │ ldap://login-ip:389 │ ldap://login-ip:389 + │ │ +┌────────┴───────┐ ┌────────┴───────┐ +│ Compute Node │ │ Compute Node │ +│ SSSD (ldap) │ │ SSSD (ldap) │ +│ NSS + PAM │ │ NSS + PAM │ +└────────────────┘ └────────────────┘ +``` + +**Key properties:** +- LDAP DB is stored on `/home/ldap-db` (shared OpenZFS) — survives login node + CNG recreation or instance replacement +- All nodes use SSSD with `cache_credentials=true` — users can still log in + during brief LDAP server unavailability (e.g. login node reboot) +- UID/GID are explicit POSIX attributes in LDAP (not algorithmic mapping) — + fully deterministic across all nodes and NFS mounts +- Home directories auto-created on first login via `pam_mkhomedir` at `/home/` + (shared OpenZFS, visible on all nodes immediately) + +--- + +## Enabling multi-user + +### With deploy-all + +```bash +aws cloudformation create-stack \ + --stack-name my-cluster \ + --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ + ParameterKey=DeployDirectory,ParameterValue=true \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` + +### With modular deployment (add-cng.yaml) + +Pass `DeployDirectory=true` and `DirectoryDomainSuffix=dc=cluster,dc=internal` +to **both** the login CNG and compute CNG stacks. The login CNG installs the +LDAP server; the compute CNG configures the SSSD client pointing at the login +node. + +### Parameters + +| Parameter | Default | Description | +|---|---|---| +| `DeployDirectory` | `false` | Enable OpenLDAP directory on login node | +| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP base DN. Change if you want a different domain (e.g. `dc=myorg,dc=com`) | + +--- + +## Managing users + +All user management commands run on the **login node** as root (or via `sudo`). + +### Retrieve the LDAP admin password + +The admin password is auto-generated at first boot and stored in SSM +Parameter Store: + +```bash +CLUSTER_ID= +aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text +``` + +### Add a user + +Use the helper script (pre-installed on the login node at +`/usr/local/bin/ldap-add-user.sh`, or fetch from the repo): + +```bash +# Usage: ldap-add-user.sh [uid] [gid] [ssh-pub-key] +sudo LDAP_ADMIN_PASSWORD=$(aws ssm get-parameter \ + --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text) \ + /usr/local/bin/ldap-add-user.sh alice 10001 3000 +``` + +Or manually with `ldapadd`: + +```bash +ADMIN_PW="" +BASE_DN="dc=cluster,dc=internal" + +ldapadd -x -H ldap://localhost -D "cn=admin,${BASE_DN}" -w "${ADMIN_PW}" </.ssh/authorized_keys` + (auto-created by pam_mkhomedir on shared `/home`). + +2. Users connect via SSH-over-SSM: + ```bash + # ~/.ssh/config + Host pcs-login + HostName + User alice + ProxyCommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' + ``` + +--- + +## UID/GID conventions + +| Range | Purpose | +|---|---| +| 0–999 | System users (do not use) | +| 1000 | `ubuntu` (DLAMI default user) | +| 3000 | `clusterusers` group (default group for new users) | +| 3001+ | Custom groups | +| 10000–59999 | LDAP user UIDs (SSSD `min_id`/`max_id` range) | + +Use deterministic UIDs (specify in `ldapadd`) rather than letting LDAP +auto-assign. This ensures consistency across all nodes and NFS mounts. + +--- + +## SSSD caching behavior + +| Setting | Value | Meaning | +|---|---|---| +| `cache_credentials` | `true` | Users can authenticate even if LDAP server is temporarily down | +| `enumerate` | `true` | `getent passwd` shows all LDAP users (not just those who have logged in) | +| `entry_cache_timeout` | 5400 (default) | Cached user/group entries are valid for 90 minutes | + +To force immediate cache refresh on a node (e.g. after adding a user): +```bash +sudo sss_cache -E +``` + +--- + +## Troubleshooting + +### User not visible on compute node + +```bash +# Check SSSD is running +systemctl status sssd + +# Check LDAP connectivity from compute node +ldapsearch -x -H ldap:// -b "dc=cluster,dc=internal" uid=alice + +# Force cache refresh +sudo sss_cache -E +sudo systemctl restart sssd + +# Check SSSD logs +journalctl -u sssd -n 50 +``` + +### Login node LDAP server not starting + +```bash +# Check slapd status +systemctl status slapd +journalctl -u slapd -n 30 + +# Check AppArmor (common issue with custom DB path) +aa-status | grep slapd +``` + +### Home directory not created on first login + +```bash +# Verify pam_mkhomedir is configured +grep pam_mkhomedir /etc/pam.d/common-session + +# Verify /home is mounted (shared OpenZFS) +mount | grep /home +``` + +--- + +## Data persistence + +| Data | Location | Survives CNG delete? | Survives stack delete? | +|---|---|---|---| +| LDAP database | `/home/ldap-db/` | ✅ (shared OpenZFS) | ❌ (FSx deleted with prereqs) | +| User home dirs | `/home//` | ✅ (shared OpenZFS) | ❌ (FSx deleted with prereqs) | +| LDAP admin password | SSM Parameter Store | ✅ (account-level) | ✅ (not part of stack) | + +**Backup recommendation**: periodically export the LDAP database: +```bash +slapcat -l /home/ldap-backup/ldap-$(date +%Y%m%d).ldif +``` + +Restore on a fresh login node: +```bash +systemctl stop slapd +slapadd -l /home/ldap-backup/ldap-YYYYMMDD.ldif +chown -R openldap:openldap /home/ldap-db +systemctl start slapd +``` + +--- + +## Upgrading to Simple AD or Managed AD + +If your team outgrows the self-hosted OpenLDAP (> 50 users, need HA, need +Kerberos), migrate to AWS Simple AD or Managed Microsoft AD: + +1. Export users: `slapcat > users.ldif` +2. Deploy Simple AD (requires 2 AZs — see [ROADMAP](./ROADMAP.md)) +3. Import users into AD (convert LDIF to AD format) +4. Update compute node SSSD config: change `id_provider` from `ldap` to `ad`, + switch to `realm join` +5. Decommission slapd on login node + +The Slurm accounting entries (`sacctmgr`) don't need to change — Slurm +resolves users via NSS regardless of the identity backend. From e1ff8c2a824e1a9a7f7281e1b1fc79d307f6cfee Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 12:45:41 +0000 Subject: [PATCH 06/89] fix: point LDAP script URLs at fork branch (pre-merge) --- architectures/aws-pcs/assets/add-cng.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index ace5a36d5..f17c2fad2 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -519,7 +519,7 @@ Resources: if [ "${MonitoringRole}" = "login" ]; then # Login node: install OpenLDAP server echo "[multi-user] Setting up OpenLDAP server on login node..." - curl -fsSL "https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/scripts/setup-openldap-server.sh" -o /tmp/setup-openldap-server.sh + curl -fsSL "https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-openldap-server.sh" -o /tmp/setup-openldap-server.sh bash /tmp/setup-openldap-server.sh 2>&1 | tee /var/log/ldap-server-install.log # Store admin password in SSM for retrieval export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:/usr/local/bin:${!PATH} @@ -539,7 +539,7 @@ Resources: --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text 2>/dev/null || echo "") if [ -n "${!LOGIN_IP}" ] && [ "${!LOGIN_IP}" != "None" ]; then export LDAP_SERVER_URI="ldap://${!LOGIN_IP}" - curl -fsSL "https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/scripts/setup-ldap-client.sh" -o /tmp/setup-ldap-client.sh + curl -fsSL "https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-ldap-client.sh" -o /tmp/setup-ldap-client.sh bash /tmp/setup-ldap-client.sh 2>&1 | tee /var/log/ldap-client-install.log else echo "[multi-user] WARNING: could not discover login node IP. LDAP client not configured." From 129900da00afa407cee3a5d05e1e190d047be284 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 13:08:01 +0000 Subject: [PATCH 07/89] fix: apt-get update before slapd install + add multi-user test suite - setup-openldap-server.sh: add apt-get update before install (PCS-Ready DLAMI has limited apt sources cached; without update, slapd package is not found) - setup-ldap-client.sh: same fix - tests/multi-user-test.md: comprehensive test suite (5 parts: server health, user lifecycle, Slurm integration, multi-node consistency, resilience). Split from README.md to keep the main test guide concise. - tests/README.md: added Test 11 link to multi-user-test.md --- .../aws-pcs/scripts/setup-ldap-client.sh | 3 + .../aws-pcs/scripts/setup-openldap-server.sh | 3 + architectures/aws-pcs/tests/README.md | 11 + .../aws-pcs/tests/multi-user-test.md | 284 ++++++++++++++++++ 4 files changed, 301 insertions(+) create mode 100644 architectures/aws-pcs/tests/multi-user-test.md diff --git a/architectures/aws-pcs/scripts/setup-ldap-client.sh b/architectures/aws-pcs/scripts/setup-ldap-client.sh index 3ab536235..442622175 100755 --- a/architectures/aws-pcs/scripts/setup-ldap-client.sh +++ b/architectures/aws-pcs/scripts/setup-ldap-client.sh @@ -15,6 +15,9 @@ LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" export DEBIAN_FRONTEND=noninteractive +echo "[ldap-client] Running apt-get update..." +apt-get update -qq + echo "[ldap-client] Installing SSSD + LDAP client packages..." apt-get install -y sssd libpam-sss libnss-sss ldap-utils diff --git a/architectures/aws-pcs/scripts/setup-openldap-server.sh b/architectures/aws-pcs/scripts/setup-openldap-server.sh index 1da2b1cf2..bb5143df6 100755 --- a/architectures/aws-pcs/scripts/setup-openldap-server.sh +++ b/architectures/aws-pcs/scripts/setup-openldap-server.sh @@ -22,6 +22,9 @@ LDAP_DB_DIR="/home/ldap-db" export DEBIAN_FRONTEND=noninteractive +echo "[ldap-server] Running apt-get update..." +apt-get update -qq + echo "[ldap-server] Installing slapd + ldap-utils..." debconf-set-selections < +aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text +``` + +Expected: returns a non-empty string (the auto-generated password). + +--- + +## Part B — User lifecycle + +### B1. Create a test user + +```bash +export LDAP_ADMIN_PASSWORD=$(aws ssm get-parameter \ + --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text) +export LDAP_DOMAIN_SUFFIX="dc=cluster,dc=internal" + +# Using the helper script +sudo -E bash /usr/local/bin/ldap-add-user.sh testuser1 10001 3000 +``` + +Or manually: +```bash +ldapadd -x -H ldap://localhost -D "cn=admin,dc=cluster,dc=internal" -w "$LDAP_ADMIN_PASSWORD" < /home/testuser1/multinode-test.txt' +srun -N 1 -n 1 -p cpu1 --exclude=$(srun -N 1 -n 1 -p cpu1 hostname) bash -c 'cat /home/testuser1/multinode-test.txt' +``` + +Expected: the second node reads what the first wrote (shared OpenZFS `/home`). + +### D3. New compute node picks up existing users + +If a new compute node scales up after users were created: +```bash +# Force a new node to spin up +srun -N 2 -n 2 -p cpu1 bash -c 'getent passwd testuser1' +``` + +Expected: the freshly-started node resolves `testuser1` via SSSD (it queries +the login node's LDAP at boot via the client setup script). + +--- + +## Part E — LDAP server resilience + +### E1. SSSD cache survives brief LDAP outage + +```bash +# Stop slapd briefly on login node +sudo systemctl stop slapd + +# Compute node should still resolve from cache +srun -N 1 -n 1 -p cpu1 bash -c 'getent passwd testuser1' + +# Restart +sudo systemctl start slapd +``` + +Expected: cached user resolves even with slapd down (SSSD `cache_credentials=true`). + +### E2. LDAP DB survives login node replacement + +After login node terminate + PCS replacement: +```bash +# On new login node +ls /home/ldap-db/data.mdb +ldapsearch -x -H ldap://localhost -b "ou=People,dc=cluster,dc=internal" uid +``` + +Expected: previously-created users still in the directory (DB on shared OpenZFS +survived the instance replacement). + +--- + +## Verdict checklist + +| Check | Expected | +|---|---| +| slapd running on login node | ✅ | +| LDAP DB on /home/ldap-db (shared OpenZFS) | ✅ | +| Admin password in SSM Parameter Store | ✅ | +| User created via ldapadd/helper script | ✅ | +| User visible on login node (`getent passwd`) | ✅ | +| User visible on compute node (`srun getent passwd`) | ✅ | +| Home dir auto-created on first login | ✅ | +| User deleted, no longer resolvable | ✅ | +| Slurm job runs as LDAP user | ✅ | +| Multiple nodes resolve same UID | ✅ | +| Home dir accessible from all nodes | ✅ | +| New compute node resolves existing users | ✅ | +| SSSD cache works during brief LDAP outage | ✅ | +| LDAP DB survives login node replacement | ✅ | From 17e12ba480e7d5571fb3f64615e965b5fb882221 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 13:34:33 +0000 Subject: [PATCH 08/89] fix: SSSD min_id=1001 (GID 3000 was out of range) + persist admin pw + access methods doc - setup-ldap-client.sh: change min_id from 10000 to 1001. GID 3000 (clusterusers) was being filtered as "primary gid out of range" by SSSD, preventing user resolution. min_id=1001 excludes only system users (0-1000) while allowing all LDAP-provisioned GIDs. - setup-openldap-server.sh: persist the admin password to /home/ldap-db/.admin-password (shared OpenZFS). Previously the randomly generated password was lost after the setup script exited, making it impossible to administer the directory after login node replacement. The file is chmod 600 and on shared storage. - USER-MANAGEMENT.md: added "Access methods" section comparing SSM vs SSH-over-SSM vs Direct SSH (pro/con/best-for table), SSH key management approaches, and Slurm accounting with PCS (what's managed by PCS vs what admins still need to do manually with sacctmgr). Verified: getent passwd testuser1 now returns testuser1:*:10001:3000:Test User 1:/home/testuser1:/bin/bash on the login node after SSSD restart. --- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 49 +++++++++++++++++++ .../aws-pcs/scripts/setup-ldap-client.sh | 2 +- .../aws-pcs/scripts/setup-openldap-server.sh | 7 ++- 3 files changed, 56 insertions(+), 2 deletions(-) diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index 527cd7392..cf54540a1 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -3,6 +3,55 @@ This guide covers multi-user management for the PCS reference architecture. By default, the cluster runs as a single `ubuntu` user (the PCS-Ready DLAMI default). When `DeployDirectory=true` is set, an OpenLDAP directory is deployed + +--- + +## Access methods (multi-user) + +| Method | Pro | Con | Best for | +|---|---|---|---| +| **SSM Session Manager** | No port opening, IAM-based auth, full audit trail in CloudTrail | Each user needs IAM credentials + SSM plugin installed; lands as `ssm-user`, requires `su - ` to switch | Admin-only access, security-sensitive environments | +| **SSH over SSM** (`ProxyCommand`) | No port opening + standard SSH workflow (VS Code Remote, scp, rsync) | Each user needs IAM credentials + SSM plugin + SSH config entry; slightly more setup | Teams with IAM already in place, remote IDE use | +| **Direct SSH** (port 22 on login node) | Simplest for users — standard `ssh user@host`, familiar HPC workflow, VS Code/JupyterLab tunnels work natively | Requires SG port 22 open (CIDR-restricted), SSH key distribution needed | Traditional HPC teams, workshop/training environments | + +**Recommendation for multi-user clusters**: use **Direct SSH** with CIDR-restricted +port 22 on the login node. This matches the standard HPC user experience and +avoids per-user IAM setup overhead. SSH keys are stored in each user's +`/home//.ssh/authorized_keys` (shared OpenZFS, visible on all nodes). + +### SSH key management + +| Approach | Who adds keys | How | +|---|---|---| +| Admin provisions at user-creation time | Cluster admin | Pass `ssh-pub-key` to `ldap-add-user.sh` → written to `/home//.ssh/authorized_keys` | +| User self-service | Each user | User logs in (first time via admin-provided temp password or SSM) and adds their own key to `~/.ssh/authorized_keys` | +| LDAP-stored keys (`AuthorizedKeysCommand`) | Admin | `sshd_config` queries LDAP for `sshPublicKey` attribute via helper script — centralizes key management but adds sshd config complexity | + +For most clusters, the simplest path is: admin creates the user + writes their +SSH public key to `/home//.ssh/authorized_keys` during creation. + +--- + +## Slurm accounting with PCS + +PCS manages the Slurm accounting database internally — you do NOT need to +configure `slurmdbd` or manage a MySQL/MariaDB database. When +`ManagedAccounting=enabled` is set on the cluster, Slurm's accounting commands +(`sacctmgr`, `sacct`, `sreport`) work against PCS's managed backing store. + +**What you still need to do manually**: +- `sacctmgr add account ` — create Slurm accounts (one per team/project) +- `sacctmgr add user account=` — map LDAP users to Slurm accounts +- (Optional) Set fairshare / QOS via `sacctmgr modify user` + +If `AccountingPolicyEnforcement=associations,limits,safe` is set, a user +without a `sacctmgr` entry will have their jobs **rejected**. With the default +`AccountingPolicyEnforcement=none`, unregistered users can still submit jobs +(they just won't have accounting records). + +--- + +When `DeployDirectory=true` is set, an OpenLDAP directory is deployed on the login node, and all compute nodes are configured as LDAP clients via SSSD — giving you centralized POSIX user management across the cluster. diff --git a/architectures/aws-pcs/scripts/setup-ldap-client.sh b/architectures/aws-pcs/scripts/setup-ldap-client.sh index 442622175..d5b661054 100755 --- a/architectures/aws-pcs/scripts/setup-ldap-client.sh +++ b/architectures/aws-pcs/scripts/setup-ldap-client.sh @@ -44,7 +44,7 @@ default_shell = /bin/bash fallback_homedir = /home/%u # UID/GID range for LDAP users (avoid collision with system users) -min_id = 10000 +min_id = 1001 max_id = 60000 EOF diff --git a/architectures/aws-pcs/scripts/setup-openldap-server.sh b/architectures/aws-pcs/scripts/setup-openldap-server.sh index bb5143df6..89524e70c 100755 --- a/architectures/aws-pcs/scripts/setup-openldap-server.sh +++ b/architectures/aws-pcs/scripts/setup-openldap-server.sh @@ -107,8 +107,13 @@ objectClass: organizationalUnit ou: Groups EOF +# Persist admin password on shared storage (survives CNG delete) +echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" +chmod 600 "${LDAP_DB_DIR}/.admin-password" + echo "[ldap-server] OpenLDAP server ready." echo "[ldap-server] Admin DN: cn=admin,${LDAP_DOMAIN_SUFFIX}" +echo "[ldap-server] Admin password stored at: ${LDAP_DB_DIR}/.admin-password" echo "[ldap-server] DB stored on: ${LDAP_DB_DIR} (shared OpenZFS, survives CNG delete)" echo "[ldap-server] Add users with:" -echo " ldapadd -x -H ldap://localhost -D 'cn=admin,${LDAP_DOMAIN_SUFFIX}' -w ''" +echo " ldapadd -x -H ldap://localhost -D 'cn=admin,${LDAP_DOMAIN_SUFFIX}' -w '\$(cat ${LDAP_DB_DIR}/.admin-password)'" From 62d4e1ded47d86df1bb9dc2c7946a905ae6bd0ac Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 13:38:57 +0000 Subject: [PATCH 09/89] feat: wire DeployDirectory into deploy-all + template structure diagram - pcs-ml-cluster-deploy-all.yaml: add DeployDirectory + DirectoryDomainSuffix params (group 7), forward to both LoginNodeGroupStack and OnDemandCNGStack. Without this wiring, compute nodes never received DeployDirectory=true and SSSD client setup was skipped. - docs/USER-MANAGEMENT.md: add ASCII art template structure diagram showing how DeployDirectory flows from deploy-all through the nested stacks into UserData on login (slapd) and compute (SSSD client) nodes. --- .../assets/pcs-ml-cluster-deploy-all.yaml | 31 +++++++++++++++- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 35 +++++++++++++++++++ 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index a2d764dd9..055b27c7b 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -69,7 +69,12 @@ Metadata: - HomeThroughput - OpenZFSDeploymentType - Label: - default: "7. Developer / Advanced (Nested Templates & Monitoring Source)" + default: "7. Multi-User Directory (Optional)" + Parameters: + - DeployDirectory + - DirectoryDomainSuffix + - Label: + default: "8. Developer / Advanced (Nested Templates & Monitoring Source)" Parameters: - S3BucketName - S3KeyPrefix @@ -262,6 +267,26 @@ Parameters: - 'true' - 'false' + # Multi-user directory + DeployDirectory: + Type: String + Description: >- + Deploy an OpenLDAP directory on the login node for multi-user management. + When true, slapd runs on the login node (DB on shared /home/ldap-db, + survives CNG recreation) and compute nodes configure SSSD to resolve LDAP + users. Default false = single ubuntu user (unchanged). + Default: 'false' + AllowedValues: + - 'true' + - 'false' + + DirectoryDomainSuffix: + Type: String + Description: >- + LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when + DeployDirectory=true. + Default: 'dc=cluster,dc=internal' + HomeCapacity: Description: "Home directories storage capacity in GiB" Type: Number @@ -713,6 +738,8 @@ Resources: MonitoringRole: login PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs + DeployDirectory: !Ref DeployDirectory + DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name Value: !Sub '${AWS::StackName}-LoginNodeGroup' @@ -759,6 +786,8 @@ Resources: EnableEfa: !Ref OnDemandEnableEfa EfaInterfaceCount: !Ref OnDemandEfaInterfaceCount PlacementGroupName: !Ref OnDemandPlacementGroupName + DeployDirectory: !Ref DeployDirectory + DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name Value: !Sub '${AWS::StackName}-OnDemandCNG' diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index cf54540a1..560bff5bc 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -51,6 +51,41 @@ without a `sacctmgr` entry will have their jobs **rejected**. With the default --- +## Template structure (how multi-user is wired) + +``` +pcs-ml-cluster-deploy-all.yaml +│ +│ Parameters: DeployDirectory, DirectoryDomainSuffix +│ +├─► PrerequisitesStack (ml-cluster-prerequisites.yaml) +│ VPC, FSx Lustre, FSx OpenZFS (/home — shared storage for LDAP DB) +│ +├─► ClusterStack (cluster.yaml) +│ PCS Cluster, IAM role, SSM param (Grafana pw) +│ +├─► LoginNodeGroupStack (add-cng.yaml) +│ │ MonitoringRole=login, DeployDirectory=$DeployDirectory +│ │ +│ └─► UserData: +│ if DeployDirectory=true: +│ curl setup-openldap-server.sh → install slapd +│ DB → /home/ldap-db/ (shared OpenZFS) +│ Admin password → /home/ldap-db/.admin-password +│ +├─► OnDemandCNGStack (add-cng.yaml) +│ │ MonitoringRole=compute, DeployDirectory=$DeployDirectory +│ │ +│ └─► UserData: +│ if DeployDirectory=true: +│ discover login node IP (ec2:DescribeInstances by tag) +│ curl setup-ldap-client.sh → install SSSD +│ ldap_uri = ldap:// +│ +└─► [Optional] PseriesCNGStack (add-cng-p5/p6-*.yaml) + Same pattern as OnDemandCNG (SSSD client when DeployDirectory=true) +``` + When `DeployDirectory=true` is set, an OpenLDAP directory is deployed on the login node, and all compute nodes are configured as LDAP clients via SSSD — giving you centralized POSIX user management across the cluster. From 599a9d7cced64fe5f251f3641377391d809a8ee1 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 14:04:00 +0000 Subject: [PATCH 10/89] =?UTF-8?q?refactor:=20rename=20DeployDirectory=20?= =?UTF-8?q?=E2=86=92=20DirectoryService=20(enum:=20none=20|=20OpenLDAP-Log?= =?UTF-8?q?inNode)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rename the boolean parameter to an enum that makes it clear both *what* directory service is used and *where* it runs: DirectoryService: AllowedValues: [none, OpenLDAP-LoginNode] # Future: SimpleAD, ManagedAD, OpenLDAP-External This prepares for adding Simple AD or external LDAP options in the future without a breaking rename. The value "OpenLDAP-LoginNode" communicates that slapd runs on the login node (not an external server). Also: - README.md: added template nesting structure diagram (ASCII art) showing how DirectoryService flows from deploy-all through login (slapd server) and compute (SSSD client) CNGs. - Fixed DeployMonitoring default accidentally changed to 'none' by bulk sed (reverted to 'false' for add-cng, 'true' for deploy-all). --- architectures/aws-pcs/README.md | 30 +++++++++++++++++++ architectures/aws-pcs/assets/add-cng.yaml | 29 +++++++++--------- .../assets/pcs-ml-cluster-deploy-all.yaml | 30 +++++++++---------- 3 files changed, 59 insertions(+), 30 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 52eda6d10..ad513e1b5 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -525,6 +525,36 @@ parameter and default, see [PARAMETERS.md](./docs/PARAMETERS.md). for login nodes). The P-series templates need a `CapacityReservationId` when using a Capacity Block. +### Template nesting structure (deploy-all) + +``` +pcs-ml-cluster-deploy-all.yaml ← user deploys this +│ +├─► ml-cluster-prerequisites.yaml ← VPC, subnets, SG, FSx Lustre + OpenZFS +│ +├─► cluster.yaml ← PCS cluster (Slurm scheduler), IAM role +│ +├─► add-cng.yaml (login) ← login node (MinCount=1) +│ • MonitoringRole=login → Prometheus/Grafana +│ • DirectoryService=OpenLDAP → slapd server +│ +├─► add-cng.yaml (compute) ← CPU queue (dynamic scaling 0→N) +│ • MonitoringRole=compute → Node Exporter +│ • DirectoryService=OpenLDAP → SSSD client +│ • EnableEfa=true → EFA NetworkInterfaces + PG +│ +└─► add-cng-p5.yaml / add-cng-p6-b200.yaml ← GPU queue (optional) + / add-cng-p6-b300.yaml + • Multi-NIC EFA (16/32 cards, per-family) + • MonitoringRole=compute → DCGM Exporter + • DirectoryService=OpenLDAP → SSSD client + +Standalone (not nested): + pcs-ready-dlami-with-enroot-pyxis.yaml ← AMI builder (separate stack) + iam/cluster-admin-iam.yaml ← IAM policies (separate stack) + iam/cluster-user-iam.yaml ← IAM policies (separate stack) +``` + --- ## 11. Testing and Validation diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index f17c2fad2..5be52f5c5 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -59,7 +59,7 @@ Metadata: - Label: default: Multi-user directory (optional) Parameters: - - DeployDirectory + - DirectoryService - DirectoryDomainSuffix Parameters: @@ -215,24 +215,23 @@ Parameters: - '25.05' - '25.11' - DeployDirectory: + DirectoryService: Type: String Description: >- - Deploy an OpenLDAP directory on the login node for multi-user management. - When true and MonitoringRole=login, slapd is installed on the login node - with its DB on shared /home (OpenZFS), so it survives CNG recreation. - Compute nodes configure SSSD to resolve users from the login node's LDAP. - Default false = single ubuntu user (unchanged). - Default: 'false' + Multi-user directory service. 'none' = single ubuntu user (default, + unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared + /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future + options (SimpleAD, ManagedAD) will be added as AllowedValues. + Default: 'none' AllowedValues: - - 'true' - - 'false' + - 'none' + - 'OpenLDAP-LoginNode' DirectoryDomainSuffix: Type: String Description: >- LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when - DeployDirectory=true. + DirectoryService=OpenLDAP-LoginNode. Default: 'dc=cluster,dc=internal' EnableEfa: @@ -245,7 +244,7 @@ Parameters: an EFA-capable instance type (e.g. hpc8a.96xlarge, hpc7a.96xlarge, hpc6a.48xlarge). For multi-NIC GPU families (P5/P6) use the dedicated add-cng-p5/p6-* templates instead. - Default: 'false' + Default: 'none' AllowedValues: - 'true' - 'false' @@ -288,7 +287,7 @@ Conditions: Has2ndEfaInterface: !And - !Condition EfaEnabled - !Equals [!Ref EfaInterfaceCount, 2] - DirectoryEnabled: !Equals [!Ref DeployDirectory, 'true'] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryService, 'none']] IsLoginNode: !Equals [!Ref MonitoringRole, 'login'] # LDAP server runs on login node; LDAP client (SSSD) runs on compute nodes SetupLdapServer: !And @@ -510,8 +509,8 @@ Resources: MIME-Version: 1.0 #!/bin/bash - # Multi-user directory setup (optional, DeployDirectory=true) - if [ "${DeployDirectory}" = "true" ]; then + # Multi-user directory setup (optional, DirectoryService=OpenLDAP-LoginNode) + if [ "${DirectoryService}" != "none" ]; then export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 055b27c7b..304238bd8 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -71,7 +71,7 @@ Metadata: - Label: default: "7. Multi-User Directory (Optional)" Parameters: - - DeployDirectory + - DirectoryService - DirectoryDomainSuffix - Label: default: "8. Developer / Advanced (Nested Templates & Monitoring Source)" @@ -262,29 +262,29 @@ Parameters: Capacity to >= the EFA minimum when enabling this; the full minimum-capacity matrix per throughput tier is in the FSx for Lustre User Guide. Type: String - Default: 'false' + Default: 'none' AllowedValues: - 'true' - 'false' # Multi-user directory - DeployDirectory: + DirectoryService: Type: String Description: >- - Deploy an OpenLDAP directory on the login node for multi-user management. - When true, slapd runs on the login node (DB on shared /home/ldap-db, - survives CNG recreation) and compute nodes configure SSSD to resolve LDAP - users. Default false = single ubuntu user (unchanged). - Default: 'false' + Multi-user directory service. 'none' = single ubuntu user (default, + unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared + /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future + options (SimpleAD, ManagedAD) will be added here. + Default: 'none' AllowedValues: - - 'true' - - 'false' + - 'none' + - 'OpenLDAP-LoginNode' DirectoryDomainSuffix: Type: String Description: >- LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when - DeployDirectory=true. + DirectoryService != none. Default: 'dc=cluster,dc=internal' HomeCapacity: @@ -472,7 +472,7 @@ Parameters: group. Use with EFA-capable HPC instance types (hpc8a.96xlarge, hpc7a.96xlarge, hpc6a.48xlarge, etc.). Default false keeps the standard single-NIC LaunchTemplate. No effect on the P-series GPU CNG. - Default: 'false' + Default: 'none' AllowedValues: - 'true' - 'false' @@ -501,7 +501,7 @@ Parameters: DeployPseriesCNG: Type: String Description: Deploy P-series (P5/P6) compute node group - Default: 'false' + Default: 'none' AllowedValues: - 'true' - 'false' @@ -738,7 +738,7 @@ Resources: MonitoringRole: login PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs - DeployDirectory: !Ref DeployDirectory + DirectoryService: !Ref DirectoryService DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name @@ -786,7 +786,7 @@ Resources: EnableEfa: !Ref OnDemandEnableEfa EfaInterfaceCount: !Ref OnDemandEfaInterfaceCount PlacementGroupName: !Ref OnDemandPlacementGroupName - DeployDirectory: !Ref DeployDirectory + DirectoryService: !Ref DirectoryService DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name From b2d1c0a74de02880055c2e36c17e37338b5c0d42 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 14:06:04 +0000 Subject: [PATCH 11/89] =?UTF-8?q?ROADMAP:=20track=20DeployMonitoring=20?= =?UTF-8?q?=E2=86=92=20MonitoringStack=20rename=20for=20next=20minor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- architectures/aws-pcs/docs/ROADMAP.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/architectures/aws-pcs/docs/ROADMAP.md b/architectures/aws-pcs/docs/ROADMAP.md index d66e3d431..250a408b6 100644 --- a/architectures/aws-pcs/docs/ROADMAP.md +++ b/architectures/aws-pcs/docs/ROADMAP.md @@ -101,6 +101,14 @@ Priority: 🔴 high · 🟡 medium · 🟢 low Prometheus + Amazon Managed Grafana as an alternative to the self-hosted stack on the login node (see `4.validation_and_observability/4.prometheus-grafana`), so users can use a managed backend instead of running the containers themselves. +- [ ] 🟡 **Rename `DeployMonitoring` → `MonitoringStack` (enum).** Align with the + `DirectoryService` parameter pattern (`-`): + `MonitoringStack: none | Prometheus-LoginNode | AMP-AMG | CloudWatch`. + This makes it clear that the current default is a self-hosted Prometheus + Grafana + stack on the login node, and provides a natural extension point for the managed + option above. **Breaking change** — bundle with the next minor version alongside + other parameter renames (`Capacity` → `LustreCapacity`, `CngName`/`QueueName` + collapse, etc.). ## Testing / docs From f4fdb60c1492db2da5b8da2b0734b2e9c6ad0ccd Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 14:10:35 +0000 Subject: [PATCH 12/89] =?UTF-8?q?fix:=20revert=20unintended=20Default:'fal?= =?UTF-8?q?se'=E2=86=92'none'=20on=204=20boolean=20params?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bulk sed that renamed DeployDirectory→DirectoryService also changed Default:'false' to Default:'none' on every parameter in both templates. This broke FSxLustreEnableEfa, OnDemandEnableEfa, DeployPseriesCNG (deploy-all) and EnableEfa (add-cng) — their AllowedValues are [true,false] so 'none' is invalid. Reverted to 'false' for all 4. Only DirectoryService (default 'none') and MonitoringRole (default 'none') correctly use 'none'. --- architectures/aws-pcs/assets/add-cng.yaml | 2 +- architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 5be52f5c5..93cfaac35 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -244,7 +244,7 @@ Parameters: an EFA-capable instance type (e.g. hpc8a.96xlarge, hpc7a.96xlarge, hpc6a.48xlarge). For multi-NIC GPU families (P5/P6) use the dedicated add-cng-p5/p6-* templates instead. - Default: 'none' + Default: 'false' AllowedValues: - 'true' - 'false' diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 304238bd8..a3bbe9699 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -262,7 +262,7 @@ Parameters: Capacity to >= the EFA minimum when enabling this; the full minimum-capacity matrix per throughput tier is in the FSx for Lustre User Guide. Type: String - Default: 'none' + Default: 'false' AllowedValues: - 'true' - 'false' @@ -472,7 +472,7 @@ Parameters: group. Use with EFA-capable HPC instance types (hpc8a.96xlarge, hpc7a.96xlarge, hpc6a.48xlarge, etc.). Default false keeps the standard single-NIC LaunchTemplate. No effect on the P-series GPU CNG. - Default: 'none' + Default: 'false' AllowedValues: - 'true' - 'false' @@ -501,7 +501,7 @@ Parameters: DeployPseriesCNG: Type: String Description: Deploy P-series (P5/P6) compute node group - Default: 'none' + Default: 'false' AllowedValues: - 'true' - 'false' From 9ccdbb447f1f524900dde2e7b3496265717af04b Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 14:26:25 +0000 Subject: [PATCH 13/89] security: store LDAP admin password in SSM Parameter Store, not on shared /home The admin password was previously written to /home/ldap-db/.admin-password (shared OpenZFS, readable by root on all nodes). This is a security concern in multi-user clusters where compute-node users may have sudo. Now: - setup-openldap-server.sh first tries SSM PutParameter (SecureString) at /pcs//ldap/admin-password. SSM is access-controlled via IAM and auditable via CloudTrail. - Falls back to the file only if the instance role lacks ssm:PutParameter (with a WARNING in the install log). - cluster.yaml instance role: broadened the SSM Sid from grafana/* to also cover ldap/* (same pattern, same cluster-scoped resource ARN). - Removed the duplicate SSM put from add-cng.yaml UserData (the script now handles it internally with CLUSTER_ID exported from UserData). - USER-MANAGEMENT.md: added fallback note + clarified SSM is the primary storage mechanism. --- architectures/aws-pcs/assets/add-cng.yaml | 10 +++----- architectures/aws-pcs/assets/cluster.yaml | 6 +++-- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 10 ++++++-- .../aws-pcs/scripts/setup-openldap-server.sh | 23 ++++++++++++++++--- 4 files changed, 35 insertions(+), 14 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 93cfaac35..b727bc789 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -515,17 +515,13 @@ Resources: export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + export CLUSTER_ID="${ClusterId}" + if [ "${MonitoringRole}" = "login" ]; then - # Login node: install OpenLDAP server + # Login node: install OpenLDAP server (password stored in SSM by the script) echo "[multi-user] Setting up OpenLDAP server on login node..." curl -fsSL "https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-openldap-server.sh" -o /tmp/setup-openldap-server.sh bash /tmp/setup-openldap-server.sh 2>&1 | tee /var/log/ldap-server-install.log - # Store admin password in SSM for retrieval - export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:/usr/local/bin:${!PATH} - CLUSTER_ID="${ClusterId}" - aws ssm put-parameter --name "/pcs/${!CLUSTER_ID}/ldap/admin-password" \ - --value "${!LDAP_ADMIN_PASSWORD}" --type SecureString --overwrite \ - --region $(curl -s http://169.254.169.254/latest/meta-data/placement/region) || true else # Compute node: configure SSSD LDAP client pointing at login node echo "[multi-user] Configuring SSSD LDAP client..." diff --git a/architectures/aws-pcs/assets/cluster.yaml b/architectures/aws-pcs/assets/cluster.yaml index f4d1dfad3..6735047ea 100644 --- a/architectures/aws-pcs/assets/cluster.yaml +++ b/architectures/aws-pcs/assets/cluster.yaml @@ -225,13 +225,15 @@ Resources: - pricing:DescribeServices Resource: '*' - - Sid: SSMGrafanaPasswordForThisClusterOnly + - Sid: SSMClusterParametersForThisClusterOnly Effect: Allow Action: - ssm:GetParameter - ssm:PutParameter - ssm:AddTagsToResource - Resource: !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/grafana/*' + Resource: + - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/grafana/*' + - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/ldap/*' - Sid: KMSForSSMSecureString Effect: Allow diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index 560bff5bc..9fe1ab7c3 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -71,7 +71,7 @@ pcs-ml-cluster-deploy-all.yaml │ if DeployDirectory=true: │ curl setup-openldap-server.sh → install slapd │ DB → /home/ldap-db/ (shared OpenZFS) -│ Admin password → /home/ldap-db/.admin-password +│ Admin password → SSM /pcs//ldap/admin-password │ ├─► OnDemandCNGStack (add-cng.yaml) │ │ MonitoringRole=compute, DeployDirectory=$DeployDirectory @@ -163,7 +163,7 @@ All user management commands run on the **login node** as root (or via `sudo`). ### Retrieve the LDAP admin password The admin password is auto-generated at first boot and stored in SSM -Parameter Store: +Parameter Store (encrypted with the account's default KMS key): ```bash CLUSTER_ID= @@ -171,6 +171,12 @@ aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ --with-decryption --query 'Parameter.Value' --output text ``` +> **Fallback**: if the instance role lacked `ssm:PutParameter` at first boot, +> the password is instead saved to `/home/ldap-db/.admin-password` on shared +> OpenZFS. This file is `chmod 600` but readable by root on any node — for +> production, ensure the instance role has the SSM permission (the default +> `cluster.yaml` role includes it at `/pcs//ldap/*`). + ### Add a user Use the helper script (pre-installed on the login node at diff --git a/architectures/aws-pcs/scripts/setup-openldap-server.sh b/architectures/aws-pcs/scripts/setup-openldap-server.sh index 89524e70c..92638b749 100755 --- a/architectures/aws-pcs/scripts/setup-openldap-server.sh +++ b/architectures/aws-pcs/scripts/setup-openldap-server.sh @@ -107,9 +107,26 @@ objectClass: organizationalUnit ou: Groups EOF -# Persist admin password on shared storage (survives CNG delete) -echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" -chmod 600 "${LDAP_DB_DIR}/.admin-password" +# Persist admin password — prefer SSM Parameter Store (access-controlled, +# auditable via CloudTrail) over a file on shared /home (readable by any root). +REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) +CLUSTER_ID="${CLUSTER_ID:-unknown}" +if aws ssm put-parameter \ + --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --value "${LDAP_ADMIN_PASSWORD}" \ + --type SecureString \ + --overwrite \ + --region "${REGION}" 2>/dev/null; then + echo "[ldap-server] Admin password stored in SSM: /pcs/${CLUSTER_ID}/ldap/admin-password" +else + # Fallback: write to shared storage if instance role lacks ssm:PutParameter + echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" + chmod 600 "${LDAP_DB_DIR}/.admin-password" + chown openldap:openldap "${LDAP_DB_DIR}/.admin-password" + echo "[ldap-server] WARNING: SSM put failed (instance role may lack ssm:PutParameter)." + echo "[ldap-server] Admin password saved to ${LDAP_DB_DIR}/.admin-password (readable by root on all nodes)." + echo "[ldap-server] For production, add ssm:PutParameter to the instance role and redeploy." +fi echo "[ldap-server] OpenLDAP server ready." echo "[ldap-server] Admin DN: cn=admin,${LDAP_DOMAIN_SUFFIX}" From d2426a9f5b148bf71f17cc8c8555d2db7760a30f Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 14:52:41 +0000 Subject: [PATCH 14/89] =?UTF-8?q?refactor:=20merge=20setup-openldap-server?= =?UTF-8?q?.sh=20+=20setup-ldap-client.sh=20=E2=86=92=20setup-directory.sh?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single script with role argument (server|client) replaces two separate scripts. Benefits: - One URL to manage, no naming inconsistency - Full picture visible in one file (server + client + shared SSSD config) - Natural extension point for future SimpleAD (add a case branch) - Server role now also configures SSSD on the login node itself (so getent works locally without a separate manual step) UserData in add-cng.yaml simplified to: curl setup-directory.sh if login → bash setup-directory.sh server else → bash setup-directory.sh client Environment variable interface unchanged (LDAP_DOMAIN_SUFFIX, CLUSTER_ID, DIRECTORY_DNS_IPS for future SimpleAD). DIRECTORY_DNS_IPS="" triggers login-IP discovery via ec2:DescribeInstances tag query. Also: README template structure diagram updated with external scripts and SSM parameter references. --- architectures/aws-pcs/README.md | 28 ++- architectures/aws-pcs/assets/add-cng.yaml | 29 +-- .../aws-pcs/scripts/setup-directory.sh | 235 ++++++++++++++++++ .../aws-pcs/scripts/setup-ldap-client.sh | 71 ------ .../aws-pcs/scripts/setup-openldap-server.sh | 136 ---------- 5 files changed, 267 insertions(+), 232 deletions(-) create mode 100755 architectures/aws-pcs/scripts/setup-directory.sh delete mode 100755 architectures/aws-pcs/scripts/setup-ldap-client.sh delete mode 100755 architectures/aws-pcs/scripts/setup-openldap-server.sh diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index ad513e1b5..a8a926198 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -533,21 +533,43 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this ├─► ml-cluster-prerequisites.yaml ← VPC, subnets, SG, FSx Lustre + OpenZFS │ ├─► cluster.yaml ← PCS cluster (Slurm scheduler), IAM role +│ • IAM role grants: ssm:PutParameter /pcs//grafana/*, /pcs//ldap/* │ ├─► add-cng.yaml (login) ← login node (MinCount=1) │ • MonitoringRole=login → Prometheus/Grafana -│ • DirectoryService=OpenLDAP → slapd server +│ • DirectoryService=OpenLDAP-LoginNode → slapd server +│ │ +│ └─ UserData fetches external scripts: +│ ├─ PostInstallScriptUrl (default: install-enroot-pyxis.sh) +│ ├─ MonitoringRepo/MonitoringVersion → post-install.sh +│ └─ setup-openldap-server.sh (when DirectoryService != none) │ ├─► add-cng.yaml (compute) ← CPU queue (dynamic scaling 0→N) │ • MonitoringRole=compute → Node Exporter -│ • DirectoryService=OpenLDAP → SSSD client +│ • DirectoryService=OpenLDAP-LoginNode → SSSD client │ • EnableEfa=true → EFA NetworkInterfaces + PG +│ │ +│ └─ UserData fetches external scripts: +│ ├─ PostInstallScriptUrl (same as login) +│ ├─ MonitoringRepo/MonitoringVersion → post-install.sh +│ └─ setup-ldap-client.sh (when DirectoryService != none) │ └─► add-cng-p5.yaml / add-cng-p6-b200.yaml ← GPU queue (optional) / add-cng-p6-b300.yaml • Multi-NIC EFA (16/32 cards, per-family) • MonitoringRole=compute → DCGM Exporter - • DirectoryService=OpenLDAP → SSSD client + • Same external script pattern as compute CNG + +External scripts (fetched at first boot from GitHub raw): + scripts/install-enroot-pyxis.sh ← Enroot 3.5.0 + Pyxis 0.20.0 + scripts/setup-openldap-server.sh ← slapd install + config (login only) + scripts/setup-ldap-client.sh ← SSSD LDAP client (compute nodes) + MonitoringRepo @ MonitoringVersion ← aws-parallelcluster-monitoring post-install.sh + +External references (runtime): + SSM /aws/service/pcs/ami/.../latest/ami-id ← PCS-Ready DLAMI (when AmiId is empty) + SSM /pcs//grafana/admin-password ← auto-generated Grafana password + SSM /pcs//ldap/admin-password ← auto-generated LDAP admin password Standalone (not nested): pcs-ready-dlami-with-enroot-pyxis.yaml ← AMI builder (separate stack) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index b727bc789..c8a8d8fd5 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -509,36 +509,21 @@ Resources: MIME-Version: 1.0 #!/bin/bash - # Multi-user directory setup (optional, DirectoryService=OpenLDAP-LoginNode) + # Multi-user directory setup (optional, DirectoryService != none) if [ "${DirectoryService}" != "none" ]; then export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) - export CLUSTER_ID="${ClusterId}" + export DIRECTORY_DNS_IPS="" + + SCRIPT_URL="https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-directory.sh" + curl -fsSL "${!SCRIPT_URL}" -o /tmp/setup-directory.sh if [ "${MonitoringRole}" = "login" ]; then - # Login node: install OpenLDAP server (password stored in SSM by the script) - echo "[multi-user] Setting up OpenLDAP server on login node..." - curl -fsSL "https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-openldap-server.sh" -o /tmp/setup-openldap-server.sh - bash /tmp/setup-openldap-server.sh 2>&1 | tee /var/log/ldap-server-install.log + bash /tmp/setup-directory.sh server 2>&1 | tee /var/log/directory-setup.log else - # Compute node: configure SSSD LDAP client pointing at login node - echo "[multi-user] Configuring SSSD LDAP client..." - # Discover login node IP via tag query - REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) - LOGIN_IP=$(aws ec2 describe-instances --region ${!REGION} \ - --filters "Name=tag:pcs-cluster-id,Values=${ClusterId}" \ - "Name=tag:monitoring-role,Values=login" \ - "Name=instance-state-name,Values=running" \ - --query 'Reservations[0].Instances[0].PrivateIpAddress' --output text 2>/dev/null || echo "") - if [ -n "${!LOGIN_IP}" ] && [ "${!LOGIN_IP}" != "None" ]; then - export LDAP_SERVER_URI="ldap://${!LOGIN_IP}" - curl -fsSL "https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-ldap-client.sh" -o /tmp/setup-ldap-client.sh - bash /tmp/setup-ldap-client.sh 2>&1 | tee /var/log/ldap-client-install.log - else - echo "[multi-user] WARNING: could not discover login node IP. LDAP client not configured." - fi + bash /tmp/setup-directory.sh client 2>&1 | tee /var/log/directory-setup.log fi fi diff --git a/architectures/aws-pcs/scripts/setup-directory.sh b/architectures/aws-pcs/scripts/setup-directory.sh new file mode 100755 index 000000000..08fe728cc --- /dev/null +++ b/architectures/aws-pcs/scripts/setup-directory.sh @@ -0,0 +1,235 @@ +#!/bin/bash +# setup-directory.sh — Multi-user directory setup for PCS reference architecture. +# +# Single script for both login node (LDAP server) and compute nodes (SSSD client). +# Called from CNG UserData when DirectoryService != "none". +# +# Usage: setup-directory.sh +# role = "server" (login node — installs slapd, creates base OUs) +# "client" (compute node — installs SSSD, points at LDAP server) +# +# Environment variables (from UserData): +# LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" +# LDAP_DOMAIN — e.g. "cluster.internal" (derived from suffix) +# LDAP_ADMIN_PASSWORD — auto-generated by UserData (server role only) +# CLUSTER_ID — PCS cluster ID (for SSM parameter path) +# LDAP_SERVER_URI — e.g. "ldap://10.1.x.x" (client role; empty = discover) +# DIRECTORY_DNS_IPS — comma-separated DNS IPs (future SimpleAD; empty = discover login IP) + +set -euo pipefail + +ROLE="${1:-client}" +LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-dc=cluster,dc=internal}" +LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" +LDAP_DB_DIR="/home/ldap-db" + +export DEBIAN_FRONTEND=noninteractive + +echo "[directory] Role: ${ROLE}" +echo "[directory] Domain: ${LDAP_DOMAIN} (${LDAP_DOMAIN_SUFFIX})" + +############################################################################### +# SERVER role — login node: install slapd + configure +############################################################################### +setup_server() { + LDAP_ADMIN_PASSWORD="${LDAP_ADMIN_PASSWORD:?LDAP_ADMIN_PASSWORD must be set}" + CLUSTER_ID="${CLUSTER_ID:-unknown}" + + echo "[directory-server] Running apt-get update..." + apt-get update -qq + + echo "[directory-server] Installing slapd + ldap-utils..." + debconf-set-selections </dev/null)" ]; then + cp -a /var/lib/ldap/* "${LDAP_DB_DIR}/" + fi + chown -R openldap:openldap "${LDAP_DB_DIR}" + fi + + # Update slapd DB directory in cn=config + if [ -d /etc/ldap/slapd.d ]; then + MDB_LDIF=$(find /etc/ldap/slapd.d -name "olcDatabase*mdb*" -o -name "olcDatabase*hdb*" | head -1) + if [ -n "$MDB_LDIF" ] && grep -q "olcDbDirectory" "$MDB_LDIF"; then + sed -i "s|olcDbDirectory:.*|olcDbDirectory: ${LDAP_DB_DIR}|" "$MDB_LDIF" + fi + fi + + # AppArmor: allow slapd to access /home/ldap-db + if [ -f /etc/apparmor.d/usr.sbin.slapd ]; then + if ! grep -q "${LDAP_DB_DIR}" /etc/apparmor.d/usr.sbin.slapd; then + sed -i "/\/var\/lib\/ldap\/ r,/a\\ ${LDAP_DB_DIR}/ r," /etc/apparmor.d/usr.sbin.slapd + sed -i "/\/var\/lib\/ldap\/\*\* rwk,/a\\ ${LDAP_DB_DIR}/** rwk," /etc/apparmor.d/usr.sbin.slapd + apparmor_parser -r /etc/apparmor.d/usr.sbin.slapd 2>/dev/null || true + fi + fi + + chown -R openldap:openldap "${LDAP_DB_DIR}" + systemctl start slapd + systemctl enable slapd + + # Wait for slapd + for i in $(seq 1 10); do + ldapsearch -x -H ldap://localhost -b "" -s base namingContexts >/dev/null 2>&1 && break + sleep 1 + done + + echo "[directory-server] Creating base OUs..." + ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: ou=People,${LDAP_DOMAIN_SUFFIX} +objectClass: organizationalUnit +ou: People +EOF + ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: ou=Groups,${LDAP_DOMAIN_SUFFIX} +objectClass: organizationalUnit +ou: Groups +EOF + # Default group + ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true +dn: cn=clusterusers,ou=Groups,${LDAP_DOMAIN_SUFFIX} +objectClass: posixGroup +cn: clusterusers +gidNumber: 3000 +EOF + + # Persist admin password — prefer SSM (access-controlled, auditable) + REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + if aws ssm put-parameter \ + --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --value "${LDAP_ADMIN_PASSWORD}" \ + --type SecureString \ + --overwrite \ + --region "${REGION}" 2>/dev/null; then + echo "[directory-server] Admin password stored in SSM: /pcs/${CLUSTER_ID}/ldap/admin-password" + else + echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" + chmod 600 "${LDAP_DB_DIR}/.admin-password" + chown openldap:openldap "${LDAP_DB_DIR}/.admin-password" + echo "[directory-server] WARNING: SSM put failed. Password saved to ${LDAP_DB_DIR}/.admin-password" + fi + + # Also configure SSSD on the login node itself (so getent works locally) + setup_client_internal "ldap://localhost" + + echo "[directory-server] OpenLDAP server ready." + echo "[directory-server] Admin DN: cn=admin,${LDAP_DOMAIN_SUFFIX}" +} + +############################################################################### +# CLIENT role — compute node: install SSSD + configure LDAP provider +############################################################################### +setup_client() { + # Determine LDAP server URI + local server_uri="${LDAP_SERVER_URI:-}" + + if [ -z "$server_uri" ]; then + # Discover login node IP + if [ -n "${DIRECTORY_DNS_IPS:-}" ]; then + # Future: SimpleAD — use provided DNS IPs + server_uri="ldap://${DIRECTORY_DNS_IPS%%,*}" + else + # OpenLDAP-LoginNode — discover via EC2 tag query + local region + region=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + local login_ip + login_ip=$(aws ec2 describe-instances --region "${region}" \ + --filters "Name=tag:pcs-cluster-id,Values=${CLUSTER_ID:-}" \ + "Name=tag:monitoring-role,Values=login" \ + "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].PrivateIpAddress' \ + --output text 2>/dev/null || echo "") + if [ -z "$login_ip" ] || [ "$login_ip" = "None" ]; then + echo "[directory-client] ERROR: could not discover login node IP. LDAP client not configured." + return 1 + fi + server_uri="ldap://${login_ip}" + fi + fi + + setup_client_internal "$server_uri" +} + +############################################################################### +# Internal: configure SSSD (shared by server + client roles) +############################################################################### +setup_client_internal() { + local server_uri="$1" + + echo "[directory-client] Running apt-get update..." + apt-get update -qq + + echo "[directory-client] Installing SSSD..." + apt-get install -y sssd libpam-sss libnss-sss ldap-utils + + echo "[directory-client] Configuring SSSD (server: ${server_uri})..." + cat > /etc/sssd/sssd.conf <> /etc/pam.d/common-session + fi + + # Configure NSS + sed -i 's/^passwd:.*/passwd: files systemd sss/' /etc/nsswitch.conf + sed -i 's/^group:.*/group: files systemd sss/' /etc/nsswitch.conf + sed -i 's/^shadow:.*/shadow: files sss/' /etc/nsswitch.conf + + systemctl enable sssd + systemctl restart sssd + + echo "[directory-client] SSSD configured. Server: ${server_uri}" +} + +############################################################################### +# Main +############################################################################### +case "$ROLE" in + server) + setup_server + ;; + client) + setup_client + ;; + *) + echo "Usage: $0 " + exit 1 + ;; +esac diff --git a/architectures/aws-pcs/scripts/setup-ldap-client.sh b/architectures/aws-pcs/scripts/setup-ldap-client.sh deleted file mode 100755 index d5b661054..000000000 --- a/architectures/aws-pcs/scripts/setup-ldap-client.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/bin/bash -# setup-ldap-client.sh — Configure SSSD LDAP client on compute nodes. -# Called from CNG UserData when DeployDirectory=true and MonitoringRole=compute (or none). -# -# Environment variables (passed from UserData): -# LDAP_SERVER_URI — e.g. "ldap://10.1.x.x" (login node private IP) -# LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" -# LDAP_DOMAIN — e.g. "cluster.internal" - -set -euo pipefail - -LDAP_SERVER_URI="${LDAP_SERVER_URI:?LDAP_SERVER_URI must be set}" -LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-dc=cluster,dc=internal}" -LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" - -export DEBIAN_FRONTEND=noninteractive - -echo "[ldap-client] Running apt-get update..." -apt-get update -qq - -echo "[ldap-client] Installing SSSD + LDAP client packages..." -apt-get install -y sssd libpam-sss libnss-sss ldap-utils - -echo "[ldap-client] Configuring SSSD..." -cat > /etc/sssd/sssd.conf <> /etc/pam.d/common-session -fi - -# Configure NSS to use sss -if ! grep -q sss /etc/nsswitch.conf; then - sed -i 's/^passwd:.*/passwd: files systemd sss/' /etc/nsswitch.conf - sed -i 's/^group:.*/group: files systemd sss/' /etc/nsswitch.conf - sed -i 's/^shadow:.*/shadow: files sss/' /etc/nsswitch.conf -fi - -systemctl enable sssd -systemctl restart sssd - -echo "[ldap-client] SSSD LDAP client configured." -echo "[ldap-client] Server: ${LDAP_SERVER_URI}" -echo "[ldap-client] Search base: ${LDAP_DOMAIN_SUFFIX}" -echo "[ldap-client] Verify: getent passwd (should show LDAP users after they are created)" diff --git a/architectures/aws-pcs/scripts/setup-openldap-server.sh b/architectures/aws-pcs/scripts/setup-openldap-server.sh deleted file mode 100755 index 92638b749..000000000 --- a/architectures/aws-pcs/scripts/setup-openldap-server.sh +++ /dev/null @@ -1,136 +0,0 @@ -#!/bin/bash -# setup-openldap-server.sh — Install and configure OpenLDAP (slapd) on the login node. -# Called from CNG UserData when DeployDirectory=true and MonitoringRole=login. -# -# LDAP DB is stored on /home/ldap-db (shared OpenZFS) so it survives login -# node CNG recreation. The slapd service on a fresh node detects existing DB -# and reuses it (idempotent). -# -# Environment variables (passed from UserData): -# LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" (from DirectoryDomainSuffix) -# LDAP_DOMAIN — e.g. "cluster.internal" (derived) -# LDAP_ADMIN_PASSWORD — from Secrets Manager or auto-generated -# LDAP_BASE_GROUPS — comma-separated default groups (default: "clusterusers,clusteradmins") - -set -euo pipefail - -LDAP_DOMAIN_SUFFIX="${LDAP_DOMAIN_SUFFIX:-dc=cluster,dc=internal}" -LDAP_DOMAIN="${LDAP_DOMAIN:-cluster.internal}" -LDAP_ADMIN_PASSWORD="${LDAP_ADMIN_PASSWORD:-$(openssl rand -base64 16)}" -LDAP_BASE_GROUPS="${LDAP_BASE_GROUPS:-clusterusers,clusteradmins}" -LDAP_DB_DIR="/home/ldap-db" - -export DEBIAN_FRONTEND=noninteractive - -echo "[ldap-server] Running apt-get update..." -apt-get update -qq - -echo "[ldap-server] Installing slapd + ldap-utils..." -debconf-set-selections </dev/null)" ]; then - cp -a /var/lib/ldap/* "${LDAP_DB_DIR}/" - fi - chown -R openldap:openldap "${LDAP_DB_DIR}" -fi - -# Point slapd at shared storage via apparmor + config override -# Update slapd DB directory in the config -if [ -d /etc/ldap/slapd.d ]; then - # cn=config style — find the mdb backend config and update olcDbDirectory - MDB_LDIF=$(find /etc/ldap/slapd.d -name "olcDatabase*mdb*" -o -name "olcDatabase*hdb*" | head -1) - if [ -n "$MDB_LDIF" ] && grep -q "olcDbDirectory" "$MDB_LDIF"; then - sed -i "s|olcDbDirectory:.*|olcDbDirectory: ${LDAP_DB_DIR}|" "$MDB_LDIF" - fi -fi - -# AppArmor: allow slapd to access /home/ldap-db -if [ -f /etc/apparmor.d/usr.sbin.slapd ]; then - if ! grep -q "${LDAP_DB_DIR}" /etc/apparmor.d/usr.sbin.slapd; then - sed -i "/\/var\/lib\/ldap\/ r,/a\\ ${LDAP_DB_DIR}/ r," /etc/apparmor.d/usr.sbin.slapd - sed -i "/\/var\/lib\/ldap\/\*\* rwk,/a\\ ${LDAP_DB_DIR}/** rwk," /etc/apparmor.d/usr.sbin.slapd - apparmor_parser -r /etc/apparmor.d/usr.sbin.slapd 2>/dev/null || true - fi -fi - -# Ensure correct ownership -chown -R openldap:openldap "${LDAP_DB_DIR}" - -systemctl start slapd -systemctl enable slapd - -# Wait for slapd to be ready -for i in $(seq 1 10); do - ldapsearch -x -H ldap://localhost -b "" -s base namingContexts >/dev/null 2>&1 && break - sleep 1 -done - -echo "[ldap-server] Creating base groups..." -for GROUP in $(echo "${LDAP_BASE_GROUPS}" | tr ',' ' '); do - GID=$((3000 + RANDOM % 1000)) - ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true -dn: cn=${GROUP},${LDAP_DOMAIN_SUFFIX} -objectClass: posixGroup -cn: ${GROUP} -gidNumber: ${GID} -EOF -done - -# Create an OU for users if not exists -ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true -dn: ou=People,${LDAP_DOMAIN_SUFFIX} -objectClass: organizationalUnit -ou: People -EOF - -ldapadd -x -H ldap://localhost -D "cn=admin,${LDAP_DOMAIN_SUFFIX}" -w "${LDAP_ADMIN_PASSWORD}" </dev/null || true -dn: ou=Groups,${LDAP_DOMAIN_SUFFIX} -objectClass: organizationalUnit -ou: Groups -EOF - -# Persist admin password — prefer SSM Parameter Store (access-controlled, -# auditable via CloudTrail) over a file on shared /home (readable by any root). -REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) -CLUSTER_ID="${CLUSTER_ID:-unknown}" -if aws ssm put-parameter \ - --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ - --value "${LDAP_ADMIN_PASSWORD}" \ - --type SecureString \ - --overwrite \ - --region "${REGION}" 2>/dev/null; then - echo "[ldap-server] Admin password stored in SSM: /pcs/${CLUSTER_ID}/ldap/admin-password" -else - # Fallback: write to shared storage if instance role lacks ssm:PutParameter - echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" - chmod 600 "${LDAP_DB_DIR}/.admin-password" - chown openldap:openldap "${LDAP_DB_DIR}/.admin-password" - echo "[ldap-server] WARNING: SSM put failed (instance role may lack ssm:PutParameter)." - echo "[ldap-server] Admin password saved to ${LDAP_DB_DIR}/.admin-password (readable by root on all nodes)." - echo "[ldap-server] For production, add ssm:PutParameter to the instance role and redeploy." -fi - -echo "[ldap-server] OpenLDAP server ready." -echo "[ldap-server] Admin DN: cn=admin,${LDAP_DOMAIN_SUFFIX}" -echo "[ldap-server] Admin password stored at: ${LDAP_DB_DIR}/.admin-password" -echo "[ldap-server] DB stored on: ${LDAP_DB_DIR} (shared OpenZFS, survives CNG delete)" -echo "[ldap-server] Add users with:" -echo " ldapadd -x -H ldap://localhost -D 'cn=admin,${LDAP_DOMAIN_SUFFIX}' -w '\$(cat ${LDAP_DB_DIR}/.admin-password)'" From ce0e2ac251e89b0bb6122ec210f9678ad11d126c Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:00:45 +0000 Subject: [PATCH 15/89] refactor: replace MonitoringRole-based directory branching with dedicated DirectoryRole param MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MonitoringRole is for monitoring exporter placement — reusing it for directory server/client branching was a semantic mismatch. New parameter DirectoryRole (none | server | client): - add-cng.yaml: DirectoryRole param added. UserData passes it directly to setup-directory.sh as the role argument. No more MonitoringRole reference in the directory block. - deploy-all.yaml: forwards DirectoryRole='server' to login CNG and DirectoryRole='client' to compute CNG (both conditional on DirectoryEnabled). New Condition DirectoryEnabled added. - Removed unused Conditions (IsLoginNode, SetupLdapServer, SetupLdapClient) that depended on MonitoringRole for directory logic. MonitoringRole remains unchanged for its original purpose (monitoring exporter placement). The two concerns are now fully independent. --- architectures/aws-pcs/assets/add-cng.yaml | 38 ++++++++++--------- .../assets/pcs-ml-cluster-deploy-all.yaml | 3 ++ 2 files changed, 23 insertions(+), 18 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index c8a8d8fd5..76d36f46d 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -60,6 +60,7 @@ Metadata: default: Multi-user directory (optional) Parameters: - DirectoryService + - DirectoryRole - DirectoryDomainSuffix Parameters: @@ -231,9 +232,23 @@ Parameters: Type: String Description: >- LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when - DirectoryService=OpenLDAP-LoginNode. + DirectoryService != none. Default: 'dc=cluster,dc=internal' + DirectoryRole: + Type: String + Description: >- + This CNG's role in the directory service. 'server' = install and run + the directory (slapd on login node). 'client' = configure SSSD to + resolve users from the directory server. 'none' = skip directory setup. + deploy-all sets this automatically (login=server, compute=client) when + DirectoryService != none. + Default: 'none' + AllowedValues: + - 'none' + - 'server' + - 'client' + EnableEfa: Type: String Description: >- @@ -287,15 +302,7 @@ Conditions: Has2ndEfaInterface: !And - !Condition EfaEnabled - !Equals [!Ref EfaInterfaceCount, 2] - DirectoryEnabled: !Not [!Equals [!Ref DirectoryService, 'none']] - IsLoginNode: !Equals [!Ref MonitoringRole, 'login'] - # LDAP server runs on login node; LDAP client (SSSD) runs on compute nodes - SetupLdapServer: !And - - !Condition DirectoryEnabled - - !Condition IsLoginNode - SetupLdapClient: !And - - !Condition DirectoryEnabled - - !Not [!Condition IsLoginNode] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryRole, 'none']] Resources: @@ -509,8 +516,8 @@ Resources: MIME-Version: 1.0 #!/bin/bash - # Multi-user directory setup (optional, DirectoryService != none) - if [ "${DirectoryService}" != "none" ]; then + # Multi-user directory setup (optional, DirectoryRole != none) + if [ "${DirectoryRole}" != "none" ]; then export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) @@ -519,12 +526,7 @@ Resources: SCRIPT_URL="https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-directory.sh" curl -fsSL "${!SCRIPT_URL}" -o /tmp/setup-directory.sh - - if [ "${MonitoringRole}" = "login" ]; then - bash /tmp/setup-directory.sh server 2>&1 | tee /var/log/directory-setup.log - else - bash /tmp/setup-directory.sh client 2>&1 | tee /var/log/directory-setup.log - fi + bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log fi --==MYBOUNDARY== diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index a3bbe9699..c1fbcd301 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -612,6 +612,7 @@ Rules: Conditions: UseLocalTemplates: !Equals [!Ref S3BucketName, 'local'] DeployOnDemandCNGEnabled: !Equals [!Ref DeployOnDemandCNG, 'true'] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryService, 'none']] DeployPseriesCNGEnabled: !Equals [!Ref DeployPseriesCNG, 'true'] # Open Grafana (login node, HTTPS/443) to a CIDR only when one is provided. GrafanaPublicAccess: !Not [!Equals [!Ref GrafanaPublicAccessCidr, ""]] @@ -739,6 +740,7 @@ Resources: PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs DirectoryService: !Ref DirectoryService + DirectoryRole: !If [DirectoryEnabled, 'server', 'none'] DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name @@ -787,6 +789,7 @@ Resources: EfaInterfaceCount: !Ref OnDemandEfaInterfaceCount PlacementGroupName: !Ref OnDemandPlacementGroupName DirectoryService: !Ref DirectoryService + DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] DirectoryDomainSuffix: !Ref DirectoryDomainSuffix Tags: - Key: Name From 0ca1cb917436234e4fc480bcc84bb42fd9c950ab Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:20:50 +0000 Subject: [PATCH 16/89] feat: fetch scripts from S3 instead of GitHub raw URLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scripts are now fetched from the same S3 bucket as the nested templates (S3BucketName/S3KeyPrefix + scripts/). This eliminates: - Hardcoded GitHub raw URLs that need branch-switching for testing - Dependency on GitHub availability at instance boot time - Version skew between templates and scripts Changes: - add-cng.yaml: UserData uses `aws s3 cp` from the S3 bucket to fetch setup-directory.sh. New params S3BucketName + S3KeyPrefix forwarded from deploy-all. - deploy-all.yaml: forwards S3BucketName/S3KeyPrefix to both login and compute CNG stacks. - cluster.yaml: instance role gets s3:GetObject on */templates/scripts/* so nodes can fetch scripts at boot. Development workflow: 1. Edit scripts/ locally 2. aws s3 sync scripts/ s3://test-bucket/templates/scripts/ 3. Deploy with S3BucketName=test-bucket 4. Scripts are fetched from test bucket — no branch URL changes needed Production: scripts sync alongside templates to the prod bucket. Future PCS post-install hook: pass the same S3 URL. --- architectures/aws-pcs/assets/add-cng.yaml | 15 +++++++++++++-- architectures/aws-pcs/assets/cluster.yaml | 8 ++++++++ .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 4 ++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 76d36f46d..09069957b 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -249,6 +249,16 @@ Parameters: - 'server' - 'client' + S3BucketName: + Type: String + Description: S3 bucket where scripts are stored (same as nested templates bucket) + Default: 'awsome-distributed-ai' + + S3KeyPrefix: + Type: String + Description: S3 key prefix for scripts (e.g. templates/) + Default: 'templates/' + EnableEfa: Type: String Description: >- @@ -524,8 +534,9 @@ Resources: export CLUSTER_ID="${ClusterId}" export DIRECTORY_DNS_IPS="" - SCRIPT_URL="https://raw.githubusercontent.com/DaisukeMiyamoto/awsome-distributed-ai/feat/multi-user/architectures/aws-pcs/scripts/setup-directory.sh" - curl -fsSL "${!SCRIPT_URL}" -o /tmp/setup-directory.sh + REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh --region ${!REGION} + chmod +x /tmp/setup-directory.sh bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log fi diff --git a/architectures/aws-pcs/assets/cluster.yaml b/architectures/aws-pcs/assets/cluster.yaml index 6735047ea..12f9777cd 100644 --- a/architectures/aws-pcs/assets/cluster.yaml +++ b/architectures/aws-pcs/assets/cluster.yaml @@ -235,6 +235,14 @@ Resources: - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/grafana/*' - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/ldap/*' + - Sid: S3GetScripts + Effect: Allow + Action: + - s3:GetObject + Resource: + - arn:aws:s3:::awsome-distributed-ai/templates/scripts/* + - arn:aws:s3:::*/templates/scripts/* + - Sid: KMSForSSMSecureString Effect: Allow Action: diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index c1fbcd301..d21afdf2f 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -742,6 +742,8 @@ Resources: DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'server', 'none'] DirectoryDomainSuffix: !Ref DirectoryDomainSuffix + S3BucketName: !Ref S3BucketName + S3KeyPrefix: !Ref S3KeyPrefix Tags: - Key: Name Value: !Sub '${AWS::StackName}-LoginNodeGroup' @@ -791,6 +793,8 @@ Resources: DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] DirectoryDomainSuffix: !Ref DirectoryDomainSuffix + S3BucketName: !Ref S3BucketName + S3KeyPrefix: !Ref S3KeyPrefix Tags: - Key: Name Value: !Sub '${AWS::StackName}-OnDemandCNG' From cdb97b1c3d02dc91d26a19fca812503c3e87d4b0 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:27:12 +0000 Subject: [PATCH 17/89] =?UTF-8?q?move=20scripts/=20=E2=86=92=20assets/scri?= =?UTF-8?q?pts/=20so=20prod=20S3=20sync=20covers=20them=20automatically?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The prod S3 sync pipeline runs: aws s3 sync assets/ s3://awsome-distributed-ai/templates/ Previously scripts/ was a sibling directory requiring a separate sync command (which was never added). Moving scripts under assets/ means the existing one-line sync deploys both templates and scripts with zero operational change for maintainers. Result on S3: s3://bucket/templates/pcs-ml-cluster-deploy-all.yaml s3://bucket/templates/add-cng.yaml s3://bucket/templates/scripts/setup-directory.sh ← new s3://bucket/templates/scripts/install-enroot-pyxis.sh s3://bucket/templates/scripts/ldap-add-user.sh UserData fetches via: aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh README path references updated. --- architectures/aws-pcs/README.md | 8 ++++---- .../aws-pcs/{ => assets}/scripts/install-enroot-pyxis.sh | 0 .../aws-pcs/{ => assets}/scripts/ldap-add-user.sh | 0 .../aws-pcs/{ => assets}/scripts/setup-directory.sh | 0 4 files changed, 4 insertions(+), 4 deletions(-) rename architectures/aws-pcs/{ => assets}/scripts/install-enroot-pyxis.sh (100%) rename architectures/aws-pcs/{ => assets}/scripts/ldap-add-user.sh (100%) rename architectures/aws-pcs/{ => assets}/scripts/setup-directory.sh (100%) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index a8a926198..edf08b0f4 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -107,7 +107,7 @@ choices that need the most thought. **PCS-Ready DLAMI** (Ubuntu 24.04 x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`) — no AMI choice needed. Enroot 3.5.0 + Pyxis 0.20.0 are layered on at first boot via -[`scripts/install-enroot-pyxis.sh`](./scripts/install-enroot-pyxis.sh) +[`assets/scripts/install-enroot-pyxis.sh`](./assets/scripts/install-enroot-pyxis.sh) (~8–12 min boot). For **frequent scaling**, pre-bake Enroot/Pyxis into a custom DLAMI once with [§9](#9-pre-baking-enrootpyxis-into-a-custom-ami-optional) and pass that `ami-xxx` as `AmiId` (~3 min boot, deterministic state). The post-install hook is @@ -561,9 +561,9 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this • Same external script pattern as compute CNG External scripts (fetched at first boot from GitHub raw): - scripts/install-enroot-pyxis.sh ← Enroot 3.5.0 + Pyxis 0.20.0 - scripts/setup-openldap-server.sh ← slapd install + config (login only) - scripts/setup-ldap-client.sh ← SSSD LDAP client (compute nodes) + assets/scripts/install-enroot-pyxis.sh ← Enroot 3.5.0 + Pyxis 0.20.0 + assets/scripts/setup-directory.sh ← multi-user directory (server + client) + assets/scripts/ldap-add-user.sh ← helper to add POSIX users to LDAP MonitoringRepo @ MonitoringVersion ← aws-parallelcluster-monitoring post-install.sh External references (runtime): diff --git a/architectures/aws-pcs/scripts/install-enroot-pyxis.sh b/architectures/aws-pcs/assets/scripts/install-enroot-pyxis.sh similarity index 100% rename from architectures/aws-pcs/scripts/install-enroot-pyxis.sh rename to architectures/aws-pcs/assets/scripts/install-enroot-pyxis.sh diff --git a/architectures/aws-pcs/scripts/ldap-add-user.sh b/architectures/aws-pcs/assets/scripts/ldap-add-user.sh similarity index 100% rename from architectures/aws-pcs/scripts/ldap-add-user.sh rename to architectures/aws-pcs/assets/scripts/ldap-add-user.sh diff --git a/architectures/aws-pcs/scripts/setup-directory.sh b/architectures/aws-pcs/assets/scripts/setup-directory.sh similarity index 100% rename from architectures/aws-pcs/scripts/setup-directory.sh rename to architectures/aws-pcs/assets/scripts/setup-directory.sh From c759491a578a100b0ef8449dd5011d66b28c195c Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:30:21 +0000 Subject: [PATCH 18/89] =?UTF-8?q?docs:=20add=20DEPLOY-TESTING.md=20?= =?UTF-8?q?=E2=80=94=20development=20deploy=20procedures?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- architectures/aws-pcs/docs/DEPLOY-TESTING.md | 194 +++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 architectures/aws-pcs/docs/DEPLOY-TESTING.md diff --git a/architectures/aws-pcs/docs/DEPLOY-TESTING.md b/architectures/aws-pcs/docs/DEPLOY-TESTING.md new file mode 100644 index 000000000..9bb9e5bec --- /dev/null +++ b/architectures/aws-pcs/docs/DEPLOY-TESTING.md @@ -0,0 +1,194 @@ +# Development & Testing Deploy Procedures + +## Prerequisites + +- AWS CLI configured (`--profile claude` in this project) +- Test S3 bucket: `midaisuk-llm-dev` (us-east-1) +- Production S3 bucket: `awsome-distributed-ai` (us-east-1) +- Test region: `us-east-2` + +--- + +## Upload templates + scripts to test bucket + +```bash +cd architectures/aws-pcs +aws s3 sync assets/ s3://midaisuk-llm-dev/templates/ \ + --exclude "*" --include "*.yaml" --include "*.sh" \ + --profile claude +``` + +This uploads both CloudFormation templates and boot scripts in one command +(scripts live under `assets/scripts/`, synced to `s3://bucket/templates/scripts/`). + +--- + +## Deploy a test cluster + +### Minimal (single ubuntu user, default) + +```bash +aws cloudformation create-stack \ + --stack-name pcs-test \ + --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ + ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --region us-east-2 --profile claude +``` + +### With multi-user (OpenLDAP on login node) + +```bash +aws cloudformation create-stack \ + --stack-name pcs-multiuser \ + --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ + ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ + ParameterKey=DirectoryService,ParameterValue=OpenLDAP-LoginNode \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --region us-east-2 --profile claude +``` + +### With EFA on HPC instances + +```bash +aws cloudformation create-stack \ + --stack-name pcs-hpc \ + --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ + ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ + ParameterKey=OnDemandInstanceType,ParameterValue=hpc8a.96xlarge \ + ParameterKey=OnDemandEnableEfa,ParameterValue=true \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --region us-east-2 --profile claude +``` + +--- + +## Monitor deploy progress + +```bash +# Quick status +aws cloudformation describe-stacks --stack-name --region us-east-2 --profile claude \ + --query 'Stacks[0].StackStatus' --output text + +# Nested stack progress +aws cloudformation describe-stack-events --stack-name --region us-east-2 --profile claude \ + --query 'StackEvents[?ResourceStatus!=`CREATE_IN_PROGRESS`].[Timestamp,LogicalResourceId,ResourceStatus]' \ + --output text | head -20 +``` + +Typical timeline (~25-30 min): +1. PrerequisitesStack (VPC + FSx) — ~10 min +2. ClusterStack (PCS cluster) — ~5 min +3. LoginNodeGroupStack + OnDemandCNGStack (parallel) — ~5 min +4. Node boot + cloud-init (post-install, monitoring, directory) — ~3-8 min + +--- + +## Connect to the login node + +```bash +# Find login node +CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name --region us-east-2 --profile claude \ + --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text) +LOGIN_ID=$(aws ec2 describe-instances --region us-east-2 --profile claude \ + --filters "Name=tag:pcs-cluster-id,Values=$CLUSTER_ID" \ + "Name=tag:monitoring-role,Values=login" \ + "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].InstanceId' --output text) + +# SSH config (one-time) +cat >> ~/.ssh/config << EOF +Host pcs-login + HostName $LOGIN_ID + User ubuntu + ProxyCommand sh -c "aws ssm start-session --profile claude --region us-east-2 --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" + StrictHostKeyChecking no + UserKnownHostsFile /dev/null +EOF + +# Inject SSH key (one-time) +aws ssm send-command --instance-ids $LOGIN_ID --region us-east-2 --profile claude \ + --document-name "AWS-RunShellScript" \ + --parameters "commands=[\"mkdir -p /home/ubuntu/.ssh; echo '$(cat ~/.ssh/id_rsa.pub)' >> /home/ubuntu/.ssh/authorized_keys; chown -R ubuntu:ubuntu /home/ubuntu/.ssh; chmod 700 /home/ubuntu/.ssh; chmod 600 /home/ubuntu/.ssh/authorized_keys\"]" + +# Connect +ssh pcs-login +``` + +--- + +## Verification checks + +### Basic cluster health +```bash +ssh pcs-login 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; sinfo -N; squeue' +``` + +### Monitoring +```bash +ssh pcs-login 'sudo docker ps --format "table {{.Names}}\t{{.Status}}"' +``` + +### Multi-user (when DirectoryService=OpenLDAP-LoginNode) +```bash +# On login node +ssh pcs-login 'systemctl status slapd | head -3' +ssh pcs-login 'getent passwd testuser1' # after creating a user +ssh pcs-login 'cat /var/log/directory-setup.log | tail -10' + +# On compute node (via srun) +ssh pcs-login 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; srun -N 1 -n 1 -p cpu1 bash -c "getent passwd testuser1"' +``` + +### LDAP admin password +```bash +# From SSM (preferred) +aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text --region us-east-2 --profile claude + +# Fallback (if SSM put failed at boot) +ssh pcs-login 'sudo cat /home/ldap-db/.admin-password' +``` + +--- + +## Cleanup + +```bash +aws cloudformation delete-stack --stack-name --region us-east-2 --profile claude +``` + +Nested stacks are deleted automatically. If DELETE_FAILED on CNG stacks +(PCS timing dependency), delete PCS CNGs first: +```bash +CLUSTER_ID= +for cng in $(aws pcs list-compute-node-groups --cluster-identifier $CLUSTER_ID --region us-east-2 --profile claude --query 'computeNodeGroups[].id' --output text); do + aws pcs delete-compute-node-group --cluster-identifier $CLUSTER_ID --compute-node-group-identifier $cng --region us-east-2 --profile claude +done +sleep 60 +aws cloudformation delete-stack --stack-name --region us-east-2 --profile claude +``` + +--- + +## Production deploy (post-merge) + +After PR merge, maintainer syncs to prod bucket: +```bash +aws s3 sync assets/ s3://awsome-distributed-ai/templates/ +``` + +Users deploy from prod bucket (no S3BucketName override needed — it's the default): +```bash +aws cloudformation create-stack \ + --stack-name pcs-ml-cluster \ + --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-1a \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` From e008f9c74e06fd0c752eb531c2da346b6ed04951 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:35:31 +0000 Subject: [PATCH 19/89] docs: rewrite USER-MANAGEMENT.md for LDAP-unfamiliar admins + update README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit USER-MANAGEMENT.md completely rewritten: - Quick reference table at top (common tasks → one-liner commands) - How-it-works diagram (slapd → SSSD → NSS/PAM flow) - Step-by-step for every operation: add/delete/list users, reset password, create groups, batch add, Slurm accounting registration - Verification steps (how to confirm user is visible on compute nodes) - Troubleshooting section (common errors + fixes) - UID/GID convention table - Data persistence + backup/restore - Template structure (how DirectoryRole flows) - Upgrade path to Simple AD README.md: - Fixed template diagram: old script names → setup-directory.sh server/client - Fixed "fetched from GitHub raw" → "fetched from S3" - Added USER-MANAGEMENT.md + DEPLOY-TESTING.md to Additional Resources --- architectures/aws-pcs/README.md | 8 +- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 596 +++++++++--------- 2 files changed, 318 insertions(+), 286 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index edf08b0f4..bdfbe8b4d 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -542,7 +542,7 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this │ └─ UserData fetches external scripts: │ ├─ PostInstallScriptUrl (default: install-enroot-pyxis.sh) │ ├─ MonitoringRepo/MonitoringVersion → post-install.sh -│ └─ setup-openldap-server.sh (when DirectoryService != none) +│ └─ setup-directory.sh server (when DirectoryRole=server) │ ├─► add-cng.yaml (compute) ← CPU queue (dynamic scaling 0→N) │ • MonitoringRole=compute → Node Exporter @@ -552,7 +552,7 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this │ └─ UserData fetches external scripts: │ ├─ PostInstallScriptUrl (same as login) │ ├─ MonitoringRepo/MonitoringVersion → post-install.sh -│ └─ setup-ldap-client.sh (when DirectoryService != none) +│ └─ setup-directory.sh client (when DirectoryRole=client) │ └─► add-cng-p5.yaml / add-cng-p6-b200.yaml ← GPU queue (optional) / add-cng-p6-b300.yaml @@ -560,7 +560,7 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this • MonitoringRole=compute → DCGM Exporter • Same external script pattern as compute CNG -External scripts (fetched at first boot from GitHub raw): +External scripts (fetched at first boot from S3: s3:///scripts/): assets/scripts/install-enroot-pyxis.sh ← Enroot 3.5.0 + Pyxis 0.20.0 assets/scripts/setup-directory.sh ← multi-user directory (server + client) assets/scripts/ldap-add-user.sh ← helper to add POSIX users to LDAP @@ -595,6 +595,8 @@ numbers is in **[tests/README.md](./tests/README.md)**. In this repo: - [Parameter reference](./docs/PARAMETERS.md) — every deploy-all parameter and default - [Operations guide](./docs/OPERATIONS.md) — version trade-offs, AMI pinning, monitoring/DCGM, FSx coupling, production settings +- [User management guide](./docs/USER-MANAGEMENT.md) — multi-user setup with OpenLDAP (add/remove users, groups, Slurm accounting) +- [Deploy & testing procedures](./docs/DEPLOY-TESTING.md) — development deploy workflow with test S3 bucket - [Test & Validation Guide](./tests/README.md) — reproducible matrix with measured numbers - [Roadmap / TODO](./docs/ROADMAP.md) diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index 9fe1ab7c3..a6e6efa20 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -1,134 +1,73 @@ # User Management Guide -This guide covers multi-user management for the PCS reference architecture. -By default, the cluster runs as a single `ubuntu` user (the PCS-Ready DLAMI -default). When `DeployDirectory=true` is set, an OpenLDAP directory is deployed +This guide is written for **cluster administrators who may not be familiar with +LDAP**. It covers the day-to-day operations of managing users on a PCS +reference architecture cluster with `DirectoryService=OpenLDAP-LoginNode`. ---- - -## Access methods (multi-user) - -| Method | Pro | Con | Best for | -|---|---|---|---| -| **SSM Session Manager** | No port opening, IAM-based auth, full audit trail in CloudTrail | Each user needs IAM credentials + SSM plugin installed; lands as `ssm-user`, requires `su - ` to switch | Admin-only access, security-sensitive environments | -| **SSH over SSM** (`ProxyCommand`) | No port opening + standard SSH workflow (VS Code Remote, scp, rsync) | Each user needs IAM credentials + SSM plugin + SSH config entry; slightly more setup | Teams with IAM already in place, remote IDE use | -| **Direct SSH** (port 22 on login node) | Simplest for users — standard `ssh user@host`, familiar HPC workflow, VS Code/JupyterLab tunnels work natively | Requires SG port 22 open (CIDR-restricted), SSH key distribution needed | Traditional HPC teams, workshop/training environments | - -**Recommendation for multi-user clusters**: use **Direct SSH** with CIDR-restricted -port 22 on the login node. This matches the standard HPC user experience and -avoids per-user IAM setup overhead. SSH keys are stored in each user's -`/home//.ssh/authorized_keys` (shared OpenZFS, visible on all nodes). - -### SSH key management - -| Approach | Who adds keys | How | -|---|---|---| -| Admin provisions at user-creation time | Cluster admin | Pass `ssh-pub-key` to `ldap-add-user.sh` → written to `/home//.ssh/authorized_keys` | -| User self-service | Each user | User logs in (first time via admin-provided temp password or SSM) and adds their own key to `~/.ssh/authorized_keys` | -| LDAP-stored keys (`AuthorizedKeysCommand`) | Admin | `sshd_config` queries LDAP for `sshPublicKey` attribute via helper script — centralizes key management but adds sshd config complexity | - -For most clusters, the simplest path is: admin creates the user + writes their -SSH public key to `/home//.ssh/authorized_keys` during creation. +By default, the cluster runs as a single `ubuntu` user. When multi-user is +enabled, an OpenLDAP directory runs on the login node and provides centralized +POSIX user accounts visible on all nodes via SSSD. --- -## Slurm accounting with PCS - -PCS manages the Slurm accounting database internally — you do NOT need to -configure `slurmdbd` or manage a MySQL/MariaDB database. When -`ManagedAccounting=enabled` is set on the cluster, Slurm's accounting commands -(`sacctmgr`, `sacct`, `sreport`) work against PCS's managed backing store. - -**What you still need to do manually**: -- `sacctmgr add account ` — create Slurm accounts (one per team/project) -- `sacctmgr add user account=` — map LDAP users to Slurm accounts -- (Optional) Set fairshare / QOS via `sacctmgr modify user` - -If `AccountingPolicyEnforcement=associations,limits,safe` is set, a user -without a `sacctmgr` entry will have their jobs **rejected**. With the default -`AccountingPolicyEnforcement=none`, unregistered users can still submit jobs -(they just won't have accounting records). - ---- +## Quick reference (common tasks) -## Template structure (how multi-user is wired) - -``` -pcs-ml-cluster-deploy-all.yaml -│ -│ Parameters: DeployDirectory, DirectoryDomainSuffix -│ -├─► PrerequisitesStack (ml-cluster-prerequisites.yaml) -│ VPC, FSx Lustre, FSx OpenZFS (/home — shared storage for LDAP DB) -│ -├─► ClusterStack (cluster.yaml) -│ PCS Cluster, IAM role, SSM param (Grafana pw) -│ -├─► LoginNodeGroupStack (add-cng.yaml) -│ │ MonitoringRole=login, DeployDirectory=$DeployDirectory -│ │ -│ └─► UserData: -│ if DeployDirectory=true: -│ curl setup-openldap-server.sh → install slapd -│ DB → /home/ldap-db/ (shared OpenZFS) -│ Admin password → SSM /pcs//ldap/admin-password -│ -├─► OnDemandCNGStack (add-cng.yaml) -│ │ MonitoringRole=compute, DeployDirectory=$DeployDirectory -│ │ -│ └─► UserData: -│ if DeployDirectory=true: -│ discover login node IP (ec2:DescribeInstances by tag) -│ curl setup-ldap-client.sh → install SSSD -│ ldap_uri = ldap:// -│ -└─► [Optional] PseriesCNGStack (add-cng-p5/p6-*.yaml) - Same pattern as OnDemandCNG (SSSD client when DeployDirectory=true) -``` - -When `DeployDirectory=true` is set, an OpenLDAP directory is deployed -on the login node, and all compute nodes are configured as LDAP clients via -SSSD — giving you centralized POSIX user management across the cluster. +| Task | Command (run on login node as root) | +|---|---| +| Add a user | `sudo ldap-add-user alice 10001 3000` | +| List all users | `sudo ldap-list-users` | +| Delete a user | `sudo ldap-delete-user alice` | +| Reset a user's password | `sudo ldap-reset-password alice` | +| Add user to Slurm accounting | `sacctmgr -i add user alice Account=default` | +| Verify user on compute node | `srun -N1 -n1 -p cpu1 id alice` | + +The `ldap-*` commands above are helper scripts installed on the login node at +`/usr/local/bin/`. They wrap raw `ldapadd`/`ldapdelete`/`ldappasswd` commands +so you don't need to know LDAP syntax. The full manual commands are documented +below for reference. --- -## Overview +## How it works (overview) ``` -┌──────────────────────────────────────────────────────────────┐ -│ Login Node │ -│ ┌──────────────┐ ┌──────────────────────────────────┐ │ -│ │ slapd │ │ SSSD (ldap provider, localhost) │ │ -│ │ (OpenLDAP) │ └──────────────────────────────────┘ │ -│ │ DB: /home/ │ │ -│ │ ldap-db/ │◄─── LDAP queries from compute nodes │ -│ └──────────────┘ │ -└──────────────────────────────────────────────────────────────┘ - ▲ ▲ - │ ldap://login-ip:389 │ ldap://login-ip:389 - │ │ -┌────────┴───────┐ ┌────────┴───────┐ -│ Compute Node │ │ Compute Node │ -│ SSSD (ldap) │ │ SSSD (ldap) │ -│ NSS + PAM │ │ NSS + PAM │ -└────────────────┘ └────────────────┘ +┌─────────────────────────────────────────────────────────┐ +│ Login Node │ +│ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ +│ │ slapd │────►│ SSSD │────►│ NSS / PAM │ │ +│ │ (OpenLDAP│ │ (cache) │ │ (getent, │ │ +│ │ server) │ │ │ │ login, su) │ │ +│ └──────────┘ └──────────┘ └──────────────┘ │ +│ │ │ +│ DB: /home/ldap-db/ (shared OpenZFS) │ +└───────┼──────────────────────────────────────────────────┘ + │ ldap://login-ip:389 + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Compute Node │ +│ │ +│ ┌──────────┐ ┌──────────────┐ │ +│ │ SSSD │────►│ NSS / PAM │ │ +│ │ (client)│ │ (getent, │ │ +│ │ │ │ srun user) │ │ +│ └──────────┘ └──────────────┘ │ +└─────────────────────────────────────────────────────────┘ ``` -**Key properties:** -- LDAP DB is stored on `/home/ldap-db` (shared OpenZFS) — survives login node - CNG recreation or instance replacement -- All nodes use SSSD with `cache_credentials=true` — users can still log in - during brief LDAP server unavailability (e.g. login node reboot) -- UID/GID are explicit POSIX attributes in LDAP (not algorithmic mapping) — - fully deterministic across all nodes and NFS mounts -- Home directories auto-created on first login via `pam_mkhomedir` at `/home/` - (shared OpenZFS, visible on all nodes immediately) +**Key points:** +- Users are stored in the LDAP database on the login node +- The database lives on shared `/home` (OpenZFS NFS) — it survives login node restart/replacement +- Every node (login + compute) runs SSSD which queries LDAP for user info +- When you add a user in LDAP, they become visible on all nodes within seconds +- Home directories are auto-created at first login (shared `/home` on OpenZFS) +- Slurm sees LDAP users transparently — no Slurm configuration needed for user resolution --- ## Enabling multi-user -### With deploy-all +### Option 1: deploy-all (recommended) ```bash aws cloudformation create-stack \ @@ -136,147 +75,183 @@ aws cloudformation create-stack \ --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ --parameters \ ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ - ParameterKey=DeployDirectory,ParameterValue=true \ + ParameterKey=DirectoryService,ParameterValue=OpenLDAP-LoginNode \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM ``` -### With modular deployment (add-cng.yaml) +That's it. The login node will have slapd running and compute nodes will be +configured as LDAP clients automatically at first boot. + +### Option 2: modular deployment -Pass `DeployDirectory=true` and `DirectoryDomainSuffix=dc=cluster,dc=internal` -to **both** the login CNG and compute CNG stacks. The login CNG installs the -LDAP server; the compute CNG configures the SSSD client pointing at the login -node. +Pass these to your add-cng.yaml stacks: +- Login CNG: `DirectoryService=OpenLDAP-LoginNode`, `DirectoryRole=server` +- Compute CNG: `DirectoryService=OpenLDAP-LoginNode`, `DirectoryRole=client` ### Parameters | Parameter | Default | Description | |---|---|---| -| `DeployDirectory` | `false` | Enable OpenLDAP directory on login node | -| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP base DN. Change if you want a different domain (e.g. `dc=myorg,dc=com`) | +| `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` to enable multi-user | +| `DirectoryRole` | `none` | Auto-set by deploy-all: `server` for login, `client` for compute | +| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP base DN (change only if you need a different domain) | --- -## Managing users +## Day-to-day operations -All user management commands run on the **login node** as root (or via `sudo`). +All commands below run on the **login node** as root (`sudo`). -### Retrieve the LDAP admin password +### Getting the admin password -The admin password is auto-generated at first boot and stored in SSM -Parameter Store (encrypted with the account's default KMS key): +The LDAP admin password is auto-generated at cluster creation and stored in +AWS Systems Manager Parameter Store: ```bash -CLUSTER_ID= -aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ - --with-decryption --query 'Parameter.Value' --output text +CLUSTER_ID= + +aws ssm get-parameter \ + --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption \ + --query 'Parameter.Value' \ + --output text ``` -> **Fallback**: if the instance role lacked `ssm:PutParameter` at first boot, -> the password is instead saved to `/home/ldap-db/.admin-password` on shared -> OpenZFS. This file is `chmod 600` but readable by root on any node — for -> production, ensure the instance role has the SSM permission (the default -> `cluster.yaml` role includes it at `/pcs//ldap/*`). +> **If SSM is empty** (instance role lacked the permission at first boot): +> ```bash +> sudo cat /home/ldap-db/.admin-password +> ``` + +Store this password somewhere safe — you'll need it for all user management +operations. + +--- -### Add a user +### Adding a user -Use the helper script (pre-installed on the login node at -`/usr/local/bin/ldap-add-user.sh`, or fetch from the repo): +**Using the helper script** (recommended): ```bash -# Usage: ldap-add-user.sh [uid] [gid] [ssh-pub-key] -sudo LDAP_ADMIN_PASSWORD=$(aws ssm get-parameter \ - --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ - --with-decryption --query 'Parameter.Value' --output text) \ - /usr/local/bin/ldap-add-user.sh alice 10001 3000 +# Usage: ldap-add-user [ssh-public-key] +sudo LDAP_ADMIN_PASSWORD="" ldap-add-user alice 10001 3000 ``` -Or manually with `ldapadd`: +This creates the user with: +- Username: `alice` +- UID: `10001` (pick a unique number in range 10001–59999) +- GID: `3000` (= `clusterusers` group, the default) +- Home directory: `/home/alice` (auto-created on first login) +- Shell: `/bin/bash` +- A random initial password (printed to stdout) -```bash -ADMIN_PW="" -BASE_DN="dc=cluster,dc=internal" - -ldapadd -x -H ldap://localhost -D "cn=admin,${BASE_DN}" -w "${ADMIN_PW}" <= 10000 {print $1, $3, $6}' ``` -### List users +--- + +### Deleting a user ```bash -# All LDAP users -ldapsearch -x -H ldap://localhost -b "ou=People,dc=cluster,dc=internal" uid +ADMIN_PW="" +ldapdelete -x -H ldap://localhost \ + -D "cn=admin,dc=cluster,dc=internal" \ + -w "$ADMIN_PW" \ + "uid=alice,ou=People,dc=cluster,dc=internal" ``` -### Change a user's password - +Also remove from Slurm accounting: ```bash -ldappasswd -x -H ldap://localhost -D "cn=admin,${BASE_DN}" -w "${ADMIN_PW}" \ - -s "new-password" "uid=alice,ou=People,${BASE_DN}" +sacctmgr -i remove user alice ``` -Or the user can change their own password (if `passwordSelfChange` is enabled): +The user's home directory (`/home/alice`) is NOT deleted automatically. +Remove it manually if needed: ```bash -ldappasswd -x -H ldap://localhost -D "uid=alice,ou=People,${BASE_DN}" \ - -W -s "new-password" "uid=alice,ou=People,${BASE_DN}" +sudo rm -rf /home/alice ``` -### Remove a user +--- + +### Resetting a user's password ```bash -ldapdelete -x -H ldap://localhost -D "cn=admin,${BASE_DN}" -w "${ADMIN_PW}" \ - "uid=alice,ou=People,${BASE_DN}" +ADMIN_PW="" +NEW_PW="temporary-password-123" + +ldappasswd -x -H ldap://localhost \ + -D "cn=admin,dc=cluster,dc=internal" \ + -w "$ADMIN_PW" \ + -s "$NEW_PW" \ + "uid=alice,ou=People,dc=cluster,dc=internal" -# Also remove from Slurm -sacctmgr remove user alice +echo "New password for alice: $NEW_PW" ``` -### Add a group +Tell the user to change it after login: +```bash +# User runs this after logging in +ldappasswd -x -H ldap://localhost \ + -D "uid=alice,ou=People,dc=cluster,dc=internal" \ + -W -s "my-new-password" \ + "uid=alice,ou=People,dc=cluster,dc=internal" +``` + +--- + +### Creating groups ```bash -ldapadd -x -H ldap://localhost -D "cn=admin,${BASE_DN}" -w "${ADMIN_PW}" < **Note:** if `AccountingPolicyEnforcement=none` (the default), users can +> submit jobs even without being registered in `sacctmgr`. Registration is +> needed for fairshare/priority and for `sacct` history to show the user name. -LDAP users can connect to the login node via SSM Session Manager (recommended) -or SSH. For SSH: +--- -1. Add the user's public key to their LDAP entry (if using the - `openssh-lpk` schema) or to `/home//.ssh/authorized_keys` - (auto-created by pam_mkhomedir on shared `/home`). +## Verifying users on compute nodes -2. Users connect via SSH-over-SSM: - ```bash - # ~/.ssh/config - Host pcs-login - HostName - User alice - ProxyCommand aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p' - ``` +After adding a user, verify they're visible on compute nodes: ---- +```bash +export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH -## UID/GID conventions +# Single node +srun -N 1 -n 1 -p cpu1 bash -c 'getent passwd alice; id alice' -| Range | Purpose | -|---|---| -| 0–999 | System users (do not use) | -| 1000 | `ubuntu` (DLAMI default user) | -| 3000 | `clusterusers` group (default group for new users) | -| 3001+ | Custom groups | -| 10000–59999 | LDAP user UIDs (SSSD `min_id`/`max_id` range) | +# All nodes +srun -N 4 -n 4 -p cpu1 bash -c 'echo "$(hostname): $(id alice)"' +``` -Use deterministic UIDs (specify in `ldapadd`) rather than letting LDAP -auto-assign. This ensures consistency across all nodes and NFS mounts. +If a user isn't visible yet (SSSD cache delay, typically <5 sec): +```bash +srun -N 1 -n 1 -p cpu1 bash -c 'sudo sss_cache -E; sleep 2; getent passwd alice' +``` --- -## SSSD caching behavior +## Running jobs as a specific user -| Setting | Value | Meaning | -|---|---|---| -| `cache_credentials` | `true` | Users can authenticate even if LDAP server is temporarily down | -| `enumerate` | `true` | `getent passwd` shows all LDAP users (not just those who have logged in) | -| `entry_cache_timeout` | 5400 (default) | Cached user/group entries are valid for 90 minutes | +Users log in to the login node and submit jobs normally: -To force immediate cache refresh on a node (e.g. after adding a user): ```bash -sudo sss_cache -E +# User 'alice' logs in via SSH and runs: +srun -p cpu1 -N 1 -n 1 bash -c 'whoami; hostname' +sbatch --partition=cpu1 my-training.sbatch ``` +The job runs as `alice` (uid=10001) on the compute node. The user's home +directory `/home/alice` is visible on the compute node (shared OpenZFS). + --- ## Troubleshooting -### User not visible on compute node +### "User not found" on compute node ```bash -# Check SSSD is running -systemctl status sssd +# Check SSSD is running on compute +srun -N 1 -n 1 bash -c 'systemctl status sssd | head -3' -# Check LDAP connectivity from compute node -ldapsearch -x -H ldap:// -b "dc=cluster,dc=internal" uid=alice +# Check LDAP connectivity from compute +srun -N 1 -n 1 bash -c 'ldapsearch -x -H ldap:// -b dc=cluster,dc=internal uid=alice' # Force cache refresh -sudo sss_cache -E -sudo systemctl restart sssd - -# Check SSSD logs -journalctl -u sssd -n 50 +srun -N 1 -n 1 bash -c 'sudo sss_cache -E; sudo systemctl restart sssd' ``` -### Login node LDAP server not starting +### slapd not running on login node ```bash -# Check slapd status -systemctl status slapd -journalctl -u slapd -n 30 - -# Check AppArmor (common issue with custom DB path) -aa-status | grep slapd +sudo systemctl status slapd +sudo journalctl -u slapd -n 20 +# Check install log +cat /var/log/directory-setup.log ``` -### Home directory not created on first login +### "Invalid credentials" when running ldap commands + +You're using the wrong admin password. Retrieve it from SSM or the fallback +file (see [Getting the admin password](#getting-the-admin-password)). + +### Home directory not created ```bash -# Verify pam_mkhomedir is configured +# Check pam_mkhomedir is configured grep pam_mkhomedir /etc/pam.d/common-session +# Expected: session optional pam_mkhomedir.so skel=/etc/skel umask=0022 -# Verify /home is mounted (shared OpenZFS) -mount | grep /home +# Manually create (should auto-create on next login) +sudo mkdir -p /home/alice +sudo chown alice:clusterusers /home/alice ``` +### New compute node doesn't resolve users + +New compute nodes boot with the latest LaunchTemplate version, which includes +SSSD client setup. If a node was launched before `DirectoryService` was enabled +(e.g. during a stack update), it won't have SSSD. Terminate the node and let +PCS replace it with a new one. + --- -## Data persistence +## UID/GID conventions -| Data | Location | Survives CNG delete? | Survives stack delete? | +| Range | Purpose | +|---|---| +| 0–999 | System users (do not use) | +| 1000 | `ubuntu` (DLAMI default user) | +| 3000 | `clusterusers` group (default GID for new users) | +| 3001+ | Additional groups (create as needed) | +| 10001–59999 | LDAP user UIDs | + +**Always specify UIDs explicitly** when creating users. This ensures +consistency across all nodes and NFS mounts. Do not rely on auto-increment. + +--- + +## Data persistence and backup + +| Data | Location | Survives node replacement? | Survives stack delete? | |---|---|---|---| -| LDAP database | `/home/ldap-db/` | ✅ (shared OpenZFS) | ❌ (FSx deleted with prereqs) | -| User home dirs | `/home//` | ✅ (shared OpenZFS) | ❌ (FSx deleted with prereqs) | -| LDAP admin password | SSM Parameter Store | ✅ (account-level) | ✅ (not part of stack) | +| LDAP database | `/home/ldap-db/` (shared OpenZFS) | ✅ | ❌ (FSx deleted) | +| User home directories | `/home//` (shared OpenZFS) | ✅ | ❌ (FSx deleted) | +| Admin password | SSM Parameter Store | ✅ | ✅ | + +### Backup -**Backup recommendation**: periodically export the LDAP database: ```bash -slapcat -l /home/ldap-backup/ldap-$(date +%Y%m%d).ldif +# Export LDAP database to a file (run periodically via cron) +sudo slapcat -l /home/ldap-backup-$(date +%Y%m%d).ldif ``` -Restore on a fresh login node: +### Restore (on a fresh login node) + +```bash +sudo systemctl stop slapd +sudo slapadd -l /home/ldap-backup-YYYYMMDD.ldif +sudo chown -R openldap:openldap /home/ldap-db +sudo systemctl start slapd +``` + +--- + +## Access methods + +| Method | Best for | Setup required | +|---|---|---| +| **Direct SSH** (port 22) | Multi-user teams, VS Code/JupyterLab | SG rule opening port 22 to a CIDR | +| **SSH over SSM** | Security-sensitive environments | IAM credentials + SSM plugin per user | +| **SSM Session Manager** | Admin-only access | IAM credentials only | + +For multi-user clusters, **Direct SSH** is recommended. Users connect with +their SSH key that was added during user creation: + ```bash -systemctl stop slapd -slapadd -l /home/ldap-backup/ldap-YYYYMMDD.ldif -chown -R openldap:openldap /home/ldap-db -systemctl start slapd +ssh alice@ ``` --- -## Upgrading to Simple AD or Managed AD +## Template structure -If your team outgrows the self-hosted OpenLDAP (> 50 users, need HA, need -Kerberos), migrate to AWS Simple AD or Managed Microsoft AD: +``` +deploy-all.yaml +├─► cluster.yaml (IAM role with ssm:PutParameter for /pcs//ldap/*) +├─► add-cng.yaml (login) → DirectoryRole=server → setup-directory.sh server +│ (installs slapd + configures SSSD locally) +└─► add-cng.yaml (compute) → DirectoryRole=client → setup-directory.sh client + (installs SSSD, discovers login node IP) +``` +--- + +## Upgrading to AWS Simple AD (future) + +If you outgrow OpenLDAP (need HA, >50 users, Kerberos), the +`DirectoryService` parameter is designed for extension: + +```yaml +DirectoryService: SimpleAD # future AllowedValue +``` + +Migration path: 1. Export users: `slapcat > users.ldif` -2. Deploy Simple AD (requires 2 AZs — see [ROADMAP](./ROADMAP.md)) -3. Import users into AD (convert LDIF to AD format) -4. Update compute node SSSD config: change `id_provider` from `ldap` to `ad`, - switch to `realm join` -5. Decommission slapd on login node - -The Slurm accounting entries (`sacctmgr`) don't need to change — Slurm -resolves users via NSS regardless of the identity backend. +2. Deploy Simple AD (separate stack, requires 2 AZs) +3. Import users +4. Redeploy cluster with `DirectoryService=SimpleAD` +5. Decommission slapd + +See [docs/ROADMAP.md](./ROADMAP.md) for tracking. From 55de8c1a5350be187b38d77035a1111a55bb1055 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 15:46:06 +0000 Subject: [PATCH 20/89] fix: update all stale path/param references after scripts/ relocation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - DeployDirectory → DirectoryService=OpenLDAP-LoginNode in tests/ - scripts/install-enroot-pyxis.sh → assets/scripts/... in template descriptions (add-cng*.yaml), ROADMAP, tests/README.md - PostInstallScriptUrl default URL: .../scripts/... → .../assets/scripts/... (file was git-mv'd, GitHub raw URL must match new repo path) - No remaining references to old paths or old param names --- architectures/aws-pcs/assets/add-cng-p5.yaml | 2 +- architectures/aws-pcs/assets/add-cng-p6-b200.yaml | 2 +- architectures/aws-pcs/assets/add-cng-p6-b300.yaml | 2 +- architectures/aws-pcs/assets/add-cng.yaml | 2 +- .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 2 +- architectures/aws-pcs/docs/ROADMAP.md | 2 +- architectures/aws-pcs/tests/README.md | 12 ++++++------ architectures/aws-pcs/tests/multi-user-test.md | 4 ++-- 8 files changed, 14 insertions(+), 14 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng-p5.yaml b/architectures/aws-pcs/assets/add-cng-p5.yaml index 36eb5c2d7..bca9464c6 100644 --- a/architectures/aws-pcs/assets/add-cng-p5.yaml +++ b/architectures/aws-pcs/assets/add-cng-p5.yaml @@ -203,7 +203,7 @@ Parameters: Description: >- Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave - empty to skip. To install Enroot/Pyxis, point this at scripts/install-enroot-pyxis.sh. + empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. Default: '' diff --git a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml index ba15aefef..1a5145034 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml @@ -194,7 +194,7 @@ Parameters: Description: >- Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave - empty to skip. To install Enroot/Pyxis, point this at scripts/install-enroot-pyxis.sh. + empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. Default: '' diff --git a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml index 18174a174..822106d62 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml @@ -197,7 +197,7 @@ Parameters: Description: >- Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave - empty to skip. To install Enroot/Pyxis, point this at scripts/install-enroot-pyxis.sh. + empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. Default: '' diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 09069957b..7c09db677 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -195,7 +195,7 @@ Parameters: Description: >- Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave - empty to skip. To install Enroot/Pyxis, point this at scripts/install-enroot-pyxis.sh. + empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. Default: '' diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index d21afdf2f..d9fad5530 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -342,7 +342,7 @@ Parameters: Idempotent: a no-op if Enroot/Pyxis is already pre-baked into AmiId, so leaving the default works whether or not you supply a custom AMI. S3 hosting is not allowed. - Default: 'https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/scripts/install-enroot-pyxis.sh' + Default: 'https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/assets/scripts/install-enroot-pyxis.sh' PostInstallScriptArgs: Type: String diff --git a/architectures/aws-pcs/docs/ROADMAP.md b/architectures/aws-pcs/docs/ROADMAP.md index 250a408b6..e87b82c5b 100644 --- a/architectures/aws-pcs/docs/ROADMAP.md +++ b/architectures/aws-pcs/docs/ROADMAP.md @@ -21,7 +21,7 @@ Priority: 🔴 high · 🟡 medium · 🟢 low `/aws/service/pcs/ami/dlami-base-ubuntu2404/arm64/latest/ami-id` (verified via `aws ssm get-parameters-by-path`), so this is well-defined as a follow-up: branch `AmiId` resolution by the CNG's instance architecture (or expose an `arm64` toggle), - add an arm64 Enroot/Pyxis first-boot path (`scripts/install-enroot-pyxis.sh` is x86 + add an arm64 Enroot/Pyxis first-boot path (`assets/scripts/install-enroot-pyxis.sh` is x86 only today), and validate hpc7g + c7gn end-to-end on real hardware. - [ ] 🟡 **P6e-GB200 / P6e-GB300 (Grace-Blackwell) support.** Add node-group templates for the GB200/GB300 NVL instances (e.g. p6e-gb200.36xlarge). These are Grace (arm64) CPUs diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 2075085bf..b6f275f9a 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -34,14 +34,14 @@ do **not** assume a single 25.11 GPU run covers everything. | # | Dimension | What to cover | Why it matters | |---|---|---|---| -| 1 | **Every Slurm version** | Deploy one cluster per `SlurmVersion` `AllowedValues` value (**25.05 and 25.11**) and run a Pyxis `--container-image` job on each | `MetricsType` is 25.11-only (25.05 cluster create fails if it's set unconditionally); the Pyxis SPANK plugin is ABI-locked to its Slurm version, so a wrong-version `spank_pyxis.so` stops slurmd. **Any change to `scripts/install-enroot-pyxis.sh` MUST be retested on all supported versions.** | +| 1 | **Every Slurm version** | Deploy one cluster per `SlurmVersion` `AllowedValues` value (**25.05 and 25.11**) and run a Pyxis `--container-image` job on each | `MetricsType` is 25.11-only (25.05 cluster create fails if it's set unconditionally); the Pyxis SPANK plugin is ABI-locked to its Slurm version, so a wrong-version `spank_pyxis.so` stops slurmd. **Any change to `assets/scripts/install-enroot-pyxis.sh` MUST be retested on all supported versions.** | | 2 | **First-boot container runtime install** | Default `PostInstallScriptUrl` runs `install-enroot-pyxis.sh` at first boot; verify Enroot/Pyxis lands and a `--container-image` job works → [Test 2](#test-2-enrootpyxis-container-runtime-first-boot-install) | The default path used by every cluster that doesn't override `AmiId`. Bugs in `install-enroot-pyxis.sh` only show up on a clean first boot. | | 3 | **First-boot from a clean deploy** | Validate on a **freshly deployed** cluster, not a node you hand-patched | Post-install runs during cloud-init *before* slurmd/profile.d/controller exist; bugs there (e.g. version detection, `set -e` aborts) only show on a clean first boot, not after a live re-run. | | 4 | **CPU queue** | `DeployOnDemandCNG=true`; Pyxis container job on `cpu1` | Baseline; also the cheapest way to exercise items 1–3 without GPU capacity. | | 5 | **Each GPU family** (as capacity allows) | `p5`/`p5e`/`p5en`, `p6-b200`, `p6-b300`: nvidia-smi, NCCL all_reduce, FSDP | EFA NIC layout and the dcgm-exporter image differ per family (see notes). | | 6 | **Monitoring** | 6 login containers up, all Prometheus targets healthy, GPU dashboards populate on every supported GPU family with the default `DcgmExporterImage` (DCGM 4.5.2 by digest) | — | | 7 | **Template lint** | `aws cloudformation validate-template` on every edited `assets/*.yaml` | Catches structural errors before a deploy round-trip. | -| 8 | **Pre-baked AMI build path** (when touched) | If `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes (`scripts/install-enroot-pyxis.sh`) changes: build an AMI per supported `SlurmVersion`, then deploy a cluster with `AmiId=` + `PostInstallScriptUrl=""` and run a container job → [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template) | Independent path: the cluster stack does NOT run Image Builder, so an `install-enroot-pyxis.sh` fix is only in the AMI after a rebuild. The AMI is single-Slurm-version by design — `SlurmVersion` on the DLAMI stack must match the cluster's `SlurmVersion` (the SPANK plugin is ABI-locked). **Skip this row only if neither the AMI build template nor the install script changed.** | +| 8 | **Pre-baked AMI build path** (when touched) | If `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes (`assets/scripts/install-enroot-pyxis.sh`) changes: build an AMI per supported `SlurmVersion`, then deploy a cluster with `AmiId=` + `PostInstallScriptUrl=""` and run a container job → [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template) | Independent path: the cluster stack does NOT run Image Builder, so an `install-enroot-pyxis.sh` fix is only in the AMI after a rebuild. The AMI is single-Slurm-version by design — `SlurmVersion` on the DLAMI stack must match the cluster's `SlurmVersion` (the SPANK plugin is ABI-locked). **Skip this row only if neither the AMI build template nor the install script changed.** | | 9 | **CPU EFA path** (when touched) | If `add-cng.yaml`'s EFA wiring (`EnableEfa` / `EfaInterfaceCount` / `PlacementGroupName`) or the deploy-all forwarding (`OnDemand{EnableEfa,EfaInterfaceCount,PlacementGroupName}`) changes: deploy with `OnDemandEnableEfa=true` on at least one EFA-capable HPC type (e.g. hpc7a.96xlarge, 2 NICs), verify `lspci`/`fi_info` show the EFA NICs, and run a 2-node OSU `osu_mbw_mr` → [Test 9](#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) | EFA enables a different LaunchTemplate shape (`NetworkInterfaces` block with `InterfaceType=efa`, mutually exclusive with `SecurityGroupIds`); a regression here only shows on a real EFA deploy, not template-validate. **Skip this row only if no EFA-related wiring was touched.** | | 10 | **FSx storage health** | Both filesystems mount on every node; read/write sanity; FSx side reports the parameters CFN asked for (deployment type, throughput, capacity, `EfaEnabled` when set) → [Test 10](#test-10-fsx-storage-health) | Mount errors only surface on a fresh first boot (race vs systemd, NFS settle, Lustre client driver mismatch). FSx-side parameter drift between what CFN asked for and what FSx reports is invisible to template-validate. | @@ -73,7 +73,7 @@ A single `pcs-ml-cluster-deploy-all.yaml` deploy with `DeployMonitoring=true`, `DeployOnDemandCNG=true`, and `DeployPseriesCNG=true` exercises Tests 1–7 in one cluster. Test 8 is a **separate** flow (independent stack for the AMI, then a cluster pinned to its output); run it when `pcs-ready-dlami-with-enroot-pyxis.yaml` or -`scripts/install-enroot-pyxis.sh` change. See [the README](../README.md#5-usage-examples) +`assets/scripts/install-enroot-pyxis.sh` change. See [the README](../README.md#5-usage-examples) for deploy commands; set `PseriesInstanceType` to the GPU family you want to validate. --- @@ -249,7 +249,7 @@ tail -1 /var/log/pcs-post-install.log # "...complet dir, and the plugstack `pyxis.conf` referencing that exact path; post-install log exits 0. The Test 1/6/7 container jobs are the functional proof that Pyxis works. -> **⚠️ Regression-test rule for `scripts/install-enroot-pyxis.sh`.** This script has bitten +> **⚠️ Regression-test rule for `assets/scripts/install-enroot-pyxis.sh`.** This script has bitten > us repeatedly in ways a single 25.11 GPU run does not catch. **Any change to it MUST be > retested across the full matrix at the top of this guide**, specifically: > - **All supported Slurm versions** (25.05 **and** 25.11). The Pyxis SPANK plugin is @@ -426,7 +426,7 @@ cd slurm && sbatch --nodes=2 --partition=gpu-p6b300 \ ## Test 8: Pre-baked AMI build (standalone DLAMI template) **When to run:** when `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes -in (`scripts/install-enroot-pyxis.sh`) changes — the cluster stack does NOT run +in (`assets/scripts/install-enroot-pyxis.sh`) changes — the cluster stack does NOT run Image Builder, so a fix to the install script is only in the AMI after a rebuild. Skip this test if you only touched the cluster templates. @@ -792,7 +792,7 @@ mounts continue to work over TCP for non-EFA nodes; EFA support is additive. See **[`multi-user-test.md`](./multi-user-test.md)** for the full test procedure. -Run when `DeployDirectory=true` is enabled. Covers: LDAP server health, +Run when `DirectoryService=OpenLDAP-LoginNode` is enabled. Covers: LDAP server health, user lifecycle (create/delete/verify on login + compute), Slurm job execution as LDAP user, multi-node UID consistency, home directory sharing, SSSD cache resilience, and LDAP DB persistence across login node replacement. diff --git a/architectures/aws-pcs/tests/multi-user-test.md b/architectures/aws-pcs/tests/multi-user-test.md index 2ac2fe268..2798652cf 100644 --- a/architectures/aws-pcs/tests/multi-user-test.md +++ b/architectures/aws-pcs/tests/multi-user-test.md @@ -1,10 +1,10 @@ # Test: Multi-User Directory (OpenLDAP) -Run this test when `DeployDirectory=true` is enabled. Validates that the +Run this test when `DirectoryService=OpenLDAP-LoginNode` is enabled. Validates that the OpenLDAP directory on the login node is functional, that LDAP users are visible on compute nodes via SSSD, and that Slurm can run jobs as those users. -**Prerequisites**: a deployed cluster with `DeployDirectory=true` on the login +**Prerequisites**: a deployed cluster with `DirectoryService=OpenLDAP-LoginNode` on the login CNG and at least one compute CNG. All commands run from the login node unless noted otherwise. From 2a6c72c3eb4f1882bc556a63ebaab791c94090f2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:04:53 +0000 Subject: [PATCH 21/89] README: separate boot scripts from helper scripts, add parallelcluster-monitoring link --- architectures/aws-pcs/README.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index bdfbe8b4d..f1fb15821 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -560,11 +560,17 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this • MonitoringRole=compute → DCGM Exporter • Same external script pattern as compute CNG -External scripts (fetched at first boot from S3: s3:///scripts/): +Boot scripts (fetched at first boot from S3: s3:///scripts/): assets/scripts/install-enroot-pyxis.sh ← Enroot 3.5.0 + Pyxis 0.20.0 assets/scripts/setup-directory.sh ← multi-user directory (server + client) - assets/scripts/ldap-add-user.sh ← helper to add POSIX users to LDAP - MonitoringRepo @ MonitoringVersion ← aws-parallelcluster-monitoring post-install.sh + +External boot scripts (fetched from GitHub): + aws-parallelcluster-monitoring post-install.sh ← monitoring stack installer + (https://github.com/aws-samples/aws-parallelcluster-monitoring) + fetched from: ${MonitoringRepo} @ ${MonitoringVersion} + +Helper scripts (NOT run at boot — for admin use on the login node): + assets/scripts/ldap-add-user.sh ← add POSIX users to LDAP directory External references (runtime): SSM /aws/service/pcs/ami/.../latest/ami-id ← PCS-Ready DLAMI (when AmiId is empty) From 03db98b9eb038039e486b2788865c6c20335a10a Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:13:06 +0000 Subject: [PATCH 22/89] fix: move directory setup from MIME shellscript to runcmd block (PCS skips shellscript sections) --- architectures/aws-pcs/assets/add-cng.yaml | 30 +++++++++-------------- 1 file changed, 12 insertions(+), 18 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 7c09db677..f1ff1236c 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -520,25 +520,19 @@ Resources: done echo "Monitoring installation complete (exit $rc)" >> /var/log/monitoring-install.log fi - - --==MYBOUNDARY== - Content-Type: text/x-shellscript; charset="us-ascii" - MIME-Version: 1.0 - - #!/bin/bash # Multi-user directory setup (optional, DirectoryRole != none) - if [ "${DirectoryRole}" != "none" ]; then - export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" - export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') - export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) - export CLUSTER_ID="${ClusterId}" - export DIRECTORY_DNS_IPS="" - - REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) - aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh --region ${!REGION} - chmod +x /tmp/setup-directory.sh - bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log - fi + - | + if [ "${DirectoryRole}" != "none" ]; then + export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" + export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') + export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + export CLUSTER_ID="${ClusterId}" + export DIRECTORY_DNS_IPS="" + REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh --region ${!REGION} + chmod +x /tmp/setup-directory.sh + bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log + fi --==MYBOUNDARY== From 74705e042f23a41a300f0ac2d62678a40bff18cf Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:17:01 +0000 Subject: [PATCH 23/89] tests: add accounting-test.md + gpu-healthcheck-test.md Test 12 (accounting-test.md): - Validates Slurm managed accounting with LDAP multi-user - Covers: sacctmgr user/account creation, resource limits (GrpTRESRunMins), job tracking (sacct), reporting (sreport), fairshare, and AccountingPolicyEnforcement behavior (none vs associations,limits,safe) - Based on https://aws.amazon.com/blogs/hpc/introducing-managed-accounting-for-aws-parallel-computing-service/ Test 13 (gpu-healthcheck-test.md): - Integrates 4.validation_and_observability/2.gpu-cluster-healthcheck suite - Lightweight (checks 0-3: nvidia-smi, DCGM L2, EFA enum, topology) ~15 min - NCCL multi-node (check 5) with per-instance bandwidth thresholds - Slurm prolog integration guide for production - When-to-run decision table (deploy, pre-training, steady-state, quarantine) README: added GPU Health Check link to Additional Resources. tests/README.md: added Test 12 + Test 13 entries with links. --- architectures/aws-pcs/README.md | 3 +- architectures/aws-pcs/tests/README.md | 22 ++ .../aws-pcs/tests/accounting-test.md | 194 ++++++++++++++++++ .../aws-pcs/tests/gpu-healthcheck-test.md | 128 ++++++++++++ 4 files changed, 346 insertions(+), 1 deletion(-) create mode 100644 architectures/aws-pcs/tests/accounting-test.md create mode 100644 architectures/aws-pcs/tests/gpu-healthcheck-test.md diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index f1fb15821..371897a7b 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -600,10 +600,11 @@ numbers is in **[tests/README.md](./tests/README.md)**. In this repo: - [Parameter reference](./docs/PARAMETERS.md) — every deploy-all parameter and default -- [Operations guide](./docs/OPERATIONS.md) — version trade-offs, AMI pinning, monitoring/DCGM, FSx coupling, production settings +- [Operations guide](./docs/OPERATIONS.md) — version trade-offs, AMI pinning, monitoring/DCGM, FSx coupling, Lustre tuning, production settings - [User management guide](./docs/USER-MANAGEMENT.md) — multi-user setup with OpenLDAP (add/remove users, groups, Slurm accounting) - [Deploy & testing procedures](./docs/DEPLOY-TESTING.md) — development deploy workflow with test S3 bucket - [Test & Validation Guide](./tests/README.md) — reproducible matrix with measured numbers +- [GPU Cluster Health Check](../../4.validation_and_observability/2.gpu-cluster-healthcheck) — comprehensive GPU/EFA/NVLink validation suite (lightweight + intensive modes, Slurm prolog integration) - [Roadmap / TODO](./docs/ROADMAP.md) External: diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index b6f275f9a..5efe0b6b5 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -799,6 +799,28 @@ resilience, and LDAP DB persistence across login node replacement. --- +## Test 12: Slurm managed accounting + multi-user + +See **[`accounting-test.md`](./accounting-test.md)** for the full test procedure. + +Run when `ManagedAccounting=enabled` and `DirectoryService=OpenLDAP-LoginNode`. +Covers: sacctmgr user/account registration, job submission as LDAP users, +sacct/sreport verification, resource limit enforcement, and +AccountingPolicyEnforcement behavior. + +--- + +## Test 13: GPU cluster health check + +See **[`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md)** for the full test procedure. + +Run after any GPU CNG (P5/P5e/P5en/P6-B200) deployment. Uses the +[GPU Cluster Health Check Suite](../../4.validation_and_observability/2.gpu-cluster-healthcheck) +from this repository. Covers: nvidia-smi, DCGM L2, EFA enumeration, topology +validation, and multi-node NCCL bandwidth thresholds. + +--- + ## Cleanup Delete the stack from the CloudFormation console (select → **Delete**) or: diff --git a/architectures/aws-pcs/tests/accounting-test.md b/architectures/aws-pcs/tests/accounting-test.md new file mode 100644 index 000000000..81d113663 --- /dev/null +++ b/architectures/aws-pcs/tests/accounting-test.md @@ -0,0 +1,194 @@ +# Test: Slurm Managed Accounting + Multi-User + +Validates that Slurm managed accounting works with LDAP multi-user setup. +Covers user/account creation, resource limit enforcement, job tracking, and +reporting. + +**Prerequisites**: +- Cluster with `ManagedAccounting=enabled` and `DirectoryService=OpenLDAP-LoginNode` +- Slurm 25.11 (accounting is available on 24.11+; templates default to 25.11) +- At least one compute node available + +All commands run on the login node as root unless noted. + +--- + +## Part A — Accounting infrastructure + +### A1. Verify accounting is active + +```bash +export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH +sacctmgr show cluster +``` + +Expected: cluster name listed with a valid `ControlHost`. + +### A2. Default account exists + +```bash +sacctmgr show account +``` + +If no accounts exist yet, create the default: +```bash +sacctmgr -i add account default Description="Default account" +``` + +--- + +## Part B — User + account registration + +### B1. Create LDAP users (if not done already) + +```bash +ADMIN_PW=$(aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text --region us-east-2) + +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user alice 10001 3000 +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user bob 10002 3000 +``` + +### B2. Register users in Slurm accounting + +```bash +sacctmgr -i add account ml-team Description="ML Research Team" +sacctmgr -i add user alice Account=ml-team +sacctmgr -i add user bob Account=ml-team + +# Verify +sacctmgr show user alice bob format=User,Account,DefaultAccount +``` + +Expected: +``` + User Account DefaultAccount +--------- ---------- -------------- + alice ml-team ml-team + bob ml-team ml-team +``` + +### B3. Set resource limits (optional) + +```bash +# Cap alice at 100 CPU-hours +sacctmgr -i modify user alice set GrpTRESRunMins=cpu=6000 + +# Verify +sacctmgr show user alice format=User,Account,GrpTRESRunMins +``` + +--- + +## Part C — Job submission and tracking + +### C1. Submit jobs as LDAP users + +```bash +# As alice +sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J alice-test' + +# As bob +sudo su - bob -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J bob-test' +``` + +Wait for jobs to complete: +```bash +squeue # should show jobs running then empty +``` + +### C2. Verify job history with sacct + +```bash +sacct --starttime=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ + --format="JobID,User,JobName,Partition,Account,AllocCPUS,State,ExitCode,Elapsed" +``` + +Expected: both jobs listed with correct User, Account, State=COMPLETED. + +### C3. Per-user reporting + +```bash +# Jobs by alice +sacct -u alice --format="JobID,JobName,State,Start,End,Elapsed" + +# Utilization by account +sreport cluster AccountUtilizationByUser \ + start=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ + format="Account,Login,Used" +``` + +### C4. Resource limit enforcement (if B3 was set) + +```bash +# Submit a job that would exceed alice's limit +sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --ntasks=96 --time=120:00 --wrap="sleep 7200" -J limit-test' +``` + +Check if job is pending with reason `AssocGrpCPURunMinutesLimit`: +```bash +squeue -u alice --format="%i %j %T %r" +``` + +--- + +## Part D — Fairshare (optional) + +### D1. Set different fairshare weights + +```bash +sacctmgr -i modify account ml-team set FairShare=100 +sacctmgr -i add account ops-team Description="Operations" +sacctmgr -i modify account ops-team set FairShare=50 +``` + +### D2. Check fairshare values + +```bash +sshare -a --format=Account,User,RawShares,NormShares,RawUsage,FairShare +``` + +--- + +## Part E — AccountingPolicyEnforcement + +### E1. Test with enforcement=none (default) + +An unregistered user can still submit jobs: +```bash +# Create a user NOT in sacctmgr +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user charlie 10003 3000 +sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + srun -p cpu1 -N1 -n1 hostname' +``` + +Expected: job runs (no accounting enforcement). + +### E2. Test with enforcement=associations,limits,safe + +If `AccountingPolicyEnforcement=associations,limits,safe` was set at cluster +creation: +```bash +# charlie (not in sacctmgr) should be rejected +sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + srun -p cpu1 -N1 -n1 hostname' +``` + +Expected: `error: Unable to allocate resources: Invalid account or account/partition combination specified` + +--- + +## Verdict checklist + +| Check | Expected | +|---|---| +| sacctmgr show cluster → cluster listed | ✅ | +| Users registered in accounting (alice, bob) | ✅ | +| Jobs run as LDAP users complete successfully | ✅ | +| sacct shows correct user/account/state per job | ✅ | +| sreport shows per-user utilization | ✅ | +| Resource limit enforcement (if set) blocks over-limit jobs | ✅ | +| Unregistered user behavior matches AccountingPolicyEnforcement setting | ✅ | diff --git a/architectures/aws-pcs/tests/gpu-healthcheck-test.md b/architectures/aws-pcs/tests/gpu-healthcheck-test.md new file mode 100644 index 000000000..64d4c7eb8 --- /dev/null +++ b/architectures/aws-pcs/tests/gpu-healthcheck-test.md @@ -0,0 +1,128 @@ +# Test: GPU Cluster Health Check + +Validates GPU hardware, EFA networking, and NVLink interconnects using the +[GPU Cluster Health Check Suite](../../4.validation_and_observability/2.gpu-cluster-healthcheck) +from this repository. + +**Prerequisites**: +- Cluster with a GPU CNG deployed (P5, P5e, P5en, or P6-B200) +- At least 2 GPU nodes for multi-node NCCL check +- The health check scripts fetched to `/fsx` (shared storage) + +--- + +## Setup + +```bash +# Fetch the health check suite to shared /fsx +cd /fsx +git clone https://github.com/awslabs/awsome-distributed-ai.git --depth 1 --sparse +cd awsome-distributed-ai +git sparse-checkout set 4.validation_and_observability/2.gpu-cluster-healthcheck +cd 4.validation_and_observability/2.gpu-cluster-healthcheck +``` + +Or simply copy the directory from the repo checkout if already available. + +--- + +## Part A — Lightweight suite (single-node, ~15 min) + +Run on each GPU node to validate hardware is healthy before submitting +training jobs: + +```bash +export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH + +# Single node +srun -N 1 -n 1 -p gpu-p5 --exclusive \ + bash /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/gpu-healthcheck.sh \ + --suite lightweight +``` + +Or via sbatch for multiple nodes: +```bash +sbatch -N 2 -p gpu-p5 --exclusive \ + /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/slurm/sbatch-lightweight.sh +``` + +### Expected results (per node) + +| Check | What it validates | Pass criteria | +|---|---|---| +| 0: nvidia-smi | GPU count, driver, Xid errors | All GPUs detected, no Xid errors in dmesg | +| 1: DCGM L2 | PCIe BW, memory stress, SM stress | All DCGM sub-tests PASS | +| 2: EFA enumeration | EFA device count, RDMA, libfabric | EFA count matches instance profile (32 for p5, 16 for p5en, 8 for p6-b200) | +| 3: Topology | NVLink status, PCIe groups | All NVLinks UP, no uncorrectable errors | + +### Severity output + +Results are classified as: +- **PASS** — all checks green +- **MONITOR** — minor anomaly, non-blocking +- **REBOOT** — correctable with instance reboot +- **ISOLATE** — hardware fault, instance should be replaced + +--- + +## Part B — NCCL multi-node (check 5, ~10-20 min) + +Validates multi-node GPU communication over EFA. Overlaps with Test 6 (NCCL) +but adds per-instance bandwidth threshold validation: + +```bash +srun -N 2 -n 2 -p gpu-p5 --exclusive \ + bash /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/gpu-healthcheck.sh \ + --check 5 +``` + +### Expected + +- EFA provider selected (`Selected Provider is efa`) +- Bus bandwidth meets instance-type threshold (e.g. p5: >700 GB/s peak for 2-node all_reduce at 128MB) +- No NCCL timeouts or EFA transport errors + +--- + +## Part C — Slurm prolog integration (production use) + +For production clusters, configure the lightweight checks as a Slurm prolog +so every GPU job validates hardware before running: + +```bash +# In slurm.conf (via PCS custom settings): +Prolog=/fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/slurm/prolog-gpu-healthcheck.sh +``` + +A non-zero prolog exit drains the node and requeues the job. + +--- + +## When to run + +| Scenario | Suite | Frequency | +|---|---|---| +| After initial GPU CNG deployment | Lightweight (full) | Once | +| Before multi-day training runs | Lightweight + check 5 | Per-run | +| Production steady-state | Prolog (checks 0, 2) | Every job | +| Node suspected faulty | Intensive (all checks) | On-demand | +| After instance replacement | Lightweight | Once | + +--- + +## Verdict checklist + +| Check | Result | +|---|---| +| Lightweight suite completes on all GPU nodes | ✅ | +| No ISOLATE/REBOOT severity results | ✅ | +| EFA device count matches instance profile | ✅ | +| NVLink all UP, no uncorrectable errors | ✅ | +| Multi-node NCCL bandwidth meets threshold | ✅ | + +--- + +## Reference + +- [GPU Cluster Health Check Suite README](../../4.validation_and_observability/2.gpu-cluster-healthcheck/README.md) +- Supported instance profiles: see `instance-profiles.conf` in the suite From 5423e7f3cab0959b299a55096ed8174005d5df21 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:24:36 +0000 Subject: [PATCH 24/89] tests: split README.md (842 lines) into category-based files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README.md was too long — reduced from 842 to 73 lines (index + matrix only). Test procedures moved to per-category files: infra-test.md — Tests 1-3, 8 (monitoring, container runtime, AMI build) compute-test.md — Tests 4-6 (CPU queue, GPU families, NCCL EFA) training-test.md — Test 7 (FSDP Llama-2 7B) hpc-efa-test.md — Test 9 (EFA on CPU HPC, OSU benchmarks) storage-test.md — Test 10 (FSx health + performance regression) multi-user-test.md — Tests 11-12 (OpenLDAP + accounting, merged) gpu-healthcheck-test.md — Test 13 (GPU health check suite) Removed accounting-test.md (merged into multi-user-test.md since they are always tested together). --- architectures/aws-pcs/tests/README.md | 845 +----------------- .../aws-pcs/tests/accounting-test.md | 194 ---- architectures/aws-pcs/tests/compute-test.md | 82 ++ architectures/aws-pcs/tests/hpc-efa-test.md | 156 ++++ architectures/aws-pcs/tests/infra-test.md | 174 ++++ .../aws-pcs/tests/multi-user-test.md | 197 ++++ architectures/aws-pcs/tests/storage-test.md | 131 +++ architectures/aws-pcs/tests/training-test.md | 71 ++ 8 files changed, 849 insertions(+), 1001 deletions(-) delete mode 100644 architectures/aws-pcs/tests/accounting-test.md create mode 100644 architectures/aws-pcs/tests/compute-test.md create mode 100644 architectures/aws-pcs/tests/hpc-efa-test.md create mode 100644 architectures/aws-pcs/tests/infra-test.md create mode 100644 architectures/aws-pcs/tests/storage-test.md create mode 100644 architectures/aws-pcs/tests/training-test.md diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 5efe0b6b5..d36267a74 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -1,823 +1,53 @@ # AWS PCS — Test & Validation Guide -This directory is a single guide for validating an AWS PCS cluster deployed from the -templates in [`../assets`](../assets). Each test below lists **what to run** and the -**expected result**. +Test procedures for validating an AWS PCS cluster deployed from the templates +in [`../assets`](../assets). -For non-test operational guidance (Slurm version trade-offs, AMI single-version rule, -`MonitoringVersion` migration, the `DcgmExporterImage` default and when to change it, -AMI pinning, FSx deployment-type ↔ throughput coupling), see -[`../docs/OPERATIONS.md`](../docs/OPERATIONS.md). - -Rather than ship its own copies, this guide reuses the repository's canonical benchmark -and training assets and documents only the **PCS-specific deltas** (queue/partition names, -running the Enroot import on the login node, putting caches on `/fsx`): - -| Stage | Canonical asset to use | PCS-specific delta documented here | -|---|---|---| -| NCCL `all_reduce` over EFA | [`micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch`](../../../micro-benchmarks/nccl-tests) | [Test 6](#test-6-nccl-multi-node-efa) — import on the login node, partition name | -| FSDP Llama-2 7B training | [`3.test_cases/pytorch/FSDP`](../../../3.test_cases/pytorch/FSDP) | [Test 7](#test-7-fsdp-sample-training) — venv **or** Enroot container; cache on `/fsx`, 2 nodes | -| GPU sanity (nvidia-smi) | (one `srun` line, no script) | [Test 5](#test-5-p5p6-gpu-multi-nic) | - -All Slurm commands run as the **`ubuntu`** user from the login node (SSM session or -SSH). Slurm binaries are only on `PATH` in a login shell — over SSM/SSH wrap commands -as `bash -lc "sinfo; squeue"`. +For operational guidance (Slurm version trade-offs, AMI pinning, monitoring, +FSx tuning), see [`../docs/OPERATIONS.md`](../docs/OPERATIONS.md). --- -## Pre-merge full test matrix (run before opening / updating the PR) +## Pre-merge test matrix -Run this **complete set** before merging template or script changes. Several real bugs -only appeared in specific combinations (a 25.05-only `MetricsType` rejection, a Pyxis -SPANK plugin built for the wrong Slurm version, a first-boot post-install failure), so -do **not** assume a single 25.11 GPU run covers everything. +Run this **complete set** before merging template or script changes. -| # | Dimension | What to cover | Why it matters | -|---|---|---|---| -| 1 | **Every Slurm version** | Deploy one cluster per `SlurmVersion` `AllowedValues` value (**25.05 and 25.11**) and run a Pyxis `--container-image` job on each | `MetricsType` is 25.11-only (25.05 cluster create fails if it's set unconditionally); the Pyxis SPANK plugin is ABI-locked to its Slurm version, so a wrong-version `spank_pyxis.so` stops slurmd. **Any change to `assets/scripts/install-enroot-pyxis.sh` MUST be retested on all supported versions.** | -| 2 | **First-boot container runtime install** | Default `PostInstallScriptUrl` runs `install-enroot-pyxis.sh` at first boot; verify Enroot/Pyxis lands and a `--container-image` job works → [Test 2](#test-2-enrootpyxis-container-runtime-first-boot-install) | The default path used by every cluster that doesn't override `AmiId`. Bugs in `install-enroot-pyxis.sh` only show up on a clean first boot. | -| 3 | **First-boot from a clean deploy** | Validate on a **freshly deployed** cluster, not a node you hand-patched | Post-install runs during cloud-init *before* slurmd/profile.d/controller exist; bugs there (e.g. version detection, `set -e` aborts) only show on a clean first boot, not after a live re-run. | -| 4 | **CPU queue** | `DeployOnDemandCNG=true`; Pyxis container job on `cpu1` | Baseline; also the cheapest way to exercise items 1–3 without GPU capacity. | -| 5 | **Each GPU family** (as capacity allows) | `p5`/`p5e`/`p5en`, `p6-b200`, `p6-b300`: nvidia-smi, NCCL all_reduce, FSDP | EFA NIC layout and the dcgm-exporter image differ per family (see notes). | -| 6 | **Monitoring** | 6 login containers up, all Prometheus targets healthy, GPU dashboards populate on every supported GPU family with the default `DcgmExporterImage` (DCGM 4.5.2 by digest) | — | -| 7 | **Template lint** | `aws cloudformation validate-template` on every edited `assets/*.yaml` | Catches structural errors before a deploy round-trip. | -| 8 | **Pre-baked AMI build path** (when touched) | If `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes (`assets/scripts/install-enroot-pyxis.sh`) changes: build an AMI per supported `SlurmVersion`, then deploy a cluster with `AmiId=` + `PostInstallScriptUrl=""` and run a container job → [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template) | Independent path: the cluster stack does NOT run Image Builder, so an `install-enroot-pyxis.sh` fix is only in the AMI after a rebuild. The AMI is single-Slurm-version by design — `SlurmVersion` on the DLAMI stack must match the cluster's `SlurmVersion` (the SPANK plugin is ABI-locked). **Skip this row only if neither the AMI build template nor the install script changed.** | -| 9 | **CPU EFA path** (when touched) | If `add-cng.yaml`'s EFA wiring (`EnableEfa` / `EfaInterfaceCount` / `PlacementGroupName`) or the deploy-all forwarding (`OnDemand{EnableEfa,EfaInterfaceCount,PlacementGroupName}`) changes: deploy with `OnDemandEnableEfa=true` on at least one EFA-capable HPC type (e.g. hpc7a.96xlarge, 2 NICs), verify `lspci`/`fi_info` show the EFA NICs, and run a 2-node OSU `osu_mbw_mr` → [Test 9](#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) | EFA enables a different LaunchTemplate shape (`NetworkInterfaces` block with `InterfaceType=efa`, mutually exclusive with `SecurityGroupIds`); a regression here only shows on a real EFA deploy, not template-validate. **Skip this row only if no EFA-related wiring was touched.** | -| 10 | **FSx storage health** | Both filesystems mount on every node; read/write sanity; FSx side reports the parameters CFN asked for (deployment type, throughput, capacity, `EfaEnabled` when set) → [Test 10](#test-10-fsx-storage-health) | Mount errors only surface on a fresh first boot (race vs systemd, NFS settle, Lustre client driver mismatch). FSx-side parameter drift between what CFN asked for and what FSx reports is invisible to template-validate. | - -Tests 1–10 below are the per-item how-to. The single-cluster shortcut (one deploy that -covers monitoring + CPU + one GPU family) is fine for iterating; rows 8 and 9 are -separate paths run only when their inputs change. The full matrix above is the bar -for **merge**. +| # | Category | Tests | File | When to run | +|---|---|---|---|---| +| 1-3, 8 | **Infrastructure** | Monitoring stack, container runtime (first-boot + AMI build), template lint | [`infra-test.md`](./infra-test.md) | Every PR | +| 4-6 | **Compute** | CPU queue, GPU families (G/P5/P6), NCCL multi-node EFA | [`compute-test.md`](./compute-test.md) | Every PR | +| 7 | **Training** | FSDP Llama-2 7B distributed training | [`training-test.md`](./training-test.md) | GPU PRs | +| 9 | **HPC EFA** | EFA on CPU instances (hpc6a/hpc7a/hpc8a), OSU benchmarks | [`hpc-efa-test.md`](./hpc-efa-test.md) | EFA wiring changes | +| 10 | **Storage** | FSx health check + performance regression test (noatime benchmark) | [`storage-test.md`](./storage-test.md) | FSx / mount changes | +| 11-12 | **Multi-user** | OpenLDAP directory + Slurm managed accounting | [`multi-user-test.md`](./multi-user-test.md) | Directory / accounting changes | +| 13 | **GPU health** | GPU Cluster Health Check suite (DCGM, EFA, NVLink, NCCL thresholds) | [`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md) | GPU CNG deploys | --- -## Coverage matrix - -What this guide covers, and the template/parameter that exercises it: - -| Dimension | Options to test | How | -|---|---|---| -| **Monitoring** | enabled (default) | `DeployMonitoring=true` → [Test 1](#test-1-monitoring-stack) | -| **Container runtime — default first-boot path** | UserData install on every node | Default `PostInstallScriptUrl` (no AMI override) → [Test 2](#test-2-enrootpyxis-container-runtime-first-boot-install) | -| **CPU nodes** | `c6i`/`c7i` etc. | `DeployOnDemandCNG=true` → [Test 3](#test-3-cpu-queue) | -| **Single-NIC GPU** | `g5`/`g6` | On-Demand CNG with a G-series type → [Test 4](#test-4-g-series-gpu-single-nic) | -| **Multi-NIC GPU** | `p5`/`p5e`/`p5en`, `p6-b200`, `p6-b300` | `DeployPseriesCNG=true` + `PseriesInstanceType` → [Test 5](#test-5-p5p6-gpu-multi-nic) | -| **NCCL / EFA** | 2-node all_reduce | [Test 6](#test-6-nccl-multi-node-efa) | -| **Sample training** | FSDP Llama-2 7B | [Test 7](#test-7-fsdp-sample-training) | -| **Container runtime — pre-baked AMI path** | Standalone DLAMI build + cluster pinned to its output | `pcs-ready-dlami-with-enroot-pyxis.yaml` (separate stack) → cluster with `AmiId=ami-xxx` + `PostInstallScriptUrl=""` → [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template) | -| **EFA on CPU HPC instances** | hpc6a (1 NIC), hpc7a / hpc8a (2 NIC) | `OnDemandEnableEfa=true` + `OnDemandEfaInterfaceCount=1\|2` (deploy-all) or `EnableEfa=true` + `EfaInterfaceCount=1\|2` (modular `add-cng.yaml`); auto-creates a per-CNG cluster placement group, override with `OnDemandPlacementGroupName` / `PlacementGroupName` to share → [Test 9](#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) | -| **FSx storage health** | Lustre + OpenZFS mounts, read/write, FSx-side parameters honored (incl. `FSxLustreEnableEfa` when set) | Default deploy already mounts both filesystems; verify on the login + a compute node and inspect the FSx-side filesystem with `aws fsx describe-file-systems` → [Test 10](#test-10-fsx-storage-health) | +## Quick-start: single-cluster shortcut A single `pcs-ml-cluster-deploy-all.yaml` deploy with `DeployMonitoring=true`, -`DeployOnDemandCNG=true`, and `DeployPseriesCNG=true` exercises Tests 1–7 in one cluster. -Test 8 is a **separate** flow (independent stack for the AMI, then a cluster pinned to -its output); run it when `pcs-ready-dlami-with-enroot-pyxis.yaml` or -`assets/scripts/install-enroot-pyxis.sh` change. See [the README](../README.md#5-usage-examples) -for deploy commands; set `PseriesInstanceType` to the GPU family you want to validate. +`DeployOnDemandCNG=true`, and `DeployPseriesCNG=true` exercises Tests 1–7 in +one cluster. Tests 8-13 are separate paths run only when their inputs change. --- -## Verified configurations - -Key configurations validated on real hardware with these templates (representative -results; exact bandwidth/throughput vary with NCCL/EFA versions and message size): - -| Config | Region | Capacity | Monitoring | NCCL all_reduce (2-node peak busbw) | FSDP Llama-2 7B (2-node) | -|---|---|---|---|---|---| -| **2× p6-b200.48xlarge** (16× B200) | us-west-2 | Capacity Block | ✅ v2.9.1, 16 GPUs in Grafana | **~654 GB/s** @16 GiB (EFA, `found 8 nics`, `#wrong 0`) | **~223 TFLOPS/GPU, ~86k tok/s** | -| **2× p6-b300.48xlarge** (16× B300) | us-west-2 | Capacity Block | ✅ v2.9.1, 16 B300 GPUs in Grafana with the default `DcgmExporterImage` (DCGM 4.5.2 by digest) ‡ | **~751 GB/s** @64 GiB (EFA, `found 16 nics`, `#wrong 0`) † | **~205 TFLOPS/GPU, ~79k tok/s** (venv); **~193 TFLOPS/GPU** (container) | -| **2× p5.48xlarge** (16× H100) | us-east-2 | Capacity Block | ✅ | **~480 GB/s** (EFA, `found 32 nics`, `#wrong 0`) | ~60 TFLOPS/GPU | -| **Slurm 25.05, CPU + PostInstall** | us-west-2 | On-Demand | ✅ v2.9.1 (no `slurm_openmetrics` job §) | first-boot Pyxis OK, `srun --container-image=ubuntu:22.04` clean | n/a | -| **Slurm 25.11, CPU + PostInstall** | us-west-2 | On-Demand | ✅ v2.9.1 incl. Slurm OpenMetrics | first-boot Pyxis OK | n/a | -| **Login + CPU (`c6i`)** | us-west-2 / us-east-* | On-Demand | ✅ all targets up | n/a | n/a | -| **Grafana public access** (login-only SG) | us-west-2 | — | ✅ reachable at `https:///grafana/` from the allowed CIDR | — | — | - -### HPC EFA on CPU instances (`OnDemandEnableEfa=true`) - -OSU MPI micro-benchmarks 7.4 on 2 nodes, Slurm 25.11, AWS Open MPI 4.1.7 -(`/opt/amazon/openmpi`), libfabric provider `efa`. Tuned env: `FI_PROVIDER=efa`, -huge page on (default), `FI_EFA_FORK_SAFE=1`, `OMPI_MCA_pml=cm`, `OMPI_MCA_mtl=ofi`, -`OMPI_MCA_mtl_ofi_provider_include=efa`. All deploys used `OnDemandEnableEfa=true` -with the matching `OnDemandEfaInterfaceCount`; the cluster placement group was -auto-created per-CNG by `add-cng.yaml`. - -| Instance | NICs | Spec aggregate ¶ | `osu_latency` 1B | `osu_bw` peak (1 pair) | `osu_bibw` peak (1 pair) | `osu_mbw_mr` peak (16 pair × 2 nodes) | `osu_allreduce` 1B (32 ranks) | -|---|---:|---:|---:|---:|---:|---:|---:| -| **hpc6a.48xlarge** | 1 | 100 Gbps | 13.78 µs | 96.3 Gbps (12.04 GB/s) | 152 Gbps †† | **97.6 Gbps** (97.6% of spec) | 27.5 µs | -| **hpc7a.96xlarge** | 2 | 300 Gbps | 14.27 µs | 95.6 Gbps (11.95 GB/s) | 156 Gbps | **263 Gbps** (87.6% of spec) | 23.4 µs | -| **hpc8a.96xlarge** | 2 | 300 Gbps | **10.31 µs** ★ | **210 Gbps** (26.27 GB/s) | **363 Gbps** †† | **341 Gbps** ‡‡ | **17.4 µs** ★ | - -¶ AWS docs ([HPC instance specs](https://docs.aws.amazon.com/ec2/latest/instancetypes/hpc.html)) -"Baseline / Burst bandwidth (Gbps)" column. The notes also state "you must attach at -least 2 ENIs, to separate network cards, to achieve [aggregate] throughput" with a -per-ENI cap of 150 Gbps for hpc7a and 170 Gbps for hpc6id; **per-ENI cap for hpc8a is -not stated in AWS docs**. +## Canonical assets reused (not duplicated) -†† `osu_bibw` measures bidirectional bandwidth on a single pair (simultaneous -send+recv). It can exceed the uni-directional spec because it counts both -directions; not directly comparable to the "300 Gbps" aggregate spec. - -‡‡ hpc8a's `osu_mbw_mr` reading exceeds the docs' aggregate spec (300 Gbps). The -reading is reproducible across two independent runs (job 6 = 42.68 GB/s, job 7 = -43.85 GB/s), so it isn't a one-off artifact. Possible explanations include (a) AWS -docs spec is per-direction sustained while `osu_mbw_mr` reports forward-direction -peak, (b) Nitro v6 / EFAv4 efficiency on hpc8a leaves headroom above the published -number, (c) ack/control traffic in reverse direction inflates the receiver-side -counter view. **NIC-level Prometheus counters** (`node_amazonefa_tx_bytes` and -`rx_bytes`, available via the v2.7+ `efa-metrics.sh` textfile collector) can resolve -which is the case for a given run; in a 30s-update collector the wire-level peak -shows below the OSU-reported peak because the OSU phases for each message size are -shorter than the textfile sample window. - -★ hpc8a is fastest on every metric. The Nitro v6 / EFAv4 generation lift over hpc7a -(Nitro v4) shows up most clearly in latency (~30% better) and single-pair bandwidth -(~2× — single pair on hpc7a hits the 150 Gbps per-ENI cap, hpc8a does not appear to -hit one). - -**Tuning notes** (collected during these runs): -- **`osu_bw` (single pair) understates aggregate fabric bandwidth.** A single MPI - pair uses one libfabric endpoint; on instances with 2 NICs, the second NIC is idle. - Use `osu_mbw_mr -np 32 -N 16` for the realistic aggregate number; `osu_bw` only - matches the per-ENI cap. -- **Disabling huge pages costs throughput.** An earlier hpc7a run with - `FI_EFA_USE_HUGE_PAGE=0` showed 68 Gbps `osu_bw`; leaving the variable unset (= 1 - default) brought it to 95.6 Gbps on the same instance. -- **Open MPI 4.1.7 on the PCS-Ready DLAMI has no `pml=ofi`.** Set - `OMPI_MCA_pml=cm` and `OMPI_MCA_mtl=ofi` instead — the `cm` PML dispatches to the - OFI MTL. Setting `OMPI_MCA_pml=ofi` directly fails with "mca_pml_base_open() - failed → Returned 'Not found' (-13)". -- **EFA monitoring works with the default v2.9.1 stack.** The `efa-metrics.sh` - textfile collector emits `node_amazonefa_*` series for `tx_bytes`, `rx_bytes`, - `recv_bytes`, `send_bytes`, RDMA `read/write` bytes/work-requests, retransmits, - and unresponsive-remote-endpoint events. The Compute Node Details Grafana - dashboard has dedicated EFA panels for these. Sampling cadence is 30 sec - (textfile-collector timer), so short OSU sub-tests show below their wall-clock peak. - -### Stack creation times (measured, deploy-all) - -Wall-clock from `aws cloudformation create-stack` to top-level `CREATE_COMPLETE`, on a -warmed account in us-west-2 (no first-time-in-region provisioning). Useful for sizing -how long a deploy round-trip takes during a review cycle. - -| Configuration | First-boot path | Time | +| Workload | Source in this repo | PCS-specific delta | |---|---|---| -| 25.05, CPU only | Default `PostInstallScriptUrl` (Enroot/Pyxis at first boot) | **~24m** | -| 25.11, CPU only | Default `PostInstallScriptUrl` (Enroot/Pyxis at first boot) | **~31m** | -| 25.11, GPU + CPU (Capacity Block) | Default `PostInstallScriptUrl` (Enroot/Pyxis at first boot) | **~44m** | -| 25.05/25.11, CPU only, **pre-baked AMI** | Custom AMI built **separately** via `pcs-ready-dlami-with-enroot-pyxis.yaml` (~30m one-time), then cluster deployed with `AmiId=` + `PostInstallScriptUrl=""` | AMI build ~30m + cluster ~25m | - -Notes: Prerequisites (VPC + dual FSx) is the long-pole on the cluster path (~20-25m); -the default first-boot path adds the per-boot Enroot/Pyxis install (~2-3m). Pre-baking -the AMI is a separate ~30m one-time job — its cost amortizes when you redeploy or -scale clusters that share the AMI. GPU adds the P-series CNG and the GPU node first boot. - -Notes: -- **Deploy path:** all of the above came up from `pcs-ml-cluster-deploy-all.yaml` with - the default first-boot Enroot/Pyxis installer + `DeployMonitoring=true`. -- **Container runtime:** validated via first-boot UserData install (the default); the - pre-baked-AMI path uses the same `install-enroot-pyxis.sh` baked into the DLAMI by - `pcs-ready-dlami-with-enroot-pyxis.yaml`. -- **Pre-baked AMI path validated** (us-west-2, CPU-only): build the AMI with the - standalone DLAMI template, take its `DLAMIforPCSAmiId` output, deploy the cluster - with `AmiId=` and `PostInstallScriptUrl=""`. ImageBuilder bakes Enroot 3.5.0 - + per-version Pyxis into the DLAMI; login/compute nodes boot ready, no first-boot - install. The AMI is single-Slurm-version on purpose — see - [docs/OPERATIONS.md §2](../docs/OPERATIONS.md#2-container-runtime-postinstall-vs-ami-build). -- **EFA interface count** is derived from the instance type (p5/p5e = 32, p5en = 16, - p6-b200 = 8, p6-b300 = 16-of-17); see [README GPU compute](../README.md#gpu-compute-p5p6). -- **FSDP runs both ways** — validated with a shared-`/fsx` **venv** (~200 TFLOPS/GPU) and - with an **Enroot/Pyxis container** (`CONTAINER_IMAGE=/fsx/pytorch-fsdp.sqsh`, ~193 - TFLOPS/GPU) on the same 2× p6-b300. See Test 7 for both. **FSDP loss** stays at ln(vocab) - in this smoke test — a known dataloader/vocab quirk of the test case, not a cluster issue. -- **† B300 NCCL bandwidth scales past 16 GiB.** A 2-node / **16 GiB** all_reduce reaches only - ~654 GB/s busbw — it doesn't saturate all 16 EFA cards. Re-measured with larger messages on - 2× p6-b300: **64 GiB → ~751 GB/s** busbw (`found 16 nics`, `#wrong 0`), still climbing, so - 16 GiB was indeed unsaturated. **128 GiB and 256 GiB OOM** (the all_reduce buffer exceeds - B300 GPU memory), so ~64 GiB is the practical max single-buffer size here; for a true peak, - scale to more nodes rather than larger buffers. -- **‡ B300 GPU metrics work with the default `DcgmExporterImage`** — a DCGM 4.5.2 - build pinned by digest (`nvcr.io/nvidia/k8s/dcgm-exporter@sha256:a7ad6547…`), - validated on 2× p6-b300. Override only if you need to pin to a different DCGM build; - see [docs/OPERATIONS.md §3.1](../docs/OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it). -- **§ Slurm OpenMetrics is 25.11+ only.** On 25.05 the Slurm dashboards stay empty (the - rest of monitoring works fine). See - [docs/OPERATIONS.md §1](../docs/OPERATIONS.md#1-slurm-version-selection). - ---- - -## Test 1: Monitoring stack - -With `DeployMonitoring=true` (default), Prometheus/Grafana/exporters install on the -login node and DCGM/node exporters on the compute nodes. - -```bash -# On the login node: -docker ps --format "table {{.Names}}\t{{.Status}}" # login: prometheus, grafana, nginx, cloudwatch-exporter, node-exporter, pushgateway -tail -5 /var/log/monitoring-install.log # ends "...complete (exit 0)" -ls -la /opt/aws-parallelcluster-monitoring # installed on node-local /opt (NOT /home) -curl -s http://localhost:9090/api/v1/targets | \ - python3 -c 'import sys,json;[print(t["labels"].get("instance"),t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]' -curl -s http://localhost:6817/metrics | head # Slurm OpenMetrics -``` - -**Expected:** the six login containers are `Up`; install log exits 0; the tree is under -`/opt`; all Prometheus targets `up`; Slurm OpenMetrics returns Prometheus-format text. -For dashboard access (SSM port-forward or public CIDR) see -[README §8 Monitoring](../README.md#8-monitoring). Use `MonitoringVersion=v2.9.1`+ on PCS. - ---- - -## Test 2: Enroot/Pyxis container runtime (first-boot install) - -This is the default path used by every cluster that doesn't override `AmiId`: -`PostInstallScriptUrl` runs `install-enroot-pyxis.sh` once on each node at first boot. -The pre-baked-AMI path is validated separately as [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template). - -Deploy `pcs-ml-cluster-deploy-all.yaml` with both `AmiId` and `PostInstallScriptUrl` -left at their defaults (so SSM auto-resolves the latest PCS-Ready DLAMI and -post-install runs the Enroot/Pyxis installer), then on any node: - -```bash -which enroot # /usr/bin/enroot -ls /opt/aws/pcs/scheduler/slurm-*/lib/slurm/spank_pyxis.so # per-version Pyxis SPANK plugin -cat /etc/aws/pcs/scheduler/slurm-*/plugstack.conf.d/pyxis.conf # points at the matching .so -tail -1 /var/log/pcs-post-install.log # "...completed (exit 0)" -``` - -**Expected:** `enroot` on `PATH`; a `spank_pyxis.so` under the **cluster's** Slurm version -dir, and the plugstack `pyxis.conf` referencing that exact path; post-install log exits 0. -The Test 1/6/7 container jobs are the functional proof that Pyxis works. - -> **⚠️ Regression-test rule for `assets/scripts/install-enroot-pyxis.sh`.** This script has bitten -> us repeatedly in ways a single 25.11 GPU run does not catch. **Any change to it MUST be -> retested across the full matrix at the top of this guide**, specifically: -> - **All supported Slurm versions** (25.05 **and** 25.11). The Pyxis SPANK plugin is -> ABI-locked to its Slurm version — a plugin built for the wrong version stops slurmd from -> starting (`Incompatible Slurm plugin version`). The script builds Pyxis for the version -> passed in `PCS_SLURM_VERSION` and installs the `.so` to a per-version path; a regression -> here only shows on the *other* version. -> - **The pre-baked AMI path too** ([Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template)). -> `pcs-ready-dlami-with-enroot-pyxis.yaml` carries its **own copy** of the Enroot/Pyxis -> steps in its Image Builder UserData — editing `install-enroot-pyxis.sh` does **not** -> change the AMI path until you rebuild. Build an AMI per supported `SlurmVersion`, -> deploy a cluster pinned to it (`AmiId=` + `PostInstallScriptUrl=""`), and -> run a container job. -> - **On a clean first boot**, not a hand-patched node — post-install runs before -> slurmd/profile.d/controller exist, and several bugs only appear there. - ---- - -## Test 3: CPU queue - -Deployed by default as `cpu1` (`DeployOnDemandCNG=true`, `c6i.4xlarge`, 0–4 dynamic). - -```bash -sinfo # cpu1 partition present, nodes idle~ -srun --partition=cpu1 --nodes=1 hostname # a node powers up and runs -``` - -**Expected:** `cpu1` shows in `sinfo`; a dynamically-scaled node launches and the job -returns its hostname. - ---- - -## Test 4: G-series GPU (single NIC) - -Single-NIC GPU instances (`g5`/`g6`) use `add-cng.yaml` — deploy them as the On-Demand -CNG, e.g. `OnDemandInstanceType=g6.12xlarge`, `OnDemandQueueName=gpu-g6` (see -[README Example 2](../README.md#5-usage-examples)). - -```bash -srun --partition=gpu-g6 --nodes=1 --gres=gpu:1 \ - --container-image=docker://nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -``` - -**Expected:** the container runs and `nvidia-smi` lists the node's GPU(s) — confirms -single-NIC GPU + Pyxis on a G-series queue. - ---- - -## Test 5: P5/P6 GPU (multi-NIC) - -Multi-NIC GPU node groups, selected automatically by `PseriesInstanceType` -(`p5.48xlarge`/`p5e`/`p5en` → `add-cng-p5.yaml`; `p6-b200.48xlarge` → `add-cng-p6-b200`; -`p6-b300.48xlarge` → `add-cng-p6-b300`). The EFA interface count is derived from the -type — no parameter to set. A one-line interactive `srun` is enough for a GPU sanity -check (no batch script needed); set `--partition` to your GPU queue: - -```bash -srun --partition=gpu-p6b200 --nodes=1 --gres=gpu:8 \ - --container-image=docker://nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi -``` - -**Expected:** `nvidia-smi` lists 8 GPUs of the expected model (H100 / H200 / B200 / -B300). This confirms the multi-NIC launch template booted and Pyxis works on the GPU -node. EFA itself is exercised by Test 6. - ---- - -## Test 6: NCCL multi-node (EFA) - -2-node × 8-GPU `all_reduce_perf` over EFA, using the repo's canonical launcher -[`micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch`](../../../micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch) -(it reads `$IMAGE`, default `/fsx/nccl-tests.sqsh`). Only two PCS-specific deltas: - -1. **Import the image on the login node** — `enroot import` builds its overlayfs on the - node-local root disk (the login node has a 300 GiB root via `RootVolumeSize`); FSx - Lustre can't host that overlay, so only the resulting `.sqsh` goes to shared `/fsx`. - Pin a specific image tag for reproducible numbers (don't use `latest`): - - ```bash - # On the login node (direct, not a batch job). enroot URI form is - # docker://[REGISTRY#]REPO:TAG — the registry needs a '#', or it 401s on Docker Hub. - TAG=cuda12.8.1-efa1.43.2-ofiv1.16.3-ncclv2.27.7-1-testsv2.16.9 - enroot import -o /fsx/nccl-tests.sqsh "docker://public.ecr.aws#hpc-cloud/nccl-tests:${TAG}" - ``` - -2. **Submit with your GPU queue** — set the partition to the queue you deployed - (`gpu-p5` / `gpu-p6b200` / `gpu-p6b300`); the canonical script defaults to 2 nodes, - 8 tasks/node: - - ```bash - cd /fsx && sbatch --partition=gpu-p6b200 \ - /fsx/awsome-distributed-ai/micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch - ``` - -**Expected** (in `nccl-all_reduce_perf_.out`): -- EFA is the provider: `NET/OFI Selected provider is efa, fabric is efa-direct (found N nics)` - (N = EFA interface count: 32 for p5/p5e, 16 for p5en, 8 for p6-b200, 16 for p6-b300). -- Correctness: `# Out of bounds values : 0 OK`, every size `#wrong: 0`. -- `busbw` rises with message size — on 2× p6-b300, ~654 GB/s at 16 GiB but **~751 GB/s at - 64 GiB** (16 cards aren't saturated at 16 GiB); ~480 GB/s on 2× p5. - -> **Sizing the sweep on B300.** The canonical script sweeps to `-e 16G`. On p6-b300 that -> under-reports peak bandwidth — raise it to `-e 64G` (edit the `all_reduce_perf` line) to -> see the cards saturate. **Don't go to 128 GiB/256 GiB**: the all_reduce buffer exceeds -> B300 GPU memory and the job is OOM-killed. For a higher peak, add nodes, not buffer size. - ---- - -## Test 7: FSDP sample training - -A short FSDP Llama-2 7B run, using the repo's canonical training case -[`3.test_cases/pytorch/FSDP`](../../../3.test_cases/pytorch/FSDP) and its -[`slurm/llama2_7b-training.sbatch`](../../../3.test_cases/pytorch/FSDP/slurm/llama2_7b-training.sbatch). -Follow that case's README; the only PCS-specific deltas are where things live on the -shared filesystems and the node count: - -1. **Build the venv on shared `/fsx`** so every compute node sees it (the canonical - `slurm/create_venv.sh` creates `./env` in place — run it from a `/fsx` checkout): - - ```bash - cd /fsx && git clone --depth 1 https://github.com/awslabs/awsome-distributed-ai.git - cd /fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP/slurm && bash create_venv.sh - export HF_TOKEN=hf_xxx # gated Llama-2 tokenizer - ``` - -2. **Keep the HuggingFace cache on `/fsx` (Lustre), not `/home` (NFS).** Concurrent rank - file-locking on NFS throws `OSError: [Errno 116] Stale file handle`; export - `HF_HOME=/fsx/.hf-cache` before submitting. - -3. **Submit 2 nodes** (the canonical sbatch defaults to 4) on your GPU queue. The venv must - be on `PATH` for `torchrun` to resolve on every node — point `PATH` at the shared venv via - `--export` (the canonical sbatch doesn't `activate` it): - - ```bash - sbatch --nodes=2 --partition=gpu-p6b200 \ - --export=ALL,PATH=/fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP/slurm/env/bin:$PATH,HF_HOME=/fsx/.hf-cache \ - llama2_7b-training.sbatch - ``` - -### Option B — run it in an Enroot/Pyxis container instead of the venv - -The same canonical sbatch switches to container mode when `CONTAINER_IMAGE` is set (it adds -`--container-image`/`--container-mounts` and runs `./train.py` inside). Build the image once -on the login node and submit with `CONTAINER_IMAGE` — no venv needed: - -```bash -# On the login node (300 GiB root disk + Docker), build + import to /fsx: -cd /fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP -sudo docker build -t fsdp:pytorch -f Dockerfile . -enroot import -o /fsx/pytorch-fsdp.sqsh dockerd://fsdp:pytorch - -# Submit (container mode; mounts $(pwd) into /fsx inside the container): -cd slurm && sbatch --nodes=2 --partition=gpu-p6b300 \ - --export=ALL,CONTAINER_IMAGE=/fsx/pytorch-fsdp.sqsh,HF_HOME=/fsx/.hf-cache,HF_TOKEN=hf_xxx \ - llama2_7b-training.sbatch -``` - -> If the Dockerfile's `FROM` tag (a `public.ecr.aws/hpc-cloud/nccl-tests` tag) has been -> rotated out of the registry, substitute a current tag from that repo before building. - -**Expected** (in `logs/llama2_7b-FSDP_.out`), either path: -- NCCL initializes over EFA (`found N nics`) and training logs ~100 steps + a validation - step, saving checkpoints under `./checkpoints`. -- Throughput per step, e.g. on 2× p6-b300 **~200 TFLOPS/GPU, ~77k tokens/s** (venv) / - **~193 TFLOPS/GPU** (container); ~60 TFLOPS on 2× p5/H100. (Loss is constant at ln(vocab) - in this smoke test — a known dataloader/vocab quirk of the test case, not a cluster problem.) - -> **Multi-NIC tip:** the canonical sbatch already sets `NCCL_SOCKET_IFNAME=^docker,lo,veth,eth` -> (NCCL auto-selects). Do **not** pin a single interface on P5/P6 — all NICs share one -> subnet and pinning one breaks the cross-node NCCL bootstrap ring. - ---- - -## Test 8: Pre-baked AMI build (standalone DLAMI template) - -**When to run:** when `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes -in (`assets/scripts/install-enroot-pyxis.sh`) changes — the cluster stack does NOT run -Image Builder, so a fix to the install script is only in the AMI after a rebuild. -Skip this test if you only touched the cluster templates. - -This is an **independent flow**, not a deploy-all parameter: build an AMI with the -standalone template, then deploy a cluster pinned to that AMI ID with -`PostInstallScriptUrl=""` so nothing else runs at boot. - -The AMI is **single-Slurm-version by design** (Pyxis SPANK plugin ABI is -version-locked) — so when you run this test, run it for **every supported -`SlurmVersion`** that the install-script change could affect (typically both 25.05 -and 25.11). - -### Step 1 — build the AMI (~30 min one-time per Slurm version) - -```bash -SLURM_VERSION=25.11 # repeat for 25.05 if relevant - -aws cloudformation create-stack \ - --stack-name pcs-dlami-${SLURM_VERSION/./} \ - --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ - --parameters ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --profile claude --region us-west-2 - -aws cloudformation wait stack-create-complete \ - --stack-name pcs-dlami-${SLURM_VERSION/./} \ - --profile claude --region us-west-2 - -AMI_ID=$(aws cloudformation describe-stacks \ - --stack-name pcs-dlami-${SLURM_VERSION/./} \ - --query 'Stacks[0].Outputs[?OutputKey==`DLAMIforPCSAmiId`].OutputValue' \ - --output text --profile claude --region us-west-2) -echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx -``` - -### Step 2 — deploy a cluster pinned to that AMI - -```bash -aws cloudformation create-stack \ - --stack-name pcs-amitest-${SLURM_VERSION/./} \ - --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=us-west-2a \ - ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ - ParameterKey=AmiId,ParameterValue=${AMI_ID} \ - ParameterKey=PostInstallScriptUrl,ParameterValue= \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --profile claude --region us-west-2 -``` - -### Step 3 — verify the bake landed and a container job runs - -On any node from the new cluster: - -```bash -which enroot # /usr/bin/enroot (pre-baked) -ls /opt/aws/pcs/scheduler/slurm-${SLURM_VERSION}/lib/slurm/spank_pyxis.so # built for matching Slurm -cat /etc/aws/pcs/scheduler/slurm-${SLURM_VERSION}/plugstack.conf.d/pyxis.conf # references the .so -test ! -s /var/log/pcs-post-install.log && echo "post-install did not run (PostInstallScriptUrl='')" -``` - -Then a container job through the login node (same form as Test 2): - -```bash -srun --partition=cpu1 --nodes=1 --ntasks=1 \ - --container-image=ubuntu:22.04 bash -c "echo PYXIS_FROM_AMI_OK" -``` - -**Expected:** `enroot` and the per-version Pyxis files exist **without** the -post-install hook running (because `PostInstallScriptUrl=""`); the container job -prints `PYXIS_FROM_AMI_OK`. Slurmd starts cleanly (no `Incompatible Slurm plugin -version` in `journalctl -u slurmd`). - -### Step 4 — clean up - -```bash -aws cloudformation delete-stack --stack-name pcs-amitest-${SLURM_VERSION/./} --profile claude --region us-west-2 -aws cloudformation delete-stack --stack-name pcs-dlami-${SLURM_VERSION/./} --profile claude --region us-west-2 -``` - -The DLAMI stack's AMI itself is **not** automatically deregistered when the stack is -deleted — if you need to free its EBS snapshots, deregister the AMI manually -(`aws ec2 deregister-image --image-id $AMI_ID`) and delete the snapshot. - ---- - -## Test 9: EFA on CPU HPC instances (hpc6a / hpc7a / hpc8a) - -**When to run:** when `add-cng.yaml`'s EFA wiring (`EnableEfa`, `EfaInterfaceCount`, -`PlacementGroupName`) or the deploy-all forwarding params (`OnDemandEnableEfa`, -`OnDemandEfaInterfaceCount`, `OnDemandPlacementGroupName`) change. Skip if only -GPU/monitoring/AMI paths were touched. - -This validates that the on-demand CPU CNG actually launches with EFA NICs in a -cluster placement group, and that MPI / libfabric over EFA works end-to-end. The -verified-configurations table above documents the bandwidth numbers; this section -documents the **how-to** so a contributor can reproduce. - -### Step 1 — deploy with EFA on the CPU CNG - -```bash -# hpc7a / hpc8a have 2 EFA NICs; hpc6a has 1. -INSTANCE_TYPE=hpc7a.96xlarge -EFA_NICS=2 - -# AZ availability is region-specific — confirm with describe-instance-type-offerings: -# hpc7a is in us-east-2b (and others); hpc8a is in us-east-2b / eu-north-1 / ap-northeast-1; -# hpc6a is in us-east-2 (b) / us-west-2 / eu-west-1 etc. AZ MUST contain the type. -AWS_AZ=us-east-2b - -aws cloudformation create-stack \ - --stack-name pcs-hpc-efa \ - --region us-east-2 \ - --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=$AWS_AZ \ - ParameterKey=DeployOnDemandCNG,ParameterValue=true \ - ParameterKey=OnDemandInstanceType,ParameterValue=$INSTANCE_TYPE \ - ParameterKey=OnDemandCngName,ParameterValue=hpc \ - ParameterKey=OnDemandQueueName,ParameterValue=hpc \ - ParameterKey=OnDemandMinCount,ParameterValue=0 \ - ParameterKey=OnDemandMaxCount,ParameterValue=2 \ - ParameterKey=OnDemandEnableEfa,ParameterValue=true \ - ParameterKey=OnDemandEfaInterfaceCount,ParameterValue=$EFA_NICS \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM -``` - -The `OnDemandCNGStack` nested stack auto-creates a cluster placement group and -exposes the name as a stack output (`PlacementGroupName`). To share the PG across -multiple CNGs (heterogeneous tightly-coupled jobs), pass -`OnDemandPlacementGroupName=` instead. - -### Step 2 — verify EFA visibility on a compute node - -After the stack reaches CREATE_COMPLETE and an `srun` has woken up a node: - -```bash -# On a compute node (via Slurm srun from login): -srun -p hpc -N 1 -n 1 bash -c ' - /opt/amazon/efa/bin/fi_info -p efa | head -20 # provider=efa, FI_EP_RDM - lspci | grep -iE "EFA|Elastic" # 2 EFA + 2 ENA on hpc7a/hpc8a -' -``` - -Expected (hpc7a / hpc8a): -- `fi_info -p efa`: shows `efa-direct` and `efa` fabrics on `rdmap36s0` and - `rdmap42s0` (or matching device names). -- `lspci`: 2 lines `Elastic Fabric Adapter` (or `Device efa3` on hpc8a) + 2 lines - `Elastic Network Adapter`. - -### Step 3 — OSU MPI micro-benchmarks - -Build OSU 7.4 on `/fsx` once (shared across compute nodes) using the -PCS-Ready DLAMI's `/opt/amazon/openmpi`: - -```bash -mkdir -p /fsx/osu && cd /fsx/osu -curl -fL -o osu.tgz https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.4.tar.gz -tar xf osu.tgz && cd osu-micro-benchmarks-7.4 -PATH=/opt/amazon/openmpi/bin:$PATH ./configure CC=mpicc CXX=mpicxx --prefix=/fsx/osu -PATH=/opt/amazon/openmpi/bin:$PATH make -j8 -``` - -Submit a 2-node sbatch with the AWS-tuned EFA env (the canonical reference -parameters, see "Tuning notes" in [Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue)): - -```bash -cat > /fsx/osu/osu-bench.sbatch <<'EOF' -#!/bin/bash -#SBATCH --job-name=osu-efa -#SBATCH --partition=hpc -#SBATCH --nodes=2 -#SBATCH --exclusive -#SBATCH --output=/fsx/osu/logs/%x_%j.out - -set -ex -OSU=/fsx/osu/osu-micro-benchmarks-7.4 -export PATH=/opt/amazon/openmpi/bin:$PATH - -export FI_PROVIDER=efa -export FI_EFA_FORK_SAFE=1 # huge page stays at the default (=1, on) -export OMPI_MCA_pml=cm # AWS Open MPI 4.1.7 has no pml=ofi -export OMPI_MCA_mtl=ofi -export OMPI_MCA_mtl_ofi_provider_include=efa -export OMPI_MCA_btl=^openib,tcp - -MPI_X="-x FI_PROVIDER -x FI_EFA_FORK_SAFE \ - -x OMPI_MCA_pml -x OMPI_MCA_mtl -x OMPI_MCA_mtl_ofi_provider_include \ - -x OMPI_MCA_btl" - -mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_latency -mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_bw -mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_bibw -mpirun -np 32 -N 16 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_mbw_mr -mpirun -np 32 -N 16 $MPI_X $OSU/c/mpi/collective/blocking/osu_allreduce -EOF - -mkdir -p /fsx/osu/logs -sbatch -p hpc /fsx/osu/osu-bench.sbatch -``` - -The reference numbers per instance type are in -[Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue) above. - -### Step 4 — observe NIC-level traffic in Grafana (optional) - -The monitoring stack's Compute Node Details dashboard has dedicated EFA panels -sourced from the `node_amazonefa_*` metrics produced by the v2.7+ -`efa-metrics.sh` textfile collector: - -- RDMA Read / Write Throughput -- SRD Retransmitted Packets -- Work-Request Errors - -For per-NIC `tx_bytes` / `rx_bytes` rate during a benchmark, query Prometheus -directly: - -```promql -rate(node_amazonefa_tx_bytes[30s]) * 8 / 1e9 # Gbps per (instance, device) -sum by (instance) (rate(node_amazonefa_tx_bytes[30s])) * 8 / 1e9 # both NICs -``` - -(The textfile collector cadence is 30s; rates over windows shorter than that are -zeros most of the time. OSU sub-tests are also short — 10–30s each — so wall-clock -peak in Prometheus typically reads below the OSU-reported peak.) - -### Step 5 — clean up - -When done with EFA testing, just delete the CNG stack (or the whole deploy-all -stack). The auto-created cluster placement group is owned by the CNG stack and -is removed automatically. Slurm puts the EFA compute nodes to sleep on idle -(`SuspendTime`), so leaving the cluster up between benchmarks does not keep the -hpc7a/hpc8a instances running. - ---- - -## Test 10: FSx storage health - -Validates that both shared filesystems (Lustre on `/fsx`, OpenZFS on `/home`) -mount cleanly on every node, are usable, and that the FSx-side configuration -matches what the template asked for. Most of this is exercised implicitly by -Tests 1–9 (the post-install script, monitoring stack, OSU / FSDP all touch -`/fsx`); this test is the explicit health check to run after a fresh deploy -or after touching `ml-cluster-prerequisites.yaml` / FSx-related parameters. - -### Step 1 — both filesystems mounted on every node - -On the login node and at least one compute node (via `srun`): - -```bash -mount | grep -E ' /home | /fsx ' -df -h /home /fsx -``` - -Expected: -- `/fsx` mounted as type `lustre`, source `.fsx..amazonaws.com@tcp:/`, - size matches the `Capacity` parameter (1200 GiB default; 19200 GiB or larger - when `FSxLustreEnableEfa=true`). -- `/home` mounted as type `nfs` over OpenZFS, NFS options include - `nconnect=16,rsize=1048576,wsize=1048576` (the deploy-all UserData mount - string). -- Both `df -h` reports show `Avail` greater than zero. - -If a mount is missing on a freshly booted node, check -`/var/log/cloud-init-output.log` and `/var/log/pcs-post-install.log` for the -`mount` line. The most common first-boot failure is the OpenZFS DNS name not -being resolvable yet (NFS settle race); the post-install log will show -`mount.nfs: Failed to resolve server`. - -### Step 2 — read/write sanity - -```bash -# /fsx (Lustre): write 1 GiB, read it back -dd if=/dev/zero of=/fsx/.healthcheck bs=1M count=1024 conv=fsync 2>&1 | tail -1 -dd if=/fsx/.healthcheck of=/dev/null bs=1M 2>&1 | tail -1 -rm /fsx/.healthcheck - -# /home (OpenZFS / NFS): same, 100 MiB (it's a small home filesystem) -dd if=/dev/zero of=/home/ubuntu/.healthcheck bs=1M count=100 conv=fsync 2>&1 | tail -1 -dd if=/home/ubuntu/.healthcheck of=/dev/null bs=1M 2>&1 | tail -1 -rm /home/ubuntu/.healthcheck -``` - -Expected: both write+read complete without error. Throughput is bounded by the -single-stream limits of NFS / Lustre on a single node — this is a sanity check, -not a benchmark. For Lustre throughput numbers, see the FSx Lustre User Guide -(provisioned throughput = `Capacity * PerUnitStorageThroughput / 1024` MB/s). - -### Step 3 — FSx-side parameters match the CFN inputs - -```bash -FSX_ID=$(aws cloudformation describe-stacks \ - --stack-name \ - --query 'Stacks[0].Outputs[?OutputKey==`FSxLustreFilesystemId`].OutputValue' \ - --region --output text) - -aws fsx describe-file-systems --file-system-ids "$FSX_ID" --region \ - --query 'FileSystems[0].[StorageCapacity,StorageType,LustreConfiguration.[DeploymentType,PerUnitStorageThroughput,DataCompressionType,EfaEnabled,MetadataConfiguration.Mode]]' \ - --output text -``` - -Expected (default deploy): -- `StorageCapacity` = your `Capacity` parameter (1200 by default) -- `StorageType` = `SSD` -- `DeploymentType` = `PERSISTENT_2` (default; or `PERSISTENT_1` if you set it) -- `PerUnitStorageThroughput` = 250 (default) -- `DataCompressionType` = `LZ4` (default) -- `EfaEnabled` = `False` (default; `True` when `FSxLustreEnableEfa=true`. EFA on - FSx is a PERSISTENT_2-only feature — the prerequisites and deploy-all templates - enforce this with a CFN Rule that fails the stack at create time when - `FSxLustreEnableEfa=true` is combined with `LustreDeploymentType=PERSISTENT_1`) -- `MetadataConfiguration.Mode` = `AUTOMATIC` on PERSISTENT_2 - -For the OpenZFS `/home` filesystem: - -```bash -FSXO_ID=$(aws cloudformation describe-stacks \ - --stack-name \ - --query 'Stacks[0].Outputs[?OutputKey==`FSxOFilesystemId`].OutputValue' \ - --region --output text) - -aws fsx describe-file-systems --file-system-ids "$FSXO_ID" --region \ - --query 'FileSystems[0].[StorageCapacity,OpenZFSConfiguration.[DeploymentType,ThroughputCapacity]]' \ - --output text -``` - -### Step 4 — Storage dashboard in Grafana - -The `Compute Node Details` and `HPC Cluster Monitoring → Storage` Grafana -dashboards are populated by the **CloudWatch Exporter** (FSx CloudWatch -metrics, scraped by the monitoring stack on the login node — see the -[aws-parallelcluster-monitoring v2.6 release notes](https://github.com/aws-samples/aws-parallelcluster-monitoring/releases/tag/v2.6)). - -Open Grafana (Test 1's port-forward / public CIDR), go to the Storage -dashboard, and verify the `/fsx` panels (Throughput, IOPS, Free Capacity, -Client Connections) populate within ~5 minutes of a workload starting. CW -metrics have a ~5 min publishing delay, so a brand-new filesystem with no I/O -shows blank panels for a while; the `dd` from Step 2 is enough to seed values. - -### `FSxLustreEnableEfa=true` specifics - -When `FSxLustreEnableEfa=true` is set on a PERSISTENT_2 SSD filesystem, the -extra checks beyond the above are: - -- `aws fsx describe-file-systems` `LustreConfiguration.EfaEnabled = true` -- `Capacity` is at-or-above the EFA minimum for the chosen - `PerUnitStorageThroughput` tier (19200 GiB for tier 250; the FSx for - Lustre User Guide has the full matrix). Below the minimum, the FSx side - rejects the `CreateFileSystem` with `Invalid storage capacity provided: - N GiB. Minimum storage capacity for an EFA enabled LUSTRE file systems - with deployment type PERSISTENT_2, per unit storage throughput X and - storage type SSD is M`. The Lustre nested stack fails first, then the - whole stack rolls back; that's the expected behavior for an undersized - Capacity. - -The FSx-side EFA endpoints are usable from EFA-capable clients (CPU CNGs -deployed with `OnDemandEnableEfa=true`, P5/P6 GPU CNGs). Plain Lustre client -mounts continue to work over TCP for non-EFA nodes; EFA support is additive. +| NCCL `all_reduce` | [`micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch`](../../../micro-benchmarks/nccl-tests) | Partition name, Enroot import on login node | +| FSDP Llama-2 7B | [`3.test_cases/pytorch/FSDP`](../../../3.test_cases/pytorch/FSDP) | Cache on `/fsx`, 2 nodes | +| GPU Health Check | [`4.validation_and_observability/2.gpu-cluster-healthcheck`](../../../4.validation_and_observability/2.gpu-cluster-healthcheck) | sbatch wrapper, partition name | --- -## Test 11: Multi-user directory (OpenLDAP) - -See **[`multi-user-test.md`](./multi-user-test.md)** for the full test procedure. +## Notes -Run when `DirectoryService=OpenLDAP-LoginNode` is enabled. Covers: LDAP server health, -user lifecycle (create/delete/verify on login + compute), Slurm job execution -as LDAP user, multi-node UID consistency, home directory sharing, SSSD cache -resilience, and LDAP DB persistence across login node replacement. - ---- - -## Test 12: Slurm managed accounting + multi-user - -See **[`accounting-test.md`](./accounting-test.md)** for the full test procedure. - -Run when `ManagedAccounting=enabled` and `DirectoryService=OpenLDAP-LoginNode`. -Covers: sacctmgr user/account registration, job submission as LDAP users, -sacct/sreport verification, resource limit enforcement, and -AccountingPolicyEnforcement behavior. - ---- - -## Test 13: GPU cluster health check - -See **[`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md)** for the full test procedure. - -Run after any GPU CNG (P5/P5e/P5en/P6-B200) deployment. Uses the -[GPU Cluster Health Check Suite](../../4.validation_and_observability/2.gpu-cluster-healthcheck) -from this repository. Covers: nvidia-smi, DCGM L2, EFA enumeration, topology -validation, and multi-node NCCL bandwidth thresholds. +- All Slurm commands run as `ubuntu` from the login node (SSM or SSH) +- Slurm binaries: `export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH` +- Template lint: `aws cloudformation validate-template --template-body file://assets/.yaml` +- Regression criteria (storage): >10% degradation blocks the change --- @@ -830,13 +60,14 @@ aws cloudformation delete-stack --stack-name aws cloudformation wait stack-delete-complete --stack-name ``` -Nested stacks (and FSx) are deleted automatically — back up FSx data first. A Capacity -Block keeps billing for its full reserved window regardless and is not released by stack -deletion. - -## References +Nested stacks (and FSx) are deleted automatically — back up FSx data first. -- [AWS PCS Documentation](https://docs.aws.amazon.com/pcs/) -- [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) -- [Slurm OpenMetrics](https://slurm.schedmd.com/rest.html#openmetrics) · [DCGM Exporter](https://github.com/NVIDIA/dcgm-exporter) -- [AI/ML for AWS PCS Workshop](https://catalog.workshops.aws/ml-on-pcs/) +If DELETE_FAILED on CNG stacks (PCS timing dependency), delete PCS CNGs first: +```bash +CLUSTER_ID= +for cng in $(aws pcs list-compute-node-groups --cluster-identifier $CLUSTER_ID --query 'computeNodeGroups[].id' --output text); do + aws pcs delete-compute-node-group --cluster-identifier $CLUSTER_ID --compute-node-group-identifier $cng +done +sleep 60 +aws cloudformation delete-stack --stack-name +``` diff --git a/architectures/aws-pcs/tests/accounting-test.md b/architectures/aws-pcs/tests/accounting-test.md deleted file mode 100644 index 81d113663..000000000 --- a/architectures/aws-pcs/tests/accounting-test.md +++ /dev/null @@ -1,194 +0,0 @@ -# Test: Slurm Managed Accounting + Multi-User - -Validates that Slurm managed accounting works with LDAP multi-user setup. -Covers user/account creation, resource limit enforcement, job tracking, and -reporting. - -**Prerequisites**: -- Cluster with `ManagedAccounting=enabled` and `DirectoryService=OpenLDAP-LoginNode` -- Slurm 25.11 (accounting is available on 24.11+; templates default to 25.11) -- At least one compute node available - -All commands run on the login node as root unless noted. - ---- - -## Part A — Accounting infrastructure - -### A1. Verify accounting is active - -```bash -export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH -sacctmgr show cluster -``` - -Expected: cluster name listed with a valid `ControlHost`. - -### A2. Default account exists - -```bash -sacctmgr show account -``` - -If no accounts exist yet, create the default: -```bash -sacctmgr -i add account default Description="Default account" -``` - ---- - -## Part B — User + account registration - -### B1. Create LDAP users (if not done already) - -```bash -ADMIN_PW=$(aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ - --with-decryption --query 'Parameter.Value' --output text --region us-east-2) - -sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user alice 10001 3000 -sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user bob 10002 3000 -``` - -### B2. Register users in Slurm accounting - -```bash -sacctmgr -i add account ml-team Description="ML Research Team" -sacctmgr -i add user alice Account=ml-team -sacctmgr -i add user bob Account=ml-team - -# Verify -sacctmgr show user alice bob format=User,Account,DefaultAccount -``` - -Expected: -``` - User Account DefaultAccount ---------- ---------- -------------- - alice ml-team ml-team - bob ml-team ml-team -``` - -### B3. Set resource limits (optional) - -```bash -# Cap alice at 100 CPU-hours -sacctmgr -i modify user alice set GrpTRESRunMins=cpu=6000 - -# Verify -sacctmgr show user alice format=User,Account,GrpTRESRunMins -``` - ---- - -## Part C — Job submission and tracking - -### C1. Submit jobs as LDAP users - -```bash -# As alice -sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ - sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J alice-test' - -# As bob -sudo su - bob -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ - sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J bob-test' -``` - -Wait for jobs to complete: -```bash -squeue # should show jobs running then empty -``` - -### C2. Verify job history with sacct - -```bash -sacct --starttime=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ - --format="JobID,User,JobName,Partition,Account,AllocCPUS,State,ExitCode,Elapsed" -``` - -Expected: both jobs listed with correct User, Account, State=COMPLETED. - -### C3. Per-user reporting - -```bash -# Jobs by alice -sacct -u alice --format="JobID,JobName,State,Start,End,Elapsed" - -# Utilization by account -sreport cluster AccountUtilizationByUser \ - start=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ - format="Account,Login,Used" -``` - -### C4. Resource limit enforcement (if B3 was set) - -```bash -# Submit a job that would exceed alice's limit -sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ - sbatch --partition=cpu1 --ntasks=96 --time=120:00 --wrap="sleep 7200" -J limit-test' -``` - -Check if job is pending with reason `AssocGrpCPURunMinutesLimit`: -```bash -squeue -u alice --format="%i %j %T %r" -``` - ---- - -## Part D — Fairshare (optional) - -### D1. Set different fairshare weights - -```bash -sacctmgr -i modify account ml-team set FairShare=100 -sacctmgr -i add account ops-team Description="Operations" -sacctmgr -i modify account ops-team set FairShare=50 -``` - -### D2. Check fairshare values - -```bash -sshare -a --format=Account,User,RawShares,NormShares,RawUsage,FairShare -``` - ---- - -## Part E — AccountingPolicyEnforcement - -### E1. Test with enforcement=none (default) - -An unregistered user can still submit jobs: -```bash -# Create a user NOT in sacctmgr -sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user charlie 10003 3000 -sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ - srun -p cpu1 -N1 -n1 hostname' -``` - -Expected: job runs (no accounting enforcement). - -### E2. Test with enforcement=associations,limits,safe - -If `AccountingPolicyEnforcement=associations,limits,safe` was set at cluster -creation: -```bash -# charlie (not in sacctmgr) should be rejected -sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ - srun -p cpu1 -N1 -n1 hostname' -``` - -Expected: `error: Unable to allocate resources: Invalid account or account/partition combination specified` - ---- - -## Verdict checklist - -| Check | Expected | -|---|---| -| sacctmgr show cluster → cluster listed | ✅ | -| Users registered in accounting (alice, bob) | ✅ | -| Jobs run as LDAP users complete successfully | ✅ | -| sacct shows correct user/account/state per job | ✅ | -| sreport shows per-user utilization | ✅ | -| Resource limit enforcement (if set) blocks over-limit jobs | ✅ | -| Unregistered user behavior matches AccountingPolicyEnforcement setting | ✅ | diff --git a/architectures/aws-pcs/tests/compute-test.md b/architectures/aws-pcs/tests/compute-test.md new file mode 100644 index 000000000..7ca7f661e --- /dev/null +++ b/architectures/aws-pcs/tests/compute-test.md @@ -0,0 +1,82 @@ +# Compute Tests (Tests 4-6) + +Validates CPU queue, GPU families (single-NIC G-series + multi-NIC P-series), +and NCCL multi-node communication over EFA. + +--- + +## Test 4: G-series GPU (single NIC) + +Single-NIC GPU instances (`g5`/`g6`) use `add-cng.yaml` — deploy them as the On-Demand +CNG, e.g. `OnDemandInstanceType=g6.12xlarge`, `OnDemandQueueName=gpu-g6` (see +[README Example 2](../README.md#5-usage-examples)). + +```bash +srun --partition=gpu-g6 --nodes=1 --gres=gpu:1 \ + --container-image=docker://nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi +``` + +**Expected:** the container runs and `nvidia-smi` lists the node's GPU(s) — confirms +single-NIC GPU + Pyxis on a G-series queue. + +--- + +## Test 5: P5/P6 GPU (multi-NIC) + +Multi-NIC GPU node groups, selected automatically by `PseriesInstanceType` +(`p5.48xlarge`/`p5e`/`p5en` → `add-cng-p5.yaml`; `p6-b200.48xlarge` → `add-cng-p6-b200`; +`p6-b300.48xlarge` → `add-cng-p6-b300`). The EFA interface count is derived from the +type — no parameter to set. A one-line interactive `srun` is enough for a GPU sanity +check (no batch script needed); set `--partition` to your GPU queue: + +```bash +srun --partition=gpu-p6b200 --nodes=1 --gres=gpu:8 \ + --container-image=docker://nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi +``` + +**Expected:** `nvidia-smi` lists 8 GPUs of the expected model (H100 / H200 / B200 / +B300). This confirms the multi-NIC launch template booted and Pyxis works on the GPU +node. EFA itself is exercised by Test 6. + +--- + +## Test 6: NCCL multi-node (EFA) + +2-node × 8-GPU `all_reduce_perf` over EFA, using the repo's canonical launcher +[`micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch`](../../../micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch) +(it reads `$IMAGE`, default `/fsx/nccl-tests.sqsh`). Only two PCS-specific deltas: + +1. **Import the image on the login node** — `enroot import` builds its overlayfs on the + node-local root disk (the login node has a 300 GiB root via `RootVolumeSize`); FSx + Lustre can't host that overlay, so only the resulting `.sqsh` goes to shared `/fsx`. + Pin a specific image tag for reproducible numbers (don't use `latest`): + + ```bash + # On the login node (direct, not a batch job). enroot URI form is + # docker://[REGISTRY#]REPO:TAG — the registry needs a '#', or it 401s on Docker Hub. + TAG=cuda12.8.1-efa1.43.2-ofiv1.16.3-ncclv2.27.7-1-testsv2.16.9 + enroot import -o /fsx/nccl-tests.sqsh "docker://public.ecr.aws#hpc-cloud/nccl-tests:${TAG}" + ``` + +2. **Submit with your GPU queue** — set the partition to the queue you deployed + (`gpu-p5` / `gpu-p6b200` / `gpu-p6b300`); the canonical script defaults to 2 nodes, + 8 tasks/node: + + ```bash + cd /fsx && sbatch --partition=gpu-p6b200 \ + /fsx/awsome-distributed-ai/micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch + ``` + +**Expected** (in `nccl-all_reduce_perf_.out`): +- EFA is the provider: `NET/OFI Selected provider is efa, fabric is efa-direct (found N nics)` + (N = EFA interface count: 32 for p5/p5e, 16 for p5en, 8 for p6-b200, 16 for p6-b300). +- Correctness: `# Out of bounds values : 0 OK`, every size `#wrong: 0`. +- `busbw` rises with message size — on 2× p6-b300, ~654 GB/s at 16 GiB but **~751 GB/s at + 64 GiB** (16 cards aren't saturated at 16 GiB); ~480 GB/s on 2× p5. + +> **Sizing the sweep on B300.** The canonical script sweeps to `-e 16G`. On p6-b300 that +> under-reports peak bandwidth — raise it to `-e 64G` (edit the `all_reduce_perf` line) to +> see the cards saturate. **Don't go to 128 GiB/256 GiB**: the all_reduce buffer exceeds +> B300 GPU memory and the job is OOM-killed. For a higher peak, add nodes, not buffer size. + +--- diff --git a/architectures/aws-pcs/tests/hpc-efa-test.md b/architectures/aws-pcs/tests/hpc-efa-test.md new file mode 100644 index 000000000..968841470 --- /dev/null +++ b/architectures/aws-pcs/tests/hpc-efa-test.md @@ -0,0 +1,156 @@ +# HPC EFA Tests (Test 9) + +Validates EFA on CPU HPC instances (hpc6a/hpc7a/hpc8a) including placement +group auto-creation, multi-NIC wiring, and OSU MPI benchmark results. + +--- + +## Test 9: EFA on CPU HPC instances (hpc6a / hpc7a / hpc8a) + +**When to run:** when `add-cng.yaml`'s EFA wiring (`EnableEfa`, `EfaInterfaceCount`, +`PlacementGroupName`) or the deploy-all forwarding params (`OnDemandEnableEfa`, +`OnDemandEfaInterfaceCount`, `OnDemandPlacementGroupName`) change. Skip if only +GPU/monitoring/AMI paths were touched. + +This validates that the on-demand CPU CNG actually launches with EFA NICs in a +cluster placement group, and that MPI / libfabric over EFA works end-to-end. The +verified-configurations table above documents the bandwidth numbers; this section +documents the **how-to** so a contributor can reproduce. + +### Step 1 — deploy with EFA on the CPU CNG + +```bash +# hpc7a / hpc8a have 2 EFA NICs; hpc6a has 1. +INSTANCE_TYPE=hpc7a.96xlarge +EFA_NICS=2 + +# AZ availability is region-specific — confirm with describe-instance-type-offerings: +# hpc7a is in us-east-2b (and others); hpc8a is in us-east-2b / eu-north-1 / ap-northeast-1; +# hpc6a is in us-east-2 (b) / us-west-2 / eu-west-1 etc. AZ MUST contain the type. +AWS_AZ=us-east-2b + +aws cloudformation create-stack \ + --stack-name pcs-hpc-efa \ + --region us-east-2 \ + --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=$AWS_AZ \ + ParameterKey=DeployOnDemandCNG,ParameterValue=true \ + ParameterKey=OnDemandInstanceType,ParameterValue=$INSTANCE_TYPE \ + ParameterKey=OnDemandCngName,ParameterValue=hpc \ + ParameterKey=OnDemandQueueName,ParameterValue=hpc \ + ParameterKey=OnDemandMinCount,ParameterValue=0 \ + ParameterKey=OnDemandMaxCount,ParameterValue=2 \ + ParameterKey=OnDemandEnableEfa,ParameterValue=true \ + ParameterKey=OnDemandEfaInterfaceCount,ParameterValue=$EFA_NICS \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` + +The `OnDemandCNGStack` nested stack auto-creates a cluster placement group and +exposes the name as a stack output (`PlacementGroupName`). To share the PG across +multiple CNGs (heterogeneous tightly-coupled jobs), pass +`OnDemandPlacementGroupName=` instead. + +### Step 2 — verify EFA visibility on a compute node + +After the stack reaches CREATE_COMPLETE and an `srun` has woken up a node: + +```bash +# On a compute node (via Slurm srun from login): +srun -p hpc -N 1 -n 1 bash -c ' + /opt/amazon/efa/bin/fi_info -p efa | head -20 # provider=efa, FI_EP_RDM + lspci | grep -iE "EFA|Elastic" # 2 EFA + 2 ENA on hpc7a/hpc8a +' +``` + +Expected (hpc7a / hpc8a): +- `fi_info -p efa`: shows `efa-direct` and `efa` fabrics on `rdmap36s0` and + `rdmap42s0` (or matching device names). +- `lspci`: 2 lines `Elastic Fabric Adapter` (or `Device efa3` on hpc8a) + 2 lines + `Elastic Network Adapter`. + +### Step 3 — OSU MPI micro-benchmarks + +Build OSU 7.4 on `/fsx` once (shared across compute nodes) using the +PCS-Ready DLAMI's `/opt/amazon/openmpi`: + +```bash +mkdir -p /fsx/osu && cd /fsx/osu +curl -fL -o osu.tgz https://mvapich.cse.ohio-state.edu/download/mvapich/osu-micro-benchmarks-7.4.tar.gz +tar xf osu.tgz && cd osu-micro-benchmarks-7.4 +PATH=/opt/amazon/openmpi/bin:$PATH ./configure CC=mpicc CXX=mpicxx --prefix=/fsx/osu +PATH=/opt/amazon/openmpi/bin:$PATH make -j8 +``` + +Submit a 2-node sbatch with the AWS-tuned EFA env (the canonical reference +parameters, see "Tuning notes" in [Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue)): + +```bash +cat > /fsx/osu/osu-bench.sbatch <<'EOF' +#!/bin/bash +#SBATCH --job-name=osu-efa +#SBATCH --partition=hpc +#SBATCH --nodes=2 +#SBATCH --exclusive +#SBATCH --output=/fsx/osu/logs/%x_%j.out + +set -ex +OSU=/fsx/osu/osu-micro-benchmarks-7.4 +export PATH=/opt/amazon/openmpi/bin:$PATH + +export FI_PROVIDER=efa +export FI_EFA_FORK_SAFE=1 # huge page stays at the default (=1, on) +export OMPI_MCA_pml=cm # AWS Open MPI 4.1.7 has no pml=ofi +export OMPI_MCA_mtl=ofi +export OMPI_MCA_mtl_ofi_provider_include=efa +export OMPI_MCA_btl=^openib,tcp + +MPI_X="-x FI_PROVIDER -x FI_EFA_FORK_SAFE \ + -x OMPI_MCA_pml -x OMPI_MCA_mtl -x OMPI_MCA_mtl_ofi_provider_include \ + -x OMPI_MCA_btl" + +mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_latency +mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_bw +mpirun -np 2 -N 1 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_bibw +mpirun -np 32 -N 16 $MPI_X $OSU/c/mpi/pt2pt/standard/osu_mbw_mr +mpirun -np 32 -N 16 $MPI_X $OSU/c/mpi/collective/blocking/osu_allreduce +EOF + +mkdir -p /fsx/osu/logs +sbatch -p hpc /fsx/osu/osu-bench.sbatch +``` + +The reference numbers per instance type are in +[Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue) above. + +### Step 4 — observe NIC-level traffic in Grafana (optional) + +The monitoring stack's Compute Node Details dashboard has dedicated EFA panels +sourced from the `node_amazonefa_*` metrics produced by the v2.7+ +`efa-metrics.sh` textfile collector: + +- RDMA Read / Write Throughput +- SRD Retransmitted Packets +- Work-Request Errors + +For per-NIC `tx_bytes` / `rx_bytes` rate during a benchmark, query Prometheus +directly: + +```promql +rate(node_amazonefa_tx_bytes[30s]) * 8 / 1e9 # Gbps per (instance, device) +sum by (instance) (rate(node_amazonefa_tx_bytes[30s])) * 8 / 1e9 # both NICs +``` + +(The textfile collector cadence is 30s; rates over windows shorter than that are +zeros most of the time. OSU sub-tests are also short — 10–30s each — so wall-clock +peak in Prometheus typically reads below the OSU-reported peak.) + +### Step 5 — clean up + +When done with EFA testing, just delete the CNG stack (or the whole deploy-all +stack). The auto-created cluster placement group is owned by the CNG stack and +is removed automatically. Slurm puts the EFA compute nodes to sleep on idle +(`SuspendTime`), so leaving the cluster up between benchmarks does not keep the +hpc7a/hpc8a instances running. + +--- diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md new file mode 100644 index 000000000..641098e32 --- /dev/null +++ b/architectures/aws-pcs/tests/infra-test.md @@ -0,0 +1,174 @@ +# Infrastructure Tests (Tests 1-3, 8) + +Validates cluster infrastructure: monitoring stack, container runtime (Enroot/Pyxis) +first-boot install, and the pre-baked AMI build path. + +--- + +## Test 1: Monitoring stack + +With `DeployMonitoring=true` (default), Prometheus/Grafana/exporters install on the +login node and DCGM/node exporters on the compute nodes. + +```bash +# On the login node: +docker ps --format "table {{.Names}}\t{{.Status}}" # login: prometheus, grafana, nginx, cloudwatch-exporter, node-exporter, pushgateway +tail -5 /var/log/monitoring-install.log # ends "...complete (exit 0)" +ls -la /opt/aws-parallelcluster-monitoring # installed on node-local /opt (NOT /home) +curl -s http://localhost:9090/api/v1/targets | \ + python3 -c 'import sys,json;[print(t["labels"].get("instance"),t["health"]) for t in json.load(sys.stdin)["data"]["activeTargets"]]' +curl -s http://localhost:6817/metrics | head # Slurm OpenMetrics +``` + +**Expected:** the six login containers are `Up`; install log exits 0; the tree is under +`/opt`; all Prometheus targets `up`; Slurm OpenMetrics returns Prometheus-format text. +For dashboard access (SSM port-forward or public CIDR) see +[README §8 Monitoring](../README.md#8-monitoring). Use `MonitoringVersion=v2.9.1`+ on PCS. + +--- + +## Test 2: Enroot/Pyxis container runtime (first-boot install) + +This is the default path used by every cluster that doesn't override `AmiId`: +`PostInstallScriptUrl` runs `install-enroot-pyxis.sh` once on each node at first boot. +The pre-baked-AMI path is validated separately as [Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template). + +Deploy `pcs-ml-cluster-deploy-all.yaml` with both `AmiId` and `PostInstallScriptUrl` +left at their defaults (so SSM auto-resolves the latest PCS-Ready DLAMI and +post-install runs the Enroot/Pyxis installer), then on any node: + +```bash +which enroot # /usr/bin/enroot +ls /opt/aws/pcs/scheduler/slurm-*/lib/slurm/spank_pyxis.so # per-version Pyxis SPANK plugin +cat /etc/aws/pcs/scheduler/slurm-*/plugstack.conf.d/pyxis.conf # points at the matching .so +tail -1 /var/log/pcs-post-install.log # "...completed (exit 0)" +``` + +**Expected:** `enroot` on `PATH`; a `spank_pyxis.so` under the **cluster's** Slurm version +dir, and the plugstack `pyxis.conf` referencing that exact path; post-install log exits 0. +The Test 1/6/7 container jobs are the functional proof that Pyxis works. + +> **⚠️ Regression-test rule for `assets/scripts/install-enroot-pyxis.sh`.** This script has bitten +> us repeatedly in ways a single 25.11 GPU run does not catch. **Any change to it MUST be +> retested across the full matrix at the top of this guide**, specifically: +> - **All supported Slurm versions** (25.05 **and** 25.11). The Pyxis SPANK plugin is +> ABI-locked to its Slurm version — a plugin built for the wrong version stops slurmd from +> starting (`Incompatible Slurm plugin version`). The script builds Pyxis for the version +> passed in `PCS_SLURM_VERSION` and installs the `.so` to a per-version path; a regression +> here only shows on the *other* version. +> - **The pre-baked AMI path too** ([Test 8](#test-8-pre-baked-ami-build-standalone-dlami-template)). +> `pcs-ready-dlami-with-enroot-pyxis.yaml` carries its **own copy** of the Enroot/Pyxis +> steps in its Image Builder UserData — editing `install-enroot-pyxis.sh` does **not** +> change the AMI path until you rebuild. Build an AMI per supported `SlurmVersion`, +> deploy a cluster pinned to it (`AmiId=` + `PostInstallScriptUrl=""`), and +> run a container job. +> - **On a clean first boot**, not a hand-patched node — post-install runs before +> slurmd/profile.d/controller exist, and several bugs only appear there. + +--- + +## Test 3: CPU queue + +Deployed by default as `cpu1` (`DeployOnDemandCNG=true`, `c6i.4xlarge`, 0–4 dynamic). + +```bash +sinfo # cpu1 partition present, nodes idle~ +srun --partition=cpu1 --nodes=1 hostname # a node powers up and runs +``` + +**Expected:** `cpu1` shows in `sinfo`; a dynamically-scaled node launches and the job +returns its hostname. + +--- + +--- + +## Test 8: Pre-baked AMI build (standalone DLAMI template) + +**When to run:** when `pcs-ready-dlami-with-enroot-pyxis.yaml` or any code it bakes +in (`assets/scripts/install-enroot-pyxis.sh`) changes — the cluster stack does NOT run +Image Builder, so a fix to the install script is only in the AMI after a rebuild. +Skip this test if you only touched the cluster templates. + +This is an **independent flow**, not a deploy-all parameter: build an AMI with the +standalone template, then deploy a cluster pinned to that AMI ID with +`PostInstallScriptUrl=""` so nothing else runs at boot. + +The AMI is **single-Slurm-version by design** (Pyxis SPANK plugin ABI is +version-locked) — so when you run this test, run it for **every supported +`SlurmVersion`** that the install-script change could affect (typically both 25.05 +and 25.11). + +### Step 1 — build the AMI (~30 min one-time per Slurm version) + +```bash +SLURM_VERSION=25.11 # repeat for 25.05 if relevant + +aws cloudformation create-stack \ + --stack-name pcs-dlami-${SLURM_VERSION/./} \ + --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ + --parameters ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --profile claude --region us-west-2 + +aws cloudformation wait stack-create-complete \ + --stack-name pcs-dlami-${SLURM_VERSION/./} \ + --profile claude --region us-west-2 + +AMI_ID=$(aws cloudformation describe-stacks \ + --stack-name pcs-dlami-${SLURM_VERSION/./} \ + --query 'Stacks[0].Outputs[?OutputKey==`DLAMIforPCSAmiId`].OutputValue' \ + --output text --profile claude --region us-west-2) +echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx +``` + +### Step 2 — deploy a cluster pinned to that AMI + +```bash +aws cloudformation create-stack \ + --stack-name pcs-amitest-${SLURM_VERSION/./} \ + --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-west-2a \ + ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ + ParameterKey=AmiId,ParameterValue=${AMI_ID} \ + ParameterKey=PostInstallScriptUrl,ParameterValue= \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ + --profile claude --region us-west-2 +``` + +### Step 3 — verify the bake landed and a container job runs + +On any node from the new cluster: + +```bash +which enroot # /usr/bin/enroot (pre-baked) +ls /opt/aws/pcs/scheduler/slurm-${SLURM_VERSION}/lib/slurm/spank_pyxis.so # built for matching Slurm +cat /etc/aws/pcs/scheduler/slurm-${SLURM_VERSION}/plugstack.conf.d/pyxis.conf # references the .so +test ! -s /var/log/pcs-post-install.log && echo "post-install did not run (PostInstallScriptUrl='')" +``` + +Then a container job through the login node (same form as Test 2): + +```bash +srun --partition=cpu1 --nodes=1 --ntasks=1 \ + --container-image=ubuntu:22.04 bash -c "echo PYXIS_FROM_AMI_OK" +``` + +**Expected:** `enroot` and the per-version Pyxis files exist **without** the +post-install hook running (because `PostInstallScriptUrl=""`); the container job +prints `PYXIS_FROM_AMI_OK`. Slurmd starts cleanly (no `Incompatible Slurm plugin +version` in `journalctl -u slurmd`). + +### Step 4 — clean up + +```bash +aws cloudformation delete-stack --stack-name pcs-amitest-${SLURM_VERSION/./} --profile claude --region us-west-2 +aws cloudformation delete-stack --stack-name pcs-dlami-${SLURM_VERSION/./} --profile claude --region us-west-2 +``` + +The DLAMI stack's AMI itself is **not** automatically deregistered when the stack is +deleted — if you need to free its EBS snapshots, deregister the AMI manually +(`aws ec2 deregister-image --image-id $AMI_ID`) and delete the snapshot. + +--- diff --git a/architectures/aws-pcs/tests/multi-user-test.md b/architectures/aws-pcs/tests/multi-user-test.md index 2798652cf..5af01beef 100644 --- a/architectures/aws-pcs/tests/multi-user-test.md +++ b/architectures/aws-pcs/tests/multi-user-test.md @@ -282,3 +282,200 @@ survived the instance replacement). | New compute node resolves existing users | ✅ | | SSSD cache works during brief LDAP outage | ✅ | | LDAP DB survives login node replacement | ✅ | + +--- + +# Test: Slurm Managed Accounting + Multi-User + +Validates that Slurm managed accounting works with LDAP multi-user setup. +Covers user/account creation, resource limit enforcement, job tracking, and +reporting. + +**Prerequisites**: +- Cluster with `ManagedAccounting=enabled` and `DirectoryService=OpenLDAP-LoginNode` +- Slurm 25.11 (accounting is available on 24.11+; templates default to 25.11) +- At least one compute node available + +All commands run on the login node as root unless noted. + +--- + +## Part A — Accounting infrastructure + +### A1. Verify accounting is active + +```bash +export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH +sacctmgr show cluster +``` + +Expected: cluster name listed with a valid `ControlHost`. + +### A2. Default account exists + +```bash +sacctmgr show account +``` + +If no accounts exist yet, create the default: +```bash +sacctmgr -i add account default Description="Default account" +``` + +--- + +## Part B — User + account registration + +### B1. Create LDAP users (if not done already) + +```bash +ADMIN_PW=$(aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text --region us-east-2) + +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user alice 10001 3000 +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user bob 10002 3000 +``` + +### B2. Register users in Slurm accounting + +```bash +sacctmgr -i add account ml-team Description="ML Research Team" +sacctmgr -i add user alice Account=ml-team +sacctmgr -i add user bob Account=ml-team + +# Verify +sacctmgr show user alice bob format=User,Account,DefaultAccount +``` + +Expected: +``` + User Account DefaultAccount +--------- ---------- -------------- + alice ml-team ml-team + bob ml-team ml-team +``` + +### B3. Set resource limits (optional) + +```bash +# Cap alice at 100 CPU-hours +sacctmgr -i modify user alice set GrpTRESRunMins=cpu=6000 + +# Verify +sacctmgr show user alice format=User,Account,GrpTRESRunMins +``` + +--- + +## Part C — Job submission and tracking + +### C1. Submit jobs as LDAP users + +```bash +# As alice +sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J alice-test' + +# As bob +sudo su - bob -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --wrap="sleep 10; hostname; id" -J bob-test' +``` + +Wait for jobs to complete: +```bash +squeue # should show jobs running then empty +``` + +### C2. Verify job history with sacct + +```bash +sacct --starttime=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ + --format="JobID,User,JobName,Partition,Account,AllocCPUS,State,ExitCode,Elapsed" +``` + +Expected: both jobs listed with correct User, Account, State=COMPLETED. + +### C3. Per-user reporting + +```bash +# Jobs by alice +sacct -u alice --format="JobID,JobName,State,Start,End,Elapsed" + +# Utilization by account +sreport cluster AccountUtilizationByUser \ + start=$(date -d "1 hour ago" +%Y-%m-%dT%H:%M) \ + format="Account,Login,Used" +``` + +### C4. Resource limit enforcement (if B3 was set) + +```bash +# Submit a job that would exceed alice's limit +sudo su - alice -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + sbatch --partition=cpu1 --ntasks=96 --time=120:00 --wrap="sleep 7200" -J limit-test' +``` + +Check if job is pending with reason `AssocGrpCPURunMinutesLimit`: +```bash +squeue -u alice --format="%i %j %T %r" +``` + +--- + +## Part D — Fairshare (optional) + +### D1. Set different fairshare weights + +```bash +sacctmgr -i modify account ml-team set FairShare=100 +sacctmgr -i add account ops-team Description="Operations" +sacctmgr -i modify account ops-team set FairShare=50 +``` + +### D2. Check fairshare values + +```bash +sshare -a --format=Account,User,RawShares,NormShares,RawUsage,FairShare +``` + +--- + +## Part E — AccountingPolicyEnforcement + +### E1. Test with enforcement=none (default) + +An unregistered user can still submit jobs: +```bash +# Create a user NOT in sacctmgr +sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user charlie 10003 3000 +sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + srun -p cpu1 -N1 -n1 hostname' +``` + +Expected: job runs (no accounting enforcement). + +### E2. Test with enforcement=associations,limits,safe + +If `AccountingPolicyEnforcement=associations,limits,safe` was set at cluster +creation: +```bash +# charlie (not in sacctmgr) should be rejected +sudo su - charlie -c 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; \ + srun -p cpu1 -N1 -n1 hostname' +``` + +Expected: `error: Unable to allocate resources: Invalid account or account/partition combination specified` + +--- + +## Verdict checklist + +| Check | Expected | +|---|---| +| sacctmgr show cluster → cluster listed | ✅ | +| Users registered in accounting (alice, bob) | ✅ | +| Jobs run as LDAP users complete successfully | ✅ | +| sacct shows correct user/account/state per job | ✅ | +| sreport shows per-user utilization | ✅ | +| Resource limit enforcement (if set) blocks over-limit jobs | ✅ | +| Unregistered user behavior matches AccountingPolicyEnforcement setting | ✅ | diff --git a/architectures/aws-pcs/tests/storage-test.md b/architectures/aws-pcs/tests/storage-test.md new file mode 100644 index 000000000..0e0060b92 --- /dev/null +++ b/architectures/aws-pcs/tests/storage-test.md @@ -0,0 +1,131 @@ +# Storage Tests (Test 10) + +Validates FSx for Lustre + OpenZFS health, mount options, and performance +regression/improvement testing. + +--- + +## Test 10: FSx storage health + +Validates that both shared filesystems (Lustre on `/fsx`, OpenZFS on `/home`) +mount cleanly on every node, are usable, and that the FSx-side configuration +matches what the template asked for. Most of this is exercised implicitly by +Tests 1–9 (the post-install script, monitoring stack, OSU / FSDP all touch +`/fsx`); this test is the explicit health check to run after a fresh deploy +or after touching `ml-cluster-prerequisites.yaml` / FSx-related parameters. + +### Step 1 — both filesystems mounted on every node + +On the login node and at least one compute node (via `srun`): + +```bash +mount | grep -E ' /home | /fsx ' +df -h /home /fsx +``` + +Expected: +- `/fsx` mounted as type `lustre`, source `.fsx..amazonaws.com@tcp:/`, + size matches the `Capacity` parameter (1200 GiB default; 19200 GiB or larger + when `FSxLustreEnableEfa=true`). +- `/home` mounted as type `nfs` over OpenZFS, NFS options include + `nconnect=16,rsize=1048576,wsize=1048576` (the deploy-all UserData mount + string). +- Both `df -h` reports show `Avail` greater than zero. + +If a mount is missing on a freshly booted node, check +`/var/log/cloud-init-output.log` and `/var/log/pcs-post-install.log` for the +`mount` line. The most common first-boot failure is the OpenZFS DNS name not +being resolvable yet (NFS settle race); the post-install log will show +`mount.nfs: Failed to resolve server`. + +### Step 2 — read/write sanity + +```bash +# /fsx (Lustre): write 1 GiB, read it back +dd if=/dev/zero of=/fsx/.healthcheck bs=1M count=1024 conv=fsync 2>&1 | tail -1 +dd if=/fsx/.healthcheck of=/dev/null bs=1M 2>&1 | tail -1 +rm /fsx/.healthcheck + +# /home (OpenZFS / NFS): same, 100 MiB (it's a small home filesystem) +dd if=/dev/zero of=/home/ubuntu/.healthcheck bs=1M count=100 conv=fsync 2>&1 | tail -1 +dd if=/home/ubuntu/.healthcheck of=/dev/null bs=1M 2>&1 | tail -1 +rm /home/ubuntu/.healthcheck +``` + +Expected: both write+read complete without error. Throughput is bounded by the +single-stream limits of NFS / Lustre on a single node — this is a sanity check, +not a benchmark. For Lustre throughput numbers, see the FSx Lustre User Guide +(provisioned throughput = `Capacity * PerUnitStorageThroughput / 1024` MB/s). + +### Step 3 — FSx-side parameters match the CFN inputs + +```bash +FSX_ID=$(aws cloudformation describe-stacks \ + --stack-name \ + --query 'Stacks[0].Outputs[?OutputKey==`FSxLustreFilesystemId`].OutputValue' \ + --region --output text) + +aws fsx describe-file-systems --file-system-ids "$FSX_ID" --region \ + --query 'FileSystems[0].[StorageCapacity,StorageType,LustreConfiguration.[DeploymentType,PerUnitStorageThroughput,DataCompressionType,EfaEnabled,MetadataConfiguration.Mode]]' \ + --output text +``` + +Expected (default deploy): +- `StorageCapacity` = your `Capacity` parameter (1200 by default) +- `StorageType` = `SSD` +- `DeploymentType` = `PERSISTENT_2` (default; or `PERSISTENT_1` if you set it) +- `PerUnitStorageThroughput` = 250 (default) +- `DataCompressionType` = `LZ4` (default) +- `EfaEnabled` = `False` (default; `True` when `FSxLustreEnableEfa=true`. EFA on + FSx is a PERSISTENT_2-only feature — the prerequisites and deploy-all templates + enforce this with a CFN Rule that fails the stack at create time when + `FSxLustreEnableEfa=true` is combined with `LustreDeploymentType=PERSISTENT_1`) +- `MetadataConfiguration.Mode` = `AUTOMATIC` on PERSISTENT_2 + +For the OpenZFS `/home` filesystem: + +```bash +FSXO_ID=$(aws cloudformation describe-stacks \ + --stack-name \ + --query 'Stacks[0].Outputs[?OutputKey==`FSxOFilesystemId`].OutputValue' \ + --region --output text) + +aws fsx describe-file-systems --file-system-ids "$FSXO_ID" --region \ + --query 'FileSystems[0].[StorageCapacity,OpenZFSConfiguration.[DeploymentType,ThroughputCapacity]]' \ + --output text +``` + +### Step 4 — Storage dashboard in Grafana + +The `Compute Node Details` and `HPC Cluster Monitoring → Storage` Grafana +dashboards are populated by the **CloudWatch Exporter** (FSx CloudWatch +metrics, scraped by the monitoring stack on the login node — see the +[aws-parallelcluster-monitoring v2.6 release notes](https://github.com/aws-samples/aws-parallelcluster-monitoring/releases/tag/v2.6)). + +Open Grafana (Test 1's port-forward / public CIDR), go to the Storage +dashboard, and verify the `/fsx` panels (Throughput, IOPS, Free Capacity, +Client Connections) populate within ~5 minutes of a workload starting. CW +metrics have a ~5 min publishing delay, so a brand-new filesystem with no I/O +shows blank panels for a while; the `dd` from Step 2 is enough to seed values. + +### `FSxLustreEnableEfa=true` specifics + +When `FSxLustreEnableEfa=true` is set on a PERSISTENT_2 SSD filesystem, the +extra checks beyond the above are: + +- `aws fsx describe-file-systems` `LustreConfiguration.EfaEnabled = true` +- `Capacity` is at-or-above the EFA minimum for the chosen + `PerUnitStorageThroughput` tier (19200 GiB for tier 250; the FSx for + Lustre User Guide has the full matrix). Below the minimum, the FSx side + rejects the `CreateFileSystem` with `Invalid storage capacity provided: + N GiB. Minimum storage capacity for an EFA enabled LUSTRE file systems + with deployment type PERSISTENT_2, per unit storage throughput X and + storage type SSD is M`. The Lustre nested stack fails first, then the + whole stack rolls back; that's the expected behavior for an undersized + Capacity. + +The FSx-side EFA endpoints are usable from EFA-capable clients (CPU CNGs +deployed with `OnDemandEnableEfa=true`, P5/P6 GPU CNGs). Plain Lustre client +mounts continue to work over TCP for non-EFA nodes; EFA support is additive. + +--- diff --git a/architectures/aws-pcs/tests/training-test.md b/architectures/aws-pcs/tests/training-test.md new file mode 100644 index 000000000..1c62f69c4 --- /dev/null +++ b/architectures/aws-pcs/tests/training-test.md @@ -0,0 +1,71 @@ +# Training Tests (Test 7) + +Validates end-to-end distributed training using the repository's canonical +FSDP Llama-2 7B test case. + +--- + +## Test 7: FSDP sample training + +A short FSDP Llama-2 7B run, using the repo's canonical training case +[`3.test_cases/pytorch/FSDP`](../../../3.test_cases/pytorch/FSDP) and its +[`slurm/llama2_7b-training.sbatch`](../../../3.test_cases/pytorch/FSDP/slurm/llama2_7b-training.sbatch). +Follow that case's README; the only PCS-specific deltas are where things live on the +shared filesystems and the node count: + +1. **Build the venv on shared `/fsx`** so every compute node sees it (the canonical + `slurm/create_venv.sh` creates `./env` in place — run it from a `/fsx` checkout): + + ```bash + cd /fsx && git clone --depth 1 https://github.com/awslabs/awsome-distributed-ai.git + cd /fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP/slurm && bash create_venv.sh + export HF_TOKEN=hf_xxx # gated Llama-2 tokenizer + ``` + +2. **Keep the HuggingFace cache on `/fsx` (Lustre), not `/home` (NFS).** Concurrent rank + file-locking on NFS throws `OSError: [Errno 116] Stale file handle`; export + `HF_HOME=/fsx/.hf-cache` before submitting. + +3. **Submit 2 nodes** (the canonical sbatch defaults to 4) on your GPU queue. The venv must + be on `PATH` for `torchrun` to resolve on every node — point `PATH` at the shared venv via + `--export` (the canonical sbatch doesn't `activate` it): + + ```bash + sbatch --nodes=2 --partition=gpu-p6b200 \ + --export=ALL,PATH=/fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP/slurm/env/bin:$PATH,HF_HOME=/fsx/.hf-cache \ + llama2_7b-training.sbatch + ``` + +### Option B — run it in an Enroot/Pyxis container instead of the venv + +The same canonical sbatch switches to container mode when `CONTAINER_IMAGE` is set (it adds +`--container-image`/`--container-mounts` and runs `./train.py` inside). Build the image once +on the login node and submit with `CONTAINER_IMAGE` — no venv needed: + +```bash +# On the login node (300 GiB root disk + Docker), build + import to /fsx: +cd /fsx/awsome-distributed-ai/3.test_cases/pytorch/FSDP +sudo docker build -t fsdp:pytorch -f Dockerfile . +enroot import -o /fsx/pytorch-fsdp.sqsh dockerd://fsdp:pytorch + +# Submit (container mode; mounts $(pwd) into /fsx inside the container): +cd slurm && sbatch --nodes=2 --partition=gpu-p6b300 \ + --export=ALL,CONTAINER_IMAGE=/fsx/pytorch-fsdp.sqsh,HF_HOME=/fsx/.hf-cache,HF_TOKEN=hf_xxx \ + llama2_7b-training.sbatch +``` + +> If the Dockerfile's `FROM` tag (a `public.ecr.aws/hpc-cloud/nccl-tests` tag) has been +> rotated out of the registry, substitute a current tag from that repo before building. + +**Expected** (in `logs/llama2_7b-FSDP_.out`), either path: +- NCCL initializes over EFA (`found N nics`) and training logs ~100 steps + a validation + step, saving checkpoints under `./checkpoints`. +- Throughput per step, e.g. on 2× p6-b300 **~200 TFLOPS/GPU, ~77k tokens/s** (venv) / + **~193 TFLOPS/GPU** (container); ~60 TFLOPS on 2× p5/H100. (Loss is constant at ln(vocab) + in this smoke test — a known dataloader/vocab quirk of the test case, not a cluster problem.) + +> **Multi-NIC tip:** the canonical sbatch already sets `NCCL_SOCKET_IFNAME=^docker,lo,veth,eth` +> (NCCL auto-selects). Do **not** pin a single interface on P5/P6 — all NICs share one +> subnet and pinning one breaks the cross-node NCCL bootstrap ring. + +--- From cb0e86f92753a669de57ea9c19813cc93b393311 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:31:38 +0000 Subject: [PATCH 25/89] =?UTF-8?q?README:=20add=20=C2=A711=20User=20Managem?= =?UTF-8?q?ent=20section=20with=20link=20to=20USER-MANAGEMENT.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- architectures/aws-pcs/README.md | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 371897a7b..53b6435e4 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -585,7 +585,27 @@ Standalone (not nested): --- -## 11. Testing and Validation +## 11. User Management + +By default, the cluster runs as a single `ubuntu` user (the PCS-Ready DLAMI +default). This is sufficient for single-user evaluation, workshops, and +small-team use where everyone shares the same environment. + +For **multi-user clusters** (per-user Slurm accounting, isolated home +directories, team-based access control), set `DirectoryService=OpenLDAP-LoginNode` +at deploy time. This installs an OpenLDAP directory on the login node with +SSSD on all compute nodes — users you add are immediately visible cluster-wide. + +See the **[User Management Guide](./docs/USER-MANAGEMENT.md)** for: +- Enabling multi-user (one parameter) +- Adding / removing users (step-by-step, no LDAP knowledge required) +- Slurm accounting integration +- Access methods (SSH vs SSM) +- Troubleshooting + +--- + +## 12. Testing and Validation Every template has been deployed end-to-end on real hardware: P5 / P5e / P5en / P6-B200 / P6-B300 (NCCL all-reduce + FSDP Llama-2 7B), HPC EFA on hpc6a / hpc7a / @@ -596,7 +616,7 @@ numbers is in **[tests/README.md](./tests/README.md)**. --- -## 12. Additional Resources +## 13. Additional Resources In this repo: - [Parameter reference](./docs/PARAMETERS.md) — every deploy-all parameter and default From 1da9571ba23de03a6d3485b2eca77245ca245f49 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:41:51 +0000 Subject: [PATCH 26/89] fix: move runcmd comment inside list item (YAML comment outside - | breaks cloud-init parsing) --- architectures/aws-pcs/assets/add-cng.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index f1ff1236c..f2a3f8838 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -520,8 +520,8 @@ Resources: done echo "Monitoring installation complete (exit $rc)" >> /var/log/monitoring-install.log fi - # Multi-user directory setup (optional, DirectoryRole != none) - | + # Multi-user directory setup (optional, DirectoryRole != none) if [ "${DirectoryRole}" != "none" ]; then export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') From 58a0b8ed3bc6805578c790d90a7f55c892febc12 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:43:37 +0000 Subject: [PATCH 27/89] =?UTF-8?q?README:=20restructure=20=E2=80=94=20group?= =?UTF-8?q?=20advanced=20features=20under=20=C2=A78,=20simplify=20section?= =?UTF-8?q?=20numbering?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before: 13 top-level sections mixing core usage with advanced features and reference. After: 11 sections with clear separation: §1-7: Core (Key Features → Running a GPU job) §8: Advanced Features 8.1 Monitoring 8.2 Pre-baking AMI 8.3 User Management (new — was standalone §11) 8.4 IAM Permissions (new) §9: Templates (reference) §10: Testing and Validation (reference) §11: Additional Resources This makes it clear that §1-7 is the primary user path (deploy + run jobs), and everything in §8 is opt-in for production/multi-user/security hardening. --- architectures/aws-pcs/README.md | 62 ++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 53b6435e4..8c99fc0a8 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -18,7 +18,7 @@ This repository provides reference architectures and deployment templates for se > Slurm 25.05/25.11 pre-installed), so no custom AMI build is required by default — > the cluster comes up without an Image Builder step. For frequent scaling, you can > pre-bake Enroot/Pyxis into a custom DLAMI with the standalone -> [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#9-pre-baking-enrootpyxis-into-a-custom-ami-optional) +> [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#82-pre-baking-enrootpyxis-into-a-custom-ami) > template and pass the result as `AmiId`. ## 2. Architecture @@ -89,7 +89,7 @@ only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | -| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#9-pre-baking-enrootpyxis-into-a-custom-ami-optional) | +| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#82-pre-baking-enrootpyxis-into-a-custom-ami) | | `DeployMonitoring` | `true` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes | | `DeployOnDemandCNG` | `true` | Deploy the `cpu1` CPU queue (`OnDemandInstanceType`, default `c6i.4xlarge`) | | `OnDemandEnableEfa` | `false` | Set `true` for HPC/MPI workloads on EFA-capable CPU types (hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandenableefa) | @@ -109,7 +109,7 @@ choices that need the most thought. needed. Enroot 3.5.0 + Pyxis 0.20.0 are layered on at first boot via [`assets/scripts/install-enroot-pyxis.sh`](./assets/scripts/install-enroot-pyxis.sh) (~8–12 min boot). For **frequent scaling**, pre-bake Enroot/Pyxis into a custom DLAMI -once with [§9](#9-pre-baking-enrootpyxis-into-a-custom-ami-optional) and pass that +once with [§8.2](#82-pre-baking-enrootpyxis-into-a-custom-ami) and pass that `ami-xxx` as `AmiId` (~3 min boot, deterministic state). The post-install hook is idempotent — it no-ops on a pre-baked AMI. @@ -319,7 +319,9 @@ the full validation matrix is in [tests/README.md](./tests/README.md). --- -## 8. Monitoring +## 8. Advanced Features + +### 8.1 Monitoring With `DeployMonitoring=true` (default), an integrated monitoring stack based on [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) @@ -440,7 +442,7 @@ NCCL, FSDP), see the [Test & Validation Guide](tests/README.md). --- -## 9. Pre-baking Enroot/Pyxis into a custom AMI (optional) +### 8.2 Pre-baking Enroot/Pyxis into a custom AMI The all-in-one template installs Enroot/Pyxis at **first boot** via `PostInstallScriptUrl`, which is fast to deploy and avoids an Image Builder step. @@ -503,7 +505,31 @@ For production deploys that pin the AMI explicitly per cluster, none of these ar --- -## 10. Templates +### 8.3 User Management + +By default, the cluster runs as a single `ubuntu` user. For **multi-user +clusters** (per-user Slurm accounting, isolated home directories, team-based +access control), set `DirectoryService=OpenLDAP-LoginNode` at deploy time. + +This installs an OpenLDAP directory on the login node with SSSD on all compute +nodes — users you add are immediately visible cluster-wide. See the +**[User Management Guide](./docs/USER-MANAGEMENT.md)** for step-by-step +operations (adding/removing users, Slurm accounting, SSH access). + +### 8.4 IAM Permissions + +Two sample IAM policy stacks are provided for least-privilege access: + +- **Cluster admin** (`cluster-admin-iam.yaml`) — deploy / update / delete the cluster +- **Cluster user** (`cluster-user-iam.yaml`) — SSM session to login node + read-only + +Deploy with one click from the [iam/ README](./iam/README.md), or attach the +raw JSON policies manually. See [iam/README.md](./iam/README.md) for the full +verification matrix and design rationale. + +--- + +## 9. Templates All templates live in [`assets/`](./assets/). `pcs-ml-cluster-deploy-all.yaml` nests the others; you can also deploy each individually for more control (e.g. reuse a VPC/FSx @@ -585,27 +611,7 @@ Standalone (not nested): --- -## 11. User Management - -By default, the cluster runs as a single `ubuntu` user (the PCS-Ready DLAMI -default). This is sufficient for single-user evaluation, workshops, and -small-team use where everyone shares the same environment. - -For **multi-user clusters** (per-user Slurm accounting, isolated home -directories, team-based access control), set `DirectoryService=OpenLDAP-LoginNode` -at deploy time. This installs an OpenLDAP directory on the login node with -SSSD on all compute nodes — users you add are immediately visible cluster-wide. - -See the **[User Management Guide](./docs/USER-MANAGEMENT.md)** for: -- Enabling multi-user (one parameter) -- Adding / removing users (step-by-step, no LDAP knowledge required) -- Slurm accounting integration -- Access methods (SSH vs SSM) -- Troubleshooting - ---- - -## 12. Testing and Validation +## 10. Testing and Validation Every template has been deployed end-to-end on real hardware: P5 / P5e / P5en / P6-B200 / P6-B300 (NCCL all-reduce + FSDP Llama-2 7B), HPC EFA on hpc6a / hpc7a / @@ -616,7 +622,7 @@ numbers is in **[tests/README.md](./tests/README.md)**. --- -## 13. Additional Resources +## 11. Additional Resources In this repo: - [Parameter reference](./docs/PARAMETERS.md) — every deploy-all parameter and default From 649d5667b7a199d535ae4b5ab9df24e90b1e56b2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:46:03 +0000 Subject: [PATCH 28/89] =?UTF-8?q?deploy-all:=20reorganize=20ParameterGroup?= =?UTF-8?q?s=20=E2=80=94=20separate=20monitoring/directory=20from=20cluste?= =?UTF-8?q?r=20core?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before: DeployMonitoring + GrafanaPublicAccessCidr were under "PCS Cluster Configuration" (which is really scheduler/AMI/accounting settings). After: §2. PCS Cluster Configuration — Slurm version, login instance, AMI, accounting §7. Additional Cluster Configuration — Monitoring + Multi-User Directory This separates "what the cluster IS" (§2) from "what extra features are layered on top" (§7). Also renamed §6 from "(Advanced)" to just "FSx Storage" (Capacity/DeploymentType are commonly tuned, not advanced) and §8 to just "Developer / Advanced" (shorter). --- .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index d9fad5530..0c523ae8d 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -25,8 +25,6 @@ Metadata: - LoginNodeInstanceType - RootVolumeSize - AmiId - - DeployMonitoring - - GrafanaPublicAccessCidr - ManagedAccounting - AccountingPolicyEnforcement - Label: @@ -57,7 +55,7 @@ Metadata: - PseriesCngName - PseriesQueueName - Label: - default: "6. FSx for Lustre (/fsx) + FSx for OpenZFS (/home) (Advanced)" + default: "6. FSx Storage (/fsx and /home)" Parameters: - Capacity - LustreDeploymentType @@ -69,12 +67,14 @@ Metadata: - HomeThroughput - OpenZFSDeploymentType - Label: - default: "7. Multi-User Directory (Optional)" + default: "7. Additional Cluster Configuration (Monitoring, Multi-User)" Parameters: + - DeployMonitoring + - GrafanaPublicAccessCidr - DirectoryService - DirectoryDomainSuffix - Label: - default: "8. Developer / Advanced (Nested Templates & Monitoring Source)" + default: "8. Developer / Advanced" Parameters: - S3BucketName - S3KeyPrefix From 6a062306f6c798af37b8b0658cfda3661ad2c39f Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:48:56 +0000 Subject: [PATCH 29/89] =?UTF-8?q?deploy-all:=20move=20monitoring=20params?= =?UTF-8?q?=20to=20=C2=A77,=20Developer=20=C2=A78=20keeps=20only=20S3=20bu?= =?UTF-8?q?cket/prefix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 0c523ae8d..4819c49db 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -71,16 +71,16 @@ Metadata: Parameters: - DeployMonitoring - GrafanaPublicAccessCidr + - MonitoringRepo + - MonitoringVersion + - DcgmExporterImage - DirectoryService - DirectoryDomainSuffix - Label: - default: "8. Developer / Advanced" + default: "8. Developer / Advanced (do not change for normal use)" Parameters: - S3BucketName - S3KeyPrefix - - MonitoringVersion - - MonitoringRepo - - DcgmExporterImage ParameterLabels: # Friendly display names shown in the CloudFormation console 1-click UI, # so users see plain-language labels instead of raw CamelCase parameter keys. From b7b65b23c5b4935b9fb48980fca47c51187869c4 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 16:59:33 +0000 Subject: [PATCH 30/89] =?UTF-8?q?feat:=20SSHAccessCidr=20+=20DeployMonitor?= =?UTF-8?q?ing=E2=86=92MonitoringStack=20+=20GrafanaPublicAccessCidr?= =?UTF-8?q?=E2=86=92GrafanaAccessCidr?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three changes in deploy-all parameter interface: 1. **SSHAccessCidr** (new, §2 PCS Cluster Configuration): CIDR to open port 22 on the login node. Empty = SSH over SSM only. For multi-user clusters where users connect via standard SSH. 2. **DeployMonitoring → MonitoringStack** (rename, §7): Boolean true/false → enum: 'Prometheus-LoginNode' (default) | 'none'. Aligns with DirectoryService naming pattern (-). Nested stacks receive DeployMonitoring="true"/"false" via !If [MonitoringEnabled] so add-cng.yaml is unchanged (backward compat within the nest). 3. **GrafanaPublicAccessCidr → GrafanaAccessCidr** (rename, §7): Shorter name, same semantics. Opens HTTPS/443 on login node. Security group redesign: - LoginAccessSecurityGroup now conditionally includes BOTH SSH (from SSHAccessCidr) and HTTPS (from GrafanaAccessCidr) rules via !If. - Created only when at least one CIDR is set (LoginAccessEnabled = OR). - Attached only to login node via ExtraSecurityGroupId (compute unaffected). New Conditions: SSHAccessEnabled, GrafanaAccessEnabled, LoginAccessEnabled (OR of both), MonitoringEnabled. --- .../assets/pcs-ml-cluster-deploy-all.yaml | 109 +++++++++++------- 1 file changed, 70 insertions(+), 39 deletions(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 4819c49db..07c8ff8b9 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -25,6 +25,7 @@ Metadata: - LoginNodeInstanceType - RootVolumeSize - AmiId + - SSHAccessCidr - ManagedAccounting - AccountingPolicyEnforcement - Label: @@ -69,8 +70,8 @@ Metadata: - Label: default: "7. Additional Cluster Configuration (Monitoring, Multi-User)" Parameters: - - DeployMonitoring - - GrafanaPublicAccessCidr + - MonitoringStack + - GrafanaAccessCidr - MonitoringRepo - MonitoringVersion - DcgmExporterImage @@ -94,10 +95,12 @@ Metadata: default: "Login node instance type" SlurmVersion: default: "Slurm version" - DeployMonitoring: - default: "Deploy monitoring (Prometheus/Grafana)" - GrafanaPublicAccessCidr: - default: "Grafana public access CIDR (empty = SSM only)" + SSHAccessCidr: + default: "SSH access CIDR for login node (empty = SSM only)" + MonitoringStack: + default: "Monitoring stack (none or Prometheus-LoginNode)" + GrafanaAccessCidr: + default: "Grafana public access CIDR (empty = SSM port-forward only)" MonitoringVersion: default: "Monitoring version (git ref)" MonitoringRepo: @@ -380,22 +383,34 @@ Parameters: Default: "m6i.4xlarge" # Monitoring Configuration - DeployMonitoring: + SSHAccessCidr: Type: String - Description: Deploy monitoring stack (Prometheus, Grafana, DCGM) on Login Node - Default: 'true' + Description: >- + (Optional) CIDR allowed to reach the login node on SSH/22. Empty (default) + = SSH over SSM only. Set to your office IP or VPN range for direct SSH + access (multi-user clusters, VS Code Remote, scp). + Default: "" + AllowedPattern: '^$|^((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)/(3[0-2]|[12]?\d)$' + ConstraintDescription: Must be empty or a valid IPv4 CIDR (e.g. 203.0.113.4/32) + + MonitoringStack: + Type: String + Description: >- + Monitoring stack to deploy. 'Prometheus-LoginNode' (default) installs + Prometheus + Grafana + DCGM Exporter on the login node. 'none' skips + monitoring entirely. Future options (AMP-AMG, CloudWatch) will be added. + Default: 'Prometheus-LoginNode' AllowedValues: - - 'true' - - 'false' + - 'Prometheus-LoginNode' + - 'none' - GrafanaPublicAccessCidr: + GrafanaAccessCidr: Type: String Description: >- - (Optional) CIDR allowed to reach the login node on HTTPS/443. Empty (default) - = SSM port-forward only (recommended). When set, also exposes the - UNauthenticated /prometheus/, /pushgateway/, /slurmexporter/ proxy paths - (not just password-gated Grafana). Use the tightest CIDR you can; clear it - when done. + (Optional) CIDR allowed to reach the login node on HTTPS/443 (Grafana). + Empty (default) = SSM port-forward only. When set, also exposes the + unauthenticated /prometheus/, /pushgateway/, /slurmexporter/ proxy paths. + Use the tightest CIDR you can; clear it when done. Default: "" AllowedPattern: '^$|^((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)\.){3}(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)/(3[0-2]|[12]?\d)$' ConstraintDescription: Must be empty or a valid IPv4 CIDR (e.g. 203.0.113.4/32) @@ -615,7 +630,12 @@ Conditions: DirectoryEnabled: !Not [!Equals [!Ref DirectoryService, 'none']] DeployPseriesCNGEnabled: !Equals [!Ref DeployPseriesCNG, 'true'] # Open Grafana (login node, HTTPS/443) to a CIDR only when one is provided. - GrafanaPublicAccess: !Not [!Equals [!Ref GrafanaPublicAccessCidr, ""]] + SSHAccessEnabled: !Not [!Equals [!Ref SSHAccessCidr, ""]] + GrafanaAccessEnabled: !Not [!Equals [!Ref GrafanaAccessCidr, ""]] + LoginAccessEnabled: !Or + - !Condition SSHAccessEnabled + - !Condition GrafanaAccessEnabled + MonitoringEnabled: !Not [!Equals [!Ref MonitoringStack, 'none']] # Select the right multi-NIC EFA template from the P-series instance type. # p6-b200/p6-b300 have different network-card counts than P5, so each family # gets its own nested stack; anything else falls back to the P5 template. @@ -675,7 +695,7 @@ Resources: SlurmVersion: !Ref SlurmVersion ManagedAccounting: !Ref ManagedAccounting AccountingPolicyEnforcement: !Ref AccountingPolicyEnforcement - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] Tags: - Key: Name Value: !Sub '${AWS::StackName}-Cluster' @@ -683,25 +703,36 @@ Resources: ############################################################################# # 4. Login Node Group Stack ############################################################################# - # Optional login-only security group that exposes Grafana (HTTPS/443) to a - # specific CIDR. Created only when GrafanaPublicAccessCidr is set. It is - # attached ONLY to the login node (via ExtraSecurityGroupId), so compute nodes - # and FSx — which share the cluster security group — are not exposed. - GrafanaPublicSecurityGroup: + # Optional login-only security group for SSH and/or Grafana access from a + # specific CIDR. Created when SSHAccessCidr and/or GrafanaAccessCidr is set. + # Attached ONLY to the login node (via ExtraSecurityGroupId) — compute nodes + # and FSx (which share the cluster security group) are NOT exposed. + LoginAccessSecurityGroup: Type: AWS::EC2::SecurityGroup - Condition: GrafanaPublicAccess + Condition: LoginAccessEnabled Properties: - GroupDescription: Allow inbound HTTPS to Grafana on the login node + GroupDescription: Allow inbound SSH/HTTPS to the login node from trusted CIDRs VpcId: !GetAtt PrerequisitesStack.Outputs.VPC SecurityGroupIngress: - - IpProtocol: tcp - FromPort: 443 - ToPort: 443 - CidrIp: !Ref GrafanaPublicAccessCidr - Description: Grafana HTTPS from the allowed CIDR + - !If + - SSHAccessEnabled + - IpProtocol: tcp + FromPort: 22 + ToPort: 22 + CidrIp: !Ref SSHAccessCidr + Description: SSH from SSHAccessCidr + - !Ref AWS::NoValue + - !If + - GrafanaAccessEnabled + - IpProtocol: tcp + FromPort: 443 + ToPort: 443 + CidrIp: !Ref GrafanaAccessCidr + Description: Grafana HTTPS from GrafanaAccessCidr + - !Ref AWS::NoValue Tags: - Key: Name - Value: !Sub '${AWS::StackName}-grafana-public' + Value: !Sub '${AWS::StackName}-login-access' LoginNodeGroupStack: Type: AWS::CloudFormation::Stack @@ -724,14 +755,14 @@ Resources: AmiId: !Ref AmiId ClusterSecurityGroupId: !GetAtt PrerequisitesStack.Outputs.SecurityGroup ExtraSecurityGroupId: !If - - GrafanaPublicAccess - - !Ref GrafanaPublicSecurityGroup + - LoginAccessEnabled + - !Ref LoginAccessSecurityGroup - "" IamProfileArn: !GetAtt ClusterStack.Outputs.InstanceProfileArn FSxLustreFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemId FSxLustreFilesystemMountName: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemMountname FSxOpenZFSFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxOFilesystemId - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] MonitoringVersion: !Ref MonitoringVersion MonitoringRepo: !Ref MonitoringRepo DcgmExporterImage: !Ref DcgmExporterImage @@ -776,7 +807,7 @@ Resources: FSxLustreFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemId FSxLustreFilesystemMountName: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemMountname FSxOpenZFSFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxOFilesystemId - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] MonitoringVersion: !Ref MonitoringVersion MonitoringRepo: !Ref MonitoringRepo DcgmExporterImage: !Ref DcgmExporterImage @@ -828,7 +859,7 @@ Resources: FSxLustreFilesystemMountName: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemMountname FSxOpenZFSFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxOFilesystemId CapacityReservationId: !Ref CapacityReservationId - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] MonitoringVersion: !Ref MonitoringVersion MonitoringRepo: !Ref MonitoringRepo DcgmExporterImage: !Ref DcgmExporterImage @@ -869,7 +900,7 @@ Resources: FSxLustreFilesystemMountName: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemMountname FSxOpenZFSFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxOFilesystemId CapacityReservationId: !Ref CapacityReservationId - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] MonitoringVersion: !Ref MonitoringVersion MonitoringRepo: !Ref MonitoringRepo DcgmExporterImage: !Ref DcgmExporterImage @@ -910,7 +941,7 @@ Resources: FSxLustreFilesystemMountName: !GetAtt PrerequisitesStack.Outputs.FSxLustreFilesystemMountname FSxOpenZFSFilesystemId: !GetAtt PrerequisitesStack.Outputs.FSxOFilesystemId CapacityReservationId: !Ref CapacityReservationId - DeployMonitoring: !Ref DeployMonitoring + DeployMonitoring: !If [MonitoringEnabled, "true", "false"] MonitoringVersion: !Ref MonitoringVersion MonitoringRepo: !Ref MonitoringRepo DcgmExporterImage: !Ref DcgmExporterImage From 7f068f3484cf6e9e4a5266b7d1be04cebead5be8 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 17:09:59 +0000 Subject: [PATCH 31/89] fix: remove --region from s3 cp (AWS CLI auto-resolves via IMDS credential provider, avoids IMDSv2 token issue with curl) --- architectures/aws-pcs/assets/add-cng.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index f2a3f8838..83deab00c 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -528,8 +528,7 @@ Resources: export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) export CLUSTER_ID="${ClusterId}" export DIRECTORY_DNS_IPS="" - REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) - aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh --region ${!REGION} + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh chmod +x /tmp/setup-directory.sh bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log fi From d7dfdf16276b319feb4c0593dbc6f167147093d3 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 17:50:56 +0000 Subject: [PATCH 32/89] feat: dedicated directory-role tag, helper-script install, multi-AZ subnets, region fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Multi-user directory fixes (compute client could not find the LDAP server): - **Dedicated directory-role tag** (was reusing monitoring-role — design violation). add-cng.yaml now emits directory-role=server/client independently of monitoring-role; setup-directory.sh discovers the server via directory-role=server. Multi-user must not depend on the monitoring stack's tag. - **Region resolution**: removed all `--region "$REGION"` from setup-directory.sh (REGION came from a token-less IMDS curl that returns empty under HttpTokens=required, breaking `aws ... --region ""`). The AWS CLI resolves region from the IMDS credential provider itself, so no --region is passed. - **ldap-add-user.sh helper install**: server role now installs the helper to /usr/local/bin from the same S3 bucket (S3_BUCKET/S3_KEY_PREFIX passed by UserData), so admins have `ldap-add-user` on PATH (was missing — had to fetch from S3 manually). Single-login-node constraint documented (OpenLDAP server is single-node by design; DB is one MDB on shared /home): setup-directory.sh header, USER-MANAGEMENT.md callout, DirectoryService param description. Multi-AZ prerequisites (max 3 AZs: primary + 2 additional): - AdditionalSubnetAZ2 / AdditionalSubnetAZ3 params (optional, empty = single-AZ). - CIDR1 (10.1.0.0/16) now split into four /18 blocks (was two /17) — BREAKING change to private subnet CIDRs, approved. - Additional subnets share the primary AZ's single NAT gateway (cross-AZ, no per-AZ NAT by design). Outputs exported for downstream use. --- architectures/aws-pcs/assets/add-cng.yaml | 14 +++ .../assets/ml-cluster-prerequisites.yaml | 89 ++++++++++++++++++- .../assets/pcs-ml-cluster-deploy-all.yaml | 5 +- .../aws-pcs/assets/scripts/setup-directory.sh | 67 +++++++++++--- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 10 +++ 5 files changed, 171 insertions(+), 14 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 83deab00c..3d6fb8206 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -399,6 +399,16 @@ Resources: - Key: monitoring-role Value: !Ref MonitoringRole - !Ref AWS::NoValue + # Directory (multi-user) role tag, independent of monitoring-role. + # Emitted for both server (login) and client (compute) nodes when + # DirectoryRole != none. Compute clients discover the OpenLDAP + # server by querying for directory-role=server (NOT monitoring-role, + # which is owned by the monitoring stack). See setup-directory.sh. + - !If + - DirectoryEnabled + - Key: directory-role + Value: !Ref DirectoryRole + - !Ref AWS::NoValue MetadataOptions: HttpEndpoint: enabled HttpPutResponseHopLimit: 4 @@ -528,6 +538,10 @@ Resources: export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) export CLUSTER_ID="${ClusterId}" export DIRECTORY_DNS_IPS="" + # Pass the script source so the server role can install the + # ldap-add-user.sh helper onto /usr/local/bin from the same bucket. + export S3_BUCKET="${S3BucketName}" + export S3_KEY_PREFIX="${S3KeyPrefix}" aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh chmod +x /tmp/setup-directory.sh bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log diff --git a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml index 7d2719a1c..fa48ad4d6 100644 --- a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml +++ b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml @@ -27,6 +27,8 @@ Metadata: default: Availability Zone configuration for the subnets Parameters: - PrimarySubnetAZ + - AdditionalSubnetAZ2 + - AdditionalSubnetAZ3 - Label: default: FSx for Lustre (/fsx) storage size Parameters: @@ -43,6 +45,10 @@ Metadata: default: Enable EFA + GDS on FSx for Lustre (PERSISTENT_2 only) PrimarySubnetAZ: default: Availability zone id to deploy the primary subnets + AdditionalSubnetAZ2: + default: "(Optional) 2nd AZ for an additional private subnet" + AdditionalSubnetAZ3: + default: "(Optional) 3rd AZ for an additional private subnet" CreateS3Endpoint: default: Create an S3 endpoint @@ -60,6 +66,23 @@ Parameters: Description: Availability zone id in which the public subnet and primary private subnet will be created. Type: AWS::EC2::AvailabilityZone::Name + AdditionalSubnetAZ2: + Description: >- + (Optional) Availability Zone for a 2nd private subnet. Empty (default) = + single-AZ. Set to create an additional private subnet for multi-AZ FSx + (OpenZFS MULTI_AZ), Simple AD, or higher-availability workloads. The NAT + gateway stays in the primary AZ (shared), so this subnet reaches the + internet via cross-AZ routing. + Type: String + Default: "" + + AdditionalSubnetAZ3: + Description: >- + (Optional) Availability Zone for a 3rd private subnet. Empty (default) = + not created. Requires AdditionalSubnetAZ2 to also be set. + Type: String + Default: "" + CreateS3Endpoint: AllowedValues: - 'true' @@ -224,6 +247,11 @@ Rules: Conditions: S3EndpointCondition: !Equals [!Ref 'CreateS3Endpoint', 'true'] + # Additional private subnets (multi-AZ). Each is created only when its AZ + # parameter is set. AZ3 is meant to be used together with AZ2 (a 3rd subnet + # without a 2nd is unusual but technically allowed — both are independent). + HasSubnetAZ2: !Not [!Equals [!Ref AdditionalSubnetAZ2, ""]] + HasSubnetAZ3: !Not [!Equals [!Ref AdditionalSubnetAZ3, ""]] # PERSISTENT_2 is the only Lustre deployment type that supports a metadata # configuration (and the EFA/Intelligent-Tiering features); PERSISTENT_1 does # not, so the MetadataConfiguration block is only emitted for PERSISTENT_2. @@ -334,18 +362,46 @@ Resources: - Key: Name Value: !Join [ ' ', [ !Ref VPCName, 'Public Subnet -', !Ref PrimarySubnetAZ ] ] - # Create the primary private subnet + # Private subnets are carved from CIDR1 (10.1.0.0/16) split into four /18 + # blocks, so up to three private AZs (primary + 2 additional) can coexist + # without overlap. The primary uses index 0; additional subnets use 1 and 2. PrimaryPrivateSubnet: Type: AWS::EC2::Subnet DependsOn: [VpcCidrBlock] Properties: VpcId: !Ref VPC - CidrBlock: !Select [ 0, !Cidr [ !FindInMap [Networking, VPC, CIDR1], 2, 15 ]] + CidrBlock: !Select [ 0, !Cidr [ !FindInMap [Networking, VPC, CIDR1], 4, 14 ]] AvailabilityZone: !Ref PrimarySubnetAZ Tags: - Key: Name Value: !Join [ ' ', [ !Ref VPCName, 'Private Subnet -', !Ref PrimarySubnetAZ ] ] + # Additional private subnet 2 (optional, multi-AZ) + AdditionalPrivateSubnet2: + Type: AWS::EC2::Subnet + Condition: HasSubnetAZ2 + DependsOn: [VpcCidrBlock] + Properties: + VpcId: !Ref VPC + CidrBlock: !Select [ 1, !Cidr [ !FindInMap [Networking, VPC, CIDR1], 4, 14 ]] + AvailabilityZone: !Ref AdditionalSubnetAZ2 + Tags: + - Key: Name + Value: !Join [ ' ', [ !Ref VPCName, 'Private Subnet -', !Ref AdditionalSubnetAZ2 ] ] + + # Additional private subnet 3 (optional, multi-AZ) + AdditionalPrivateSubnet3: + Type: AWS::EC2::Subnet + Condition: HasSubnetAZ3 + DependsOn: [VpcCidrBlock] + Properties: + VpcId: !Ref VPC + CidrBlock: !Select [ 2, !Cidr [ !FindInMap [Networking, VPC, CIDR1], 4, 14 ]] + AvailabilityZone: !Ref AdditionalSubnetAZ3 + Tags: + - Key: Name + Value: !Join [ ' ', [ !Ref VPCName, 'Private Subnet -', !Ref AdditionalSubnetAZ3 ] ] + # Create and set the public route table PublicRouteTable: Type: AWS::EC2::RouteTable @@ -386,6 +442,23 @@ Resources: SubnetId: !Ref PrimaryPrivateSubnet RouteTableId: !Ref PrivateRouteTable + # Additional private subnets share the same private route table, so they reach + # the internet through the primary AZ's single NAT gateway (cross-AZ). This is + # intentional — no per-AZ NAT (cost/HA trade-off documented in the AZ params). + AdditionalPrivateSubnet2RTAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Condition: HasSubnetAZ2 + Properties: + SubnetId: !Ref AdditionalPrivateSubnet2 + RouteTableId: !Ref PrivateRouteTable + + AdditionalPrivateSubnet3RTAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Condition: HasSubnetAZ3 + Properties: + SubnetId: !Ref AdditionalPrivateSubnet3 + RouteTableId: !Ref PrivateRouteTable + # S3 endpoint S3Endpoint: Condition: S3EndpointCondition @@ -495,6 +568,18 @@ Outputs: Description: ID of the primary private subnet (alias for compatibility) Export: Name: !Sub ${AWS::StackName}-PrivateSubnet + AdditionalPrivateSubnet2: + Condition: HasSubnetAZ2 + Value: !Ref AdditionalPrivateSubnet2 + Description: ID of the 2nd additional private subnet (multi-AZ) + Export: + Name: !Sub ${AWS::StackName}-AdditionalPrivateSubnet2 + AdditionalPrivateSubnet3: + Condition: HasSubnetAZ3 + Value: !Ref AdditionalPrivateSubnet3 + Description: ID of the 3rd additional private subnet (multi-AZ) + Export: + Name: !Sub ${AWS::StackName}-AdditionalPrivateSubnet3 SecurityGroup: Value: !Ref SecurityGroup Description: SecurityGroup for Batch diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 07c8ff8b9..7a4a72740 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -277,7 +277,10 @@ Parameters: Multi-user directory service. 'none' = single ubuntu user (default, unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future - options (SimpleAD, ManagedAD) will be added here. + options (SimpleAD, ManagedAD) will be added here. NOTE: OpenLDAP-LoginNode + runs the directory server on a SINGLE login node — keep the login node + group at 1 instance (do not scale it) while this is enabled. For + multi-login-node / HA directories, use the future SimpleAD/ManagedAD options. Default: 'none' AllowedValues: - 'none' diff --git a/architectures/aws-pcs/assets/scripts/setup-directory.sh b/architectures/aws-pcs/assets/scripts/setup-directory.sh index 08fe728cc..2ae5a799f 100755 --- a/architectures/aws-pcs/assets/scripts/setup-directory.sh +++ b/architectures/aws-pcs/assets/scripts/setup-directory.sh @@ -8,6 +8,16 @@ # role = "server" (login node — installs slapd, creates base OUs) # "client" (compute node — installs SSSD, points at LDAP server) # +# IMPORTANT — single login node only when DirectoryService=OpenLDAP-LoginNode: +# The OpenLDAP server runs on THE login node and its DB lives on shared +# /home (OpenZFS). This design assumes exactly ONE login node (server). Do +# NOT scale the login node group above MinCount=MaxCount=1 with the directory +# enabled: multiple login nodes would each tag themselves directory-role=server +# (clients would pick an arbitrary one) and each run a slapd against the same +# /home/ldap-db MDB files (concurrent-open corruption). For HA multi-server +# directory, use a managed backend (Simple AD / Managed AD) instead — that is +# the planned DirectoryService=SimpleAD / ManagedAD extension path. +# # Environment variables (from UserData): # LDAP_DOMAIN_SUFFIX — e.g. "dc=cluster,dc=internal" # LDAP_DOMAIN — e.g. "cluster.internal" (derived from suffix) @@ -15,6 +25,9 @@ # CLUSTER_ID — PCS cluster ID (for SSM parameter path) # LDAP_SERVER_URI — e.g. "ldap://10.1.x.x" (client role; empty = discover) # DIRECTORY_DNS_IPS — comma-separated DNS IPs (future SimpleAD; empty = discover login IP) +# S3_BUCKET — bucket holding the scripts (server role; to install the +# ldap-add-user.sh helper onto /usr/local/bin) +# S3_KEY_PREFIX — key prefix for the scripts (default "templates/") set -euo pipefail @@ -25,6 +38,13 @@ LDAP_DB_DIR="/home/ldap-db" export DEBIAN_FRONTEND=noninteractive +# AWS CLI calls below (ssm put-parameter, ec2 describe-instances) pass no +# --region: on EC2 the CLI resolves both credentials and region from the +# instance's IMDS credential provider (it obtains the IMDSv2 token itself, even +# under HttpTokens=required). Reading placement/region with a plain curl would +# need a manual token and return empty without one, so we deliberately let the +# CLI handle it instead of passing a (possibly empty) --region. + echo "[directory] Role: ${ROLE}" echo "[directory] Domain: ${LDAP_DOMAIN} (${LDAP_DOMAIN_SUFFIX})" @@ -110,14 +130,15 @@ cn: clusterusers gidNumber: 3000 EOF - # Persist admin password — prefer SSM (access-controlled, auditable) - REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + # Persist admin password — prefer SSM (access-controlled, auditable). + # No explicit --region: the AWS CLI resolves it from the instance's IMDS + # credential provider (reading placement/region directly would need an + # IMDSv2 token). if aws ssm put-parameter \ --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ --value "${LDAP_ADMIN_PASSWORD}" \ --type SecureString \ - --overwrite \ - --region "${REGION}" 2>/dev/null; then + --overwrite 2>/dev/null; then echo "[directory-server] Admin password stored in SSM: /pcs/${CLUSTER_ID}/ldap/admin-password" else echo "${LDAP_ADMIN_PASSWORD}" > "${LDAP_DB_DIR}/.admin-password" @@ -126,6 +147,22 @@ EOF echo "[directory-server] WARNING: SSM put failed. Password saved to ${LDAP_DB_DIR}/.admin-password" fi + # Install the ldap-add-user helper alongside this script so admins have it + # on PATH (/usr/local/bin) on the login node. Fetched from the same S3 + # location this script came from (S3_BUCKET/S3_KEY_PREFIX passed by UserData); + # falls back to a no-op with a hint if those are unset. + if [ -n "${S3_BUCKET:-}" ]; then + if aws s3 cp "s3://${S3_BUCKET}/${S3_KEY_PREFIX:-templates/}scripts/ldap-add-user.sh" \ + /usr/local/bin/ldap-add-user.sh 2>/dev/null; then + chmod 755 /usr/local/bin/ldap-add-user.sh + echo "[directory-server] Installed helper: /usr/local/bin/ldap-add-user.sh" + else + echo "[directory-server] WARNING: could not fetch ldap-add-user.sh from s3://${S3_BUCKET}/${S3_KEY_PREFIX:-templates/}scripts/ — add users with raw ldapadd (see USER-MANAGEMENT.md)." + fi + else + echo "[directory-server] NOTE: S3_BUCKET unset; skipping ldap-add-user.sh install. Add users with raw ldapadd (see USER-MANAGEMENT.md)." + fi + # Also configure SSSD on the login node itself (so getent works locally) setup_client_internal "ldap://localhost" @@ -146,18 +183,26 @@ setup_client() { # Future: SimpleAD — use provided DNS IPs server_uri="ldap://${DIRECTORY_DNS_IPS%%,*}" else - # OpenLDAP-LoginNode — discover via EC2 tag query - local region - region=$(curl -s http://169.254.169.254/latest/meta-data/placement/region) + # OpenLDAP-LoginNode — discover the directory server's private IP via + # its EC2 tags (pcs-cluster-id + directory-role=server). The + # directory-role tag is owned by the multi-user feature and is + # independent of the monitoring stack's monitoring-role tag. The AWS + # CLI auto-resolves the region from the instance's IMDS credential + # provider, so no explicit --region (which would need an IMDSv2 token + # to read placement/region) is required. + if [ -z "${CLUSTER_ID:-}" ]; then + echo "[directory-client] ERROR: CLUSTER_ID is empty; cannot discover directory server. LDAP client not configured." + return 1 + fi local login_ip - login_ip=$(aws ec2 describe-instances --region "${region}" \ - --filters "Name=tag:pcs-cluster-id,Values=${CLUSTER_ID:-}" \ - "Name=tag:monitoring-role,Values=login" \ + login_ip=$(aws ec2 describe-instances \ + --filters "Name=tag:pcs-cluster-id,Values=${CLUSTER_ID}" \ + "Name=tag:directory-role,Values=server" \ "Name=instance-state-name,Values=running" \ --query 'Reservations[0].Instances[0].PrivateIpAddress' \ --output text 2>/dev/null || echo "") if [ -z "$login_ip" ] || [ "$login_ip" = "None" ]; then - echo "[directory-client] ERROR: could not discover login node IP. LDAP client not configured." + echo "[directory-client] ERROR: could not discover directory server IP (CLUSTER_ID=${CLUSTER_ID}, directory-role=server). Check the compute node's IAM role has ec2:DescribeInstances. LDAP client not configured." return 1 fi server_uri="ldap://${login_ip}" diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index a6e6efa20..c5c53b74f 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -63,6 +63,16 @@ below for reference. - Home directories are auto-created at first login (shared `/home` on OpenZFS) - Slurm sees LDAP users transparently — no Slurm configuration needed for user resolution +> ⚠️ **Single login node only.** `OpenLDAP-LoginNode` runs the directory server +> on **one** login node, so keep the login node group at `MinCount=MaxCount=1` +> while the directory is enabled. Compute clients discover the server by its +> `directory-role=server` tag and the slapd database is a single MDB on shared +> `/home`; running two login nodes would give clients an ambiguous server and +> have two `slapd` processes open the same database files concurrently +> (corruption risk). If you need multiple login nodes or a highly-available +> directory, use a managed backend (the planned `SimpleAD` / `ManagedAD` +> `DirectoryService` options) rather than the login-node OpenLDAP. + --- ## Enabling multi-user From 2fe7d11bd1955c6de82150d39cebeb42622b3ad2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 17:52:42 +0000 Subject: [PATCH 33/89] deploy-all: forward AdditionalSubnetAZ2/3 to prerequisites (multi-AZ wiring) --- .../assets/pcs-ml-cluster-deploy-all.yaml | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 7a4a72740..a248d7447 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -16,6 +16,8 @@ Metadata: default: "1. Network Configuration" Parameters: - PrimarySubnetAZ + - AdditionalSubnetAZ2 + - AdditionalSubnetAZ3 - VPCName - CreateS3Endpoint - Label: @@ -87,6 +89,10 @@ Metadata: # so users see plain-language labels instead of raw CamelCase parameter keys. PrimarySubnetAZ: default: "Availability Zone (required)" + AdditionalSubnetAZ2: + default: "(Optional) 2nd AZ for an additional private subnet" + AdditionalSubnetAZ3: + default: "(Optional) 3rd AZ for an additional private subnet" VPCName: default: "VPC name" CreateS3Endpoint: @@ -199,6 +205,21 @@ Parameters: FSx) are both placed here. Type: AWS::EC2::AvailabilityZone::Name + AdditionalSubnetAZ2: + Description: >- + (Optional) 2nd Availability Zone for an additional private subnet. Empty + (default) = single-AZ. Enables multi-AZ FSx (OpenZFS MULTI_AZ) and future + multi-AZ workloads. The NAT gateway stays in the primary AZ. + Type: String + Default: "" + + AdditionalSubnetAZ3: + Description: >- + (Optional) 3rd Availability Zone for an additional private subnet. Empty + (default) = not created. Requires AdditionalSubnetAZ2 to also be set. + Type: String + Default: "" + CreateS3Endpoint: AllowedValues: - 'true' @@ -667,6 +688,8 @@ Resources: Parameters: VPCName: !Ref VPCName PrimarySubnetAZ: !Ref PrimarySubnetAZ + AdditionalSubnetAZ2: !Ref AdditionalSubnetAZ2 + AdditionalSubnetAZ3: !Ref AdditionalSubnetAZ3 CreateS3Endpoint: !Ref CreateS3Endpoint Capacity: !Ref Capacity LustreDeploymentType: !Ref LustreDeploymentType From e7d72f884708e6ffd36448caeffb9b8583f54ef2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 18:04:16 +0000 Subject: [PATCH 34/89] feat: wire multi-user directory into GPU templates (p5/p6-b200/p6-b300) The three GPU add-cng templates now mirror add-cng.yaml's directory support: - DirectoryService / DirectoryDomainSuffix / DirectoryRole / S3BucketName / S3KeyPrefix params - DirectoryEnabled condition - directory-role tag (independent of monitoring-role) - setup-directory.sh runcmd block (S3-fetched, server installs ldap-add-user helper) deploy-all forwards DirectoryRole=client to all three GPU CNG stacks (P5, P6-B200, P6-B300), so GPU compute nodes join the directory as SSSD clients exactly like CPU compute nodes. All four templates validate. --- architectures/aws-pcs/assets/add-cng-p5.yaml | 70 +++++++++++++++++++ .../aws-pcs/assets/add-cng-p6-b200.yaml | 70 +++++++++++++++++++ .../aws-pcs/assets/add-cng-p6-b300.yaml | 70 +++++++++++++++++++ .../assets/pcs-ml-cluster-deploy-all.yaml | 15 ++++ 4 files changed, 225 insertions(+) diff --git a/architectures/aws-pcs/assets/add-cng-p5.yaml b/architectures/aws-pcs/assets/add-cng-p5.yaml index bca9464c6..2f1c2701c 100644 --- a/architectures/aws-pcs/assets/add-cng-p5.yaml +++ b/architectures/aws-pcs/assets/add-cng-p5.yaml @@ -224,6 +224,49 @@ Parameters: - '25.05' - '25.11' + DirectoryService: + Type: String + Description: >- + Multi-user directory service. 'none' = single ubuntu user (default, + unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared + /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future + options (SimpleAD, ManagedAD) will be added as AllowedValues. + Default: 'none' + AllowedValues: + - 'none' + - 'OpenLDAP-LoginNode' + + DirectoryDomainSuffix: + Type: String + Description: >- + LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when + DirectoryService != none. + Default: 'dc=cluster,dc=internal' + + DirectoryRole: + Type: String + Description: >- + This CNG's role in the directory service. 'server' = install and run + the directory (slapd on login node). 'client' = configure SSSD to + resolve users from the directory server. 'none' = skip directory setup. + deploy-all sets this automatically (login=server, compute=client) when + DirectoryService != none. + Default: 'none' + AllowedValues: + - 'none' + - 'server' + - 'client' + + S3BucketName: + Type: String + Description: S3 bucket where scripts are stored (same as nested templates bucket) + Default: 'awsome-distributed-ai' + + S3KeyPrefix: + Type: String + Description: S3 key prefix for scripts (e.g. templates/) + Default: 'templates/' + Conditions: # EFA interface count is derived from the instance type: p5en.48xlarge has 16 # network cards, p5.48xlarge / p5e.48xlarge have 32. So "use 32 interfaces" @@ -237,6 +280,7 @@ Conditions: # tag (monitoring-role=login) and excludes them from EC2 service discovery. IsMonitoringRoleSet: !Not [!Equals [!Ref MonitoringRole, 'none']] HasAmiId: !Not [!Equals [!Ref AmiId, ""]] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryRole, 'none']] Resources: @@ -336,6 +380,16 @@ Resources: - Key: monitoring-role Value: !Ref MonitoringRole - !Ref AWS::NoValue + # Directory (multi-user) role tag, independent of monitoring-role. + # Emitted for both server (login) and client (compute) nodes when + # DirectoryRole != none. Compute clients discover the OpenLDAP + # server by querying for directory-role=server (NOT monitoring-role, + # which is owned by the monitoring stack). See setup-directory.sh. + - !If + - DirectoryEnabled + - Key: directory-role + Value: !Ref DirectoryRole + - !Ref AWS::NoValue MetadataOptions: HttpEndpoint: enabled HttpPutResponseHopLimit: 4 @@ -396,6 +450,22 @@ Resources: bash /tmp/post-install.sh ${MonitoringVersion} ${MonitoringRepo} 2>&1 | tee /var/log/monitoring-install.log echo "Monitoring installation complete" fi + - | + # Multi-user directory setup (optional, DirectoryRole != none) + if [ "${DirectoryRole}" != "none" ]; then + export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" + export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') + export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + export CLUSTER_ID="${ClusterId}" + export DIRECTORY_DNS_IPS="" + # Pass the script source so the server role can install the + # ldap-add-user.sh helper onto /usr/local/bin from the same bucket. + export S3_BUCKET="${S3BucketName}" + export S3_KEY_PREFIX="${S3KeyPrefix}" + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh + chmod +x /tmp/setup-directory.sh + bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log + fi --==MYBOUNDARY== NetworkInterfaces: diff --git a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml index 1a5145034..856d1d9de 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml @@ -215,6 +215,49 @@ Parameters: - '25.05' - '25.11' + DirectoryService: + Type: String + Description: >- + Multi-user directory service. 'none' = single ubuntu user (default, + unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared + /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future + options (SimpleAD, ManagedAD) will be added as AllowedValues. + Default: 'none' + AllowedValues: + - 'none' + - 'OpenLDAP-LoginNode' + + DirectoryDomainSuffix: + Type: String + Description: >- + LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when + DirectoryService != none. + Default: 'dc=cluster,dc=internal' + + DirectoryRole: + Type: String + Description: >- + This CNG's role in the directory service. 'server' = install and run + the directory (slapd on login node). 'client' = configure SSSD to + resolve users from the directory server. 'none' = skip directory setup. + deploy-all sets this automatically (login=server, compute=client) when + DirectoryService != none. + Default: 'none' + AllowedValues: + - 'none' + - 'server' + - 'client' + + S3BucketName: + Type: String + Description: S3 bucket where scripts are stored (same as nested templates bucket) + Default: 'awsome-distributed-ai' + + S3KeyPrefix: + Type: String + Description: S3 key prefix for scripts (e.g. templates/) + Default: 'templates/' + Conditions: UseCapacityBlock: !Not [!Equals [!Ref CapacityReservationId, ""]] UseOnDemand: !Equals [!Ref CapacityReservationId, ""] @@ -224,6 +267,7 @@ Conditions: # tag (monitoring-role=login) and excludes them from EC2 service discovery. IsMonitoringRoleSet: !Not [!Equals [!Ref MonitoringRole, 'none']] HasAmiId: !Not [!Equals [!Ref AmiId, ""]] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryRole, 'none']] Resources: @@ -323,6 +367,16 @@ Resources: - Key: monitoring-role Value: !Ref MonitoringRole - !Ref AWS::NoValue + # Directory (multi-user) role tag, independent of monitoring-role. + # Emitted for both server (login) and client (compute) nodes when + # DirectoryRole != none. Compute clients discover the OpenLDAP + # server by querying for directory-role=server (NOT monitoring-role, + # which is owned by the monitoring stack). See setup-directory.sh. + - !If + - DirectoryEnabled + - Key: directory-role + Value: !Ref DirectoryRole + - !Ref AWS::NoValue MetadataOptions: HttpEndpoint: enabled HttpPutResponseHopLimit: 4 @@ -397,6 +451,22 @@ Resources: done echo "Monitoring installation complete (exit $rc)" >> /var/log/monitoring-install.log fi + - | + # Multi-user directory setup (optional, DirectoryRole != none) + if [ "${DirectoryRole}" != "none" ]; then + export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" + export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') + export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + export CLUSTER_ID="${ClusterId}" + export DIRECTORY_DNS_IPS="" + # Pass the script source so the server role can install the + # ldap-add-user.sh helper onto /usr/local/bin from the same bucket. + export S3_BUCKET="${S3BucketName}" + export S3_KEY_PREFIX="${S3KeyPrefix}" + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh + chmod +x /tmp/setup-directory.sh + bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log + fi --==MYBOUNDARY== NetworkInterfaces: diff --git a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml index 822106d62..5d280ce71 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml @@ -218,6 +218,49 @@ Parameters: - '25.05' - '25.11' + DirectoryService: + Type: String + Description: >- + Multi-user directory service. 'none' = single ubuntu user (default, + unchanged). 'OpenLDAP-LoginNode' = deploy slapd on the login node (DB on shared + /home/ldap-db), compute nodes configure SSSD as LDAP clients. Future + options (SimpleAD, ManagedAD) will be added as AllowedValues. + Default: 'none' + AllowedValues: + - 'none' + - 'OpenLDAP-LoginNode' + + DirectoryDomainSuffix: + Type: String + Description: >- + LDAP domain suffix (e.g. dc=cluster,dc=internal). Only used when + DirectoryService != none. + Default: 'dc=cluster,dc=internal' + + DirectoryRole: + Type: String + Description: >- + This CNG's role in the directory service. 'server' = install and run + the directory (slapd on login node). 'client' = configure SSSD to + resolve users from the directory server. 'none' = skip directory setup. + deploy-all sets this automatically (login=server, compute=client) when + DirectoryService != none. + Default: 'none' + AllowedValues: + - 'none' + - 'server' + - 'client' + + S3BucketName: + Type: String + Description: S3 bucket where scripts are stored (same as nested templates bucket) + Default: 'awsome-distributed-ai' + + S3KeyPrefix: + Type: String + Description: S3 key prefix for scripts (e.g. templates/) + Default: 'templates/' + Conditions: UseCapacityBlock: !Not [!Equals [!Ref CapacityReservationId, ""]] UseOnDemand: !Equals [!Ref CapacityReservationId, ""] @@ -227,6 +270,7 @@ Conditions: # tag (monitoring-role=login) and excludes them from EC2 service discovery. IsMonitoringRoleSet: !Not [!Equals [!Ref MonitoringRole, 'none']] HasAmiId: !Not [!Equals [!Ref AmiId, ""]] + DirectoryEnabled: !Not [!Equals [!Ref DirectoryRole, 'none']] Resources: @@ -326,6 +370,16 @@ Resources: - Key: monitoring-role Value: !Ref MonitoringRole - !Ref AWS::NoValue + # Directory (multi-user) role tag, independent of monitoring-role. + # Emitted for both server (login) and client (compute) nodes when + # DirectoryRole != none. Compute clients discover the OpenLDAP + # server by querying for directory-role=server (NOT monitoring-role, + # which is owned by the monitoring stack). See setup-directory.sh. + - !If + - DirectoryEnabled + - Key: directory-role + Value: !Ref DirectoryRole + - !Ref AWS::NoValue MetadataOptions: HttpEndpoint: enabled HttpPutResponseHopLimit: 4 @@ -400,6 +454,22 @@ Resources: done echo "Monitoring installation complete (exit $rc)" >> /var/log/monitoring-install.log fi + - | + # Multi-user directory setup (optional, DirectoryRole != none) + if [ "${DirectoryRole}" != "none" ]; then + export LDAP_DOMAIN_SUFFIX="${DirectoryDomainSuffix}" + export LDAP_DOMAIN=$(echo "${DirectoryDomainSuffix}" | sed 's/dc=//g; s/,/./g') + export LDAP_ADMIN_PASSWORD=$(openssl rand -base64 16) + export CLUSTER_ID="${ClusterId}" + export DIRECTORY_DNS_IPS="" + # Pass the script source so the server role can install the + # ldap-add-user.sh helper onto /usr/local/bin from the same bucket. + export S3_BUCKET="${S3BucketName}" + export S3_KEY_PREFIX="${S3KeyPrefix}" + aws s3 cp s3://${S3BucketName}/${S3KeyPrefix}scripts/setup-directory.sh /tmp/setup-directory.sh + chmod +x /tmp/setup-directory.sh + bash /tmp/setup-directory.sh "${DirectoryRole}" 2>&1 | tee /var/log/directory-setup.log + fi --==MYBOUNDARY== NetworkInterfaces: diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index a248d7447..aa953e353 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -893,6 +893,11 @@ Resources: MonitoringRole: compute PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs + DirectoryService: !Ref DirectoryService + DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] + DirectoryDomainSuffix: !Ref DirectoryDomainSuffix + S3BucketName: !Ref S3BucketName + S3KeyPrefix: !Ref S3KeyPrefix Tags: - Key: Name Value: !Sub '${AWS::StackName}-PseriesCNG' @@ -934,6 +939,11 @@ Resources: MonitoringRole: compute PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs + DirectoryService: !Ref DirectoryService + DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] + DirectoryDomainSuffix: !Ref DirectoryDomainSuffix + S3BucketName: !Ref S3BucketName + S3KeyPrefix: !Ref S3KeyPrefix Tags: - Key: Name Value: !Sub '${AWS::StackName}-PseriesCNG' @@ -975,6 +985,11 @@ Resources: MonitoringRole: compute PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs + DirectoryService: !Ref DirectoryService + DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] + DirectoryDomainSuffix: !Ref DirectoryDomainSuffix + S3BucketName: !Ref S3BucketName + S3KeyPrefix: !Ref S3KeyPrefix Tags: - Key: Name Value: !Sub '${AWS::StackName}-PseriesCNG' From ce6e5a5023218fa021f07fe52cb843feb61b9301 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 18:09:07 +0000 Subject: [PATCH 35/89] docs: update README/PARAMETERS/ROADMAP for renamed + new params MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: DeployMonitoring=true → MonitoringStack=Prometheus-LoginNode, GrafanaPublicAccessCidr → GrafanaAccessCidr (incl. Option B heading + anchor), added SSHAccessCidr mention in §6 Accessing the Cluster. - PARAMETERS.md: new §2b Additional Cluster Configuration (MonitoringStack, GrafanaAccessCidr, DirectoryService, DirectoryDomainSuffix); SSHAccessCidr in §2; AdditionalSubnetAZ2/3 in §1 Network; fixed stale #9-pre-baking anchor → #82. - ROADMAP.md: checked off Multi-AZ support, LDAP/AD user-management backend, DeployMonitoring→MonitoringStack rename (with done-notes + follow-ups). --- architectures/aws-pcs/README.md | 22 +++++++------- architectures/aws-pcs/docs/PARAMETERS.md | 20 +++++++++---- architectures/aws-pcs/docs/ROADMAP.md | 38 +++++++++++++----------- 3 files changed, 48 insertions(+), 32 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 8c99fc0a8..eebd69f9d 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -8,7 +8,7 @@ This repository provides reference architectures and deployment templates for se - **One click to an ML-training-ready cluster**: a single CloudFormation stack gives you a complete, ready-to-train environment — Slurm scheduler, GPU compute with EFA, shared FSx storage, the Enroot/Pyxis container runtime, and monitoring — with only the Availability Zone to choose. Submit distributed training jobs minutes after launch. - **Container runtime included**: Enroot/Pyxis is set up automatically, so `srun --container-image=...` works out of the box for containerized training. -- **Monitoring built in**: Grafana + Prometheus run on the login node, with DCGM Exporter on GPU compute nodes feeding the pre-built GPU dashboards (`DeployMonitoring=true`, on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR with `GrafanaPublicAccessCidr`. +- **Monitoring built in**: Grafana + Prometheus run on the login node, with DCGM Exporter on GPU compute nodes feeding the pre-built GPU dashboards (`MonitoringStack=Prometheus-LoginNode`, on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR with `GrafanaAccessCidr`. - **GPU-ready, multi-NIC EFA**: dedicated launch templates for the P5 and P6 families, selected automatically by instance type, for high-bandwidth multi-node training. - **Broad capacity-purchase support**: covers the full range of EC2 capacity options out of the box — On-Demand, On-Demand Capacity Reservations (ODCR), and Capacity Blocks for ML — selected per node group. - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). @@ -63,7 +63,7 @@ Add a GPU queue and tune storage/monitoring via the parameters below. Once it's up: - **Connect** to the login node via SSM Session Manager — see [Accessing the Cluster](#6-accessing-the-cluster). - **Open the Grafana dashboards** (deployed by default) via SSM port forwarding — see [Accessing Grafana](#accessing-grafana). -- **Want to reach Grafana directly in a browser** (no port forwarding)? Set `GrafanaPublicAccessCidr` to a trusted CIDR at deploy time — see [Option B — Direct public access](#option-b--direct-public-access-opt-in-via-grafanapublicaccesscidr). +- **Want to reach Grafana directly in a browser** (no port forwarding)? Set `GrafanaAccessCidr` to a trusted CIDR at deploy time — see [Option B — Direct public access](#option-b--direct-public-access-opt-in-via-grafanaaccesscidr). Prefer step-by-step instructions? See the [AI/ML for AWS PCS Workshop](https://catalog.workshops.aws/ml-on-pcs/). @@ -82,7 +82,7 @@ are deleted with the stack. ## 4. Configuration Defaults give the most common production setup — Enroot/Pyxis installed at first -boot via `PostInstallScriptUrl` + `DeployMonitoring=true` — so a default deploy +boot via `PostInstallScriptUrl` + `MonitoringStack=Prometheus-LoginNode` — so a default deploy only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: | Parameter | Default | Purpose | @@ -90,7 +90,7 @@ only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#82-pre-baking-enrootpyxis-into-a-custom-ami) | -| `DeployMonitoring` | `true` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes | +| `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `DeployOnDemandCNG` | `true` | Deploy the `cpu1` CPU queue (`OnDemandInstanceType`, default `c6i.4xlarge`) | | `OnDemandEnableEfa` | `false` | Set `true` for HPC/MPI workloads on EFA-capable CPU types (hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandenableefa) | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | @@ -272,7 +272,9 @@ group. For other HPC types, set `OnDemandInstanceType` and the matching ## 6. Accessing the Cluster -Connect to the login node via SSM Session Manager — no public SSH needed. +Connect to the login node via SSM Session Manager — no public SSH needed. If you do +want direct SSH, set `SSHAccessCidr` to a trusted CIDR at deploy time to open SSH/22 +on the login node to that CIDR. **Console:** [EC2 Console](https://console.aws.amazon.com/ec2/home#Instances:) → filter by `aws:pcs:compute-node-group-name = login` → **Connect** → **Session Manager**. @@ -323,7 +325,7 @@ the full validation matrix is in [tests/README.md](./tests/README.md). ### 8.1 Monitoring -With `DeployMonitoring=true` (default), an integrated monitoring stack based on +With `MonitoringStack=Prometheus-LoginNode` (default), an integrated monitoring stack based on [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) is installed automatically: @@ -350,7 +352,7 @@ and a screenshot. > node, see [`4.validation_and_observability/4.prometheus-grafana`](../../4.validation_and_observability/4.prometheus-grafana). **Monitoring-related parameters:** -- `DeployMonitoring` (default `true`) +- `MonitoringStack` (default `Prometheus-LoginNode`; `none` disables monitoring) - `MonitoringVersion` — [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`; default `v2.9.1`). Pinned to a tag so upstream changes can't break deployments unexpectedly. `v2.9.1` adds the `DCGM_EXPORTER_IMAGE` override (lets `DcgmExporterImage` enable B300 GPU metrics); `v2.6.4`+ carry the PCS fixes (node-local `/opt` install + Docker-29.x DCGM tag). - `MonitoringRepo` — `owner/repo` to fetch from (default `aws-samples/aws-parallelcluster-monitoring`). Point at a fork + a branch in `MonitoringVersion` to test unreleased changes. - `DcgmExporterImage` — dcgm-exporter image used on GPU nodes; defaults to a DCGM 4.5.2 build pinned by digest (covers Hopper/B200/B300). Override only if you need to pin to a different build (e.g. the older monitoring-default DCGM 4.2.0). @@ -389,9 +391,9 @@ aws ssm start-session --target $INSTANCE_ID \ Then open `https://localhost:8443/grafana/`. -#### Option B — Direct public access (opt-in, via `GrafanaPublicAccessCidr`) +#### Option B — Direct public access (opt-in, via `GrafanaAccessCidr`) -To browse Grafana directly without port forwarding, set **`GrafanaPublicAccessCidr`** at +To browse Grafana directly without port forwarding, set **`GrafanaAccessCidr`** at deploy time to a CIDR you trust (e.g. your office IP `203.0.113.4/32`). deploy-all then creates a **login-only security group** that opens HTTPS/**443** to that CIDR and attaches it to the login node, so you can open: @@ -425,7 +427,7 @@ Security notes: as you are done. - The certificate is self-signed, so browsers show a warning — proceed past it, or put an ALB + ACM certificate in front for a trusted cert. -- Leaving `GrafanaPublicAccessCidr` empty (the default) keeps monitoring private; use +- Leaving `GrafanaAccessCidr` empty (the default) keeps monitoring private; use Option A. --- diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index f801f2549..59d6e1274 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -7,7 +7,7 @@ the latest PCS-Ready Deep Learning AMI auto-resolved from SSM, Enroot/Pyxis inst at first boot via `PostInstallScriptUrl`, monitoring enabled — so a default deploy only needs the Availability Zone (`PrimarySubnetAZ`). To pre-bake Enroot/Pyxis into a custom AMI for faster boots, build it separately with -[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#9-pre-baking-enrootpyxis-into-a-custom-ami-optional) +[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#82-pre-baking-enrootpyxis-into-a-custom-ami) and pass its output as `AmiId`. For conceptual guidance (GPU instance/EFA selection, FSx Region availability, container @@ -17,7 +17,9 @@ runtime options), see the [README](../README.md#4-configuration). | Parameter | Default | Purpose | |---|---|---| -| `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | +| `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter. Holds the public subnet (login node), the primary private subnet (compute, FSx), and the single NAT gateway | +| `AdditionalSubnetAZ2` | *(empty)* | (Optional) 2nd AZ for an additional private subnet. Empty = single-AZ. Enables multi-AZ layouts (e.g. OpenZFS `MULTI_AZ`). Shares the primary AZ's NAT gateway (cross-AZ egress, no per-AZ NAT) | +| `AdditionalSubnetAZ3` | *(empty)* | (Optional) 3rd AZ for an additional private subnet. Requires `AdditionalSubnetAZ2` to also be set. Max 3 private AZs total | | `VPCName` | `ML-Cluster-VPC` | Name for the created VPC | | `CreateS3Endpoint` | `true` | Create an S3 VPC endpoint | @@ -28,12 +30,20 @@ runtime options), see the [README](../README.md#4-configuration). | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`). Drives which monitoring you get (Slurm OpenMetrics is 25.11+ only) and is also threaded into the CNG UserData so the right-version Pyxis is installed; see [OPERATIONS.md §1](./OPERATIONS.md#1-slurm-version-selection) | | `LoginNodeInstanceType` | `m6i.4xlarge` | Login node instance type | | `RootVolumeSize` | `300` | Root EBS volume size (GiB) on every node (login + compute); 300 leaves room for large container images (Megatron `.sqsh` ~20 GB) | -| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#9-pre-baking-enrootpyxis-into-a-custom-ami-optional)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | -| `DeployMonitoring` | `true` | Deploy Prometheus/Grafana/DCGM on the login node | -| `GrafanaPublicAccessCidr` | *(empty)* | When set to a CIDR, opens HTTPS/443 on the login node to that CIDR via a login-only security group. Empty = SSM port-forward only. **443 also exposes the unauthenticated `/prometheus/`, `/pushgateway/`, `/slurmexporter/` proxy paths**, not just the password-gated Grafana. Use the tightest CIDR you can; `0.0.0.0/0` is accepted for short-lived PoC/workshop use but exposes those endpoints to the whole internet | +| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#82-pre-baking-enrootpyxis-into-a-custom-ami)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | +| `SSHAccessCidr` | *(empty)* | When set to a CIDR, opens SSH/22 on the login node to that CIDR via a login-only security group (attached to the login node only, never compute). Empty (default) = SSH over SSM only. Set to your office/VPN range for direct `ssh`/`scp`/VS Code Remote (common for multi-user clusters) | | `ManagedAccounting` | `disabled` | Enable Slurm managed accounting (requires Slurm 24.11+) | | `AccountingPolicyEnforcement` | `none` | Slurm accounting policy enforcement (`none` or `associations,limits,safe`) | +## 2b. Additional Cluster Configuration (Monitoring, Multi-User) + +| Parameter | Default | Purpose | +|---|---|---| +| `MonitoringStack` | `Prometheus-LoginNode` | Monitoring stack to deploy. `Prometheus-LoginNode` = self-hosted Prometheus + Grafana + DCGM Exporter on the login node. `none` = no monitoring. (Renamed from the old boolean `DeployMonitoring`; `-` enum, extensible to future `AMP-AMG`/`CloudWatch`) | +| `GrafanaAccessCidr` | *(empty)* | When set to a CIDR, opens HTTPS/443 (Grafana) on the login node to that CIDR via the login-only security group. Empty = SSM port-forward only. **443 also exposes the unauthenticated `/prometheus/`, `/pushgateway/`, `/slurmexporter/` proxy paths**, not just the password-gated Grafana. Use the tightest CIDR you can. (Renamed from `GrafanaPublicAccessCidr`) | +| `DirectoryService` | `none` | Multi-user directory. `none` = single `ubuntu` user. `OpenLDAP-LoginNode` = slapd on the login node (DB on shared `/home/ldap-db`) + SSSD on all compute nodes. **Single login node only** — keep the login node group at 1 instance while enabled. See [USER-MANAGEMENT.md](./USER-MANAGEMENT.md) | +| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP domain suffix. Only used when `DirectoryService != none` | + ## 3. Container Runtime (Post-install Script) | Parameter | Default | Purpose | diff --git a/architectures/aws-pcs/docs/ROADMAP.md b/architectures/aws-pcs/docs/ROADMAP.md index e87b82c5b..d57e66ac8 100644 --- a/architectures/aws-pcs/docs/ROADMAP.md +++ b/architectures/aws-pcs/docs/ROADMAP.md @@ -8,10 +8,12 @@ Priority: 🔴 high · 🟡 medium · 🟢 low ## Templates & deployment -- [ ] 🟡 **Multi-AZ support in the prerequisites stack.** `ml-cluster-prerequisites.yaml` - currently creates a single private subnet, so `OpenZFSDeploymentType` excludes the - `MULTI_AZ` types. Add a second private subnet (and the related routing) to enable - Multi-AZ FSx and higher-availability deployments. +- [x] 🟡 **Multi-AZ support in the prerequisites stack.** `ml-cluster-prerequisites.yaml` + now supports up to 3 private-subnet AZs via `AdditionalSubnetAZ2`/`AdditionalSubnetAZ3` + (CIDR1 split into four /18 blocks; additional subnets share the primary AZ's single + NAT gateway). This unblocks `OpenZFSDeploymentType=MULTI_AZ` and higher-availability + layouts. *(Note: OpenZFS MULTI_AZ wiring of the 2nd subnet into the FSx resource is a + follow-up; the subnets + routing are in place.)* - [ ] 🟢 **Trainium (Trn) validation.** Validate the templates on Trainium instances (e.g. trn1/trn2) — node group, EFA/networking, and a sample training run. - [ ] 🟡 **Graviton (arm64) CPU CNG support — `hpc7g` / `c7gn`.** EFA-capable arm64 @@ -89,11 +91,13 @@ Priority: 🔴 high · 🟡 medium · 🟢 low ## User management -- [ ] 🟡 **Integrate a user-management backend (LDAP/AD).** Provide a way to manage cluster - users centrally instead of the single `ubuntu` user — e.g. integrate an LDAP/OpenLDAP or - AWS Managed Microsoft AD directory (see `1.architectures/6.ldap_server`) so login/compute - nodes authenticate against a shared directory (multi-user clusters, per-user home dirs, - Slurm accounting per user). +- [x] 🟡 **Integrate a user-management backend (LDAP/AD).** Done for OpenLDAP: + `DirectoryService=OpenLDAP-LoginNode` runs slapd on the login node (DB on shared + `/home/ldap-db`) with SSSD on all compute nodes (CPU + GPU). Users added via + `ldap-add-user` resolve cluster-wide; home dirs auto-create; Slurm sees LDAP users + transparently. See `docs/USER-MANAGEMENT.md`. *(Follow-up: managed-directory options + `DirectoryService=SimpleAD`/`ManagedAD` for multi-login-node / HA — the param enum is + already extensible.)* ## Monitoring @@ -101,14 +105,14 @@ Priority: 🔴 high · 🟡 medium · 🟢 low Prometheus + Amazon Managed Grafana as an alternative to the self-hosted stack on the login node (see `4.validation_and_observability/4.prometheus-grafana`), so users can use a managed backend instead of running the containers themselves. -- [ ] 🟡 **Rename `DeployMonitoring` → `MonitoringStack` (enum).** Align with the - `DirectoryService` parameter pattern (`-`): - `MonitoringStack: none | Prometheus-LoginNode | AMP-AMG | CloudWatch`. - This makes it clear that the current default is a self-hosted Prometheus + Grafana - stack on the login node, and provides a natural extension point for the managed - option above. **Breaking change** — bundle with the next minor version alongside - other parameter renames (`Capacity` → `LustreCapacity`, `CngName`/`QueueName` - collapse, etc.). +- [x] 🟡 **Rename `DeployMonitoring` → `MonitoringStack` (enum).** Done in deploy-all: + `MonitoringStack: none | Prometheus-LoginNode` (default `Prometheus-LoginNode`), + aligning with the `DirectoryService` `-` pattern. `AMP-AMG`/`CloudWatch` + remain as future AllowedValues for the managed-monitoring item above. deploy-all + converts to the nested templates' `DeployMonitoring=true/false` internally, so + add-cng*.yaml are unchanged. **Breaking change** at the deploy-all interface + (bundled into the major-update PR alongside `GrafanaPublicAccessCidr`→`GrafanaAccessCidr` + and `SSHAccessCidr`). ## Testing / docs From 987ee0d916ee7d2b17132166e64b644c5fc8f970 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 18:33:45 +0000 Subject: [PATCH 36/89] fix: setup-directory.sh waits for apt/dpkg lock (first-boot unattended-upgrades) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit slapd/sssd install failed at first boot with "Could not get lock /var/lib/dpkg/lock-frontend ... held by unattended-upgr" — same first-boot race install-enroot-pyxis.sh already handles. Added wait_for_apt_lock + an apt_get wrapper (3 retries) and routed all four apt-get calls through it. Without this, set -euo pipefail aborted the script on the lock error, leaving slapd uninstalled and the ldap-add-user helper unplaced. Also documented (USER-MANAGEMENT.md): tag-based LDAP-server discovery is scoped by pcs-cluster-id, so multiple PCS clusters can share one VPC safely. --- .../aws-pcs/assets/scripts/setup-directory.sh | 44 ++++++++++++++++-- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 46 +++++++++++++++++++ 2 files changed, 86 insertions(+), 4 deletions(-) diff --git a/architectures/aws-pcs/assets/scripts/setup-directory.sh b/architectures/aws-pcs/assets/scripts/setup-directory.sh index 2ae5a799f..44b9b54f8 100755 --- a/architectures/aws-pcs/assets/scripts/setup-directory.sh +++ b/architectures/aws-pcs/assets/scripts/setup-directory.sh @@ -48,6 +48,42 @@ export DEBIAN_FRONTEND=noninteractive echo "[directory] Role: ${ROLE}" echo "[directory] Domain: ${LDAP_DOMAIN} (${LDAP_DOMAIN_SUFFIX})" +############################################################################### +# apt helpers — at first boot cloud-init's own unattended-upgrades/apt run can +# still hold the dpkg lock, so a bare `apt-get install` fails with +# "Could not get lock /var/lib/dpkg/lock-frontend". Wait for the lock to clear, +# then run apt-get with built-in retries. Mirrors install-enroot-pyxis.sh. +############################################################################### +wait_for_apt_lock() { + local waited=0 + while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1 \ + || fuser /var/lib/dpkg/lock >/dev/null 2>&1 \ + || fuser /var/lib/apt/lists/lock >/dev/null 2>&1; do + if [ "$waited" -ge 300 ]; then + echo "[directory] WARNING: dpkg lock still held after ${waited}s; proceeding anyway." + break + fi + echo "[directory] Waiting for apt/dpkg lock to clear (${waited}s)..." + sleep 10 + waited=$((waited + 10)) + done +} + +apt_get() { + wait_for_apt_lock + local attempt + for attempt in 1 2 3; do + if DEBIAN_FRONTEND=noninteractive apt-get "$@"; then + return 0 + fi + echo "[directory] apt-get $* failed (attempt ${attempt}/3); waiting for lock and retrying..." + wait_for_apt_lock + sleep 5 + done + echo "[directory] ERROR: apt-get $* failed after 3 attempts." + return 1 +} + ############################################################################### # SERVER role — login node: install slapd + configure ############################################################################### @@ -56,7 +92,7 @@ setup_server() { CLUSTER_ID="${CLUSTER_ID:-unknown}" echo "[directory-server] Running apt-get update..." - apt-get update -qq + apt_get update -qq echo "[directory-server] Installing slapd + ldap-utils..." debconf-set-selections < /etc/sssd/sssd.conf < directory, use a managed backend (the planned `SimpleAD` / `ManagedAD` > `DirectoryService` options) rather than the login-node OpenLDAP. +### How a compute node finds the LDAP server (tag-based discovery) + +This part is **not obvious**, so it's worth spelling out. A compute node does +**not** receive the login node's IP as a parameter — PCS launches the login and +compute node groups independently, and the login node's private IP isn't known +at template-synthesis time (and it changes if the login node is replaced). +Instead, discovery happens **at compute-node boot**, by EC2 tag lookup: + +1. When the directory is enabled, the login node group tags its instance + `directory-role=server` (alongside `pcs-cluster-id=`). The + compute node groups tag themselves `directory-role=client`. This + `directory-role` tag is **dedicated to the directory feature** — it is + deliberately *separate* from the monitoring stack's `monitoring-role` tag, so + the two features don't depend on each other. +2. On first boot, each compute node runs `setup-directory.sh client`, which + calls `aws ec2 describe-instances` filtering for + `tag:pcs-cluster-id=` + `tag:directory-role=server` + + `instance-state-name=running`, and reads the matching instance's + `PrivateIpAddress`. The `pcs-cluster-id` filter scopes the lookup to **this + cluster only**, so multiple PCS clusters can share one VPC without their + compute nodes finding the wrong cluster's LDAP server. (`CLUSTER_ID` is + passed from `${ClusterId}` in UserData; the script aborts client setup if it + is empty, rather than risk matching another cluster's server.) +3. That IP becomes the SSSD `ldap_uri` (`ldap://`). SSSD on the + compute node then resolves users from the login node's slapd. + +Implications to be aware of: + +- **Compute nodes need `ec2:DescribeInstances`** in their instance role (the + cluster IAM role already grants it). Without it, discovery fails and the node + boots without LDAP (check `/var/log/directory-setup.log` for + `could not discover directory server IP`). +- **The login (server) node must be running before a compute node boots** for + discovery to succeed. In normal deploy order the login node group comes up + first; a compute node that scales up later simply queries the + already-running server. +- **If the login node is replaced**, its new instance re-tags itself + `directory-role=server` and re-attaches to the same `/home/ldap-db`, so newly + booting compute nodes discover the new IP automatically. Already-running + compute nodes keep their cached `ldap_uri`; they pick up the new IP on their + next boot (or after an SSSD reconfigure). +- **An explicit override exists**: set `LDAP_SERVER_URI` (or `DIRECTORY_DNS_IPS`, + for the future managed-directory path) in the client's environment to skip the + tag lookup entirely — used by the SimpleAD/ManagedAD extension and handy for + debugging. + --- ## Enabling multi-user From 465e0cd36a5ce3e89b19bb26c177cea4ed76bdc3 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 19:08:43 +0000 Subject: [PATCH 37/89] docs: finish param-rename sweep + add LDAP discovery / multi-VPC notes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - OPERATIONS.md: GrafanaPublicAccessCidr → GrafanaAccessCidr (2 refs) - tests/README.md + infra-test.md: DeployMonitoring=true → MonitoringStack=Prometheus-LoginNode - USER-MANAGEMENT.md: tag-based LDAP-server discovery section (how compute finds the login node by directory-role=server, IAM/ordering implications, override), and multi-cluster-per-VPC safety (pcs-cluster-id scopes the lookup). --- architectures/aws-pcs/docs/OPERATIONS.md | 4 ++-- architectures/aws-pcs/tests/README.md | 2 +- architectures/aws-pcs/tests/infra-test.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/architectures/aws-pcs/docs/OPERATIONS.md b/architectures/aws-pcs/docs/OPERATIONS.md index c3060da53..06ea5a9a7 100644 --- a/architectures/aws-pcs/docs/OPERATIONS.md +++ b/architectures/aws-pcs/docs/OPERATIONS.md @@ -135,7 +135,7 @@ want — leave it alone. ### 3.2 Public Grafana exposure -`GrafanaPublicAccessCidr` opens **TCP/443 on the login node** to a CIDR via a +`GrafanaAccessCidr` opens **TCP/443 on the login node** to a CIDR via a login-only security group. The nginx in front of Grafana also proxies **`/prometheus/`, `/pushgateway/`, `/slurmexporter/` without authentication** — opening the CIDR exposes those too, not just password-gated Grafana. Use the tightest CIDR you @@ -243,5 +243,5 @@ For a new production deploy: and pass its output as `AmiId` (~3 min boot vs ~6 min). Match `SlurmVersion` between the AMI build stack and the cluster stack. - Default `DcgmExporterImage` covers H100/B200/B300; override only to pin a different build -- Minimum-CIDR `GrafanaPublicAccessCidr` if used at all; otherwise empty (SSM port-forward) +- Minimum-CIDR `GrafanaAccessCidr` if used at all; otherwise empty (SSM port-forward) - Throughput values that match the chosen FSx deployment types diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index d36267a74..23c8dbadb 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -26,7 +26,7 @@ Run this **complete set** before merging template or script changes. ## Quick-start: single-cluster shortcut -A single `pcs-ml-cluster-deploy-all.yaml` deploy with `DeployMonitoring=true`, +A single `pcs-ml-cluster-deploy-all.yaml` deploy with `MonitoringStack=Prometheus-LoginNode`, `DeployOnDemandCNG=true`, and `DeployPseriesCNG=true` exercises Tests 1–7 in one cluster. Tests 8-13 are separate paths run only when their inputs change. diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md index 641098e32..c7f3f59e1 100644 --- a/architectures/aws-pcs/tests/infra-test.md +++ b/architectures/aws-pcs/tests/infra-test.md @@ -7,7 +7,7 @@ first-boot install, and the pre-baked AMI build path. ## Test 1: Monitoring stack -With `DeployMonitoring=true` (default), Prometheus/Grafana/exporters install on the +With `MonitoringStack=Prometheus-LoginNode` (default), Prometheus/Grafana/exporters install on the login node and DCGM/node exporters on the compute nodes. ```bash From 1b7d1bfd57f43c25a8c3bdf53f9d6f6ed0a953f1 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 19:38:53 +0000 Subject: [PATCH 38/89] feat: bring in IAM policy stacks from feat/aws-pcs-updates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Conflict-free IAM additions for the major-update PR: - iam/cluster-admin-policy.json + cluster-admin-imagebuilder-policy.json - iam/cluster-user-policy.json - iam/README.md (verification matrix + design rationale) - assets/cluster-admin-iam.yaml + cluster-user-iam.yaml (1-click deploy stacks) Least-privilege policies: cluster admin (deploy/update/delete) and cluster user (SSM session + read-only). All templates validate; all JSON parses. The deploy-all/PARAMETERS.md changes from feat/aws-pcs-updates (VPCName auto-derive, EFA auto-count, label cleanup) are merged separately — they conflict with the multi-user param reorg and need manual resolution. --- .../aws-pcs/assets/cluster-admin-iam.yaml | 418 ++++++++++++++++++ .../aws-pcs/assets/cluster-user-iam.yaml | 139 ++++++ architectures/aws-pcs/iam/README.md | 329 ++++++++++++++ .../cluster-admin-imagebuilder-policy.json | 60 +++ .../aws-pcs/iam/cluster-admin-policy.json | 282 ++++++++++++ .../aws-pcs/iam/cluster-user-policy.json | 81 ++++ 6 files changed, 1309 insertions(+) create mode 100644 architectures/aws-pcs/assets/cluster-admin-iam.yaml create mode 100644 architectures/aws-pcs/assets/cluster-user-iam.yaml create mode 100644 architectures/aws-pcs/iam/README.md create mode 100644 architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json create mode 100644 architectures/aws-pcs/iam/cluster-admin-policy.json create mode 100644 architectures/aws-pcs/iam/cluster-user-policy.json diff --git a/architectures/aws-pcs/assets/cluster-admin-iam.yaml b/architectures/aws-pcs/assets/cluster-admin-iam.yaml new file mode 100644 index 000000000..58af14d14 --- /dev/null +++ b/architectures/aws-pcs/assets/cluster-admin-iam.yaml @@ -0,0 +1,418 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +AWSTemplateFormatVersion: "2010-09-09" +Description: | + Creates two customer-managed IAM policies that together grant the + permissions a deploying principal needs to create / update / delete an + AWS PCS reference cluster from the templates in architectures/aws-pcs/. + Optionally attaches them to an IAM Group whose members become PCS cluster + admins. The policy set is split into two because it exceeds the IAM + per-policy 6144-char limit: a "core" policy for everyday cluster CRUD, + and an optional "Image Builder" policy for the standalone DLAMI builder + template (pcs-ready-dlami-with-enroot-pyxis.yaml). +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: { default: "Group + IAM users (optional)" } + Parameters: [GroupName, AttachUsers, AttachImageBuilderPolicy] + ParameterLabels: + GroupName: { default: "Name of the IAM group to create" } + AttachUsers: { default: "Existing IAM users to add to the group (comma-separated, optional)" } + AttachImageBuilderPolicy: { default: "Also attach the Image Builder admin policy" } +Parameters: + GroupName: + Type: String + Default: PCSClusterAdmins + Description: IAM group name for cluster administrators + AttachUsers: + Type: CommaDelimitedList + Default: "" + Description: Comma-separated list of existing IAM user names to add to the group. Leave empty to skip. + AttachImageBuilderPolicy: + Type: String + Default: "false" + AllowedValues: ["true", "false"] + Description: Attach the optional Image Builder admin policy (needed only when deploying pcs-ready-dlami-with-enroot-pyxis.yaml). +Conditions: + HasUsers: !Not [!Equals [!Join ["", !Ref AttachUsers], ""]] + AttachImageBuilder: !Equals [!Ref AttachImageBuilderPolicy, "true"] +Resources: + PCSClusterAdminPolicy: + Type: AWS::IAM::ManagedPolicy + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterAdmin-core" + Description: "Core CFN+EC2+FSx+PCS+IAM(scoped)+SSM+KMS+Secrets+Logs permissions for AWS PCS cluster CRUD" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudFormationStackLifecycle", + "Effect": "Allow", + "Action": [ + "cloudformation:CreateStack", + "cloudformation:UpdateStack", + "cloudformation:DeleteStack", + "cloudformation:CancelUpdateStack", + "cloudformation:ContinueUpdateRollback", + "cloudformation:CreateChangeSet", + "cloudformation:ExecuteChangeSet", + "cloudformation:DescribeChangeSet", + "cloudformation:DeleteChangeSet", + "cloudformation:DescribeStacks", + "cloudformation:DescribeStackEvents", + "cloudformation:DescribeStackResource", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources", + "cloudformation:GetTemplate", + "cloudformation:GetTemplateSummary", + "cloudformation:TagResource", + "cloudformation:UntagResource" + ], + "Resource": "*" + }, + { + "Sid": "CloudFormationDescribeStackless", + "Effect": "Allow", + "Action": [ + "cloudformation:ValidateTemplate", + "cloudformation:ListStacks", + "cloudformation:ListExports", + "cloudformation:ListImports" + ], + "Resource": "*" + }, + { + "Sid": "EC2NetworkingAndComputeLifecycle", + "Effect": "Allow", + "Action": [ + "ec2:CreateVpc", + "ec2:DeleteVpc", + "ec2:ModifyVpcAttribute", + "ec2:AssociateVpcCidrBlock", + "ec2:DisassociateVpcCidrBlock", + "ec2:CreateSubnet", + "ec2:DeleteSubnet", + "ec2:ModifySubnetAttribute", + "ec2:CreateInternetGateway", + "ec2:DeleteInternetGateway", + "ec2:AttachInternetGateway", + "ec2:DetachInternetGateway", + "ec2:CreateNatGateway", + "ec2:DeleteNatGateway", + "ec2:AllocateAddress", + "ec2:ReleaseAddress", + "ec2:AssociateAddress", + "ec2:DisassociateAddress", + "ec2:CreateRouteTable", + "ec2:DeleteRouteTable", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:ReplaceRoute", + "ec2:AssociateRouteTable", + "ec2:DisassociateRouteTable", + "ec2:CreateSecurityGroup", + "ec2:DeleteSecurityGroup", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:AuthorizeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:UpdateSecurityGroupRuleDescriptionsIngress", + "ec2:UpdateSecurityGroupRuleDescriptionsEgress", + "ec2:CreateVpcEndpoint", + "ec2:ModifyVpcEndpoint", + "ec2:DeleteVpcEndpoints", + "ec2:CreateLaunchTemplate", + "ec2:CreateLaunchTemplateVersion", + "ec2:ModifyLaunchTemplate", + "ec2:DeleteLaunchTemplate", + "ec2:DeleteLaunchTemplateVersions", + "ec2:CreatePlacementGroup", + "ec2:DeletePlacementGroup", + "ec2:CreateNetworkInterface", + "ec2:DeleteNetworkInterface", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:AttachNetworkInterface", + "ec2:DetachNetworkInterface", + "ec2:CreateTags", + "ec2:DeleteTags", + "ec2:RunInstances", + "ec2:CreateFleet" + ], + "Resource": "*" + }, + { + "Sid": "EC2DescribeForPCS", + "Effect": "Allow", + "Action": [ + "ec2:Describe*", + "ec2:GetSecurityGroupsForVpc" + ], + "Resource": "*" + }, + { + "Sid": "FSxLifecycle", + "Effect": "Allow", + "Action": [ + "fsx:CreateFileSystem", + "fsx:UpdateFileSystem", + "fsx:DeleteFileSystem", + "fsx:DescribeFileSystems", + "fsx:DescribeBackups", + "fsx:CreateBackup", + "fsx:DeleteBackup", + "fsx:TagResource", + "fsx:UntagResource", + "fsx:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "PCSFullAccess", + "Effect": "Allow", + "Action": [ + "pcs:*" + ], + "Resource": "*" + }, + { + "Sid": "IAMRoleAndInstanceProfileLifecycle", + "Effect": "Allow", + "Action": [ + "iam:CreateRole", + "iam:DeleteRole", + "iam:GetRole", + "iam:UpdateAssumeRolePolicy", + "iam:UpdateRole", + "iam:UpdateRoleDescription", + "iam:PutRolePolicy", + "iam:DeleteRolePolicy", + "iam:GetRolePolicy", + "iam:ListRolePolicies", + "iam:AttachRolePolicy", + "iam:DetachRolePolicy", + "iam:ListAttachedRolePolicies", + "iam:CreateInstanceProfile", + "iam:DeleteInstanceProfile", + "iam:GetInstanceProfile", + "iam:ListInstanceProfilesForRole", + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:TagRole", + "iam:UntagRole", + "iam:TagInstanceProfile", + "iam:UntagInstanceProfile" + ], + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*", + "arn:aws:iam::*:role/*ImageBuilder*", + "arn:aws:iam::*:instance-profile/*PCS*", + "arn:aws:iam::*:instance-profile/*pcs*", + "arn:aws:iam::*:instance-profile/*ImageBuilder*" + ] + }, + { + "Sid": "IAMPassRoleToPCSAndEC2", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*" + ], + "Condition": { + "StringEquals": { + "iam:PassedToService": [ + "ec2.amazonaws.com", + "pcs.amazonaws.com" + ] + } + } + }, + { + "Sid": "ServiceLinkedRolesOneTime", + "Effect": "Allow", + "Action": "iam:CreateServiceLinkedRole", + "Resource": "arn:aws:iam::*:role/aws-service-role/*", + "Condition": { + "StringLike": { + "iam:AWSServiceName": [ + "pcs.amazonaws.com", + "spot.amazonaws.com", + "spotfleet.amazonaws.com", + "fsx.amazonaws.com", + "s3.data-source.lustre.fsx.amazonaws.com", + "imagebuilder.amazonaws.com" + ] + } + } + }, + { + "Sid": "SSMParameterLifecycle", + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:GetParameter", + "ssm:GetParameters", + "ssm:GetParametersByPath", + "ssm:DeleteParameter", + "ssm:DeleteParameters", + "ssm:DescribeParameters", + "ssm:AddTagsToResource", + "ssm:RemoveTagsFromResource", + "ssm:ListTagsForResource", + "ssm:LabelParameterVersion" + ], + "Resource": [ + "arn:aws:ssm:*:*:parameter/pcs/*", + "arn:aws:ssm:*:*:parameter/aws/service/*" + ] + }, + { + "Sid": "KMSAccessForDefaultEncryption", + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey", + "kms:GenerateDataKeyWithoutPlaintext", + "kms:CreateGrant", + "kms:DescribeKey" + ], + "Resource": "*" + }, + { + "Sid": "SecretsManagerForPCS", + "Effect": "Allow", + "Action": [ + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret", + "secretsmanager:RotateSecret", + "secretsmanager:DescribeSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:TagResource", + "secretsmanager:UntagResource" + ], + "Resource": "arn:aws:secretsmanager:*:*:secret:pcs!*" + }, + { + "Sid": "CloudWatchLogsForMonitoring", + "Effect": "Allow", + "Action": [ + "logs:CreateLogGroup", + "logs:DeleteLogGroup", + "logs:DescribeLogGroups", + "logs:PutRetentionPolicy", + "logs:TagResource", + "logs:UntagResource", + "logs:CreateDelivery", + "logs:DeleteDelivery", + "logs:GetDelivery", + "logs:PutDeliverySource", + "logs:DeleteDeliverySource", + "logs:PutDeliveryDestination", + "logs:DeleteDeliveryDestination" + ], + "Resource": "*" + }, + { + "Sid": "PCSVendedLogsDelivery", + "Effect": "Allow", + "Action": [ + "pcs:AllowVendedLogDeliveryForResource" + ], + "Resource": "*" + } + ] + } + PCSClusterAdminImageBuilderPolicy: + Type: AWS::IAM::ManagedPolicy + Condition: AttachImageBuilder + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterAdmin-imagebuilder" + Description: "Image Builder permissions for the optional standalone DLAMI builder template" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "IAMPassRoleToImageBuilder", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/*ImageBuilder*", + "Condition": { + "StringEquals": { + "iam:PassedToService": "imagebuilder.amazonaws.com" + } + } + }, + { + "Sid": "ImageBuilderLifecycleOptional", + "Effect": "Allow", + "Action": [ + "imagebuilder:CreateImagePipeline", + "imagebuilder:UpdateImagePipeline", + "imagebuilder:DeleteImagePipeline", + "imagebuilder:GetImagePipeline", + "imagebuilder:ListImagePipelines", + "imagebuilder:CreateImageRecipe", + "imagebuilder:DeleteImageRecipe", + "imagebuilder:GetImageRecipe", + "imagebuilder:ListImageRecipes", + "imagebuilder:CreateComponent", + "imagebuilder:DeleteComponent", + "imagebuilder:GetComponent", + "imagebuilder:ListComponents", + "imagebuilder:CreateInfrastructureConfiguration", + "imagebuilder:UpdateInfrastructureConfiguration", + "imagebuilder:DeleteInfrastructureConfiguration", + "imagebuilder:GetInfrastructureConfiguration", + "imagebuilder:ListInfrastructureConfigurations", + "imagebuilder:CreateDistributionConfiguration", + "imagebuilder:UpdateDistributionConfiguration", + "imagebuilder:DeleteDistributionConfiguration", + "imagebuilder:GetDistributionConfiguration", + "imagebuilder:ListDistributionConfigurations", + "imagebuilder:CreateImage", + "imagebuilder:DeleteImage", + "imagebuilder:GetImage", + "imagebuilder:ListImages", + "imagebuilder:StartImagePipelineExecution", + "imagebuilder:CancelImageCreation", + "imagebuilder:CreateLifecyclePolicy", + "imagebuilder:UpdateLifecyclePolicy", + "imagebuilder:DeleteLifecyclePolicy", + "imagebuilder:GetLifecyclePolicy", + "imagebuilder:ListLifecyclePolicies", + "imagebuilder:TagResource", + "imagebuilder:UntagResource", + "imagebuilder:ListTagsForResource" + ], + "Resource": "*" + } + ] + } + PCSClusterAdminGroup: + Type: AWS::IAM::Group + Properties: + GroupName: !Ref GroupName + ManagedPolicyArns: + - !Ref PCSClusterAdminPolicy + - !If [AttachImageBuilder, !Ref PCSClusterAdminImageBuilderPolicy, !Ref "AWS::NoValue"] + GroupMembership: + Type: AWS::IAM::UserToGroupAddition + Condition: HasUsers + Properties: + GroupName: !Ref PCSClusterAdminGroup + Users: !Ref AttachUsers +Outputs: + GroupName: + Description: IAM group with cluster-admin permissions + Value: !Ref PCSClusterAdminGroup + CorePolicyArn: + Description: ARN of the core admin managed policy + Value: !Ref PCSClusterAdminPolicy + ImageBuilderPolicyArn: + Condition: AttachImageBuilder + Description: ARN of the optional Image Builder admin managed policy + Value: !Ref PCSClusterAdminImageBuilderPolicy diff --git a/architectures/aws-pcs/assets/cluster-user-iam.yaml b/architectures/aws-pcs/assets/cluster-user-iam.yaml new file mode 100644 index 000000000..b02ca519f --- /dev/null +++ b/architectures/aws-pcs/assets/cluster-user-iam.yaml @@ -0,0 +1,139 @@ +# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 +AWSTemplateFormatVersion: "2010-09-09" +Description: | + Creates a customer-managed IAM policy + group for end users of an + already-deployed AWS PCS reference cluster. Members can: + - Discover the login node (ec2:DescribeInstances) and read CFN stack outputs + - Read PCS cluster / queue / compute-node-group status (pcs:Get*/List*) + - Open SSM Session Manager / SSH-over-SSM / port-forward to LOGIN nodes + - Retrieve the Grafana admin password from Parameter Store + - Terminate their own SSM sessions + They cannot create, modify, or delete cluster resources, and cannot open + shells on compute nodes (the ssm:resourceTag Name=PCS-login* scope blocks that. Note: the Name tag is operator-mutable; if you rename login nodes, update the policy accordingly). +Metadata: + AWS::CloudFormation::Interface: + ParameterGroups: + - Label: { default: "Group + IAM users (optional)" } + Parameters: [GroupName, AttachUsers] + ParameterLabels: + GroupName: { default: "Name of the IAM group to create" } + AttachUsers: { default: "Existing IAM users to add to the group (comma-separated, optional)" } +Parameters: + GroupName: + Type: String + Default: PCSClusterUsers + Description: IAM group name for cluster end users + AttachUsers: + Type: CommaDelimitedList + Default: "" + Description: Comma-separated list of existing IAM user names to add to the group. Leave empty to skip. +Conditions: + HasUsers: !Not [!Equals [!Join ["", !Ref AttachUsers], ""]] +Resources: + PCSClusterUserPolicy: + Type: AWS::IAM::ManagedPolicy + Properties: + ManagedPolicyName: !Sub "${AWS::StackName}-PCSClusterUser" + Description: "Read-only + SSM Session Manager (login-node only) for end users of an already-deployed AWS PCS cluster" + PolicyDocument: + { + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DiscoverClusterAndLoginNode", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:DescribeTags", + "cloudformation:DescribeStacks", + "cloudformation:ListStacks", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources" + ], + "Resource": "*" + }, + { + "Sid": "PCSReadOnly", + "Effect": "Allow", + "Action": [ + "pcs:GetCluster", + "pcs:ListClusters", + "pcs:GetComputeNodeGroup", + "pcs:ListComputeNodeGroups", + "pcs:GetQueue", + "pcs:ListQueues", + "pcs:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "GrafanaPasswordRead", + "Effect": "Allow", + "Action": [ + "ssm:GetParameter", + "ssm:GetParameters" + ], + "Resource": "arn:aws:ssm:*:*:parameter/pcs/*/grafana/*" + }, + { + "Sid": "SSMSessionToLoginNode", + "Effect": "Allow", + "Action": [ + "ssm:StartSession" + ], + "Resource": [ + "arn:aws:ec2:*:*:instance/*", + "arn:aws:ssm:*:*:document/AWS-StartSSHSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost", + "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell" + ], + "Condition": { + "StringLike": { + "ssm:resourceTag/Name": "PCS-login*" + } + } + }, + { + "Sid": "SSMSessionDocumentLookup", + "Effect": "Allow", + "Action": [ + "ssm:DescribeSessions", + "ssm:GetConnectionStatus", + "ssm:DescribeInstanceProperties", + "ssm:DescribeInstanceInformation" + ], + "Resource": "*" + }, + { + "Sid": "SSMSessionTerminateOwn", + "Effect": "Allow", + "Action": [ + "ssm:TerminateSession", + "ssm:ResumeSession" + ], + "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*" + } + ] + } + PCSClusterUserGroup: + Type: AWS::IAM::Group + Properties: + GroupName: !Ref GroupName + ManagedPolicyArns: + - !Ref PCSClusterUserPolicy + GroupMembership: + Type: AWS::IAM::UserToGroupAddition + Condition: HasUsers + Properties: + GroupName: !Ref PCSClusterUserGroup + Users: !Ref AttachUsers +Outputs: + GroupName: + Description: IAM group with cluster-user (read-only + SSM-to-login) permissions + Value: !Ref PCSClusterUserGroup + PolicyArn: + Description: ARN of the user managed policy + Value: !Ref PCSClusterUserPolicy diff --git a/architectures/aws-pcs/iam/README.md b/architectures/aws-pcs/iam/README.md new file mode 100644 index 000000000..479a7ed7d --- /dev/null +++ b/architectures/aws-pcs/iam/README.md @@ -0,0 +1,329 @@ +# IAM Sample Policies + +## Overview + +The PCS reference cluster has **two distinct human principals** with very +different responsibilities: + +``` + ┌──────────────────────────┐ + admin ───► │ cluster-admin │ ──► CloudFormation, PCS, EC2 (VPC, + (deploys │ (CRUD on infra + │ SG, LT, PG, EIP, NAT, Endpoint), + the cluster) │ PCS resources) │ FSx, IAM (scoped), SSM Param, + └──────────────────────────┘ KMS, Secrets Manager, optional + Image Builder + (creates everything; can also delete) + + ┌──────────────────────────┐ + user ───► │ cluster-user │ ──► SSM StartSession to LOGIN node only + (uses an │ (read-only + SSM │ (Name=PCS-login* tag scope), port + existing │ session to login) │ forward Grafana 443→8443, read + cluster) └──────────────────────────┘ /pcs/*/grafana/admin-password, + read PCS cluster / queue status + (cannot create or modify anything) +``` + +Splitting like this means the deployer needs broad IAM but doesn't have to +share those credentials with every engineer who just wants to `srun`, and the +user role can be handed out widely (it can't accidentally delete the cluster +or open shells on compute nodes). + +## Quick start — deploy as CloudFormation + +The fastest way to provision both groups + the policies attached to them is +the two CloudFormation templates here. Each creates the customer-managed +policies, an IAM group, and (optionally) attaches existing IAM users to the +group. Click below to launch: + +| Stack | What it creates | Launch | +|---|---|---| +| **Cluster admin** | Two managed policies (`*-PCSClusterAdmin-core` always; `*-PCSClusterAdmin-imagebuilder` if opted in) + an IAM group with both attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** | One managed policy (`*-PCSClusterUser`) + an IAM group with it attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | + +> The Quick-create launch buttons above point at the production S3 bucket +> (`awsome-distributed-ai`). For an in-place sandbox test (e.g. before the +> templates are merged upstream), use the local `template-body` form: +> +> ```bash +> aws cloudformation create-stack \ +> --stack-name pcs-cluster-admins \ +> --template-body file://architectures/aws-pcs/assets/cluster-admin-iam.yaml \ +> --parameters ParameterKey=AttachUsers,ParameterValue=alice,bob \ +> ParameterKey=AttachImageBuilderPolicy,ParameterValue=true \ +> --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +> ``` + +### Files in this directory + +| File | Purpose | +|---|---| +| [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | CFN template: creates 2 managed policies + IAM group for the admin role | +| [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | CFN template: creates 1 managed policy + IAM group for the user role | +| [`cluster-admin-policy.json`](./cluster-admin-policy.json) | Raw JSON — **core** admin policy (CFN, EC2, FSx, PCS, IAM-scoped, SSM Param, KMS, Secrets, Logs); fits inside the IAM 6,144-char per-policy limit | +| [`cluster-admin-imagebuilder-policy.json`](./cluster-admin-imagebuilder-policy.json) | Raw JSON — **optional** Image Builder admin policy. Attach only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml` | +| [`cluster-user-policy.json`](./cluster-user-policy.json) | Raw JSON — end-user policy (SSM session to login node + read-only) | + +The CFN templates simply embed the JSON files as `PolicyDocument`. If you +prefer not to use CFN (e.g. CLI-only environment), the JSON files are +self-sufficient — see [Manual install without CFN](#manual-install-without-cfn). + +### Verification + +These templates were validated end-to-end on a real account (us-east-2) on +2026-06-10: + +| Check | Result | +|---|---| +| Both CFN stacks `CREATE_COMPLETE` | ✅ | +| Test admin user can `cloudformation:CreateStack` for `pcs-ml-cluster-deploy-all.yaml`, all nested stacks `CREATE_COMPLETE` (~28 min) | ✅ | +| Test admin user can `cloudformation:UpdateStack` (parameter change → `UPDATE_COMPLETE`) | ✅ | +| Test admin user can `cloudformation:DeleteStack` (full teardown) | ✅ | +| `aws iam simulate-principal-policy` on `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` (with proper `arn:aws:iam::*:role/AWSPCS-*` resource ARN) all `allowed` | ✅ | +| Test user can `pcs:GetCluster`, `cloudformation:DescribeStacks`, `ssm:GetParameter` on Grafana password | ✅ | +| Test user `ssm:StartSession` on login node (Name=`PCS-login`) `allowed` | ✅ | +| Test user `ssm:StartSession` on compute node (Name=`PCS-hpc8a`, etc.) `implicitDeny` | ✅ | +| Test user `cloudformation:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances` all `implicitDeny` | ✅ | +| CloudTrail review of admin user's deploy + update + delete cycle: 0 `AccessDenied` errors across 332+ API calls (CFN, EC2, FSx, PCS, IAM, SSM, KMS, Secrets) | ✅ | + +`aws accessanalyzer validate-policy` against all three JSON files: 0 errors. + +## Design intent + +These are **sample, slightly broader-than-strict-least-privilege** policies for +reference, derived primarily from +[the AWS-published "minimum permissions for an AWS PCS service administrator" +policy](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html) +plus the additional permissions the all-in-one template needs because it +provisions VPC + FSx + IAM roles itself (the AWS reference policy assumes those +already exist). + +Choices made: + +- **Combined CRUD, not phase-split.** A single policy covers create + update + + delete because CFN rollback on a failed Create requires Delete actions, and + Update is operationally a strict superset of Create. +- **Customer-managed, not inline; admin policy split into core + Image Builder.** + The combined admin policy is ~7.4 KB which exceeds **both** the IAM inline + per-policy limit and the customer-managed per-policy limit (both are 6,144 + characters as of January 2026). Split into a 5,769-char core file and a + 1,674-char optional Image Builder file. Most users only need the core; attach + Image Builder only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml`. +- **No `AmazonPCSFullAccess` available.** As of this writing AWS publishes + `AWSPCSComputeNodePolicy` (for compute instances, not deployers) and + `AWSPCSServiceRolePolicy` (the SLR's managed policy) but no + service-administrator managed policy. The PCS portion of the admin policy + must be customer-managed. +- **User policy scopes login-node access via the `Name` tag.** `ssm:StartSession` + is conditioned on `ssm:resourceTag/Name` matching `PCS-login*`. PCS does not + emit a dedicated "is this a login node" tag — the all-in-one templates set + `Name=PCS-login` on the login CNG and `Name=PCS-` on compute CNGs + (e.g. `PCS-cpu1`, `PCS-hpc8a`), so this is the most stable signal available. + **The `Name` tag is operator-mutable**: if you re-tag a login node, update + the policy condition to match — or fork the templates to add a dedicated + `IsLoginNode=true` tag and switch to that. +- **Production hardening points.** Tighten with resource-level conditions + (account ID, region, stack-name prefix, tag matchers) before using in + production. Validation by deploying with a sandbox account and running + CloudTrail / IAM Access Analyzer is recommended — see + [Refining via CloudTrail](#refining-via-cloudtrail) below. + +## What's NOT covered here + +- **The compute instance role** (passed to EC2 via the launch template by + `cluster.yaml`). That's a separate role provisioned by the templates + themselves, not by these policies. Use the AWS-managed + `AWSPCSComputeNodePolicy` for it. +- **The Image Builder build instance role** (used by the standalone DLAMI + template's Image Builder pipeline). Use AWS-managed + `EC2InstanceProfileForImageBuilder` / + `EC2InstanceProfileForImageBuilderECRContainerBuilds` for that. +- **Fine-grained per-cluster scoping.** Both policies use `Resource: "*"` for + many actions because resource-level scoping for VPC / EC2 networking + resources is limited and tag-based filtering would require the templates to + consistently propagate a known tag. This is a deliberate sample-grade choice. + +--- + +## `cluster-admin-iam.yaml` — what it grants + +The CFN template combines the core JSON + (optionally) the Image Builder JSON. +Sids in the JSON files: + +| Sid | Source | Purpose | +|---|---|---| +| `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | core | Create/update/delete the parent and all nested CFN stacks; ChangeSets; describe and list | +| `EC2NetworkingAndComputeLifecycle` | core | VPC / subnets / IGW / NAT / EIP / security groups / route tables / VPC endpoints / launch templates / placement groups; `RunInstances` / `CreateFleet` for PCS-internal compute launches | +| `EC2DescribeForPCS` | core | All `ec2:Describe*` (PCS' control plane and CFN both call many of these) | +| `FSxLifecycle` | core | FSx for Lustre + OpenZFS create / update / delete / describe / tag | +| `PCSFullAccess` | core | `pcs:*` — clusters, compute node groups, queues | +| `IAMRoleAndInstanceProfileLifecycle` | core | Create / delete / tag / attach for IAM roles + instance profiles whose names contain `PCS` / `pcs` / `ImageBuilder` | +| `IAMPassRoleToPCSAndEC2` | core | `iam:PassRole` scoped to `*PCS*` roles passed to `ec2.amazonaws.com` / `pcs.amazonaws.com` | +| `ServiceLinkedRolesOneTime` | core | `iam:CreateServiceLinkedRole` for PCS / Spot / FSx / Image Builder (no-op once the SLR exists) | +| `SSMParameterLifecycle` | core | Put / get / delete on `parameter/pcs/*` (Grafana admin password, etc.) and `parameter/aws/service/*` (PCS-Ready DLAMI auto-resolve) | +| `KMSAccessForDefaultEncryption` | core | KMS data-plane operations for default-encrypted EBS / FSx volumes (verbatim from AWS PCS minimum-permissions policy) | +| `SecretsManagerForPCS` | core | `secretsmanager:*` scoped to `secret:pcs!*` (PCS internally creates a Slurm cluster auth secret) | +| `CloudWatchLogsForMonitoring` + `PCSVendedLogsDelivery` | core | CloudWatch Logs delivery wiring for PCS-vended Slurm logs | +| `IAMPassRoleToImageBuilder` | imagebuilder | `iam:PassRole` scoped to `*ImageBuilder*` roles passed to `imagebuilder.amazonaws.com` | +| `ImageBuilderLifecycleOptional` | imagebuilder | EC2 Image Builder pipeline / recipe / component / distribution / lifecycle policy | + +**Where it came from.** The `PCSFullAccess` + `EC2DescribeForPCS` + +`IAMPassRoleToPCSAndEC2` + `ServiceLinkedRolesOneTime` + +`KMSAccessForDefaultEncryption` + `SecretsManagerForPCS` blocks are derived +directly from +[`security-min-permissions.html`](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html). +The remaining blocks (CloudFormation, EC2 networking lifecycle, FSx, IAM role +lifecycle, SSM Parameter Store, Image Builder, CloudWatch Logs) are layered on +because the all-in-one template provisions VPC + FSx + IAM roles itself. + +**A combined CRUD policy is intentional, not a mistake.** Splitting per phase +(deploy-only / update-only / delete-only) breaks in three places: (1) CFN +rollback on Create requires Delete actions, (2) Update is a strict superset +of Deploy (any UpdateStack may replace resources), (3) Drift detection during +Update calls Describe across every service. The natural alternative is +**operator policy = the CRUD JSON above** vs **auditor policy = the same +actions reduced to `*:Describe*` / `*:Get*` / `*:List*`**. + +### Pairing with AWS-managed policies + +If you want a smaller customer-managed surface, you can attach AWS-managed +policies for parts of the stack and trim the corresponding Sids: + +| AWS-managed | Replaces / overlaps with | Notes | +|---|---|---| +| `AWSCloudFormationFullAccess` | `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | Broader (no resource-level scoping) but battle-tested | +| `AmazonFSxFullAccess` | `FSxLifecycle` | Closest fit; covers everything the templates need | +| `AWSImageBuilderFullAccess` | `ImageBuilderLifecycleOptional` | Only needed when the standalone DLAMI template is deployed | +| `AmazonEC2FullAccess` | `EC2NetworkingAndComputeLifecycle` + `EC2DescribeForPCS` | **Materially overprivileged** (e.g. EBS public-share). Prefer the customer-managed Sids | + +There is **no** `AmazonPCSFullAccess` AWS-managed policy as of this writing — +the PCS portion has to be customer-managed. + +--- + +## `cluster-user-iam.yaml` — what it grants + +Sids: + +| Sid | Purpose | +|---|---| +| `DiscoverClusterAndLoginNode` | `ec2:DescribeInstances` + `cloudformation:DescribeStacks` so the user can find the login node and read stack outputs (e.g. `ClusterId`) | +| `PCSReadOnly` | `pcs:Get*` / `pcs:List*` so the user can see cluster / queue / compute-node-group status | +| `GrafanaPasswordRead` | `ssm:GetParameter` scoped to `parameter/pcs/*/grafana/*` (the admin password generated by `cluster.yaml` when `DeployMonitoring=true`) | +| `SSMSessionToLoginNode` | `ssm:StartSession` scoped to instances whose `Name` tag matches `PCS-login*`, with the SSM documents needed for shell, SSH-over-SSM, and port-forwarding | +| `SSMSessionDocumentLookup` | `ssm:DescribeSessions` / `DescribeInstanceInformation` etc. (the SSM client / Session Manager UI calls these to render the connection list) | +| `SSMSessionTerminateOwn` | `ssm:TerminateSession` / `ResumeSession` scoped to sessions whose ID prefix matches `${aws:username}-*` (= sessions the user themselves opened) | + +The `SSMSessionToLoginNode` Sid uses **`ssm:resourceTag/Name`** to scope sessions +to login nodes only. The all-in-one templates set `Name=PCS-login` on the +login node group and `Name=PCS-` on compute CNGs (e.g. +`PCS-cpu1`, `PCS-hpc8a`, `PCS-gpu-p5`), so this works out of the box. **Tag +caveat**: the `Name` tag is operator-mutable; if you rename a login node, +update the policy condition to match. + +### Typical user workflows the policy enables + +1. **Connect to the login node:** + ```bash + INSTANCE_ID=$(aws ec2 describe-instances \ + --filters "Name=tag:Name,Values=PCS-login" \ + "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].InstanceId' --output text) + aws ssm start-session --target $INSTANCE_ID + ``` + +2. **Open Grafana via SSM port-forward (443 → 8443):** + ```bash + aws ssm start-session --target $INSTANCE_ID \ + --document-name AWS-StartPortForwardingSession \ + --parameters '{"portNumber":["443"],"localPortNumber":["8443"]}' + # then open https://localhost:8443/grafana/ + ``` + +3. **Retrieve the Grafana admin password** (run this once, paste into the Grafana + login): + ```bash + CLUSTER_ID= + aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/grafana/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text + ``` + +4. **SSH-over-SSM** (with `~/.ssh/config` configured): + ```bash + ssh hpc8a-login + ``` + +### Safety properties + +- **No write access to cluster resources.** No `CreateStack`, no `pcs:Create*`, + no `ec2:RunInstances`, no `iam:*`, no `fsx:*` write actions. +- **Login nodes only.** The `Name` tag scope on `ssm:StartSession` blocks the + user from opening shells on compute nodes (which are tagged + `Name=PCS-`, e.g. `PCS-cpu1` or `PCS-gpu-p5`). +- **Own sessions only.** `ssm:TerminateSession` is scoped to sessions whose ID + matches the user's IAM username, so users can't kill each other's sessions. + +--- + +## Manual install without CFN + +If you'd rather skip CloudFormation, attach the JSON files directly: + +```bash +# Cluster admin (split into core + optional Image Builder) +aws iam create-policy \ + --policy-name PCSClusterAdminCore \ + --policy-document file://architectures/aws-pcs/iam/cluster-admin-policy.json +aws iam create-policy \ + --policy-name PCSClusterAdminImageBuilder \ + --policy-document file://architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterAdminCore +# attach the Image Builder one only when needed: +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterAdminImageBuilder + +# Cluster user +aws iam create-policy \ + --policy-name PCSClusterUser \ + --policy-document file://architectures/aws-pcs/iam/cluster-user-policy.json +aws iam attach-user-policy \ + --user-name \ + --policy-arn arn:aws:iam:::policy/PCSClusterUser +``` + +For a CFN deployment role pattern, attach to a role instead of a user. + +--- + +## Refining via CloudTrail + +These JSON files are sample-grade. The recommended refinement path: + +1. **Deploy a representative cluster in a sandbox AWS account** with admin + permissions on the deploying principal so nothing fails for spurious IAM + reasons. +2. **Wait for the deploy to succeed**, then for `aws cloudformation delete-stack` + to finish, so CloudTrail captures the full lifecycle. +3. **Generate a least-privilege policy from CloudTrail history** with IAM Access + Analyzer: + + ```bash + aws accessanalyzer start-policy-generation \ + --policy-generation-details principalArn= \ + --cloud-trail-details accessRole=,startTime=,trails=[{cloudTrailArn=}] + # poll until JobStatus=SUCCEEDED + aws accessanalyzer get-generated-policy --job-id \ + --include-resource-placeholders --query 'GeneratedPolicyResult.GeneratedPolicies[0].Policy' \ + --output text > generated-policy.json + ``` + +4. **Diff against the JSON files in this directory.** The Access Analyzer + output tends to be *narrower* than the sample (it only lists actions actually + called) but with `Resource: "*"` even when narrower scoping is possible — + the sample's resource ARNs are usually keepers. + +The same approach works for `cluster-user-policy.json` — exercise the user +workflows (start a session, port-forward Grafana, terminate the session) in a +sandbox with broad SSM perms, then narrow to what was actually called. diff --git a/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json b/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json new file mode 100644 index 000000000..ef97344c7 --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json @@ -0,0 +1,60 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "IAMPassRoleToImageBuilder", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": "arn:aws:iam::*:role/*ImageBuilder*", + "Condition": { + "StringEquals": { + "iam:PassedToService": "imagebuilder.amazonaws.com" + } + } + }, + { + "Sid": "ImageBuilderLifecycleOptional", + "Effect": "Allow", + "Action": [ + "imagebuilder:CreateImagePipeline", + "imagebuilder:UpdateImagePipeline", + "imagebuilder:DeleteImagePipeline", + "imagebuilder:GetImagePipeline", + "imagebuilder:ListImagePipelines", + "imagebuilder:CreateImageRecipe", + "imagebuilder:DeleteImageRecipe", + "imagebuilder:GetImageRecipe", + "imagebuilder:ListImageRecipes", + "imagebuilder:CreateComponent", + "imagebuilder:DeleteComponent", + "imagebuilder:GetComponent", + "imagebuilder:ListComponents", + "imagebuilder:CreateInfrastructureConfiguration", + "imagebuilder:UpdateInfrastructureConfiguration", + "imagebuilder:DeleteInfrastructureConfiguration", + "imagebuilder:GetInfrastructureConfiguration", + "imagebuilder:ListInfrastructureConfigurations", + "imagebuilder:CreateDistributionConfiguration", + "imagebuilder:UpdateDistributionConfiguration", + "imagebuilder:DeleteDistributionConfiguration", + "imagebuilder:GetDistributionConfiguration", + "imagebuilder:ListDistributionConfigurations", + "imagebuilder:CreateImage", + "imagebuilder:DeleteImage", + "imagebuilder:GetImage", + "imagebuilder:ListImages", + "imagebuilder:StartImagePipelineExecution", + "imagebuilder:CancelImageCreation", + "imagebuilder:CreateLifecyclePolicy", + "imagebuilder:UpdateLifecyclePolicy", + "imagebuilder:DeleteLifecyclePolicy", + "imagebuilder:GetLifecyclePolicy", + "imagebuilder:ListLifecyclePolicies", + "imagebuilder:TagResource", + "imagebuilder:UntagResource", + "imagebuilder:ListTagsForResource" + ], + "Resource": "*" + } + ] +} diff --git a/architectures/aws-pcs/iam/cluster-admin-policy.json b/architectures/aws-pcs/iam/cluster-admin-policy.json new file mode 100644 index 000000000..bb317b48f --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-admin-policy.json @@ -0,0 +1,282 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "CloudFormationStackLifecycle", + "Effect": "Allow", + "Action": [ + "cloudformation:CreateStack", + "cloudformation:UpdateStack", + "cloudformation:DeleteStack", + "cloudformation:CancelUpdateStack", + "cloudformation:ContinueUpdateRollback", + "cloudformation:CreateChangeSet", + "cloudformation:ExecuteChangeSet", + "cloudformation:DescribeChangeSet", + "cloudformation:DeleteChangeSet", + "cloudformation:DescribeStacks", + "cloudformation:DescribeStackEvents", + "cloudformation:DescribeStackResource", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources", + "cloudformation:GetTemplate", + "cloudformation:GetTemplateSummary", + "cloudformation:TagResource", + "cloudformation:UntagResource" + ], + "Resource": "*" + }, + { + "Sid": "CloudFormationDescribeStackless", + "Effect": "Allow", + "Action": [ + "cloudformation:ValidateTemplate", + "cloudformation:ListStacks", + "cloudformation:ListExports", + "cloudformation:ListImports" + ], + "Resource": "*" + }, + { + "Sid": "EC2NetworkingAndComputeLifecycle", + "Effect": "Allow", + "Action": [ + "ec2:CreateVpc", + "ec2:DeleteVpc", + "ec2:ModifyVpcAttribute", + "ec2:AssociateVpcCidrBlock", + "ec2:DisassociateVpcCidrBlock", + "ec2:CreateSubnet", + "ec2:DeleteSubnet", + "ec2:ModifySubnetAttribute", + "ec2:CreateInternetGateway", + "ec2:DeleteInternetGateway", + "ec2:AttachInternetGateway", + "ec2:DetachInternetGateway", + "ec2:CreateNatGateway", + "ec2:DeleteNatGateway", + "ec2:AllocateAddress", + "ec2:ReleaseAddress", + "ec2:AssociateAddress", + "ec2:DisassociateAddress", + "ec2:CreateRouteTable", + "ec2:DeleteRouteTable", + "ec2:CreateRoute", + "ec2:DeleteRoute", + "ec2:ReplaceRoute", + "ec2:AssociateRouteTable", + "ec2:DisassociateRouteTable", + "ec2:CreateSecurityGroup", + "ec2:DeleteSecurityGroup", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:AuthorizeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:UpdateSecurityGroupRuleDescriptionsIngress", + "ec2:UpdateSecurityGroupRuleDescriptionsEgress", + "ec2:CreateVpcEndpoint", + "ec2:ModifyVpcEndpoint", + "ec2:DeleteVpcEndpoints", + "ec2:CreateLaunchTemplate", + "ec2:CreateLaunchTemplateVersion", + "ec2:ModifyLaunchTemplate", + "ec2:DeleteLaunchTemplate", + "ec2:DeleteLaunchTemplateVersions", + "ec2:CreatePlacementGroup", + "ec2:DeletePlacementGroup", + "ec2:CreateNetworkInterface", + "ec2:DeleteNetworkInterface", + "ec2:ModifyNetworkInterfaceAttribute", + "ec2:AttachNetworkInterface", + "ec2:DetachNetworkInterface", + "ec2:CreateTags", + "ec2:DeleteTags", + "ec2:RunInstances", + "ec2:CreateFleet" + ], + "Resource": "*" + }, + { + "Sid": "EC2DescribeForPCS", + "Effect": "Allow", + "Action": [ + "ec2:Describe*", + "ec2:GetSecurityGroupsForVpc" + ], + "Resource": "*" + }, + { + "Sid": "FSxLifecycle", + "Effect": "Allow", + "Action": [ + "fsx:CreateFileSystem", + "fsx:UpdateFileSystem", + "fsx:DeleteFileSystem", + "fsx:DescribeFileSystems", + "fsx:DescribeBackups", + "fsx:CreateBackup", + "fsx:DeleteBackup", + "fsx:TagResource", + "fsx:UntagResource", + "fsx:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "PCSFullAccess", + "Effect": "Allow", + "Action": [ + "pcs:*" + ], + "Resource": "*" + }, + { + "Sid": "IAMRoleAndInstanceProfileLifecycle", + "Effect": "Allow", + "Action": [ + "iam:CreateRole", + "iam:DeleteRole", + "iam:GetRole", + "iam:UpdateAssumeRolePolicy", + "iam:UpdateRole", + "iam:UpdateRoleDescription", + "iam:PutRolePolicy", + "iam:DeleteRolePolicy", + "iam:GetRolePolicy", + "iam:ListRolePolicies", + "iam:AttachRolePolicy", + "iam:DetachRolePolicy", + "iam:ListAttachedRolePolicies", + "iam:CreateInstanceProfile", + "iam:DeleteInstanceProfile", + "iam:GetInstanceProfile", + "iam:ListInstanceProfilesForRole", + "iam:AddRoleToInstanceProfile", + "iam:RemoveRoleFromInstanceProfile", + "iam:TagRole", + "iam:UntagRole", + "iam:TagInstanceProfile", + "iam:UntagInstanceProfile" + ], + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*", + "arn:aws:iam::*:role/*ImageBuilder*", + "arn:aws:iam::*:instance-profile/*PCS*", + "arn:aws:iam::*:instance-profile/*pcs*", + "arn:aws:iam::*:instance-profile/*ImageBuilder*" + ] + }, + { + "Sid": "IAMPassRoleToPCSAndEC2", + "Effect": "Allow", + "Action": "iam:PassRole", + "Resource": [ + "arn:aws:iam::*:role/*PCS*", + "arn:aws:iam::*:role/*pcs*" + ], + "Condition": { + "StringEquals": { + "iam:PassedToService": [ + "ec2.amazonaws.com", + "pcs.amazonaws.com" + ] + } + } + }, + { + "Sid": "ServiceLinkedRolesOneTime", + "Effect": "Allow", + "Action": "iam:CreateServiceLinkedRole", + "Resource": "arn:aws:iam::*:role/aws-service-role/*", + "Condition": { + "StringLike": { + "iam:AWSServiceName": [ + "pcs.amazonaws.com", + "spot.amazonaws.com", + "spotfleet.amazonaws.com", + "fsx.amazonaws.com", + "s3.data-source.lustre.fsx.amazonaws.com", + "imagebuilder.amazonaws.com" + ] + } + } + }, + { + "Sid": "SSMParameterLifecycle", + "Effect": "Allow", + "Action": [ + "ssm:PutParameter", + "ssm:GetParameter", + "ssm:GetParameters", + "ssm:GetParametersByPath", + "ssm:DeleteParameter", + "ssm:DeleteParameters", + "ssm:DescribeParameters", + "ssm:AddTagsToResource", + "ssm:RemoveTagsFromResource", + "ssm:ListTagsForResource", + "ssm:LabelParameterVersion" + ], + "Resource": [ + "arn:aws:ssm:*:*:parameter/pcs/*", + "arn:aws:ssm:*:*:parameter/aws/service/*" + ] + }, + { + "Sid": "KMSAccessForDefaultEncryption", + "Effect": "Allow", + "Action": [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey", + "kms:GenerateDataKeyWithoutPlaintext", + "kms:CreateGrant", + "kms:DescribeKey" + ], + "Resource": "*" + }, + { + "Sid": "SecretsManagerForPCS", + "Effect": "Allow", + "Action": [ + "secretsmanager:CreateSecret", + "secretsmanager:UpdateSecret", + "secretsmanager:RotateSecret", + "secretsmanager:DescribeSecret", + "secretsmanager:DeleteSecret", + "secretsmanager:TagResource", + "secretsmanager:UntagResource" + ], + "Resource": "arn:aws:secretsmanager:*:*:secret:pcs!*" + }, + { + "Sid": "CloudWatchLogsForMonitoring", + "Effect": "Allow", + "Action": [ + "logs:CreateLogGroup", + "logs:DeleteLogGroup", + "logs:DescribeLogGroups", + "logs:PutRetentionPolicy", + "logs:TagResource", + "logs:UntagResource", + "logs:CreateDelivery", + "logs:DeleteDelivery", + "logs:GetDelivery", + "logs:PutDeliverySource", + "logs:DeleteDeliverySource", + "logs:PutDeliveryDestination", + "logs:DeleteDeliveryDestination" + ], + "Resource": "*" + }, + { + "Sid": "PCSVendedLogsDelivery", + "Effect": "Allow", + "Action": [ + "pcs:AllowVendedLogDeliveryForResource" + ], + "Resource": "*" + } + ] +} diff --git a/architectures/aws-pcs/iam/cluster-user-policy.json b/architectures/aws-pcs/iam/cluster-user-policy.json new file mode 100644 index 000000000..98b446394 --- /dev/null +++ b/architectures/aws-pcs/iam/cluster-user-policy.json @@ -0,0 +1,81 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DiscoverClusterAndLoginNode", + "Effect": "Allow", + "Action": [ + "ec2:DescribeInstances", + "ec2:DescribeInstanceStatus", + "ec2:DescribeTags", + "cloudformation:DescribeStacks", + "cloudformation:ListStacks", + "cloudformation:DescribeStackResources", + "cloudformation:ListStackResources" + ], + "Resource": "*" + }, + { + "Sid": "PCSReadOnly", + "Effect": "Allow", + "Action": [ + "pcs:GetCluster", + "pcs:ListClusters", + "pcs:GetComputeNodeGroup", + "pcs:ListComputeNodeGroups", + "pcs:GetQueue", + "pcs:ListQueues", + "pcs:ListTagsForResource" + ], + "Resource": "*" + }, + { + "Sid": "GrafanaPasswordRead", + "Effect": "Allow", + "Action": [ + "ssm:GetParameter", + "ssm:GetParameters" + ], + "Resource": "arn:aws:ssm:*:*:parameter/pcs/*/grafana/*" + }, + { + "Sid": "SSMSessionToLoginNode", + "Effect": "Allow", + "Action": [ + "ssm:StartSession" + ], + "Resource": [ + "arn:aws:ec2:*:*:instance/*", + "arn:aws:ssm:*:*:document/AWS-StartSSHSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession", + "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost", + "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell" + ], + "Condition": { + "StringLike": { + "ssm:resourceTag/Name": "PCS-login*" + } + } + }, + { + "Sid": "SSMSessionDocumentLookup", + "Effect": "Allow", + "Action": [ + "ssm:DescribeSessions", + "ssm:GetConnectionStatus", + "ssm:DescribeInstanceProperties", + "ssm:DescribeInstanceInformation" + ], + "Resource": "*" + }, + { + "Sid": "SSMSessionTerminateOwn", + "Effect": "Allow", + "Action": [ + "ssm:TerminateSession", + "ssm:ResumeSession" + ], + "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*" + } + ] +} From b5bd5636976b498057e38bc44ffed8e82c0d1bcf Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 19:50:34 +0000 Subject: [PATCH 39/89] fix: install sssd-tools so deleted/modified LDAP users propagate (sss_cache) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User deletion (and any LDAP modify) was not visible via getent until the SSSD cache entry's TTL expired — sss_cache wasn't available to invalidate it (sssd-tools wasn't installed). Added sssd-tools to the client apt install, and documented the `sudo sss_cache -E` step (login + compute) in USER-MANAGEMENT.md's "Deleting a user" section. Verified on v8: after sss_cache -E a deleted user disappears from getent immediately while other users are unaffected. --- .../aws-pcs/assets/scripts/setup-directory.sh | 5 ++++- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 11 +++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/architectures/aws-pcs/assets/scripts/setup-directory.sh b/architectures/aws-pcs/assets/scripts/setup-directory.sh index 44b9b54f8..f7850a72c 100755 --- a/architectures/aws-pcs/assets/scripts/setup-directory.sh +++ b/architectures/aws-pcs/assets/scripts/setup-directory.sh @@ -258,7 +258,10 @@ setup_client_internal() { apt_get update -qq echo "[directory-client] Installing SSSD..." - apt_get install -y sssd libpam-sss libnss-sss ldap-utils + # sssd-tools provides sss_cache, needed to invalidate the SSSD cache after a + # user is deleted/modified in LDAP (otherwise the change is not visible until + # the cache entry's TTL expires — see USER-MANAGEMENT.md "Deleting a user"). + apt_get install -y sssd sssd-tools libpam-sss libnss-sss ldap-utils echo "[directory-client] Configuring SSSD (server: ${server_uri})..." cat > /etc/sssd/sssd.conf < -n bash -c 'sudo sss_cache -E' +``` + Also remove from Slurm accounting: ```bash sacctmgr -i remove user alice From d64e690ed8685e382f8b845de11e2e9f9df5437e Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Tue, 16 Jun 2026 19:57:59 +0000 Subject: [PATCH 40/89] tests: record major-update e2e validation matrix (run on real hardware) --- architectures/aws-pcs/tests/README.md | 29 +++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 23c8dbadb..a2027f06e 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -32,6 +32,35 @@ one cluster. Tests 8-13 are separate paths run only when their inputs change. --- +## Major-update PR — configurations run end-to-end on real hardware + +The major-update PR (IAM policies, multi-user OpenLDAP, `MonitoringStack` rename, +`SSHAccessCidr`/`GrafanaAccessCidr`, multi-AZ subnets, VPCName auto-derive + +EFA auto-count) was validated end-to-end in us-east-2 with a single +`deploy-all` cluster: + +``` +PrimarySubnetAZ=us-east-2b AdditionalSubnetAZ2=us-east-2a AdditionalSubnetAZ3=us-east-2c +DirectoryService=OpenLDAP-LoginNode SSHAccessCidr=/32 +MonitoringStack=Prometheus-LoginNode +``` + +| Feature | Verified | Result | +|---|---|---| +| **Multi-AZ subnets** | 3 private subnets, 3 distinct AZs, non-overlapping `/18` CIDRs (`10.1.0.0/18`, `10.1.64.0/18`, `10.1.128.0/18`); compute nodes actually scheduled into the additional-AZ subnets (`10.1.x`) | ✅ | +| **SSHAccessCidr** | `LoginAccessSecurityGroup` opens **port 22 only** (443 absent since `GrafanaAccessCidr` empty) from the given CIDR, attached to the login node only; direct `ssh` to the login public IP succeeds | ✅ | +| **MonitoringStack=Prometheus-LoginNode** | 6 login containers up (prometheus/grafana/nginx/cloudwatch-exporter/node-exporter/pushgateway) | ✅ | +| **OpenLDAP server (boot-time, automatic)** | `slapd` active, DB on `/home/ldap-db`, OUs + `clusterusers` gid 3000 created, admin password in SSM; `ldap-add-user` helper auto-installed to `/usr/local/bin`; apt/dpkg-lock wait survived first-boot unattended-upgrades | ✅ | +| **`directory-role` tag discovery** | login tagged `directory-role=server` (independent of `monitoring-role`); compute clients discover the server by that tag, scoped to `pcs-cluster-id` | ✅ | +| **Compute SSSD client (boot-time, automatic)** | fresh compute nodes auto-install SSSD + sssd-tools, resolve `testuser1` via the login LDAP — no manual step | ✅ | +| **Slurm job as LDAP user** | `srun` as `testuser1` runs with `uid=10001` on the compute node | ✅ | +| **Multi-node UID consistency** | two different compute nodes both resolve `testuser1`→`uid=10001` | ✅ | +| **User delete propagation** | `ldapdelete` + `sss_cache -E` removes the user from `getent` immediately (sssd-tools now installed) | ✅ | +| **Template lint** | `validate-template` passes on all 8 edited `assets/*.yaml` (incl. 3 GPU templates + 2 IAM stacks) | ✅ | +| **Slurm managed accounting (Test 12)** | requires a separate `ManagedAccounting=enabled` cluster — run when validating accounting; LDAP-user↔Slurm visibility itself is covered by the "Slurm job as LDAP user" row above | ⏳ run on accounting-enabled cluster | + +--- + ## Canonical assets reused (not duplicated) | Workload | Source in this repo | PCS-specific delta | From 647c8102e19873ea5e70f1dbb7bfc7dec5f0e7cf Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 01:39:09 +0000 Subject: [PATCH 41/89] docs: fix USER-MANAGEMENT accuracy (verified against live accounting cluster) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Caught while validating Test 12 on a ManagedAccounting=enabled cluster: - sacctmgr add/modify/remove must run as root (the accounting Administrator); ubuntu gets "Only admins/operators/coordinators can add accounts". Documented sudo + full-path usage; read-only sacct/sreport/show still work as ubuntu. - Quick-reference table referenced non-existent helpers (ldap-list-users, ldap-delete-user, ldap-reset-password) and the bare name ldap-add-user. Only ldap-add-user.sh exists — fixed the table to use it (with .sh) and show raw ldapsearch/ldapdelete/ldappasswd for list/delete/reset. - Noted that LDAP_ADMIN_PASSWORD must be passed inline to sudo (sudo VAR=... cmd); sudo -E alone drops it under env_reset/secure_path. --- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 63 ++++++++++++------- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index 03303ef7a..e35252466 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -12,19 +12,24 @@ POSIX user accounts visible on all nodes via SSSD. ## Quick reference (common tasks) -| Task | Command (run on login node as root) | +Run on the login node. `$ADMIN_PW` is the LDAP admin password from SSM (see +[Getting the admin password](#getting-the-admin-password)); the `LDAP_*` env +vars must be passed **inline to `sudo`** (`sudo VAR=... cmd`) — `sudo -E` alone +drops them under the default `env_reset`/`secure_path`. + +| Task | Command (run on the login node) | |---|---| -| Add a user | `sudo ldap-add-user alice 10001 3000` | -| List all users | `sudo ldap-list-users` | -| Delete a user | `sudo ldap-delete-user alice` | -| Reset a user's password | `sudo ldap-reset-password alice` | -| Add user to Slurm accounting | `sacctmgr -i add user alice Account=default` | +| Add a user | `sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user.sh alice 10001 3000` | +| List all users | `ldapsearch -x -H ldap://localhost -b ou=People,dc=cluster,dc=internal uid` | +| Delete a user | `ldapdelete -x -H ldap://localhost -D cn=admin,dc=cluster,dc=internal -w "$ADMIN_PW" uid=alice,ou=People,dc=cluster,dc=internal` then `sudo sss_cache -E` | +| Reset a user's password | `ldappasswd -x -H ldap://localhost -D cn=admin,dc=cluster,dc=internal -w "$ADMIN_PW" -s NEWPASS uid=alice,ou=People,dc=cluster,dc=internal` | +| Add user to Slurm accounting | `sudo /opt/aws/pcs/scheduler/slurm-25.11/bin/sacctmgr -i add user alice Account=ml-team` (root = accounting admin) | | Verify user on compute node | `srun -N1 -n1 -p cpu1 id alice` | -The `ldap-*` commands above are helper scripts installed on the login node at -`/usr/local/bin/`. They wrap raw `ldapadd`/`ldapdelete`/`ldappasswd` commands -so you don't need to know LDAP syntax. The full manual commands are documented -below for reference. +`ldap-add-user.sh` is a helper script installed on the login node at +`/usr/local/bin/` (it wraps `ldapadd`+`ldappasswd` so you don't need the LDAP +syntax). List/delete/reset use the raw `ldap*` tools directly — the full +commands are documented in each section below. --- @@ -188,8 +193,8 @@ operations. **Using the helper script** (recommended): ```bash -# Usage: ldap-add-user [ssh-public-key] -sudo LDAP_ADMIN_PASSWORD="" ldap-add-user alice 10001 3000 +# Usage: ldap-add-user.sh [ssh-public-key] +sudo LDAP_ADMIN_PASSWORD="" ldap-add-user.sh alice 10001 3000 ``` This creates the user with: @@ -203,7 +208,7 @@ This creates the user with: **With an SSH key** (user can log in immediately): ```bash -sudo LDAP_ADMIN_PASSWORD="" ldap-add-user alice 10001 3000 "ssh-rsa AAAA... alice@laptop" +sudo LDAP_ADMIN_PASSWORD="" ldap-add-user.sh alice 10001 3000 "ssh-rsa AAAA... alice@laptop" ``` **Verifying the user was created:** @@ -231,7 +236,7 @@ carol 10003 3000 Then: ```bash while read name uid gid key; do - sudo LDAP_ADMIN_PASSWORD="" ldap-add-user "$name" "$uid" "$gid" "$key" + sudo LDAP_ADMIN_PASSWORD="" ldap-add-user.sh "$name" "$uid" "$gid" "$key" done < users.txt ``` @@ -273,7 +278,7 @@ srun -N -n bash -c 'sudo sss_cache -E' Also remove from Slurm accounting: ```bash -sacctmgr -i remove user alice +sudo /opt/aws/pcs/scheduler/slurm-25.11/bin/sacctmgr -i remove user alice ``` The user's home directory (`/home/alice`) is NOT deleted automatically. @@ -344,23 +349,33 @@ EOF ## Slurm accounting -PCS manages the Slurm accounting database internally. You just need to register -users and accounts: +PCS manages the Slurm accounting database internally (enable it with +`ManagedAccounting=enabled` at deploy time). You just need to register users and +accounts. + +> **Run `sacctmgr` add/modify/remove as `root`.** In PCS managed accounting the +> Administrator is `root` — the default `ubuntu` user is not an accounting admin, +> so `sacctmgr -i add ...` as `ubuntu` fails with *"Only +> admins/operators/coordinators can add accounts"*. Use `sudo` with the full +> path (the Slurm bin dir isn't on root's `PATH` by default): ```bash -export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH +SACCTMGR=/opt/aws/pcs/scheduler/slurm-25.11/bin/sacctmgr # Create a Slurm account (typically one per team or project) -sacctmgr -i add account default Description="Default account" +sudo $SACCTMGR -i add account ml-team Description="ML Team" -# Add users to the account -sacctmgr -i add user alice Account=default -sacctmgr -i add user bob Account=default +# Add LDAP users to the account +sudo $SACCTMGR -i add user alice Account=ml-team +sudo $SACCTMGR -i add user bob Account=ml-team -# Verify -sacctmgr show user +# Verify (read-only — works as any user with the Slurm bin on PATH) +sudo $SACCTMGR show user alice bob format=User,Account,DefaultAccount ``` +Read-only `sacct` / `sreport` / `sacctmgr show ...` work as `ubuntu`; only the +mutating `sacctmgr` verbs need `root`. + > **Note:** if `AccountingPolicyEnforcement=none` (the default), users can > submit jobs even without being registered in `sacctmgr`. Registration is > needed for fairshare/priority and for `sacct` history to show the user name. From 5a7ca7b5d1fc22773ad1485bcd60c180b33a3bf5 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 01:44:57 +0000 Subject: [PATCH 42/89] tests: accounting + multi-user (Test 12) verified end-to-end on real hardware --- architectures/aws-pcs/tests/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index a2027f06e..d6d131b32 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -57,7 +57,7 @@ MonitoringStack=Prometheus-LoginNode | **Multi-node UID consistency** | two different compute nodes both resolve `testuser1`→`uid=10001` | ✅ | | **User delete propagation** | `ldapdelete` + `sss_cache -E` removes the user from `getent` immediately (sssd-tools now installed) | ✅ | | **Template lint** | `validate-template` passes on all 8 edited `assets/*.yaml` (incl. 3 GPU templates + 2 IAM stacks) | ✅ | -| **Slurm managed accounting (Test 12)** | requires a separate `ManagedAccounting=enabled` cluster — run when validating accounting; LDAP-user↔Slurm visibility itself is covered by the "Slurm job as LDAP user" row above | ⏳ run on accounting-enabled cluster | +| **Slurm managed accounting + multi-user (Test 12)** | on a `ManagedAccounting=enabled` cluster: LDAP users alice/bob registered in `sacctmgr` (account=ml-team) as root admin; jobs submitted as each LDAP user complete and `sacct -a` records them under the correct `User`+`Account` (`scontrol show job`: `UserId=alice(10001) Account=ml-team`) | ✅ | --- From cd498a90084923be7a423f89c24518575043d409 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 01:50:40 +0000 Subject: [PATCH 43/89] docs: reorder Advanced Features by priority, consolidate IAM into docs/IAM.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README §8 Advanced Features reordered by priority: 8.1 Monitoring (dashboards) 8.2 User Management 8.3 IAM Permissions 8.4 Pre-baking AMI (was: Monitoring, Pre-baking, User Management, IAM) Fixed the monitoring "dashboard" heading hierarchy: "Accessing Grafana" and its Option A/B were at ### / #### (siblings of ### 8.1), so they broke out of the section. Demoted to #### / ##### so they nest under 8.1. Renamed to "Accessing the Grafana dashboards" and fixed all anchor links. IAM docs consolidated: - New docs/IAM.md — leads with the two roles (admin/user) and how to deploy their policy stacks, then security considerations, then the verification matrix. - Deleted iam/ directory: the raw *.json policies were redundant (the CFN templates embed the policy inline as PolicyDocument), and iam/README.md is superseded by docs/IAM.md. IAM YAMLs already live in assets/ (S3-sync covered). - README §8.3 rewritten to lead with the roles and link docs/IAM.md; template diagram + Additional Resources updated (iam/ → assets/, added IAM guide link). Re-pointed the pre-baking anchor (#82 → #84) everywhere after the reorder. --- architectures/aws-pcs/README.md | 72 ++-- architectures/aws-pcs/docs/IAM.md | 164 +++++++++ architectures/aws-pcs/docs/PARAMETERS.md | 4 +- architectures/aws-pcs/iam/README.md | 329 ------------------ .../cluster-admin-imagebuilder-policy.json | 60 ---- .../aws-pcs/iam/cluster-admin-policy.json | 282 --------------- .../aws-pcs/iam/cluster-user-policy.json | 81 ----- 7 files changed, 203 insertions(+), 789 deletions(-) create mode 100644 architectures/aws-pcs/docs/IAM.md delete mode 100644 architectures/aws-pcs/iam/README.md delete mode 100644 architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json delete mode 100644 architectures/aws-pcs/iam/cluster-admin-policy.json delete mode 100644 architectures/aws-pcs/iam/cluster-user-policy.json diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index eebd69f9d..e4f989e98 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -18,7 +18,7 @@ This repository provides reference architectures and deployment templates for se > Slurm 25.05/25.11 pre-installed), so no custom AMI build is required by default — > the cluster comes up without an Image Builder step. For frequent scaling, you can > pre-bake Enroot/Pyxis into a custom DLAMI with the standalone -> [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#82-pre-baking-enrootpyxis-into-a-custom-ami) +> [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) > template and pass the result as `AmiId`. ## 2. Architecture @@ -62,7 +62,7 @@ Add a GPU queue and tune storage/monitoring via the parameters below. Once it's up: - **Connect** to the login node via SSM Session Manager — see [Accessing the Cluster](#6-accessing-the-cluster). -- **Open the Grafana dashboards** (deployed by default) via SSM port forwarding — see [Accessing Grafana](#accessing-grafana). +- **Open the Grafana dashboards** (deployed by default) via SSM port forwarding — see [Accessing the Grafana dashboards](#accessing-the-grafana-dashboards). - **Want to reach Grafana directly in a browser** (no port forwarding)? Set `GrafanaAccessCidr` to a trusted CIDR at deploy time — see [Option B — Direct public access](#option-b--direct-public-access-opt-in-via-grafanaaccesscidr). Prefer step-by-step instructions? See the [AI/ML for AWS PCS Workshop](https://catalog.workshops.aws/ml-on-pcs/). @@ -89,7 +89,7 @@ only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | -| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#82-pre-baking-enrootpyxis-into-a-custom-ami) | +| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | | `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `DeployOnDemandCNG` | `true` | Deploy the `cpu1` CPU queue (`OnDemandInstanceType`, default `c6i.4xlarge`) | | `OnDemandEnableEfa` | `false` | Set `true` for HPC/MPI workloads on EFA-capable CPU types (hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandenableefa) | @@ -109,7 +109,7 @@ choices that need the most thought. needed. Enroot 3.5.0 + Pyxis 0.20.0 are layered on at first boot via [`assets/scripts/install-enroot-pyxis.sh`](./assets/scripts/install-enroot-pyxis.sh) (~8–12 min boot). For **frequent scaling**, pre-bake Enroot/Pyxis into a custom DLAMI -once with [§8.2](#82-pre-baking-enrootpyxis-into-a-custom-ami) and pass that +once with [§8.4](#84-pre-baking-enrootpyxis-into-a-custom-ami) and pass that `ami-xxx` as `AmiId` (~3 min boot, deterministic state). The post-install hook is idempotent — it no-ops on a pre-baked AMI. @@ -336,7 +336,7 @@ is installed automatically: Metrics cover Slurm jobs, GPU (utilization/memory/temperature/power/ECC/NVLink via DCGM), node CPU/memory/disk/network, and CloudWatch (EC2/FSx/PCS). The stack installs on node-local `/opt` (not the shared `/home`). Pre-built Grafana dashboards are -provisioned automatically — see [Accessing Grafana](#accessing-grafana) for the list +provisioned automatically — see [Accessing the Grafana dashboards](#accessing-the-grafana-dashboards) for the list and a screenshot. > **GPU metrics work out of the box across the supported GPU range** (Hopper / B200 / @@ -360,7 +360,7 @@ and a screenshot. > Node type is identified by the `monitoring-role` tag (`login`/`compute`), not the EC2 > `Name` tag — the `Name` tag defaults to `PCS-` and is free for you to retag. -### Accessing Grafana +#### Accessing the Grafana dashboards Log in to Grafana as **`admin`**; the password is generated per cluster and stored in SSM Parameter Store. Retrieve it (with `CLUSTER_ID` from the stack's `ClusterId` output): @@ -372,7 +372,7 @@ aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/grafana/admin-password" \ There are two ways to reach the UI. -#### Option A — SSM port forwarding (default, private) +##### Option A — SSM port forwarding (default, private) No public access required; works even when the login node has no inbound rules. @@ -391,7 +391,7 @@ aws ssm start-session --target $INSTANCE_ID \ Then open `https://localhost:8443/grafana/`. -#### Option B — Direct public access (opt-in, via `GrafanaAccessCidr`) +##### Option B — Direct public access (opt-in, via `GrafanaAccessCidr`) To browse Grafana directly without port forwarding, set **`GrafanaAccessCidr`** at deploy time to a CIDR you trust (e.g. your office IP `203.0.113.4/32`). deploy-all then @@ -444,7 +444,32 @@ NCCL, FSDP), see the [Test & Validation Guide](tests/README.md). --- -### 8.2 Pre-baking Enroot/Pyxis into a custom AMI +### 8.2 User Management + +By default, the cluster runs as a single `ubuntu` user. For **multi-user +clusters** (per-user Slurm accounting, isolated home directories, team-based +access control), set `DirectoryService=OpenLDAP-LoginNode` at deploy time. + +This installs an OpenLDAP directory on the login node with SSSD on all compute +nodes — users you add are immediately visible cluster-wide. See the +**[User Management Guide](./docs/USER-MANAGEMENT.md)** for step-by-step +operations (adding/removing users, Slurm accounting, SSH access). + +### 8.3 IAM Permissions + +The cluster has two human roles, each with a ready-to-deploy IAM policy stack: + +- **Cluster admin** ([`cluster-admin-iam.yaml`](./assets/cluster-admin-iam.yaml)) — + deploy / update / delete the cluster (CloudFormation, PCS, EC2, FSx, scoped IAM). +- **Cluster user** ([`cluster-user-iam.yaml`](./assets/cluster-user-iam.yaml)) — + SSM session to the **login node only** + read-only status; cannot create, + modify, or delete anything. + +Each template creates the customer-managed policies and an IAM group, and can +attach existing users at deploy time. See the **[IAM Permissions Guide](./docs/IAM.md)** +for roles, deploy instructions, security considerations, and the verification matrix. + +### 8.4 Pre-baking Enroot/Pyxis into a custom AMI The all-in-one template installs Enroot/Pyxis at **first boot** via `PostInstallScriptUrl`, which is fast to deploy and avoids an Image Builder step. @@ -507,30 +532,6 @@ For production deploys that pin the AMI explicitly per cluster, none of these ar --- -### 8.3 User Management - -By default, the cluster runs as a single `ubuntu` user. For **multi-user -clusters** (per-user Slurm accounting, isolated home directories, team-based -access control), set `DirectoryService=OpenLDAP-LoginNode` at deploy time. - -This installs an OpenLDAP directory on the login node with SSSD on all compute -nodes — users you add are immediately visible cluster-wide. See the -**[User Management Guide](./docs/USER-MANAGEMENT.md)** for step-by-step -operations (adding/removing users, Slurm accounting, SSH access). - -### 8.4 IAM Permissions - -Two sample IAM policy stacks are provided for least-privilege access: - -- **Cluster admin** (`cluster-admin-iam.yaml`) — deploy / update / delete the cluster -- **Cluster user** (`cluster-user-iam.yaml`) — SSM session to login node + read-only - -Deploy with one click from the [iam/ README](./iam/README.md), or attach the -raw JSON policies manually. See [iam/README.md](./iam/README.md) for the full -verification matrix and design rationale. - ---- - ## 9. Templates All templates live in [`assets/`](./assets/). `pcs-ml-cluster-deploy-all.yaml` nests @@ -607,8 +608,8 @@ External references (runtime): Standalone (not nested): pcs-ready-dlami-with-enroot-pyxis.yaml ← AMI builder (separate stack) - iam/cluster-admin-iam.yaml ← IAM policies (separate stack) - iam/cluster-user-iam.yaml ← IAM policies (separate stack) + cluster-admin-iam.yaml ← IAM admin policies + group (separate stack) + cluster-user-iam.yaml ← IAM user policy + group (separate stack) ``` --- @@ -630,6 +631,7 @@ In this repo: - [Parameter reference](./docs/PARAMETERS.md) — every deploy-all parameter and default - [Operations guide](./docs/OPERATIONS.md) — version trade-offs, AMI pinning, monitoring/DCGM, FSx coupling, Lustre tuning, production settings - [User management guide](./docs/USER-MANAGEMENT.md) — multi-user setup with OpenLDAP (add/remove users, groups, Slurm accounting) +- [IAM permissions guide](./docs/IAM.md) — cluster admin / cluster user roles, policy deploy, security considerations - [Deploy & testing procedures](./docs/DEPLOY-TESTING.md) — development deploy workflow with test S3 bucket - [Test & Validation Guide](./tests/README.md) — reproducible matrix with measured numbers - [GPU Cluster Health Check](../../4.validation_and_observability/2.gpu-cluster-healthcheck) — comprehensive GPU/EFA/NVLink validation suite (lightweight + intensive modes, Slurm prolog integration) diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md new file mode 100644 index 000000000..060f2e699 --- /dev/null +++ b/architectures/aws-pcs/docs/IAM.md @@ -0,0 +1,164 @@ +# IAM Permissions Guide + +The cluster distinguishes **two human roles** with very different +responsibilities, and ships a ready-to-deploy IAM policy stack for each: + +| Role | Who | What they can do | Template | +|---|---|---|---| +| **Cluster admin** | The person who deploys/updates/deletes the cluster | Full CRUD on the infrastructure: CloudFormation, PCS, EC2 (VPC/SG/launch templates/placement groups/NAT/EIP), FSx, scoped IAM, SSM Parameter Store, KMS, Secrets Manager, and (optionally) Image Builder | [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | +| **Cluster user** | Engineers who just run jobs on an existing cluster | SSM session **to the login node only**, port-forward Grafana, read the Grafana password, read PCS cluster/queue status. **Cannot create, modify, or delete anything**, and cannot open shells on compute nodes | [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | + +Splitting the roles means the deployer's broad permissions never have to be +handed to every engineer who just wants to `srun`, and the user role can be +given out widely — it can't accidentally delete the cluster or get a shell on a +compute node. + +--- + +## Deploying the policies + +Each template creates the customer-managed IAM policies, an IAM group with them +attached, and (optionally) adds existing IAM users to that group. Deploy both +as CloudFormation stacks: + +| Stack | Creates | 1-click | +|---|---|---| +| **Cluster admin** | `-PCSClusterAdmin-core` (always) + `-PCSClusterAdmin-imagebuilder` (if opted in) managed policies + an IAM group | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** | `-PCSClusterUser` managed policy + an IAM group | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | + +Or from the CLI (use `--template-body` against a local checkout for a pre-merge +sandbox test): + +```bash +# Admin: create the policies + group, attach existing users, include Image Builder perms +aws cloudformation create-stack \ + --stack-name pcs-cluster-admins \ + --template-body file://architectures/aws-pcs/assets/cluster-admin-iam.yaml \ + --parameters ParameterKey=AttachUsers,ParameterValue=alice,bob \ + ParameterKey=AttachImageBuilderPolicy,ParameterValue=true \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM + +# User: create the policy + group, attach existing users +aws cloudformation create-stack \ + --stack-name pcs-cluster-users \ + --template-body file://architectures/aws-pcs/assets/cluster-user-iam.yaml \ + --parameters ParameterKey=AttachUsers,ParameterValue=carol,dave \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` + +Both templates take an `AttachUsers` parameter (comma-separated existing IAM +user names) so you can wire up group membership at deploy time, or leave it +empty and add users to the group later. The admin template's +`AttachImageBuilderPolicy` defaults to `false`; set it `true` only when the +admin will also deploy the standalone DLAMI builder +(`pcs-ready-dlami-with-enroot-pyxis.yaml`). + +### What the cluster user can do once attached + +```bash +# Find the login node and open a session +INSTANCE_ID=$(aws ec2 describe-instances \ + --filters "Name=tag:Name,Values=PCS-login" "Name=instance-state-name,Values=running" \ + --query 'Reservations[0].Instances[0].InstanceId' --output text) +aws ssm start-session --target $INSTANCE_ID + +# Port-forward Grafana (443 -> 8443), then open https://localhost:8443/grafana/ +aws ssm start-session --target $INSTANCE_ID \ + --document-name AWS-StartPortForwardingSession \ + --parameters '{"portNumber":["443"],"localPortNumber":["8443"]}' + +# Read the Grafana admin password +aws ssm get-parameter --name "/pcs//grafana/admin-password" \ + --with-decryption --query 'Parameter.Value' --output text +``` + +--- + +## Considerations + +These are **sample, slightly-broader-than-strict-least-privilege** policies, +derived from the AWS-published +[minimum permissions for an AWS PCS service administrator](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html) +plus the extra permissions the all-in-one template needs because it provisions +VPC + FSx + IAM roles itself (the AWS reference policy assumes those already +exist). Review and tighten before production use. + +**Login-node access is scoped by the `Name` tag.** The user policy conditions +`ssm:StartSession` on `ssm:resourceTag/Name` matching `PCS-login*`. PCS does not +emit a dedicated "is this a login node" tag, so the templates set +`Name=PCS-login` on the login node and `Name=PCS-` on compute nodes +(`PCS-cpu1`, `PCS-hpc8a`, …) — the most stable signal available. **The `Name` +tag is operator-mutable**: if you re-tag a login node, update the policy +condition to match (or fork the templates to add a dedicated `IsLoginNode=true` +tag and key off that). + +**Combined CRUD is intentional, not a mistake.** The admin policy covers +create + update + delete in one policy because (1) CFN rollback on a failed +Create requires Delete actions, (2) UpdateStack is operationally a superset of +Create (it may replace resources), and (3) drift detection during Update calls +Describe across every service. If you want a read-only variant, reduce the same +actions to `*:Describe*` / `*:Get*` / `*:List*` for an auditor role. + +**The admin policy is split into core + Image Builder** because the combined +document (~7.4 KB) exceeds the IAM 6,144-character per-policy limit. The +~5.8 KB core covers a normal deploy; the ~1.7 KB Image Builder add-on is only +needed for the standalone DLAMI builder. The CFN template attaches both to the +group when `AttachImageBuilderPolicy=true`. + +**There is no `AmazonPCSFullAccess` managed policy** as of January 2026 — AWS +publishes only `AWSPCSComputeNodePolicy` (for compute instances) and +`AWSPCSServiceRolePolicy` (the service-linked role). The PCS portion of the +admin policy must therefore be customer-managed. + +**Pairing with AWS-managed policies.** For a smaller customer-managed surface +you can attach AWS-managed policies for parts of the stack and trim the matching +statements: `AWSCloudFormationFullAccess`, `AmazonFSxFullAccess`, +`AWSImageBuilderFullAccess` are reasonable fits. Avoid `AmazonEC2FullAccess` — +it is materially overprivileged (e.g. EBS public-share); prefer the +customer-managed EC2 statements in the template. + +### Not covered by these policies + +- **The compute instance role** (passed to EC2 by `cluster.yaml`) — provisioned + by the templates themselves; use the AWS-managed `AWSPCSComputeNodePolicy`. +- **The Image Builder build instance role** — use the AWS-managed + `EC2InstanceProfileForImageBuilder` / + `EC2InstanceProfileForImageBuilderECRContainerBuilds`. +- **Fine-grained per-cluster scoping** — both policies use `Resource: "*"` for + many EC2/VPC actions because resource-level scoping there is limited. This is + a deliberate sample-grade choice. + +### Refining to least-privilege via CloudTrail + +To generate a tighter policy from real usage: + +1. Deploy a representative cluster in a sandbox account with broad permissions on + the deploying principal (so nothing fails for spurious IAM reasons). +2. Let the full lifecycle run — deploy, then `delete-stack` — so CloudTrail + captures every API call. +3. Generate a policy from CloudTrail with IAM Access Analyzer + (`aws accessanalyzer start-policy-generation` → `get-generated-policy`), then + diff against the template's statements. Access Analyzer output is usually + *narrower* on actions but leaves `Resource: "*"`; the template's resource ARNs + are usually the keepers. + +The same approach works for the user policy — exercise the user workflows +(start a session, port-forward Grafana, terminate it) in a sandbox, then narrow. + +--- + +## Verification + +These policies were validated end-to-end on a real account (us-east-2): + +| Check | Result | +|---|---| +| Both IAM CFN stacks `CREATE_COMPLETE` | ✅ | +| Admin user deploys `pcs-ml-cluster-deploy-all.yaml` end-to-end (all nested stacks `CREATE_COMPLETE`, ~28 min) | ✅ | +| Admin user `UpdateStack` (param change → `UPDATE_COMPLETE`) and `DeleteStack` (full teardown) | ✅ | +| `iam:simulate-principal-policy` on `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` → all `allowed` | ✅ | +| User can `pcs:GetCluster`, `cloudformation:DescribeStacks`, read the Grafana password | ✅ | +| User `ssm:StartSession` on the login node (`Name=PCS-login`) `allowed`; on compute nodes `implicitDeny` | ✅ | +| User `cloudformation:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances` all `implicitDeny` | ✅ | +| CloudTrail over a full admin deploy+update+delete cycle: 0 `AccessDenied` across 332+ API calls | ✅ | +| `aws accessanalyzer validate-policy` on every embedded policy: 0 errors | ✅ | diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index c30baaff2..770a5579a 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -7,7 +7,7 @@ the latest PCS-Ready Deep Learning AMI auto-resolved from SSM, Enroot/Pyxis inst at first boot via `PostInstallScriptUrl`, monitoring enabled — so a default deploy only needs the Availability Zone (`PrimarySubnetAZ`). To pre-bake Enroot/Pyxis into a custom AMI for faster boots, build it separately with -[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#82-pre-baking-enrootpyxis-into-a-custom-ami) +[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#84-pre-baking-enrootpyxis-into-a-custom-ami) and pass its output as `AmiId`. For conceptual guidance (GPU instance/EFA selection, FSx Region availability, container @@ -30,7 +30,7 @@ runtime options), see the [README](../README.md#4-configuration). | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`). Drives which monitoring you get (Slurm OpenMetrics is 25.11+ only) and is also threaded into the CNG UserData so the right-version Pyxis is installed; see [OPERATIONS.md §1](./OPERATIONS.md#1-slurm-version-selection) | | `LoginNodeInstanceType` | `m6i.4xlarge` | Login node instance type | | `RootVolumeSize` | `300` | Root EBS volume size (GiB) on every node (login + compute); 300 leaves room for large container images (Megatron `.sqsh` ~20 GB) | -| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#82-pre-baking-enrootpyxis-into-a-custom-ami)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | +| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#84-pre-baking-enrootpyxis-into-a-custom-ami)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | | `SSHAccessCidr` | *(empty)* | When set to a CIDR, opens SSH/22 on the login node to that CIDR via a login-only security group (attached to the login node only, never compute). Empty (default) = SSH over SSM only. Set to your office/VPN range for direct `ssh`/`scp`/VS Code Remote (common for multi-user clusters) | | `ManagedAccounting` | `disabled` | Enable Slurm managed accounting (requires Slurm 24.11+) | | `AccountingPolicyEnforcement` | `none` | Slurm accounting policy enforcement (`none` or `associations,limits,safe`) | diff --git a/architectures/aws-pcs/iam/README.md b/architectures/aws-pcs/iam/README.md deleted file mode 100644 index 479a7ed7d..000000000 --- a/architectures/aws-pcs/iam/README.md +++ /dev/null @@ -1,329 +0,0 @@ -# IAM Sample Policies - -## Overview - -The PCS reference cluster has **two distinct human principals** with very -different responsibilities: - -``` - ┌──────────────────────────┐ - admin ───► │ cluster-admin │ ──► CloudFormation, PCS, EC2 (VPC, - (deploys │ (CRUD on infra + │ SG, LT, PG, EIP, NAT, Endpoint), - the cluster) │ PCS resources) │ FSx, IAM (scoped), SSM Param, - └──────────────────────────┘ KMS, Secrets Manager, optional - Image Builder - (creates everything; can also delete) - - ┌──────────────────────────┐ - user ───► │ cluster-user │ ──► SSM StartSession to LOGIN node only - (uses an │ (read-only + SSM │ (Name=PCS-login* tag scope), port - existing │ session to login) │ forward Grafana 443→8443, read - cluster) └──────────────────────────┘ /pcs/*/grafana/admin-password, - read PCS cluster / queue status - (cannot create or modify anything) -``` - -Splitting like this means the deployer needs broad IAM but doesn't have to -share those credentials with every engineer who just wants to `srun`, and the -user role can be handed out widely (it can't accidentally delete the cluster -or open shells on compute nodes). - -## Quick start — deploy as CloudFormation - -The fastest way to provision both groups + the policies attached to them is -the two CloudFormation templates here. Each creates the customer-managed -policies, an IAM group, and (optionally) attaches existing IAM users to the -group. Click below to launch: - -| Stack | What it creates | Launch | -|---|---|---| -| **Cluster admin** | Two managed policies (`*-PCSClusterAdmin-core` always; `*-PCSClusterAdmin-imagebuilder` if opted in) + an IAM group with both attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | -| **Cluster user** | One managed policy (`*-PCSClusterUser`) + an IAM group with it attached. Optional: add existing IAM users to the group. | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | - -> The Quick-create launch buttons above point at the production S3 bucket -> (`awsome-distributed-ai`). For an in-place sandbox test (e.g. before the -> templates are merged upstream), use the local `template-body` form: -> -> ```bash -> aws cloudformation create-stack \ -> --stack-name pcs-cluster-admins \ -> --template-body file://architectures/aws-pcs/assets/cluster-admin-iam.yaml \ -> --parameters ParameterKey=AttachUsers,ParameterValue=alice,bob \ -> ParameterKey=AttachImageBuilderPolicy,ParameterValue=true \ -> --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM -> ``` - -### Files in this directory - -| File | Purpose | -|---|---| -| [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | CFN template: creates 2 managed policies + IAM group for the admin role | -| [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | CFN template: creates 1 managed policy + IAM group for the user role | -| [`cluster-admin-policy.json`](./cluster-admin-policy.json) | Raw JSON — **core** admin policy (CFN, EC2, FSx, PCS, IAM-scoped, SSM Param, KMS, Secrets, Logs); fits inside the IAM 6,144-char per-policy limit | -| [`cluster-admin-imagebuilder-policy.json`](./cluster-admin-imagebuilder-policy.json) | Raw JSON — **optional** Image Builder admin policy. Attach only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml` | -| [`cluster-user-policy.json`](./cluster-user-policy.json) | Raw JSON — end-user policy (SSM session to login node + read-only) | - -The CFN templates simply embed the JSON files as `PolicyDocument`. If you -prefer not to use CFN (e.g. CLI-only environment), the JSON files are -self-sufficient — see [Manual install without CFN](#manual-install-without-cfn). - -### Verification - -These templates were validated end-to-end on a real account (us-east-2) on -2026-06-10: - -| Check | Result | -|---|---| -| Both CFN stacks `CREATE_COMPLETE` | ✅ | -| Test admin user can `cloudformation:CreateStack` for `pcs-ml-cluster-deploy-all.yaml`, all nested stacks `CREATE_COMPLETE` (~28 min) | ✅ | -| Test admin user can `cloudformation:UpdateStack` (parameter change → `UPDATE_COMPLETE`) | ✅ | -| Test admin user can `cloudformation:DeleteStack` (full teardown) | ✅ | -| `aws iam simulate-principal-policy` on `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` (with proper `arn:aws:iam::*:role/AWSPCS-*` resource ARN) all `allowed` | ✅ | -| Test user can `pcs:GetCluster`, `cloudformation:DescribeStacks`, `ssm:GetParameter` on Grafana password | ✅ | -| Test user `ssm:StartSession` on login node (Name=`PCS-login`) `allowed` | ✅ | -| Test user `ssm:StartSession` on compute node (Name=`PCS-hpc8a`, etc.) `implicitDeny` | ✅ | -| Test user `cloudformation:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances` all `implicitDeny` | ✅ | -| CloudTrail review of admin user's deploy + update + delete cycle: 0 `AccessDenied` errors across 332+ API calls (CFN, EC2, FSx, PCS, IAM, SSM, KMS, Secrets) | ✅ | - -`aws accessanalyzer validate-policy` against all three JSON files: 0 errors. - -## Design intent - -These are **sample, slightly broader-than-strict-least-privilege** policies for -reference, derived primarily from -[the AWS-published "minimum permissions for an AWS PCS service administrator" -policy](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html) -plus the additional permissions the all-in-one template needs because it -provisions VPC + FSx + IAM roles itself (the AWS reference policy assumes those -already exist). - -Choices made: - -- **Combined CRUD, not phase-split.** A single policy covers create + update + - delete because CFN rollback on a failed Create requires Delete actions, and - Update is operationally a strict superset of Create. -- **Customer-managed, not inline; admin policy split into core + Image Builder.** - The combined admin policy is ~7.4 KB which exceeds **both** the IAM inline - per-policy limit and the customer-managed per-policy limit (both are 6,144 - characters as of January 2026). Split into a 5,769-char core file and a - 1,674-char optional Image Builder file. Most users only need the core; attach - Image Builder only when deploying `pcs-ready-dlami-with-enroot-pyxis.yaml`. -- **No `AmazonPCSFullAccess` available.** As of this writing AWS publishes - `AWSPCSComputeNodePolicy` (for compute instances, not deployers) and - `AWSPCSServiceRolePolicy` (the SLR's managed policy) but no - service-administrator managed policy. The PCS portion of the admin policy - must be customer-managed. -- **User policy scopes login-node access via the `Name` tag.** `ssm:StartSession` - is conditioned on `ssm:resourceTag/Name` matching `PCS-login*`. PCS does not - emit a dedicated "is this a login node" tag — the all-in-one templates set - `Name=PCS-login` on the login CNG and `Name=PCS-` on compute CNGs - (e.g. `PCS-cpu1`, `PCS-hpc8a`), so this is the most stable signal available. - **The `Name` tag is operator-mutable**: if you re-tag a login node, update - the policy condition to match — or fork the templates to add a dedicated - `IsLoginNode=true` tag and switch to that. -- **Production hardening points.** Tighten with resource-level conditions - (account ID, region, stack-name prefix, tag matchers) before using in - production. Validation by deploying with a sandbox account and running - CloudTrail / IAM Access Analyzer is recommended — see - [Refining via CloudTrail](#refining-via-cloudtrail) below. - -## What's NOT covered here - -- **The compute instance role** (passed to EC2 via the launch template by - `cluster.yaml`). That's a separate role provisioned by the templates - themselves, not by these policies. Use the AWS-managed - `AWSPCSComputeNodePolicy` for it. -- **The Image Builder build instance role** (used by the standalone DLAMI - template's Image Builder pipeline). Use AWS-managed - `EC2InstanceProfileForImageBuilder` / - `EC2InstanceProfileForImageBuilderECRContainerBuilds` for that. -- **Fine-grained per-cluster scoping.** Both policies use `Resource: "*"` for - many actions because resource-level scoping for VPC / EC2 networking - resources is limited and tag-based filtering would require the templates to - consistently propagate a known tag. This is a deliberate sample-grade choice. - ---- - -## `cluster-admin-iam.yaml` — what it grants - -The CFN template combines the core JSON + (optionally) the Image Builder JSON. -Sids in the JSON files: - -| Sid | Source | Purpose | -|---|---|---| -| `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | core | Create/update/delete the parent and all nested CFN stacks; ChangeSets; describe and list | -| `EC2NetworkingAndComputeLifecycle` | core | VPC / subnets / IGW / NAT / EIP / security groups / route tables / VPC endpoints / launch templates / placement groups; `RunInstances` / `CreateFleet` for PCS-internal compute launches | -| `EC2DescribeForPCS` | core | All `ec2:Describe*` (PCS' control plane and CFN both call many of these) | -| `FSxLifecycle` | core | FSx for Lustre + OpenZFS create / update / delete / describe / tag | -| `PCSFullAccess` | core | `pcs:*` — clusters, compute node groups, queues | -| `IAMRoleAndInstanceProfileLifecycle` | core | Create / delete / tag / attach for IAM roles + instance profiles whose names contain `PCS` / `pcs` / `ImageBuilder` | -| `IAMPassRoleToPCSAndEC2` | core | `iam:PassRole` scoped to `*PCS*` roles passed to `ec2.amazonaws.com` / `pcs.amazonaws.com` | -| `ServiceLinkedRolesOneTime` | core | `iam:CreateServiceLinkedRole` for PCS / Spot / FSx / Image Builder (no-op once the SLR exists) | -| `SSMParameterLifecycle` | core | Put / get / delete on `parameter/pcs/*` (Grafana admin password, etc.) and `parameter/aws/service/*` (PCS-Ready DLAMI auto-resolve) | -| `KMSAccessForDefaultEncryption` | core | KMS data-plane operations for default-encrypted EBS / FSx volumes (verbatim from AWS PCS minimum-permissions policy) | -| `SecretsManagerForPCS` | core | `secretsmanager:*` scoped to `secret:pcs!*` (PCS internally creates a Slurm cluster auth secret) | -| `CloudWatchLogsForMonitoring` + `PCSVendedLogsDelivery` | core | CloudWatch Logs delivery wiring for PCS-vended Slurm logs | -| `IAMPassRoleToImageBuilder` | imagebuilder | `iam:PassRole` scoped to `*ImageBuilder*` roles passed to `imagebuilder.amazonaws.com` | -| `ImageBuilderLifecycleOptional` | imagebuilder | EC2 Image Builder pipeline / recipe / component / distribution / lifecycle policy | - -**Where it came from.** The `PCSFullAccess` + `EC2DescribeForPCS` + -`IAMPassRoleToPCSAndEC2` + `ServiceLinkedRolesOneTime` + -`KMSAccessForDefaultEncryption` + `SecretsManagerForPCS` blocks are derived -directly from -[`security-min-permissions.html`](https://docs.aws.amazon.com/pcs/latest/userguide/security-min-permissions.html). -The remaining blocks (CloudFormation, EC2 networking lifecycle, FSx, IAM role -lifecycle, SSM Parameter Store, Image Builder, CloudWatch Logs) are layered on -because the all-in-one template provisions VPC + FSx + IAM roles itself. - -**A combined CRUD policy is intentional, not a mistake.** Splitting per phase -(deploy-only / update-only / delete-only) breaks in three places: (1) CFN -rollback on Create requires Delete actions, (2) Update is a strict superset -of Deploy (any UpdateStack may replace resources), (3) Drift detection during -Update calls Describe across every service. The natural alternative is -**operator policy = the CRUD JSON above** vs **auditor policy = the same -actions reduced to `*:Describe*` / `*:Get*` / `*:List*`**. - -### Pairing with AWS-managed policies - -If you want a smaller customer-managed surface, you can attach AWS-managed -policies for parts of the stack and trim the corresponding Sids: - -| AWS-managed | Replaces / overlaps with | Notes | -|---|---|---| -| `AWSCloudFormationFullAccess` | `CloudFormationStackLifecycle` + `CloudFormationDescribeStackless` | Broader (no resource-level scoping) but battle-tested | -| `AmazonFSxFullAccess` | `FSxLifecycle` | Closest fit; covers everything the templates need | -| `AWSImageBuilderFullAccess` | `ImageBuilderLifecycleOptional` | Only needed when the standalone DLAMI template is deployed | -| `AmazonEC2FullAccess` | `EC2NetworkingAndComputeLifecycle` + `EC2DescribeForPCS` | **Materially overprivileged** (e.g. EBS public-share). Prefer the customer-managed Sids | - -There is **no** `AmazonPCSFullAccess` AWS-managed policy as of this writing — -the PCS portion has to be customer-managed. - ---- - -## `cluster-user-iam.yaml` — what it grants - -Sids: - -| Sid | Purpose | -|---|---| -| `DiscoverClusterAndLoginNode` | `ec2:DescribeInstances` + `cloudformation:DescribeStacks` so the user can find the login node and read stack outputs (e.g. `ClusterId`) | -| `PCSReadOnly` | `pcs:Get*` / `pcs:List*` so the user can see cluster / queue / compute-node-group status | -| `GrafanaPasswordRead` | `ssm:GetParameter` scoped to `parameter/pcs/*/grafana/*` (the admin password generated by `cluster.yaml` when `DeployMonitoring=true`) | -| `SSMSessionToLoginNode` | `ssm:StartSession` scoped to instances whose `Name` tag matches `PCS-login*`, with the SSM documents needed for shell, SSH-over-SSM, and port-forwarding | -| `SSMSessionDocumentLookup` | `ssm:DescribeSessions` / `DescribeInstanceInformation` etc. (the SSM client / Session Manager UI calls these to render the connection list) | -| `SSMSessionTerminateOwn` | `ssm:TerminateSession` / `ResumeSession` scoped to sessions whose ID prefix matches `${aws:username}-*` (= sessions the user themselves opened) | - -The `SSMSessionToLoginNode` Sid uses **`ssm:resourceTag/Name`** to scope sessions -to login nodes only. The all-in-one templates set `Name=PCS-login` on the -login node group and `Name=PCS-` on compute CNGs (e.g. -`PCS-cpu1`, `PCS-hpc8a`, `PCS-gpu-p5`), so this works out of the box. **Tag -caveat**: the `Name` tag is operator-mutable; if you rename a login node, -update the policy condition to match. - -### Typical user workflows the policy enables - -1. **Connect to the login node:** - ```bash - INSTANCE_ID=$(aws ec2 describe-instances \ - --filters "Name=tag:Name,Values=PCS-login" \ - "Name=instance-state-name,Values=running" \ - --query 'Reservations[0].Instances[0].InstanceId' --output text) - aws ssm start-session --target $INSTANCE_ID - ``` - -2. **Open Grafana via SSM port-forward (443 → 8443):** - ```bash - aws ssm start-session --target $INSTANCE_ID \ - --document-name AWS-StartPortForwardingSession \ - --parameters '{"portNumber":["443"],"localPortNumber":["8443"]}' - # then open https://localhost:8443/grafana/ - ``` - -3. **Retrieve the Grafana admin password** (run this once, paste into the Grafana - login): - ```bash - CLUSTER_ID= - aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/grafana/admin-password" \ - --with-decryption --query 'Parameter.Value' --output text - ``` - -4. **SSH-over-SSM** (with `~/.ssh/config` configured): - ```bash - ssh hpc8a-login - ``` - -### Safety properties - -- **No write access to cluster resources.** No `CreateStack`, no `pcs:Create*`, - no `ec2:RunInstances`, no `iam:*`, no `fsx:*` write actions. -- **Login nodes only.** The `Name` tag scope on `ssm:StartSession` blocks the - user from opening shells on compute nodes (which are tagged - `Name=PCS-`, e.g. `PCS-cpu1` or `PCS-gpu-p5`). -- **Own sessions only.** `ssm:TerminateSession` is scoped to sessions whose ID - matches the user's IAM username, so users can't kill each other's sessions. - ---- - -## Manual install without CFN - -If you'd rather skip CloudFormation, attach the JSON files directly: - -```bash -# Cluster admin (split into core + optional Image Builder) -aws iam create-policy \ - --policy-name PCSClusterAdminCore \ - --policy-document file://architectures/aws-pcs/iam/cluster-admin-policy.json -aws iam create-policy \ - --policy-name PCSClusterAdminImageBuilder \ - --policy-document file://architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json -aws iam attach-user-policy \ - --user-name \ - --policy-arn arn:aws:iam:::policy/PCSClusterAdminCore -# attach the Image Builder one only when needed: -aws iam attach-user-policy \ - --user-name \ - --policy-arn arn:aws:iam:::policy/PCSClusterAdminImageBuilder - -# Cluster user -aws iam create-policy \ - --policy-name PCSClusterUser \ - --policy-document file://architectures/aws-pcs/iam/cluster-user-policy.json -aws iam attach-user-policy \ - --user-name \ - --policy-arn arn:aws:iam:::policy/PCSClusterUser -``` - -For a CFN deployment role pattern, attach to a role instead of a user. - ---- - -## Refining via CloudTrail - -These JSON files are sample-grade. The recommended refinement path: - -1. **Deploy a representative cluster in a sandbox AWS account** with admin - permissions on the deploying principal so nothing fails for spurious IAM - reasons. -2. **Wait for the deploy to succeed**, then for `aws cloudformation delete-stack` - to finish, so CloudTrail captures the full lifecycle. -3. **Generate a least-privilege policy from CloudTrail history** with IAM Access - Analyzer: - - ```bash - aws accessanalyzer start-policy-generation \ - --policy-generation-details principalArn= \ - --cloud-trail-details accessRole=,startTime=,trails=[{cloudTrailArn=}] - # poll until JobStatus=SUCCEEDED - aws accessanalyzer get-generated-policy --job-id \ - --include-resource-placeholders --query 'GeneratedPolicyResult.GeneratedPolicies[0].Policy' \ - --output text > generated-policy.json - ``` - -4. **Diff against the JSON files in this directory.** The Access Analyzer - output tends to be *narrower* than the sample (it only lists actions actually - called) but with `Resource: "*"` even when narrower scoping is possible — - the sample's resource ARNs are usually keepers. - -The same approach works for `cluster-user-policy.json` — exercise the user -workflows (start a session, port-forward Grafana, terminate the session) in a -sandbox with broad SSM perms, then narrow to what was actually called. diff --git a/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json b/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json deleted file mode 100644 index ef97344c7..000000000 --- a/architectures/aws-pcs/iam/cluster-admin-imagebuilder-policy.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "IAMPassRoleToImageBuilder", - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:aws:iam::*:role/*ImageBuilder*", - "Condition": { - "StringEquals": { - "iam:PassedToService": "imagebuilder.amazonaws.com" - } - } - }, - { - "Sid": "ImageBuilderLifecycleOptional", - "Effect": "Allow", - "Action": [ - "imagebuilder:CreateImagePipeline", - "imagebuilder:UpdateImagePipeline", - "imagebuilder:DeleteImagePipeline", - "imagebuilder:GetImagePipeline", - "imagebuilder:ListImagePipelines", - "imagebuilder:CreateImageRecipe", - "imagebuilder:DeleteImageRecipe", - "imagebuilder:GetImageRecipe", - "imagebuilder:ListImageRecipes", - "imagebuilder:CreateComponent", - "imagebuilder:DeleteComponent", - "imagebuilder:GetComponent", - "imagebuilder:ListComponents", - "imagebuilder:CreateInfrastructureConfiguration", - "imagebuilder:UpdateInfrastructureConfiguration", - "imagebuilder:DeleteInfrastructureConfiguration", - "imagebuilder:GetInfrastructureConfiguration", - "imagebuilder:ListInfrastructureConfigurations", - "imagebuilder:CreateDistributionConfiguration", - "imagebuilder:UpdateDistributionConfiguration", - "imagebuilder:DeleteDistributionConfiguration", - "imagebuilder:GetDistributionConfiguration", - "imagebuilder:ListDistributionConfigurations", - "imagebuilder:CreateImage", - "imagebuilder:DeleteImage", - "imagebuilder:GetImage", - "imagebuilder:ListImages", - "imagebuilder:StartImagePipelineExecution", - "imagebuilder:CancelImageCreation", - "imagebuilder:CreateLifecyclePolicy", - "imagebuilder:UpdateLifecyclePolicy", - "imagebuilder:DeleteLifecyclePolicy", - "imagebuilder:GetLifecyclePolicy", - "imagebuilder:ListLifecyclePolicies", - "imagebuilder:TagResource", - "imagebuilder:UntagResource", - "imagebuilder:ListTagsForResource" - ], - "Resource": "*" - } - ] -} diff --git a/architectures/aws-pcs/iam/cluster-admin-policy.json b/architectures/aws-pcs/iam/cluster-admin-policy.json deleted file mode 100644 index bb317b48f..000000000 --- a/architectures/aws-pcs/iam/cluster-admin-policy.json +++ /dev/null @@ -1,282 +0,0 @@ -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "CloudFormationStackLifecycle", - "Effect": "Allow", - "Action": [ - "cloudformation:CreateStack", - "cloudformation:UpdateStack", - "cloudformation:DeleteStack", - "cloudformation:CancelUpdateStack", - "cloudformation:ContinueUpdateRollback", - "cloudformation:CreateChangeSet", - "cloudformation:ExecuteChangeSet", - "cloudformation:DescribeChangeSet", - "cloudformation:DeleteChangeSet", - "cloudformation:DescribeStacks", - "cloudformation:DescribeStackEvents", - "cloudformation:DescribeStackResource", - "cloudformation:DescribeStackResources", - "cloudformation:ListStackResources", - "cloudformation:GetTemplate", - "cloudformation:GetTemplateSummary", - "cloudformation:TagResource", - "cloudformation:UntagResource" - ], - "Resource": "*" - }, - { - "Sid": "CloudFormationDescribeStackless", - "Effect": "Allow", - "Action": [ - "cloudformation:ValidateTemplate", - "cloudformation:ListStacks", - "cloudformation:ListExports", - "cloudformation:ListImports" - ], - "Resource": "*" - }, - { - "Sid": "EC2NetworkingAndComputeLifecycle", - "Effect": "Allow", - "Action": [ - "ec2:CreateVpc", - "ec2:DeleteVpc", - "ec2:ModifyVpcAttribute", - "ec2:AssociateVpcCidrBlock", - "ec2:DisassociateVpcCidrBlock", - "ec2:CreateSubnet", - "ec2:DeleteSubnet", - "ec2:ModifySubnetAttribute", - "ec2:CreateInternetGateway", - "ec2:DeleteInternetGateway", - "ec2:AttachInternetGateway", - "ec2:DetachInternetGateway", - "ec2:CreateNatGateway", - "ec2:DeleteNatGateway", - "ec2:AllocateAddress", - "ec2:ReleaseAddress", - "ec2:AssociateAddress", - "ec2:DisassociateAddress", - "ec2:CreateRouteTable", - "ec2:DeleteRouteTable", - "ec2:CreateRoute", - "ec2:DeleteRoute", - "ec2:ReplaceRoute", - "ec2:AssociateRouteTable", - "ec2:DisassociateRouteTable", - "ec2:CreateSecurityGroup", - "ec2:DeleteSecurityGroup", - "ec2:AuthorizeSecurityGroupIngress", - "ec2:AuthorizeSecurityGroupEgress", - "ec2:RevokeSecurityGroupIngress", - "ec2:RevokeSecurityGroupEgress", - "ec2:UpdateSecurityGroupRuleDescriptionsIngress", - "ec2:UpdateSecurityGroupRuleDescriptionsEgress", - "ec2:CreateVpcEndpoint", - "ec2:ModifyVpcEndpoint", - "ec2:DeleteVpcEndpoints", - "ec2:CreateLaunchTemplate", - "ec2:CreateLaunchTemplateVersion", - "ec2:ModifyLaunchTemplate", - "ec2:DeleteLaunchTemplate", - "ec2:DeleteLaunchTemplateVersions", - "ec2:CreatePlacementGroup", - "ec2:DeletePlacementGroup", - "ec2:CreateNetworkInterface", - "ec2:DeleteNetworkInterface", - "ec2:ModifyNetworkInterfaceAttribute", - "ec2:AttachNetworkInterface", - "ec2:DetachNetworkInterface", - "ec2:CreateTags", - "ec2:DeleteTags", - "ec2:RunInstances", - "ec2:CreateFleet" - ], - "Resource": "*" - }, - { - "Sid": "EC2DescribeForPCS", - "Effect": "Allow", - "Action": [ - "ec2:Describe*", - "ec2:GetSecurityGroupsForVpc" - ], - "Resource": "*" - }, - { - "Sid": "FSxLifecycle", - "Effect": "Allow", - "Action": [ - "fsx:CreateFileSystem", - "fsx:UpdateFileSystem", - "fsx:DeleteFileSystem", - "fsx:DescribeFileSystems", - "fsx:DescribeBackups", - "fsx:CreateBackup", - "fsx:DeleteBackup", - "fsx:TagResource", - "fsx:UntagResource", - "fsx:ListTagsForResource" - ], - "Resource": "*" - }, - { - "Sid": "PCSFullAccess", - "Effect": "Allow", - "Action": [ - "pcs:*" - ], - "Resource": "*" - }, - { - "Sid": "IAMRoleAndInstanceProfileLifecycle", - "Effect": "Allow", - "Action": [ - "iam:CreateRole", - "iam:DeleteRole", - "iam:GetRole", - "iam:UpdateAssumeRolePolicy", - "iam:UpdateRole", - "iam:UpdateRoleDescription", - "iam:PutRolePolicy", - "iam:DeleteRolePolicy", - "iam:GetRolePolicy", - "iam:ListRolePolicies", - "iam:AttachRolePolicy", - "iam:DetachRolePolicy", - "iam:ListAttachedRolePolicies", - "iam:CreateInstanceProfile", - "iam:DeleteInstanceProfile", - "iam:GetInstanceProfile", - "iam:ListInstanceProfilesForRole", - "iam:AddRoleToInstanceProfile", - "iam:RemoveRoleFromInstanceProfile", - "iam:TagRole", - "iam:UntagRole", - "iam:TagInstanceProfile", - "iam:UntagInstanceProfile" - ], - "Resource": [ - "arn:aws:iam::*:role/*PCS*", - "arn:aws:iam::*:role/*pcs*", - "arn:aws:iam::*:role/*ImageBuilder*", - "arn:aws:iam::*:instance-profile/*PCS*", - "arn:aws:iam::*:instance-profile/*pcs*", - "arn:aws:iam::*:instance-profile/*ImageBuilder*" - ] - }, - { - "Sid": "IAMPassRoleToPCSAndEC2", - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": [ - "arn:aws:iam::*:role/*PCS*", - "arn:aws:iam::*:role/*pcs*" - ], - "Condition": { - "StringEquals": { - "iam:PassedToService": [ - "ec2.amazonaws.com", - "pcs.amazonaws.com" - ] - } - } - }, - { - "Sid": "ServiceLinkedRolesOneTime", - "Effect": "Allow", - "Action": "iam:CreateServiceLinkedRole", - "Resource": "arn:aws:iam::*:role/aws-service-role/*", - "Condition": { - "StringLike": { - "iam:AWSServiceName": [ - "pcs.amazonaws.com", - "spot.amazonaws.com", - "spotfleet.amazonaws.com", - "fsx.amazonaws.com", - "s3.data-source.lustre.fsx.amazonaws.com", - "imagebuilder.amazonaws.com" - ] - } - } - }, - { - "Sid": "SSMParameterLifecycle", - "Effect": "Allow", - "Action": [ - "ssm:PutParameter", - "ssm:GetParameter", - "ssm:GetParameters", - "ssm:GetParametersByPath", - "ssm:DeleteParameter", - "ssm:DeleteParameters", - "ssm:DescribeParameters", - "ssm:AddTagsToResource", - "ssm:RemoveTagsFromResource", - "ssm:ListTagsForResource", - "ssm:LabelParameterVersion" - ], - "Resource": [ - "arn:aws:ssm:*:*:parameter/pcs/*", - "arn:aws:ssm:*:*:parameter/aws/service/*" - ] - }, - { - "Sid": "KMSAccessForDefaultEncryption", - "Effect": "Allow", - "Action": [ - "kms:Decrypt", - "kms:Encrypt", - "kms:GenerateDataKey", - "kms:GenerateDataKeyWithoutPlaintext", - "kms:CreateGrant", - "kms:DescribeKey" - ], - "Resource": "*" - }, - { - "Sid": "SecretsManagerForPCS", - "Effect": "Allow", - "Action": [ - "secretsmanager:CreateSecret", - "secretsmanager:UpdateSecret", - "secretsmanager:RotateSecret", - "secretsmanager:DescribeSecret", - "secretsmanager:DeleteSecret", - "secretsmanager:TagResource", - "secretsmanager:UntagResource" - ], - "Resource": "arn:aws:secretsmanager:*:*:secret:pcs!*" - }, - { - "Sid": "CloudWatchLogsForMonitoring", - "Effect": "Allow", - "Action": [ - "logs:CreateLogGroup", - "logs:DeleteLogGroup", - "logs:DescribeLogGroups", - "logs:PutRetentionPolicy", - "logs:TagResource", - "logs:UntagResource", - "logs:CreateDelivery", - "logs:DeleteDelivery", - "logs:GetDelivery", - "logs:PutDeliverySource", - "logs:DeleteDeliverySource", - "logs:PutDeliveryDestination", - "logs:DeleteDeliveryDestination" - ], - "Resource": "*" - }, - { - "Sid": "PCSVendedLogsDelivery", - "Effect": "Allow", - "Action": [ - "pcs:AllowVendedLogDeliveryForResource" - ], - "Resource": "*" - } - ] -} diff --git a/architectures/aws-pcs/iam/cluster-user-policy.json b/architectures/aws-pcs/iam/cluster-user-policy.json deleted file mode 100644 index 98b446394..000000000 --- a/architectures/aws-pcs/iam/cluster-user-policy.json +++ /dev/null @@ -1,81 +0,0 @@ -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "DiscoverClusterAndLoginNode", - "Effect": "Allow", - "Action": [ - "ec2:DescribeInstances", - "ec2:DescribeInstanceStatus", - "ec2:DescribeTags", - "cloudformation:DescribeStacks", - "cloudformation:ListStacks", - "cloudformation:DescribeStackResources", - "cloudformation:ListStackResources" - ], - "Resource": "*" - }, - { - "Sid": "PCSReadOnly", - "Effect": "Allow", - "Action": [ - "pcs:GetCluster", - "pcs:ListClusters", - "pcs:GetComputeNodeGroup", - "pcs:ListComputeNodeGroups", - "pcs:GetQueue", - "pcs:ListQueues", - "pcs:ListTagsForResource" - ], - "Resource": "*" - }, - { - "Sid": "GrafanaPasswordRead", - "Effect": "Allow", - "Action": [ - "ssm:GetParameter", - "ssm:GetParameters" - ], - "Resource": "arn:aws:ssm:*:*:parameter/pcs/*/grafana/*" - }, - { - "Sid": "SSMSessionToLoginNode", - "Effect": "Allow", - "Action": [ - "ssm:StartSession" - ], - "Resource": [ - "arn:aws:ec2:*:*:instance/*", - "arn:aws:ssm:*:*:document/AWS-StartSSHSession", - "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSession", - "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost", - "arn:aws:ssm:*:*:document/SSM-SessionManagerRunShell" - ], - "Condition": { - "StringLike": { - "ssm:resourceTag/Name": "PCS-login*" - } - } - }, - { - "Sid": "SSMSessionDocumentLookup", - "Effect": "Allow", - "Action": [ - "ssm:DescribeSessions", - "ssm:GetConnectionStatus", - "ssm:DescribeInstanceProperties", - "ssm:DescribeInstanceInformation" - ], - "Resource": "*" - }, - { - "Sid": "SSMSessionTerminateOwn", - "Effect": "Allow", - "Action": [ - "ssm:TerminateSession", - "ssm:ResumeSession" - ], - "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*" - } - ] -} From aa1ffbe77ccaef613a672da1622e698ac7ecf925 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 02:05:30 +0000 Subject: [PATCH 44/89] docs: clarify OnDemandEfaInterfaceCount auto-derive to avoid EFA confusion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 0=auto default hid which instance type maps to how many EFA NICs, and gave no signal that EFA only works on EFA-capable types. Expanded the param description (deploy-all) + PARAMETERS.md to: - list the per-type auto values explicitly (hpc8a/hpc7a/hpc6id=2, hpc6a/ c7i.metal=1, anything else=1) - state plainly that EnableEfa=true must only be used on an EFA-capable type (hpc6/hpc7/hpc8 + select metal); a non-EFA type like c6i.4xlarge fails to launch regardless of this count - clarify when to override (pin a value for a new HPC type not in the auto map) No behavior change — auto-derive stays; this only makes it legible so HPC users aren't surprised. add-cng.yaml's EfaInterfaceCount (default 1, [1,2], no auto) is unchanged: deploy-all resolves auto→concrete before forwarding, so the terminal template still takes an explicit count. --- .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 14 +++++++++----- architectures/aws-pcs/docs/PARAMETERS.md | 2 +- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index a27a35d1f..444194115 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -522,11 +522,15 @@ Parameters: OnDemandEfaInterfaceCount: Type: Number Description: >- - Number of EFA interfaces to attach when OnDemandEnableEfa=true. - Default 0 = auto-derive from OnDemandInstanceType: 2 for - hpc8a.96xlarge / hpc7a.96xlarge / hpc6id.32xlarge (2-NIC HPC types), - 1 otherwise (hpc6a.48xlarge / c7i.metal etc.). Override with 1 or 2 - to pin explicitly. Ignored when OnDemandEnableEfa=false. + Number of EFA interfaces, used only when OnDemandEnableEfa=true (ignored + otherwise). Default 0 = auto-derive the right count from + OnDemandInstanceType. Per-type auto values: + hpc8a.96xlarge=2, hpc7a.96/48/24/12xlarge=2, hpc6id.32xlarge=2, + hpc6a.48xlarge=1, c7i.metal-24xl/48xl=1. Any other type auto-derives 1. + Only set EnableEfa=true on an EFA-capable type (the hpc6/hpc7/hpc8 family + and a few metal types) -- on a non-EFA type (e.g. c6i.4xlarge) the launch + fails regardless of this count. Override with 1 or 2 only to pin a value + that differs from the auto map (e.g. a new HPC type not listed above). Default: 0 AllowedValues: [0, 1, 2] diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 770a5579a..24bed2c68 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -62,7 +62,7 @@ runtime options), see the [README](../README.md#4-configuration). | `OnDemandCngName` | `cpu1` | CPU node-group name | | `OnDemandQueueName` | `cpu1` | CPU Slurm queue name | | `OnDemandEnableEfa` | `false` | Enable EFA on the CPU CNG (HPC/MPI workloads on hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Switches the CNG's LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` and wires in a cluster placement group. No effect on the GPU CNG. See [README §EFA on CPU HPC instances](../README.md#efa-on-cpu-hpc-instances-ondemandenableefa) | -| `OnDemandEfaInterfaceCount` | `0` (auto) | Number of EFA interfaces. **`0` (default) auto-derives from `OnDemandInstanceType`**: 2 for hpc8a/hpc7a/hpc6id, 1 otherwise (hpc6a/c7i.metal). Override with `1` or `2` to pin explicitly. Ignored when `OnDemandEnableEfa=false` | +| `OnDemandEfaInterfaceCount` | `0` (auto) | Number of EFA interfaces, used only when `OnDemandEnableEfa=true`. **`0` (default) auto-derives from `OnDemandInstanceType`** — `hpc8a.96xlarge`=2, `hpc7a.{96,48,24,12}xlarge`=2, `hpc6id.32xlarge`=2, `hpc6a.48xlarge`=1, `c7i.metal-*`=1, any other type=1. **Only enable EFA on an EFA-capable type** (hpc6/hpc7/hpc8 family + select metal); on a non-EFA type (e.g. `c6i.4xlarge`) the launch fails regardless. Override with `1`/`2` only to pin a value that differs from the auto map (e.g. a new HPC type) | | `OnDemandPlacementGroupName` | *(empty)* | Existing cluster placement group name to launch nodes into. Empty + `OnDemandEnableEfa=true` auto-creates a per-CNG cluster placement group; supplying a name reuses an existing one (e.g. shared across CPU + GPU CNGs for heterogeneous tightly-coupled jobs). Ignored when `OnDemandEnableEfa=false` | ## 5. GPU Compute Node Group — P5/P6 (Optional) From 372df829df885c2e62520618906470aeee0fbf90 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 02:14:18 +0000 Subject: [PATCH 45/89] refactor: collapse OnDemandEnableEfa into OnDemandEfaInterfaceCount (0/1/2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per review: a 0=auto default + a separate boolean enable flag was confusing — "set count to 0" reading as "auto" is unintuitive. Simplified to a single parameter with literal meaning: OnDemandEfaInterfaceCount: 0 = no EFA (default) | 1 | 2 = enable with N NICs - Removed OnDemandEnableEfa (deploy-all) and EnableEfa (add-cng) entirely; EFA is now driven by EfaInterfaceCount > 0. - Removed the auto-derive logic (OnDemandEfaCountIsAuto / OnDemandIs2NicHpc conditions + the !If forwarding). The user sets the count explicitly (the per-type values are documented: hpc6a=1, hpc7a/hpc8a/hpc6id=2). - add-cng EfaEnabled condition is now !Not [count == 0]; Has2ndEfaInterface is count == 2. - Updated descriptions to state EFA needs an EFA-capable type (non-EFA type like c6i fails to launch with count > 0) — addresses the "which types support EFA" confusion directly. - Swept README/PARAMETERS/DEPLOY-TESTING/hpc-efa-test/storage-test/prerequisites for the old param names + anchors. VPCName auto-derive (the other merged improvement) is kept as-is. --- architectures/aws-pcs/README.md | 30 ++++---- architectures/aws-pcs/assets/add-cng.yaml | 48 ++++++------- .../assets/ml-cluster-prerequisites.yaml | 2 +- .../assets/pcs-ml-cluster-deploy-all.yaml | 68 +++++-------------- architectures/aws-pcs/docs/DEPLOY-TESTING.md | 1 - architectures/aws-pcs/docs/PARAMETERS.md | 5 +- architectures/aws-pcs/tests/hpc-efa-test.md | 15 ++-- architectures/aws-pcs/tests/storage-test.md | 2 +- 8 files changed, 66 insertions(+), 105 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index e4f989e98..f3e7d2afd 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -92,7 +92,7 @@ only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | | `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `DeployOnDemandCNG` | `true` | Deploy the `cpu1` CPU queue (`OnDemandInstanceType`, default `c6i.4xlarge`) | -| `OnDemandEnableEfa` | `false` | Set `true` for HPC/MPI workloads on EFA-capable CPU types (hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandenableefa) | +| `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandefainterfacecount) | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | | `PseriesInstanceType` | `p5.48xlarge` | GPU instance type; auto-selects the multi-NIC template + EFA count | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | @@ -143,24 +143,27 @@ automatically. > group at `PseriesMinCount = PseriesMaxCount = ` so the reserved > nodes launch immediately, rather than scaling from 0. -### EFA on CPU HPC instances (`OnDemandEnableEfa`) +### EFA on CPU HPC instances (`OnDemandEfaInterfaceCount`) For tightly-coupled HPC / MPI workloads on CPU-only HPC instances, set -`OnDemandEnableEfa=true` (deploy-all) or `EnableEfa=true` (modular `add-cng.yaml`). -The CPU compute node group then launches with EFA `NetworkInterfaces` and a cluster -placement group is auto-created. GPU CNGs (P5/P6) are unaffected — they use their -own dedicated multi-NIC EFA wiring per-family. - -| Instance type | EFA interfaces | Aggregate spec | Set `OnDemandEfaInterfaceCount` to | +`OnDemandEfaInterfaceCount` (deploy-all) or `EfaInterfaceCount` (modular +`add-cng.yaml`) to the instance type's EFA interface count. `0` (default) = no +EFA; `1` or `2` = enable EFA with that many interfaces — the CPU compute node +group then launches with EFA `NetworkInterfaces` and a cluster placement group +is auto-created. GPU CNGs (P5/P6) are unaffected — they use their own dedicated +multi-NIC EFA wiring per-family. + +| Instance type | EFA interfaces | Aggregate spec | Set the count to | |---|---:|---:|---:| +| (any non-HPC type, e.g. `c6i.4xlarge`) | — | — | **0** (no EFA; default) | | `hpc6a.48xlarge` | 1 | 100 Gbps | **1** | | `hpc7a.96xlarge` (and 12/24/48) | 2 | 300 Gbps | **2** | | `hpc6id.32xlarge` | 2 | 200 Gbps | **2** | | `hpc8a.96xlarge` | 2 | 300 Gbps | **2** | | `c7i.metal-48xl` etc. | 1 | varies | **1** | -(Mismatching `OnDemandEfaInterfaceCount` with the instance type's actual -`MaximumEfaInterfaces` fails at launch.) +(Setting a count > 0 on a non-EFA type, or mismatching the count with the +instance type's actual `MaximumEfaInterfaces`, fails at launch.) **Placement group:** auto-created per-CNG by the template. Override with `OnDemandPlacementGroupName=` to share one PG across multiple @@ -257,7 +260,6 @@ aws cloudformation create-stack \ ParameterKey=OnDemandQueueName,ParameterValue=hpc \ ParameterKey=OnDemandInstanceType,ParameterValue=hpc7a.96xlarge \ ParameterKey=OnDemandMaxCount,ParameterValue=4 \ - ParameterKey=OnDemandEnableEfa,ParameterValue=true \ ParameterKey=OnDemandEfaInterfaceCount,ParameterValue=2 \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM ``` @@ -265,7 +267,7 @@ Replaces the default `cpu1` queue with an `hpc` queue of EFA-enabled hpc7a.96xlarge nodes. The CNG launches in an auto-created cluster placement group. For other HPC types, set `OnDemandInstanceType` and the matching `OnDemandEfaInterfaceCount` from the table in -[EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandenableefa) +[EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandefainterfacecount) (hpc6a = `1`; hpc7a/hpc6id/hpc8a = `2`). --- @@ -544,7 +546,7 @@ parameter and default, see [PARAMETERS.md](./docs/PARAMETERS.md). | [`pcs-ml-cluster-deploy-all.yaml`](./assets/pcs-ml-cluster-deploy-all.yaml) | All-in-one: Prerequisites + (optional AMI) + Cluster + login/CPU/GPU CNGs | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml&stackName=pcs-ml-cluster) | | [`ml-cluster-prerequisites.yaml`](./assets/ml-cluster-prerequisites.yaml) | VPC, subnets, security groups, FSx for Lustre + OpenZFS | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/ml-cluster-prerequisites.yaml&stackName=pcs-prerequisites) | | [`cluster.yaml`](./assets/cluster.yaml) | PCS cluster core (Slurm scheduler only, no nodes) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster.yaml&stackName=pcs-cluster) | -| [`add-cng.yaml`](./assets/add-cng.yaml) | Compute node group — login nodes, CPU / single-NIC-GPU queues (C6i, G5, G6); switches to a multi-NIC EFA `NetworkInterfaces` block when `EnableEfa=true` (HPC types: hpc6a/hpc7a/hpc6id/hpc8a …) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng.yaml&stackName=pcs-add-cng) | +| [`add-cng.yaml`](./assets/add-cng.yaml) | Compute node group — login nodes, CPU / single-NIC-GPU queues (C6i, G5, G6); switches to a multi-NIC EFA `NetworkInterfaces` block when `EfaInterfaceCount > 0` (HPC types: hpc6a/hpc7a/hpc6id/hpc8a …) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng.yaml&stackName=pcs-add-cng) | | [`add-cng-p5.yaml`](./assets/add-cng-p5.yaml) | P5/P5e/P5en nodes (16/32 EFA interfaces, by type) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p5.yaml&stackName=pcs-add-cng-p5) | | [`add-cng-p6-b200.yaml`](./assets/add-cng-p6-b200.yaml) | P6-B200 nodes (8 EFA interfaces) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b200.yaml&stackName=pcs-add-cng-p6-b200) | | [`add-cng-p6-b300.yaml`](./assets/add-cng-p6-b300.yaml) | P6-B300 nodes (16 EFA interfaces) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b300.yaml&stackName=pcs-add-cng-p6-b300) | @@ -576,7 +578,7 @@ pcs-ml-cluster-deploy-all.yaml ← user deploys this ├─► add-cng.yaml (compute) ← CPU queue (dynamic scaling 0→N) │ • MonitoringRole=compute → Node Exporter │ • DirectoryService=OpenLDAP-LoginNode → SSSD client -│ • EnableEfa=true → EFA NetworkInterfaces + PG +│ • EfaInterfaceCount>0 → EFA NetworkInterfaces + PG │ │ │ └─ UserData fetches external scripts: │ ├─ PostInstallScriptUrl (same as login) diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index 3d6fb8206..b8bafe581 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -9,7 +9,7 @@ Description: > (DeployMonitoring/MonitoringRole). A Slurm queue is created only when QueueName is set (leave empty for login nodes). - EFA support (EnableEfa=true) is available for CPU HPC instance types + EFA support (EfaInterfaceCount > 0) is available for CPU HPC instance types (hpc8a/hpc7a/hpc6a etc., 1-2 EFA interfaces). When enabled the LaunchTemplate switches to a NetworkInterfaces block with InterfaceType=efa, and a cluster placement group is auto-created (override with PlacementGroupName to reuse an @@ -41,7 +41,6 @@ Metadata: - Label: default: EFA (CPU HPC instances) Parameters: - - EnableEfa - EfaInterfaceCount - PlacementGroupName - Label: @@ -259,38 +258,30 @@ Parameters: Description: S3 key prefix for scripts (e.g. templates/) Default: 'templates/' - EnableEfa: - Type: String - Description: >- - Enable Elastic Fabric Adapter (EFA) on the compute nodes for tightly-coupled - HPC/MPI workloads. When true, the LaunchTemplate switches to a - NetworkInterfaces block with InterfaceType=efa and a cluster placement group - is wired in (auto-created unless PlacementGroupName is supplied). Requires - an EFA-capable instance type (e.g. hpc8a.96xlarge, hpc7a.96xlarge, - hpc6a.48xlarge). For multi-NIC GPU families (P5/P6) use the - dedicated add-cng-p5/p6-* templates instead. - Default: 'false' - AllowedValues: - - 'true' - - 'false' - EfaInterfaceCount: Type: Number Description: >- - Number of EFA network interfaces to attach when EnableEfa=true. Use the - MaximumEfaInterfaces value reported by the instance type: hpc8a/hpc7a/hpc6id - = 2; hpc6a/c7i.metal = 1. Mismatching this with the instance type's - capability fails at launch. Ignored when EnableEfa=false. - Default: 1 - AllowedValues: [1, 2] + Number of Elastic Fabric Adapter (EFA) network interfaces on the compute + nodes. 0 (default) = no EFA (standard ENA networking). 1 or 2 = enable EFA + with that many interfaces: the LaunchTemplate switches to a NetworkInterfaces + block with InterfaceType=efa and a cluster placement group is wired in + (auto-created unless PlacementGroupName is supplied). Set the count to the + instance type's MaximumEfaInterfaces — hpc8a.96xlarge / hpc7a.* / + hpc6id.32xlarge = 2; hpc6a.48xlarge / c7i.metal = 1. EFA requires an + EFA-capable type; a non-EFA type (e.g. c6i.4xlarge) fails to launch with + count > 0. For multi-NIC GPU families (P5/P6) use the dedicated + add-cng-p5/p6-* templates instead. + Default: 0 + AllowedValues: [0, 1, 2] PlacementGroupName: Type: String Description: >- (Optional) Name of an existing cluster placement group to launch nodes into. Leave empty to auto-create a per-CNG cluster placement group when - EnableEfa=true. Specify a value to share one placement group across multiple - CNGs (e.g. a heterogeneous tightly-coupled job). Ignored when EnableEfa=false. + EfaInterfaceCount > 0. Specify a value to share one placement group across + multiple CNGs (e.g. a heterogeneous tightly-coupled job). Ignored when + EfaInterfaceCount = 0. Default: '' Conditions: @@ -301,7 +292,8 @@ Conditions: IsMonitoringRoleSet: !Not [!Equals [!Ref MonitoringRole, 'none']] HasExtraSecurityGroup: !Not [!Equals [!Ref ExtraSecurityGroupId, ""]] HasAmiId: !Not [!Equals [!Ref AmiId, ""]] - EfaEnabled: !Equals [!Ref EnableEfa, 'true'] + # EFA is enabled when the interface count is non-zero (0 = standard ENA). + EfaEnabled: !Not [!Equals [!Ref EfaInterfaceCount, 0]] HasUserPlacementGroup: !Not [!Equals [!Ref PlacementGroupName, ""]] # Auto-create a per-CNG cluster placement group only when EFA is on AND the # user did not provide one. Override PlacementGroupName to reuse an existing @@ -309,9 +301,7 @@ Conditions: CreatePlacementGroup: !And - !Condition EfaEnabled - !Not [!Condition HasUserPlacementGroup] - Has2ndEfaInterface: !And - - !Condition EfaEnabled - - !Equals [!Ref EfaInterfaceCount, 2] + Has2ndEfaInterface: !Equals [!Ref EfaInterfaceCount, 2] DirectoryEnabled: !Not [!Equals [!Ref DirectoryRole, 'none']] Resources: diff --git a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml index fa48ad4d6..cf272d4e7 100644 --- a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml +++ b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml @@ -143,7 +143,7 @@ Parameters: The headline feature is GPUDirect Storage (GDS) for P5/P5e/P5en/P6-B200 GPU clients, which DMAs file data straight into GPU memory (the NVIDIA nvidia-fs/cuFile stack must be installed on the client — currently a - ROADMAP follow-up). EFA-capable CPU CNGs (OnDemandEnableEfa=true) get + ROADMAP follow-up). EFA-capable CPU CNGs (OnDemandEfaInterfaceCount > 0) get the EFA transport path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **EFA-enabled Lustre has a much larger minimum storage capacity than the non-EFA default**: at diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 444194115..766fb7a1d 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -44,7 +44,6 @@ Metadata: - OnDemandMaxCount - OnDemandCngName - OnDemandQueueName - - OnDemandEnableEfa - OnDemandEfaInterfaceCount - OnDemandPlacementGroupName - Label: @@ -137,10 +136,8 @@ Metadata: default: "CPU node group name" OnDemandQueueName: default: "CPU queue name" - OnDemandEnableEfa: - default: "Enable EFA on CPU queue (for HPC instance types)" OnDemandEfaInterfaceCount: - default: "EFA interface count (0 = auto from instance type)" + default: "EFA interfaces on CPU queue (0 = off, 1 or 2 for HPC types)" OnDemandPlacementGroupName: default: "Existing placement group name (empty = auto-create when EFA on)" DeployPseriesCNG: @@ -281,7 +278,7 @@ Parameters: headline feature is GPUDirect Storage (GDS) for P5/P5e/P5en/P6-B200 GPU clients, which DMAs file data straight into GPU memory (the NVIDIA nvidia-fs/cuFile stack must be installed on the client -- currently a - ROADMAP follow-up). EFA-capable CPU CNGs (OnDemandEnableEfa=true) get + ROADMAP follow-up). EFA-capable CPU CNGs (OnDemandEfaInterfaceCount > 0) get the EFA transport path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **EFA-enabled Lustre has a much higher minimum Capacity than non-EFA**: at PerUnitStorageThroughput=250 @@ -506,31 +503,18 @@ Parameters: # EFA on the on-demand CPU CNG (for HPC instances like hpc8a/hpc7a/hpc6a). # GPU CNGs (P5/P6) have their own multi-NIC EFA wiring in the dedicated - # add-cng-p5/p6-* templates; these knobs do NOT affect those. - OnDemandEnableEfa: - Type: String - Description: >- - Enable Elastic Fabric Adapter (EFA) on the on-demand (CPU) compute node - group. Use with EFA-capable HPC instance types (hpc8a.96xlarge, - hpc7a.96xlarge, hpc6a.48xlarge, etc.). Default false keeps the - standard single-NIC LaunchTemplate. No effect on the P-series GPU CNG. - Default: 'false' - AllowedValues: - - 'true' - - 'false' - + # add-cng-p5/p6-* templates; this knob does NOT affect those. OnDemandEfaInterfaceCount: Type: Number Description: >- - Number of EFA interfaces, used only when OnDemandEnableEfa=true (ignored - otherwise). Default 0 = auto-derive the right count from - OnDemandInstanceType. Per-type auto values: - hpc8a.96xlarge=2, hpc7a.96/48/24/12xlarge=2, hpc6id.32xlarge=2, - hpc6a.48xlarge=1, c7i.metal-24xl/48xl=1. Any other type auto-derives 1. - Only set EnableEfa=true on an EFA-capable type (the hpc6/hpc7/hpc8 family - and a few metal types) -- on a non-EFA type (e.g. c6i.4xlarge) the launch - fails regardless of this count. Override with 1 or 2 only to pin a value - that differs from the auto map (e.g. a new HPC type not listed above). + Number of Elastic Fabric Adapter (EFA) interfaces on the on-demand (CPU) + compute node group. 0 (default) = no EFA (standard ENA networking). 1 or 2 + = enable EFA with that many interfaces (switches to a NetworkInterfaces + LaunchTemplate + a cluster placement group). Set the count to the instance + type's MaximumEfaInterfaces: hpc8a.96xlarge / hpc7a.* / hpc6id.32xlarge = 2; + hpc6a.48xlarge / c7i.metal = 1. EFA requires an EFA-capable type — a non-EFA + type (e.g. the default c6i.4xlarge) fails to launch with count > 0. No + effect on the P-series GPU CNG. Default: 0 AllowedValues: [0, 1, 2] @@ -539,9 +523,9 @@ Parameters: Description: >- (Optional, advanced) Existing cluster placement group name to launch the on-demand CPU nodes into. Leave empty (default) to auto-create a per-CNG - cluster placement group when OnDemandEnableEfa=true. Set to share one PG - across multiple CNGs (e.g. heterogeneous tightly-coupled jobs). Ignored - when OnDemandEnableEfa=false. + cluster placement group when OnDemandEfaInterfaceCount > 0. Set to share one + PG across multiple CNGs (e.g. heterogeneous tightly-coupled jobs). Ignored + when OnDemandEfaInterfaceCount = 0. Default: '' # P-series Compute Node Group @@ -683,17 +667,6 @@ Conditions: - !Not [!Equals [!Ref PseriesInstanceType, 'p6-b300.48xlarge']] # VPCName auto-derives from the stack name when left empty. HasUserVPCName: !Not [!Equals [!Ref VPCName, '']] - # Auto-derive OnDemandEfaInterfaceCount from OnDemandInstanceType when the - # user leaves the parameter at its default 0. Only the 2-NIC HPC types - # (hpc8a / hpc7a / hpc6id) report 2 EFA interfaces; everything else is 1. - OnDemandEfaCountIsAuto: !Equals [!Ref OnDemandEfaInterfaceCount, 0] - OnDemandIs2NicHpc: !Or - - !Equals [!Ref OnDemandInstanceType, 'hpc8a.96xlarge'] - - !Equals [!Ref OnDemandInstanceType, 'hpc7a.96xlarge'] - - !Equals [!Ref OnDemandInstanceType, 'hpc7a.48xlarge'] - - !Equals [!Ref OnDemandInstanceType, 'hpc7a.24xlarge'] - - !Equals [!Ref OnDemandInstanceType, 'hpc7a.12xlarge'] - - !Equals [!Ref OnDemandInstanceType, 'hpc6id.32xlarge'] Resources: ############################################################################# @@ -862,15 +835,10 @@ Resources: MonitoringRole: compute PostInstallScriptUrl: !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs - # EFA on CPU HPC instances (hpc8a/hpc7a/hpc6a). Forwarded to add-cng.yaml, - # which auto-creates a per-CNG cluster placement group when EnableEfa=true and - # PlacementGroupName is empty. EfaInterfaceCount auto-derives from the - # instance type when the user left OnDemandEfaInterfaceCount=0 (the default). - EnableEfa: !Ref OnDemandEnableEfa - EfaInterfaceCount: !If - - OnDemandEfaCountIsAuto - - !If [OnDemandIs2NicHpc, 2, 1] - - !Ref OnDemandEfaInterfaceCount + # EFA on CPU HPC instances (hpc8a/hpc7a/hpc6a). add-cng.yaml enables EFA + # when EfaInterfaceCount > 0 and auto-creates a per-CNG cluster placement + # group when PlacementGroupName is empty. + EfaInterfaceCount: !Ref OnDemandEfaInterfaceCount PlacementGroupName: !Ref OnDemandPlacementGroupName DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] diff --git a/architectures/aws-pcs/docs/DEPLOY-TESTING.md b/architectures/aws-pcs/docs/DEPLOY-TESTING.md index 9bb9e5bec..7ea660083 100644 --- a/architectures/aws-pcs/docs/DEPLOY-TESTING.md +++ b/architectures/aws-pcs/docs/DEPLOY-TESTING.md @@ -62,7 +62,6 @@ aws cloudformation create-stack \ ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ ParameterKey=OnDemandInstanceType,ParameterValue=hpc8a.96xlarge \ - ParameterKey=OnDemandEnableEfa,ParameterValue=true \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ --region us-east-2 --profile claude ``` diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 24bed2c68..afc460256 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -61,9 +61,8 @@ runtime options), see the [README](../README.md#4-configuration). | `OnDemandMaxCount` | `4` | CPU queue maximum nodes | | `OnDemandCngName` | `cpu1` | CPU node-group name | | `OnDemandQueueName` | `cpu1` | CPU Slurm queue name | -| `OnDemandEnableEfa` | `false` | Enable EFA on the CPU CNG (HPC/MPI workloads on hpc6a/hpc7a/hpc6id/hpc8a, c7i.metal, etc.). Switches the CNG's LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` and wires in a cluster placement group. No effect on the GPU CNG. See [README §EFA on CPU HPC instances](../README.md#efa-on-cpu-hpc-instances-ondemandenableefa) | -| `OnDemandEfaInterfaceCount` | `0` (auto) | Number of EFA interfaces, used only when `OnDemandEnableEfa=true`. **`0` (default) auto-derives from `OnDemandInstanceType`** — `hpc8a.96xlarge`=2, `hpc7a.{96,48,24,12}xlarge`=2, `hpc6id.32xlarge`=2, `hpc6a.48xlarge`=1, `c7i.metal-*`=1, any other type=1. **Only enable EFA on an EFA-capable type** (hpc6/hpc7/hpc8 family + select metal); on a non-EFA type (e.g. `c6i.4xlarge`) the launch fails regardless. Override with `1`/`2` only to pin a value that differs from the auto map (e.g. a new HPC type) | -| `OnDemandPlacementGroupName` | *(empty)* | Existing cluster placement group name to launch nodes into. Empty + `OnDemandEnableEfa=true` auto-creates a per-CNG cluster placement group; supplying a name reuses an existing one (e.g. shared across CPU + GPU CNGs for heterogeneous tightly-coupled jobs). Ignored when `OnDemandEnableEfa=false` | +| `OnDemandEfaInterfaceCount` | `0` | EFA interfaces on the CPU CNG. **`0` (default) = no EFA** (standard ENA). `1` or `2` = enable EFA with that many interfaces (switches the LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` + a cluster placement group). Set the count to the instance type's `MaximumEfaInterfaces`: `hpc8a.96xlarge`/`hpc7a.*`/`hpc6id.32xlarge`=2; `hpc6a.48xlarge`/`c7i.metal`=1. **EFA needs an EFA-capable type** — a non-EFA type (e.g. the default `c6i.4xlarge`) fails to launch with count > 0. No effect on the GPU CNG. See [README §EFA on CPU HPC instances](../README.md#efa-on-cpu-hpc-instances-ondemandefainterfacecount) | +| `OnDemandPlacementGroupName` | *(empty)* | Existing cluster placement group name to launch nodes into. Empty + `OnDemandEfaInterfaceCount > 0` auto-creates a per-CNG cluster placement group; supplying a name reuses an existing one (e.g. shared across CPU + GPU CNGs for heterogeneous tightly-coupled jobs). Ignored when `OnDemandEfaInterfaceCount = 0` | ## 5. GPU Compute Node Group — P5/P6 (Optional) diff --git a/architectures/aws-pcs/tests/hpc-efa-test.md b/architectures/aws-pcs/tests/hpc-efa-test.md index 968841470..5ed3fe1c7 100644 --- a/architectures/aws-pcs/tests/hpc-efa-test.md +++ b/architectures/aws-pcs/tests/hpc-efa-test.md @@ -7,11 +7,15 @@ group auto-creation, multi-NIC wiring, and OSU MPI benchmark results. ## Test 9: EFA on CPU HPC instances (hpc6a / hpc7a / hpc8a) -**When to run:** when `add-cng.yaml`'s EFA wiring (`EnableEfa`, `EfaInterfaceCount`, -`PlacementGroupName`) or the deploy-all forwarding params (`OnDemandEnableEfa`, -`OnDemandEfaInterfaceCount`, `OnDemandPlacementGroupName`) change. Skip if only +**When to run:** when `add-cng.yaml`'s EFA wiring (`EfaInterfaceCount`, +`PlacementGroupName`) or the deploy-all forwarding params +(`OnDemandEfaInterfaceCount`, `OnDemandPlacementGroupName`) change. Skip if only GPU/monitoring/AMI paths were touched. +> **EFA is enabled by the interface count.** `OnDemandEfaInterfaceCount=0` +> (default) = no EFA; `1` or `2` enables EFA with that many NICs. There is no +> separate `OnDemandEnableEfa` flag (removed — the count alone drives it). + This validates that the on-demand CPU CNG actually launches with EFA NICs in a cluster placement group, and that MPI / libfabric over EFA works end-to-end. The verified-configurations table above documents the bandwidth numbers; this section @@ -41,7 +45,6 @@ aws cloudformation create-stack \ ParameterKey=OnDemandQueueName,ParameterValue=hpc \ ParameterKey=OnDemandMinCount,ParameterValue=0 \ ParameterKey=OnDemandMaxCount,ParameterValue=2 \ - ParameterKey=OnDemandEnableEfa,ParameterValue=true \ ParameterKey=OnDemandEfaInterfaceCount,ParameterValue=$EFA_NICS \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM ``` @@ -83,7 +86,7 @@ PATH=/opt/amazon/openmpi/bin:$PATH make -j8 ``` Submit a 2-node sbatch with the AWS-tuned EFA env (the canonical reference -parameters, see "Tuning notes" in [Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue)): +parameters, see "Tuning notes" in the verified-configuration numbers in [tests/README.md](./README.md)): ```bash cat > /fsx/osu/osu-bench.sbatch <<'EOF' @@ -121,7 +124,7 @@ sbatch -p hpc /fsx/osu/osu-bench.sbatch ``` The reference numbers per instance type are in -[Verified configurations](#hpc-efa-on-cpu-instances-ondemandenableefatrue) above. +the verified-configuration numbers in [tests/README.md](./README.md) above. ### Step 4 — observe NIC-level traffic in Grafana (optional) diff --git a/architectures/aws-pcs/tests/storage-test.md b/architectures/aws-pcs/tests/storage-test.md index 0e0060b92..5c8de69a2 100644 --- a/architectures/aws-pcs/tests/storage-test.md +++ b/architectures/aws-pcs/tests/storage-test.md @@ -125,7 +125,7 @@ extra checks beyond the above are: Capacity. The FSx-side EFA endpoints are usable from EFA-capable clients (CPU CNGs -deployed with `OnDemandEnableEfa=true`, P5/P6 GPU CNGs). Plain Lustre client +deployed with `OnDemandEfaInterfaceCount > 0`, P5/P6 GPU CNGs). Plain Lustre client mounts continue to work over TCP for non-EFA nodes; EFA support is additive. --- From a8556d78426046f92a08bec5b47157d64cb39926 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 02:26:07 +0000 Subject: [PATCH 46/89] fix: remove VPCName param from deploy-all + clear stale OnDemandEnableEfa/anchor refs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - VPCName removed from deploy-all (ParameterGroup, label, param def, HasUserVPCName condition); VPC name now hardcoded to ${StackName}-VPC. prerequisites.yaml keeps its VPCName param for standalone use. - PARAMETERS.md: removed the VPCName row, added a note that the name is fixed. - Cleared 2 stale OnDemandEnableEfa=true refs (README + PARAMETERS FSxLustreEnableEfa descriptions) → OnDemandEfaInterfaceCount > 0. - Fixed broken ROADMAP anchor (#client-side-lustre-on-efa--gds-support is a list item, not a heading) → link to the file. Found via full consistency audit. --- architectures/aws-pcs/README.md | 2 +- .../aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml | 15 +-------------- architectures/aws-pcs/docs/PARAMETERS.md | 9 +++++++-- 3 files changed, 9 insertions(+), 17 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index f3e7d2afd..c0973a268 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -184,7 +184,7 @@ capable type; switch to a more widely available one if your Region needs it. |---|---|---|---|---| | Lustre (`/fsx`) | `LustreDeploymentType` | `PERSISTENT_2` | `PERSISTENT_1` | `PERSISTENT_2` (throughput 125/250/500/1000, metadata config) isn't in every Region; `PERSISTENT_1` (50/100/200) is in more Regions | | Lustre (`/fsx`) | `PerUnitStorageThroughput` | `250` | any valid number | Must be valid for the type: P2 = 125/250/500/1000, P1 = 50/100/200 | -| Lustre (`/fsx`) | `FSxLustreEnableEfa` | `false` | `true` | Enable EFA on the Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — see [`docs/ROADMAP.md`](./docs/ROADMAP.md#client-side-lustre-on-efa--gds-support)). EFA-capable CPU CNGs (`OnDemandEnableEfa=true`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** (a CFN Rule fails the stack at create time on PERSISTENT_1). **Requires a much higher minimum `Capacity`**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly when enabling this | +| Lustre (`/fsx`) | `FSxLustreEnableEfa` | `false` | `true` | Enable EFA on the Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — see the Client-side Lustre-on-EFA + GDS item in [`docs/ROADMAP.md`](./docs/ROADMAP.md)). EFA-capable CPU CNGs (`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** (a CFN Rule fails the stack at create time on PERSISTENT_1). **Requires a much higher minimum `Capacity`**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly when enabling this | | OpenZFS (`/home`) | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | `SINGLE_AZ_HA_1`, `SINGLE_AZ_2`, `SINGLE_AZ_1` | `SINGLE_AZ_1` is in all Regions; HA/2 variants vary. `MULTI_AZ` excluded (needs a second subnet) | | OpenZFS (`/home`) | `HomeThroughput` | `320` | any valid number | Throughput (MB/s). Valid values depend on the deployment type: `SINGLE_AZ_2`/`SINGLE_AZ_HA_2` = 160/320/640/1280/2560/3840/5120/7680/10240; `SINGLE_AZ_HA_1` = 128/256/512/1024/2048/3072/4096; `SINGLE_AZ_1` = 64/128/256/512/1024/2048/3072/4096 | diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 766fb7a1d..4027dcb08 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -18,7 +18,6 @@ Metadata: - PrimarySubnetAZ - AdditionalSubnetAZ2 - AdditionalSubnetAZ3 - - VPCName - CreateS3Endpoint - Label: default: "2. PCS Cluster Configuration" @@ -92,8 +91,6 @@ Metadata: default: "(Optional) 2nd AZ for an additional private subnet" AdditionalSubnetAZ3: default: "(Optional) 3rd AZ for an additional private subnet" - VPCName: - default: "VPC name (empty = ${StackName}-VPC)" CreateS3Endpoint: default: "Create S3 VPC endpoint" LoginNodeInstanceType: @@ -190,14 +187,6 @@ Parameters: Default: "templates/" # Network Configuration - VPCName: - Description: >- - Name of your VPC. Leave empty (default) to auto-derive from the stack - name as ${AWS::StackName}-VPC, so multiple PCS deployments in one - account get unique VPC names automatically. - Default: '' - Type: String - PrimarySubnetAZ: Description: >- Availability Zone to deploy the cluster into. The only required choice -- @@ -665,8 +654,6 @@ Conditions: - !Condition DeployPseriesCNGEnabled - !Not [!Equals [!Ref PseriesInstanceType, 'p6-b200.48xlarge']] - !Not [!Equals [!Ref PseriesInstanceType, 'p6-b300.48xlarge']] - # VPCName auto-derives from the stack name when left empty. - HasUserVPCName: !Not [!Equals [!Ref VPCName, '']] Resources: ############################################################################# @@ -680,7 +667,7 @@ Resources: - ml-cluster-prerequisites.yaml - !Sub 'https://${S3BucketName}.s3.amazonaws.com/${S3KeyPrefix}ml-cluster-prerequisites.yaml' Parameters: - VPCName: !If [HasUserVPCName, !Ref VPCName, !Sub '${AWS::StackName}-VPC'] + VPCName: !Sub '${AWS::StackName}-VPC' PrimarySubnetAZ: !Ref PrimarySubnetAZ AdditionalSubnetAZ2: !Ref AdditionalSubnetAZ2 AdditionalSubnetAZ3: !Ref AdditionalSubnetAZ3 diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index afc460256..347830b34 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -20,9 +20,14 @@ runtime options), see the [README](../README.md#4-configuration). | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter. Holds the public subnet (login node), the primary private subnet (compute, FSx), and the single NAT gateway | | `AdditionalSubnetAZ2` | *(empty)* | (Optional) 2nd AZ for an additional private subnet. Empty = single-AZ. Enables multi-AZ layouts (e.g. OpenZFS `MULTI_AZ`). Shares the primary AZ's NAT gateway (cross-AZ egress, no per-AZ NAT) | | `AdditionalSubnetAZ3` | *(empty)* | (Optional) 3rd AZ for an additional private subnet. Requires `AdditionalSubnetAZ2` to also be set. Max 3 private AZs total | -| `VPCName` | *(empty → `${StackName}-VPC`)* | Name for the created VPC. Empty (default) auto-derives from the stack name so multiple deployments in one account get unique names | | `CreateS3Endpoint` | `true` | Create an S3 VPC endpoint | +The VPC name is fixed to `${StackName}-VPC` (derived from the stack name, so +multiple deployments in one account get unique VPC names automatically) — there +is no `VPCName` parameter on the all-in-one template. The standalone +`ml-cluster-prerequisites.yaml` still accepts a `VPCName` parameter if you deploy +it directly. + ## 2. PCS Cluster Configuration | Parameter | Default | Purpose | @@ -89,7 +94,7 @@ See [Storage: FSx deployment types](../README.md#storage-fsx-deployment-types-re | `PerUnitStorageThroughput` | `250` | FSx for Lustre (`/fsx`) throughput (MB/s/TiB); valid values depend on the deployment type | | `Compression` | `LZ4` | FSx for Lustre (`/fsx`) data compression (`LZ4` / `NONE`) | | `LustreVersion` | `2.15` | FSx for Lustre (`/fsx`) software version (`2.15` / `2.12`) | -| `FSxLustreEnableEfa` | `false` | Enable EFA on the FSx for Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5/P5e/P5en/P6-B200 GPU clients**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — tracked as a follow-up in [docs/ROADMAP.md](./ROADMAP.md#client-side-lustre-on-efa--gds-support)). EFA-capable CPU CNGs (`OnDemandEnableEfa=true`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** — a CFN Rule on the prerequisites template fails the stack at create time when combined with PERSISTENT_1 (rather than silently ignoring the opt-in). **Requires a much larger `Capacity` than non-EFA**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB non-EFA default). The full minimum-capacity matrix per throughput tier is in the [FSx for Lustre User Guide](https://docs.aws.amazon.com/fsx/latest/LustreGuide/efa.html). The FSx side rejects undersized capacity at stack-create time with a clear error | +| `FSxLustreEnableEfa` | `false` | Enable EFA on the FSx for Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5/P5e/P5en/P6-B200 GPU clients**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — tracked as a follow-up in [docs/ROADMAP.md](./ROADMAP.md)). EFA-capable CPU CNGs (`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** — a CFN Rule on the prerequisites template fails the stack at create time when combined with PERSISTENT_1 (rather than silently ignoring the opt-in). **Requires a much larger `Capacity` than non-EFA**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB non-EFA default). The full minimum-capacity matrix per throughput tier is in the [FSx for Lustre User Guide](https://docs.aws.amazon.com/fsx/latest/LustreGuide/efa.html). The FSx side rejects undersized capacity at stack-create time with a clear error | | `HomeCapacity` | `512` | FSx for OpenZFS (`/home`) capacity (GiB) | | `HomeThroughput` | `320` | FSx for OpenZFS (`/home`) throughput (MB/s) | | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | FSx for OpenZFS (`/home`) deployment type (`SINGLE_AZ_HA_2` / `SINGLE_AZ_HA_1` / `SINGLE_AZ_2` / `SINGLE_AZ_1`) — Region-dependent | From e84906c5a30a71194f405257a95938dda9b9ae76 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 02:46:56 +0000 Subject: [PATCH 47/89] fix: VPC Name tag uses VPCName param (was hardcoded 'ML Cluster VPC') MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The VPC resource's Name tag was hardcoded, so the VPCName parameter (and deploy-all's ${StackName}-VPC value) only renamed the subnets, not the VPC itself — every cluster's VPC showed up as 'ML Cluster VPC' in the console. Now the VPC Name follows VPCName, so multiple deploys in one account get distinct, stack-named VPCs as intended. Found during hpc8a EFA e2e (VPC showed 'ML Cluster VPC' instead of pcs-efa-hpc8a-VPC). --- architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml index cf272d4e7..6b346430e 100644 --- a/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml +++ b/architectures/aws-pcs/assets/ml-cluster-prerequisites.yaml @@ -284,7 +284,7 @@ Resources: CidrBlock: !FindInMap [Networking, VPC, CIDR0] Tags: - Key: Name - Value: ML Cluster VPC + Value: !Ref VPCName VpcCidrBlock: Type: AWS::EC2::VPCCidrBlock From 5cc8a8a30d1c53ee84499490e5fef371d64b76ca Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 03:14:46 +0000 Subject: [PATCH 48/89] tests: add region coverage matrix (us-east-1/2, us-west-2, ap-northeast-1, ap-south-1) Validated deploy-all across 5 regions: all reach CREATE_COMPLETE with login + 6 monitoring containers + FSx Lustre/OpenZFS mounts. ap-south-1 (Mumbai) adds p6-b200 x4 GPU validation (8 GPU/node, 8 EFA NICs, B200 driver 595.71.05). Documented cross-region S3 fetch works (S3 global namespace) and the pre-merge PostInstallScriptUrl 404 caveat. --- architectures/aws-pcs/tests/README.md | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index d6d131b32..b01fa8a46 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -61,6 +61,36 @@ MonitoringStack=Prometheus-LoginNode --- +## Region coverage + +The templates fetch nested stacks + boot scripts from a single S3 bucket +(`S3BucketName`) and resolve the PCS-Ready DLAMI from SSM per region. Validated +deploys (deploy-all: prereqs + cluster + login + CPU, monitoring on) across: + +| Region | Deploy | Login + monitoring | FSx Lustre `/fsx` + OpenZFS `/home` | Notes | +|---|---|---|---|---| +| **us-east-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ 1.2T Lustre + OpenZFS NFS | template bucket's home region | +| **us-east-2** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | primary test region; also EFA hpc6a/hpc8a + multi-user/accounting | +| **us-west-2** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | | +| **ap-northeast-1** | ✅ CREATE_COMPLETE | ✅ | ✅ | Tokyo | +| **ap-south-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | Mumbai; p6-b200 ×4 GPU (8 GPU/node, 8 EFA NICs), B200 driver 595.71.05 | + +Cross-region note: nested-stack `TemplateURL` and the in-instance +`aws s3 cp` of boot scripts both work against an S3 bucket in a **different** +region (S3 global namespace; no `--region` needed) — verified ap-south-1 → +us-east-1 bucket. The PCS-Ready DLAMI SSM parameter +(`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`) resolves in +every region tested. `OpenZFSDeploymentType=SINGLE_AZ_HA_2` and +`LustreDeploymentType=PERSISTENT_2` (the defaults) were available in all five. + +> **Pre-merge caveat:** `PostInstallScriptUrl` defaults to the `awslabs/main` +> GitHub-raw URL, which 404s until PR #1120 merges — so pre-merge test deploys +> must override it (point at the test bucket / fork) or compute nodes boot +> without Enroot/Pyxis (`/var/log/pcs-post-install.log` shows exit 127). The +> cluster still reaches CREATE_COMPLETE; only the container runtime is missing. + +--- + ## Canonical assets reused (not duplicated) | Workload | Source in this repo | PCS-specific delta | From 98b8d9feca4b108bbe31a81b9d94e9b785a814b7 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 03:29:35 +0000 Subject: [PATCH 49/89] tests: record p6-b200 x4 NCCL all_reduce busbw (peak 377 GB/s over EFA, 32 GPU, Mumbai) --- architectures/aws-pcs/tests/README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index b01fa8a46..29a2eb534 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -75,6 +75,30 @@ deploys (deploy-all: prereqs + cluster + login + CPU, monitoring on) across: | **ap-northeast-1** | ✅ CREATE_COMPLETE | ✅ | ✅ | Tokyo | | **ap-south-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | Mumbai; p6-b200 ×4 GPU (8 GPU/node, 8 EFA NICs), B200 driver 595.71.05 | +### p6-b200 ×4 NCCL all_reduce (ap-south-1, 32 GPU) + +NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl-tests`, +4× p6-b200.48xlarge = 32 B200 GPUs, NCCL auto-selected NICs): + +- **NET/OFI provider: `efa`, fabric `efa-direct`, 8 NICs per node** (all EFA NICs in use) +- **Peak bus bandwidth: 377.4 GB/s** (16 GiB message, in/out-of-place) +- Avg bus bandwidth across the 8 B → 16 GiB sweep: 106 GB/s; 0 wrong / 0 out-of-bounds + +| Message size | busbw (GB/s) | +|---|---| +| 128 MiB | 295 | +| 256 MiB | 325 | +| 1 GiB | 367 | +| 4 GiB | 375 | +| 16 GiB | **377** | + +> Pre-merge gotcha hit here: the GPU nodes booted without Enroot/Pyxis (the +> `PostInstallScriptUrl` 404, see caveat below). After installing +> `install-enroot-pyxis.sh` manually on the **login node and all compute nodes** +> and `systemctl restart slurmd` on the compute nodes (so the freshly-dropped +> `spank_pyxis.so` loads), `--container-image` jobs ran. A correct +> `PostInstallScriptUrl` at deploy time avoids all of this. + Cross-region note: nested-stack `TemplateURL` and the in-instance `aws s3 cp` of boot scripts both work against an S3 bucket in a **different** region (S3 global namespace; no `--region` needed) — verified ap-south-1 → From be8d968499da1ea8880223005d6b9d85bd87e694 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 03:52:55 +0000 Subject: [PATCH 50/89] tests: record FSDP distributed-training readiness on 32 B200 GPUs (Mumbai) NCCL 32-GPU init + FSDP2 model wrap (0.41B) + optimizer succeed across 4x p6-b200; 8 GPUs/node confirmed. Training step loop blocked only by HuggingFace 429/shard-FileNotFoundError on streaming allenai/c4 (external rate limit, not a cluster issue). --- architectures/aws-pcs/tests/README.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 29a2eb534..8c33c8893 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -99,6 +99,25 @@ NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl- > `spank_pyxis.so` loads), `--container-image` jobs ran. A correct > `PostInstallScriptUrl` at deploy time avoids all of this. +### FSDP distributed training (ap-south-1, 32 B200 GPUs) + +FSDP Llama-3 1B (`3.test_cases/pytorch/FSDP`, venv on shared `/home`, 4× +p6-b200 = 32 GPUs, `torchrun --nnodes=4 --nproc_per_node=8`): + +- **Distributed init across 32 GPUs succeeds**: NCCL rendezvous (c10d on the + master node's resolved IP — Slurm node names like `gpu-b200-1` are not in DNS, + so the sbatch resolves `hostname -i`), FSDP2 model wrap (`410,143,488` params + / 0.41 B), and optimizer creation all complete on every rank. +- Each node exposes all **8 B200 GPUs** to its single `torchrun` (verified + `nvidia-smi -L` = 8 per node); `--ntasks-per-node=1` so torchrun (not srun) + spawns the 8 ranks per node. +- The training step loop is gated on streaming the `allenai/c4` dataset from + HuggingFace, which intermittently returns `429 Too Many Requests` / + shard `FileNotFoundError` from 32 concurrent readers — an **external HF + rate-limit**, not a cluster/template issue. The distributed-training stack + (the part the architecture is responsible for) is proven up to the data + pipeline; a cached/mirrored dataset or an HF token avoids the throttling. + Cross-region note: nested-stack `TemplateURL` and the in-instance `aws s3 cp` of boot scripts both work against an S3 bucket in a **different** region (S3 global namespace; no `--region` needed) — verified ap-south-1 → From 422ed2f14fdb8bf0721a9c7a3f5591d8b16d1063 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 04:06:31 +0000 Subject: [PATCH 51/89] fix: split PCS instance-role perms inline by feature; drop ManagedPolicy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two problems with the old MonitoringPolicy (AWS::IAM::ManagedPolicy, gated on IsMonitoringEnabled): 1. It required the DEPLOYER to have iam:CreatePolicy — which the cluster-admin policy intentionally does not grant — so a cluster-admin principal could not actually run deploy-all (the ideal is: admin policy ⇒ deploy-all works). The only AWS::IAM::ManagedPolicy in the entire nest was this one. 2. It bundled multi-user (SSM /pcs//ldap/*), S3 script fetch, and KMS-decrypt perms together with monitoring perms, all behind IsMonitoringEnabled. So MonitoringStack=none + DirectoryService=OpenLDAP-LoginNode left the instance WITHOUT permission to store the LDAP admin password in SSM (silently fell back to a file). Fix: instance-role perms are now inline AWS::IAM::Policy (PutRolePolicy, which the admin policy grants), split by feature: - PcsInstanceBasePolicy (always): pcs:RegisterComputeNodeGroupInstance, S3 GetObject on templates/scripts/*, SSM Get/Put on this cluster's grafana+ldap params, KMS Decrypt via SSM. Baseline every instance needs — independent of monitoring or directory. - MonitoringInstancePolicy (AWS::IAM::Policy, IsMonitoringEnabled only): just the monitoring reads (EC2/FSx describe, CFN describe, pricing, CloudWatch, logs, pcs:GetCluster). Net effect: deploy-all needs no iam:CreatePolicy (admin policy already covers CreateRole+PutRolePolicy), multi-user works regardless of MonitoringStack, and the admin policy stays at its current size (no CreatePolicy/policy-ARN additions). --- architectures/aws-pcs/assets/cluster.yaml | 83 ++++++++++++++--------- 1 file changed, 50 insertions(+), 33 deletions(-) diff --git a/architectures/aws-pcs/assets/cluster.yaml b/architectures/aws-pcs/assets/cluster.yaml index 12f9777cd..032830e1e 100644 --- a/architectures/aws-pcs/assets/cluster.yaml +++ b/architectures/aws-pcs/assets/cluster.yaml @@ -159,16 +159,55 @@ Resources: ManagedPolicyArns: - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" - !Sub "arn:${AWS::Partition}:iam::aws:policy/AmazonS3ReadOnlyAccess" + # Instance-role permissions are inline (PutRolePolicy), NOT a separate + # AWS::IAM::ManagedPolicy — so deploying this stack needs only + # iam:CreateRole + iam:PutRolePolicy (which the cluster-admin policy + # grants), never iam:CreatePolicy. Split by feature so monitoring and + # multi-user don't depend on each other: Policies: - - PolicyName: PcsRegisterInstancePolicy + # Always present (every cluster): register with PCS + read boot scripts + # from S3 + read this cluster's own SSM parameters (grafana password, + # ldap admin password) + decrypt SSM SecureStrings. None of this is + # monitoring- or directory-specific; it is the baseline an instance + # needs to register, fetch its post-install/directory scripts, and + # store/read the per-cluster secrets it generates. + - PolicyName: PcsInstanceBasePolicy PolicyDocument: Version: "2012-10-17" Statement: - - Effect: Allow + - Sid: PcsRegisterInstance + Effect: Allow Action: - pcs:RegisterComputeNodeGroupInstance Resource: "*" + - Sid: S3GetScripts + Effect: Allow + Action: + - s3:GetObject + Resource: + - arn:aws:s3:::awsome-distributed-ai/templates/scripts/* + - arn:aws:s3:::*/templates/scripts/* + + - Sid: SSMClusterParametersForThisClusterOnly + Effect: Allow + Action: + - ssm:GetParameter + - ssm:PutParameter + - ssm:AddTagsToResource + Resource: + - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/grafana/*' + - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/ldap/*' + + - Sid: KMSForSSMSecureString + Effect: Allow + Action: + - kms:Decrypt + Resource: '*' + Condition: + StringEquals: + kms:ViaService: !Sub 'ssm.${AWS::Region}.amazonaws.com' + # IAM Instance Profile PcsInstanceProfile: Type: AWS::IAM::InstanceProfile @@ -179,12 +218,17 @@ Resources: Roles: - !Ref PcsInstanceIamRole - # Monitoring IAM Policy (optional, attached to the instance role) - MonitoringPolicy: - Type: AWS::IAM::ManagedPolicy + # Monitoring instance permissions — inline policy added to the instance role + # only when monitoring is enabled. Inline (RoleName + PutRolePolicy via an + # AWS::IAM::Policy) instead of a ManagedPolicy, so the deployer never needs + # iam:CreatePolicy. Holds ONLY monitoring-specific reads (Prometheus service + # discovery + cost/CloudWatch metrics); multi-user/SSM/S3 perms live in the + # always-on base policy above. + MonitoringInstancePolicy: + Type: AWS::IAM::Policy Condition: IsMonitoringEnabled Properties: - ManagedPolicyName: !Sub '${PCSCluster.Id}-monitoring-policy' + PolicyName: !Sub '${PCSCluster.Id}-monitoring' Roles: - !Ref PcsInstanceIamRole PolicyDocument: @@ -225,33 +269,6 @@ Resources: - pricing:DescribeServices Resource: '*' - - Sid: SSMClusterParametersForThisClusterOnly - Effect: Allow - Action: - - ssm:GetParameter - - ssm:PutParameter - - ssm:AddTagsToResource - Resource: - - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/grafana/*' - - !Sub 'arn:${AWS::Partition}:ssm:${AWS::Region}:${AWS::AccountId}:parameter/pcs/${PCSCluster.Id}/ldap/*' - - - Sid: S3GetScripts - Effect: Allow - Action: - - s3:GetObject - Resource: - - arn:aws:s3:::awsome-distributed-ai/templates/scripts/* - - arn:aws:s3:::*/templates/scripts/* - - - Sid: KMSForSSMSecureString - Effect: Allow - Action: - - kms:Decrypt - Resource: '*' - Condition: - StringEquals: - kms:ViaService: !Sub 'ssm.${AWS::Region}.amazonaws.com' - - Sid: CloudWatchGetMetricData Effect: Allow Action: From fe49d0035d665c26c348dc61ebfdce51a1feecad Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 04:34:52 +0000 Subject: [PATCH 52/89] =?UTF-8?q?tests:=20refine=20FSDP=20result=20?= =?UTF-8?q?=E2=80=94=20distributed=20stack=20proven;=20c4=20streaming=20bl?= =?UTF-8?q?ocked=20by=20HF=20rate-limit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ran the canonical llama3_2_1b-training.sbatch unchanged. NCCL c10d rendezvous + FSDP2 wrap (1.15B) + optimizer succeed on 32 B200 GPUs (8/node). Training step loop unreachable purely due to external HF 429 on allenai/c4 streaming (tree API, 1024 shards) — confirmed external by exhausting token/cache/offline/ long-timeout/single-node mitigations. Cluster is training-ready (NCCL 377 GB/s); c4-at-scale needs HF higher-rate account / mirror / pre-tokenized /fsx data. --- architectures/aws-pcs/tests/README.md | 44 ++++++++++++++++----------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 8c33c8893..e44475b2b 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -99,24 +99,32 @@ NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl- > `spank_pyxis.so` loads), `--container-image` jobs ran. A correct > `PostInstallScriptUrl` at deploy time avoids all of this. -### FSDP distributed training (ap-south-1, 32 B200 GPUs) - -FSDP Llama-3 1B (`3.test_cases/pytorch/FSDP`, venv on shared `/home`, 4× -p6-b200 = 32 GPUs, `torchrun --nnodes=4 --nproc_per_node=8`): - -- **Distributed init across 32 GPUs succeeds**: NCCL rendezvous (c10d on the - master node's resolved IP — Slurm node names like `gpu-b200-1` are not in DNS, - so the sbatch resolves `hostname -i`), FSDP2 model wrap (`410,143,488` params - / 0.41 B), and optimizer creation all complete on every rank. -- Each node exposes all **8 B200 GPUs** to its single `torchrun` (verified - `nvidia-smi -L` = 8 per node); `--ntasks-per-node=1` so torchrun (not srun) - spawns the 8 ranks per node. -- The training step loop is gated on streaming the `allenai/c4` dataset from - HuggingFace, which intermittently returns `429 Too Many Requests` / - shard `FileNotFoundError` from 32 concurrent readers — an **external HF - rate-limit**, not a cluster/template issue. The distributed-training stack - (the part the architecture is responsible for) is proven up to the data - pipeline; a cached/mirrored dataset or an HF token avoids the throttling. +### FSDP distributed training (ap-south-1, p6-b200) + +Ran the repo's canonical `3.test_cases/pytorch/FSDP/slurm/llama3_2_1b-training.sbatch` +unchanged (venv per `create_venv.sh`, `srun … torchrun --nproc_per_node=8`, +`--dataset=allenai/c4`). + +- **The distributed-training stack the architecture is responsible for works.** + Across 4× p6-b200 (32 GPUs): NCCL c10d rendezvous, FSDP2 wrap of the Llama-3 + 1B model (`1,154,549,760` params), and optimizer creation complete on every + rank. Each node exposes all 8 B200 GPUs to its `torchrun` (`nvidia-smi -L` = 8/node). +- **The training step loop could not be reached because of an external + HuggingFace rate-limit on `allenai/c4`**, not a cluster/template issue. Every + rank's dataloader does `load_dataset("allenai/c4", streaming=True)`, which + hits the c4 *tree* API (1024-shard listing); HF returns `429 Too Many Requests`. + Confirmed it is purely external by exhausting the mitigations: + - HF token (authenticated) — still 429 + - shared `HF_HOME=/fsx` cache + single-process prefetch of shards — still 429 + (streaming re-lists shards via the tree API regardless of cache) + - `HF_HUB_OFFLINE=1` — flips to `OfflineModeIsEnabled` (streaming needs the Hub) + - long `HF_HUB_ETAG_TIMEOUT`/`DOWNLOAD_TIMEOUT` + the loader's 20× retry — still 429 + - dropping to a single node (8 concurrent readers instead of 32) — still 429 +- **Conclusion:** the cluster is training-ready (NCCL/EFA/FSDP all proven; NCCL + all_reduce hit 377 GB/s above). Driving `allenai/c4` through the streaming + loader at scale needs an HF account with a higher rate limit, an HF mirror, or + pre-tokenized data staged on `/fsx` (non-streaming) — an operational/data + choice, independent of these templates. Cross-region note: nested-stack `TemplateURL` and the in-instance `aws s3 cp` of boot scripts both work against an S3 bucket in a **different** From 5f5fbf6f7c75ad1f610fdc85af1c294856fe173d Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 06:01:47 +0000 Subject: [PATCH 53/89] docs(IAM): refresh verification matrix against major-update templates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-validated with a role attached to ONLY the admin (resp. user) managed policies on us-east-2: - admin-only role runs deploy-all (multi-user + SSH + monitoring) to full CREATE_COMPLETE with no AccessDenied, and needs NO iam:CreatePolicy (instance perms are now inline) — confirms "cluster-admin policy ⇒ deploy-all works". - user-only role: read-only allowed, all writes implicitDeny, login-node SSM session allowed but compute-node denied, grafana password readable but the new LDAP admin password (/pcs/*/ldap/*) NOT readable. - Documented that the admin policy intentionally omits s3:GetObject (prod bucket is public; private template buckets need it granted separately). --- architectures/aws-pcs/docs/IAM.md | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md index 060f2e699..018f88600 100644 --- a/architectures/aws-pcs/docs/IAM.md +++ b/architectures/aws-pcs/docs/IAM.md @@ -149,16 +149,23 @@ The same approach works for the user policy — exercise the user workflows ## Verification -These policies were validated end-to-end on a real account (us-east-2): +Re-validated end-to-end on a real account (us-east-2) against the major-update +templates — a dedicated role attached to **only** the admin (or user) managed +policies, nothing else: | Check | Result | |---|---| | Both IAM CFN stacks `CREATE_COMPLETE` | ✅ | -| Admin user deploys `pcs-ml-cluster-deploy-all.yaml` end-to-end (all nested stacks `CREATE_COMPLETE`, ~28 min) | ✅ | -| Admin user `UpdateStack` (param change → `UPDATE_COMPLETE`) and `DeleteStack` (full teardown) | ✅ | -| `iam:simulate-principal-policy` on `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` → all `allowed` | ✅ | -| User can `pcs:GetCluster`, `cloudformation:DescribeStacks`, read the Grafana password | ✅ | -| User `ssm:StartSession` on the login node (`Name=PCS-login`) `allowed`; on compute nodes `implicitDeny` | ✅ | -| User `cloudformation:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances` all `implicitDeny` | ✅ | -| CloudTrail over a full admin deploy+update+delete cycle: 0 `AccessDenied` across 332+ API calls | ✅ | -| `aws accessanalyzer validate-policy` on every embedded policy: 0 errors | ✅ | +| **A role with only the admin policies runs `pcs-ml-cluster-deploy-all.yaml`** (DirectoryService=OpenLDAP-LoginNode + SSHAccessCidr + MonitoringStack=Prometheus-LoginNode) — every nested stack `CREATE_COMPLETE`, no `AccessDenied` | ✅ | +| Admin role needs **no `iam:CreatePolicy`** — the instance role's perms are inline (`PutRolePolicy`), so deploy-all works with the admin policy as-is (`iam:CreatePolicy` simulates `implicitDeny`, and deploy-all still completes) | ✅ | +| `simulate-principal-policy` (with proper resource ARNs / `iam:PassedToService` context): `cfn:CreateStack`, `ec2:RunInstances/CreateSubnet/CreateSecurityGroup`, `fsx:CreateFileSystem`, `pcs:CreateCluster`, `iam:CreateRole/PutRolePolicy/CreateInstanceProfile/PassRole(ec2,pcs)`, `ssm:Put/GetParameter` on `/pcs/*/ldap/*`, `kms:Decrypt` → all `allowed` | ✅ | +| User role: `pcs:GetCluster`/`ListComputeNodeGroups`, `cloudformation:DescribeStacks`, `ec2:DescribeInstances` → `allowed`; `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` → `implicitDeny` | ✅ | +| User `ssm:StartSession` on login node (`Name=PCS-login`) `allowed`; on compute node (`Name=PCS-cpu1`) `implicitDeny` | ✅ | +| **Security: user reads `/pcs/*/grafana/*` (`allowed`) but NOT `/pcs/*/ldap/*`** (the OpenLDAP admin password) → `implicitDeny` | ✅ | + +> **Note on the template-source bucket.** The admin policy grants no `s3:GetObject`, +> because the production templates live in the public `awsome-distributed-ai` +> bucket (CFN fetches `--template-url` anonymously). If you host the templates in +> a **private** bucket, the deploying principal additionally needs `s3:GetObject` +> on that bucket — grant it separately; it is intentionally out of the +> cluster-admin policy. From 04dc0b2f339f3e6afc05b851f294bd6947298213 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 07:15:59 +0000 Subject: [PATCH 54/89] tests: record EFA OSU real-traffic numbers (hpc8a: osu_bw 26.3 GB/s, osu_mbw_mr 42.9 GB/s) + fix major-update summary --- architectures/aws-pcs/tests/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index e44475b2b..401ef4c4f 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -35,9 +35,9 @@ one cluster. Tests 8-13 are separate paths run only when their inputs change. ## Major-update PR — configurations run end-to-end on real hardware The major-update PR (IAM policies, multi-user OpenLDAP, `MonitoringStack` rename, -`SSHAccessCidr`/`GrafanaAccessCidr`, multi-AZ subnets, VPCName auto-derive + -EFA auto-count) was validated end-to-end in us-east-2 with a single -`deploy-all` cluster: +`SSHAccessCidr`/`GrafanaAccessCidr`, multi-AZ subnets, `OnDemandEfaInterfaceCount` +0/1/2 collapse, VPCName fixed to `${StackName}-VPC`, instance-role perms split +inline) was validated end-to-end in us-east-2 with a single `deploy-all` cluster: ``` PrimarySubnetAZ=us-east-2b AdditionalSubnetAZ2=us-east-2a AdditionalSubnetAZ3=us-east-2c @@ -57,6 +57,7 @@ MonitoringStack=Prometheus-LoginNode | **Multi-node UID consistency** | two different compute nodes both resolve `testuser1`→`uid=10001` | ✅ | | **User delete propagation** | `ldapdelete` + `sss_cache -E` removes the user from `getent` immediately (sssd-tools now installed) | ✅ | | **Template lint** | `validate-template` passes on all 8 edited `assets/*.yaml` (incl. 3 GPU templates + 2 IAM stacks) | ✅ | +| **`OnDemandEfaInterfaceCount` (0/1/2 collapse)** | `=1` → hpc6a launch template has 1 EFA NIC; `=2` → hpc8a has 2 EFA NICs (`DeviceIndex 0/1`, `InterfaceType: efa`). Real EFA traffic on hpc8a (2 nodes, `FI_PROVIDER=efa`): `osu_bw` peak **26.3 GB/s** (~210 Gbps, 1 pair); `osu_mbw_mr` peak **42.9 GB/s** (~343 Gbps, 16 pairs/node multi-rail) | ✅ | | **Slurm managed accounting + multi-user (Test 12)** | on a `ManagedAccounting=enabled` cluster: LDAP users alice/bob registered in `sacctmgr` (account=ml-team) as root admin; jobs submitted as each LDAP user complete and `sacct -a` records them under the correct `User`+`Account` (`scontrol show job`: `UserId=alice(10001) Account=ml-team`) | ✅ | --- From 6318d4de6b4e455f476bff518989c512d106b57a Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 07:23:27 +0000 Subject: [PATCH 55/89] docs(IAM): record admin-only-role delete-stack teardown (DELETE_COMPLETE, no AccessDenied) --- architectures/aws-pcs/docs/IAM.md | 1 + 1 file changed, 1 insertion(+) diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md index 018f88600..499672475 100644 --- a/architectures/aws-pcs/docs/IAM.md +++ b/architectures/aws-pcs/docs/IAM.md @@ -157,6 +157,7 @@ policies, nothing else: |---|---| | Both IAM CFN stacks `CREATE_COMPLETE` | ✅ | | **A role with only the admin policies runs `pcs-ml-cluster-deploy-all.yaml`** (DirectoryService=OpenLDAP-LoginNode + SSHAccessCidr + MonitoringStack=Prometheus-LoginNode) — every nested stack `CREATE_COMPLETE`, no `AccessDenied` | ✅ | +| Same admin-only role tears it down: `delete-stack` → `DELETE_COMPLETE` with no `AccessDenied` (delete perms cover the full teardown) | ✅ | | Admin role needs **no `iam:CreatePolicy`** — the instance role's perms are inline (`PutRolePolicy`), so deploy-all works with the admin policy as-is (`iam:CreatePolicy` simulates `implicitDeny`, and deploy-all still completes) | ✅ | | `simulate-principal-policy` (with proper resource ARNs / `iam:PassedToService` context): `cfn:CreateStack`, `ec2:RunInstances/CreateSubnet/CreateSecurityGroup`, `fsx:CreateFileSystem`, `pcs:CreateCluster`, `iam:CreateRole/PutRolePolicy/CreateInstanceProfile/PassRole(ec2,pcs)`, `ssm:Put/GetParameter` on `/pcs/*/ldap/*`, `kms:Decrypt` → all `allowed` | ✅ | | User role: `pcs:GetCluster`/`ListComputeNodeGroups`, `cloudformation:DescribeStacks`, `ec2:DescribeInstances` → `allowed`; `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` → `implicitDeny` | ✅ | From c2a325e8fb7944d24b1ffb5cfb4348257ce1509f Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 07:24:14 +0000 Subject: [PATCH 56/89] tests: record GPU health check 4/4 PASS on B200 (Mumbai p6-b200) --- architectures/aws-pcs/tests/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 401ef4c4f..f8085044f 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -22,6 +22,11 @@ Run this **complete set** before merging template or script changes. | 11-12 | **Multi-user** | OpenLDAP directory + Slurm managed accounting | [`multi-user-test.md`](./multi-user-test.md) | Directory / accounting changes | | 13 | **GPU health** | GPU Cluster Health Check suite (DCGM, EFA, NVLink, NCCL thresholds) | [`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md) | GPU CNG deploys | +> **GPU health check verified on B200** (ap-south-1 p6-b200): lightweight suite +> **4/4 PASS** — nvidia-smi (8 GPUs, no Xid/ECC), DCGM L2 diagnostics, EFA +> enumeration (8 PCI / 10 RDMA devices), topology (8 GPUs, connectivity validated; +> a B200-specific "unsupported P2P path" warning is non-blocking). + --- ## Quick-start: single-cluster shortcut From d32b60622f8af11e56fcad0d8ad664a14143c4c5 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 07:25:31 +0000 Subject: [PATCH 57/89] tests: list all 18 PCS launch regions in coverage matrix (5 tested, 13 not-run) --- architectures/aws-pcs/tests/README.md | 31 +++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index f8085044f..caf8ce835 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -81,6 +81,37 @@ deploys (deploy-all: prereqs + cluster + login + CPU, monitoring on) across: | **ap-northeast-1** | ✅ CREATE_COMPLETE | ✅ | ✅ | Tokyo | | **ap-south-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | Mumbai; p6-b200 ×4 GPU (8 GPU/node, 8 EFA NICs), B200 driver 595.71.05 | +### All PCS launch regions + +AWS PCS is available in the 18 regions below (per +`/aws/service/global-infrastructure/services/pcs/regions`). The 5 marked +**tested** above were validated end-to-end; the rest are expected to work (the +templates resolve the PCS-Ready DLAMI from SSM and FSx deployment types per +region) but have not been run. Confirm `LustreDeploymentType` / +`OpenZFSDeploymentType` availability in a new region before relying on the +defaults (`PERSISTENT_2` / `SINGLE_AZ_HA_2`). + +| Region | Status | +|---|---| +| us-east-1 (N. Virginia) | ✅ tested | +| us-east-2 (Ohio) | ✅ tested | +| us-west-2 (Oregon) | ✅ tested | +| ap-northeast-1 (Tokyo) | ✅ tested | +| ap-south-1 (Mumbai) | ✅ tested (GPU/B200) | +| ap-northeast-3 (Osaka) | ⬜ not run | +| ap-southeast-1 (Singapore) | ⬜ not run | +| ap-southeast-2 (Sydney) | ⬜ not run | +| eu-central-1 (Frankfurt) | ⬜ not run | +| eu-north-1 (Stockholm) | ⬜ not run | +| eu-south-1 (Milan) | ⬜ not run | +| eu-south-2 (Spain) | ⬜ not run | +| eu-west-1 (Ireland) | ⬜ not run | +| eu-west-2 (London) | ⬜ not run | +| eu-west-3 (Paris) | ⬜ not run | +| sa-east-1 (São Paulo) | ⬜ not run | +| us-gov-east-1 (GovCloud East) | ⬜ not run | +| us-gov-west-1 (GovCloud West) | ⬜ not run | + ### p6-b200 ×4 NCCL all_reduce (ap-south-1, 32 GPU) NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl-tests`, From a4bcf89061043b54a7b0864923e6664bafbc79d3 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 07:58:57 +0000 Subject: [PATCH 58/89] =?UTF-8?q?feat:=20PostInstallScriptUrl=20accepts=20?= =?UTF-8?q?s3://=20(instance-role=20fetch)=20=E2=80=94=20works=20with=20pr?= =?UTF-8?q?ivate=20buckets?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Dev accounts can't use public S3, and the post-install hook fetched its script with curl (anonymous HTTP) — so a private-bucket S3 https URL 403'd and Enroot/Pyxis silently didn't install. Now the runcmd dispatches by scheme: s3://* → aws s3 cp (instance role; private bucket OK) http(s) → curl (public only; GitHub raw etc.) in add-cng.yaml + all 3 GPU templates. deploy-all: PostInstallScriptUrl default is now empty and, when empty, forwards s3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh — so the default install path uses the same bucket (and same IAM-authenticated fetch) as setup-directory.sh. This unifies all first-party boot scripts on aws-s3-cp (private-bucket friendly); only the monitoring post-install.sh stays on GitHub raw, since it pulls from the external aws-parallelcluster-monitoring repo. Updated the param descriptions (4 templates) and PARAMETERS.md: dropped the stale "S3 hosting is not allowed" / "empty = skip" wording; empty now = auto Enroot/Pyxis from the templates bucket, single space = skip. Also: gpu-healthcheck-test.md trimmed to PCS-specific deltas (defer suite mechanics to the suite's own README, per the reuse-don't-duplicate convention). --- architectures/aws-pcs/assets/add-cng-p5.yaml | 7 +- .../aws-pcs/assets/add-cng-p6-b200.yaml | 7 +- .../aws-pcs/assets/add-cng-p6-b300.yaml | 7 +- architectures/aws-pcs/assets/add-cng.yaml | 17 +- .../assets/pcs-ml-cluster-deploy-all.yaml | 46 +++-- architectures/aws-pcs/docs/PARAMETERS.md | 2 +- .../aws-pcs/tests/gpu-healthcheck-test.md | 160 +++++------------- 7 files changed, 110 insertions(+), 136 deletions(-) diff --git a/architectures/aws-pcs/assets/add-cng-p5.yaml b/architectures/aws-pcs/assets/add-cng-p5.yaml index 2f1c2701c..292ebc080 100644 --- a/architectures/aws-pcs/assets/add-cng-p5.yaml +++ b/architectures/aws-pcs/assets/add-cng-p5.yaml @@ -204,7 +204,7 @@ Parameters: Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. - Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. + Accepts an s3:// URL (instance-role fetch; private bucket OK) or an http(s):// URL (curl; public only). For Enroot/Pyxis point at scripts/install-enroot-pyxis.sh in the templates bucket. Default: '' PostInstallScriptArgs: @@ -410,7 +410,10 @@ Resources: - | if [ -n "${PostInstallScriptUrl}" ]; then echo "Downloading post-install script from ${PostInstallScriptUrl}..." | tee /var/log/pcs-post-install.log - curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh + case "${PostInstallScriptUrl}" in + s3://*) aws s3 cp "${PostInstallScriptUrl}" /tmp/pcs-post-install.sh >> /var/log/pcs-post-install.log 2>&1 ;; + *) curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh ;; + esac chmod +x /tmp/pcs-post-install.sh echo "Executing post-install script..." | tee -a /var/log/pcs-post-install.log # Tell the script this cluster's Slurm version (it can't discover it at diff --git a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml index 856d1d9de..132943b4c 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b200.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b200.yaml @@ -195,7 +195,7 @@ Parameters: Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. - Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. + Accepts an s3:// URL (instance-role fetch; private bucket OK) or an http(s):// URL (curl; public only). For Enroot/Pyxis point at scripts/install-enroot-pyxis.sh in the templates bucket. Default: '' PostInstallScriptArgs: @@ -397,7 +397,10 @@ Resources: - | if [ -n "${PostInstallScriptUrl}" ]; then echo "Downloading post-install script from ${PostInstallScriptUrl}..." | tee /var/log/pcs-post-install.log - curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh + case "${PostInstallScriptUrl}" in + s3://*) aws s3 cp "${PostInstallScriptUrl}" /tmp/pcs-post-install.sh >> /var/log/pcs-post-install.log 2>&1 ;; + *) curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh ;; + esac chmod +x /tmp/pcs-post-install.sh echo "Executing post-install script..." | tee -a /var/log/pcs-post-install.log # Tell the script this cluster's Slurm version (it can't discover it at diff --git a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml index 5d280ce71..b4672b28c 100644 --- a/architectures/aws-pcs/assets/add-cng-p6-b300.yaml +++ b/architectures/aws-pcs/assets/add-cng-p6-b300.yaml @@ -198,7 +198,7 @@ Parameters: Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. - Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. + Accepts an s3:// URL (instance-role fetch; private bucket OK) or an http(s):// URL (curl; public only). For Enroot/Pyxis point at scripts/install-enroot-pyxis.sh in the templates bucket. Default: '' PostInstallScriptArgs: @@ -400,7 +400,10 @@ Resources: - | if [ -n "${PostInstallScriptUrl}" ]; then echo "Downloading post-install script from ${PostInstallScriptUrl}..." | tee /var/log/pcs-post-install.log - curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh + case "${PostInstallScriptUrl}" in + s3://*) aws s3 cp "${PostInstallScriptUrl}" /tmp/pcs-post-install.sh >> /var/log/pcs-post-install.log 2>&1 ;; + *) curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh ;; + esac chmod +x /tmp/pcs-post-install.sh echo "Executing post-install script..." | tee -a /var/log/pcs-post-install.log # Tell the script this cluster's Slurm version (it can't discover it at diff --git a/architectures/aws-pcs/assets/add-cng.yaml b/architectures/aws-pcs/assets/add-cng.yaml index b8bafe581..963910326 100644 --- a/architectures/aws-pcs/assets/add-cng.yaml +++ b/architectures/aws-pcs/assets/add-cng.yaml @@ -194,8 +194,11 @@ Parameters: Description: >- Optional URL of a post-install script to download and run on each node at first boot (PCS equivalent of ParallelCluster OnNodeConfigured). Leave - empty to skip. To install Enroot/Pyxis, point this at assets/scripts/install-enroot-pyxis.sh. - Must be an HTTP(S) URL (e.g. a GitHub raw URL); S3 public hosting is not allowed. + empty to skip. Accepts an s3:// URL (fetched with the instance role — works + with a private bucket) or an http(s):// URL (fetched with curl — public + only, e.g. a GitHub raw URL). To install Enroot/Pyxis, point this at + scripts/install-enroot-pyxis.sh in your templates bucket + (s3:///scripts/install-enroot-pyxis.sh). Default: '' PostInstallScriptArgs: @@ -463,10 +466,18 @@ Resources: runcmd: # Post-install script (optional) - generic OnNodeConfigured-style hook. # Downloads and runs a user-supplied script (e.g. Enroot/Pyxis install). + # Accepts BOTH an s3:// URL (fetched with `aws s3 cp` using the instance + # role — works with a PRIVATE bucket, the default for Enroot/Pyxis) and an + # http(s):// URL (fetched anonymously with curl — for public GitHub raw or + # any user-hosted script). Pick by scheme so dev accounts without public S3 + # still install Enroot/Pyxis from their own private templates bucket. - | if [ -n "${PostInstallScriptUrl}" ]; then echo "Downloading post-install script from ${PostInstallScriptUrl}..." | tee /var/log/pcs-post-install.log - curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh + case "${PostInstallScriptUrl}" in + s3://*) aws s3 cp "${PostInstallScriptUrl}" /tmp/pcs-post-install.sh >> /var/log/pcs-post-install.log 2>&1 ;; + *) curl -fsSL "${PostInstallScriptUrl}" -o /tmp/pcs-post-install.sh ;; + esac chmod +x /tmp/pcs-post-install.sh echo "Executing post-install script..." | tee -a /var/log/pcs-post-install.log # Tell the script this cluster's Slurm version (it can't discover it at diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 4027dcb08..09d5b1c05 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -352,13 +352,15 @@ Parameters: PostInstallScriptUrl: Type: String Description: >- - HTTP(S) URL of a script run on every node at first boot (PCS equivalent of - ParallelCluster OnNodeConfigured). Default installs Enroot/Pyxis. Set empty - to skip, or point at any other HTTP(S) script for custom first-boot setup. - Idempotent: a no-op if Enroot/Pyxis is already pre-baked into AmiId, so - leaving the default works whether or not you supply a custom AMI. - S3 hosting is not allowed. - Default: 'https://raw.githubusercontent.com/awslabs/awsome-distributed-ai/main/architectures/aws-pcs/assets/scripts/install-enroot-pyxis.sh' + URL of a script run on every node at first boot (PCS equivalent of + ParallelCluster OnNodeConfigured). Accepts an s3:// URL (fetched with the + instance role — works with a private bucket) or an http(s):// URL (fetched + with curl — public only). Leave EMPTY (default) to auto-install Enroot/Pyxis + from the templates bucket (s3:///scripts/install-enroot-pyxis.sh), + so it works in dev accounts without public S3. Set to your own s3://or + https:// script for custom first-boot setup, or to a single space to skip. + Idempotent: a no-op if Enroot/Pyxis is already pre-baked into AmiId. + Default: '' PostInstallScriptArgs: Type: String @@ -631,6 +633,11 @@ Rules: Conditions: UseLocalTemplates: !Equals [!Ref S3BucketName, 'local'] + # When PostInstallScriptUrl is left empty, default it to the Enroot/Pyxis + # installer in the templates bucket (s3://, fetched via the instance role — + # works with a private bucket). A non-empty value (s3:// or https://) is used + # verbatim; a single space disables the post-install hook. + PostInstallScriptUrlIsDefault: !Equals [!Ref PostInstallScriptUrl, ''] DeployOnDemandCNGEnabled: !Equals [!Ref DeployOnDemandCNG, 'true'] DirectoryEnabled: !Not [!Equals [!Ref DirectoryService, 'none']] DeployPseriesCNGEnabled: !Equals [!Ref DeployPseriesCNG, 'true'] @@ -775,7 +782,10 @@ Resources: DcgmExporterImage: !Ref DcgmExporterImage SlurmVersion: !Ref SlurmVersion MonitoringRole: login - PostInstallScriptUrl: !Ref PostInstallScriptUrl + PostInstallScriptUrl: !If + - PostInstallScriptUrlIsDefault + - !Sub 's3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh' + - !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'server', 'none'] @@ -820,7 +830,10 @@ Resources: DcgmExporterImage: !Ref DcgmExporterImage SlurmVersion: !Ref SlurmVersion MonitoringRole: compute - PostInstallScriptUrl: !Ref PostInstallScriptUrl + PostInstallScriptUrl: !If + - PostInstallScriptUrlIsDefault + - !Sub 's3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh' + - !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs # EFA on CPU HPC instances (hpc8a/hpc7a/hpc6a). add-cng.yaml enables EFA # when EfaInterfaceCount > 0 and auto-creates a per-CNG cluster placement @@ -871,7 +884,10 @@ Resources: DcgmExporterImage: !Ref DcgmExporterImage SlurmVersion: !Ref SlurmVersion MonitoringRole: compute - PostInstallScriptUrl: !Ref PostInstallScriptUrl + PostInstallScriptUrl: !If + - PostInstallScriptUrlIsDefault + - !Sub 's3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh' + - !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] @@ -917,7 +933,10 @@ Resources: DcgmExporterImage: !Ref DcgmExporterImage SlurmVersion: !Ref SlurmVersion MonitoringRole: compute - PostInstallScriptUrl: !Ref PostInstallScriptUrl + PostInstallScriptUrl: !If + - PostInstallScriptUrlIsDefault + - !Sub 's3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh' + - !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] @@ -963,7 +982,10 @@ Resources: DcgmExporterImage: !Ref DcgmExporterImage SlurmVersion: !Ref SlurmVersion MonitoringRole: compute - PostInstallScriptUrl: !Ref PostInstallScriptUrl + PostInstallScriptUrl: !If + - PostInstallScriptUrlIsDefault + - !Sub 's3://${S3BucketName}/${S3KeyPrefix}scripts/install-enroot-pyxis.sh' + - !Ref PostInstallScriptUrl PostInstallScriptArgs: !Ref PostInstallScriptArgs DirectoryService: !Ref DirectoryService DirectoryRole: !If [DirectoryEnabled, 'client', 'none'] diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 347830b34..0437d811e 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -53,7 +53,7 @@ it directly. | Parameter | Default | Purpose | |---|---|---| -| `PostInstallScriptUrl` | Enroot/Pyxis installer | HTTP(S) script run on every node at first boot (PCS equivalent of ParallelCluster `OnNodeConfigured`). Empty = skip; or override with any other HTTP(S) script. Idempotent: a no-op if Enroot/Pyxis is already pre-baked into `AmiId` | +| `PostInstallScriptUrl` | *(empty → auto)* | Script run on every node at first boot (PCS equivalent of ParallelCluster `OnNodeConfigured`). **Empty (default) auto-installs Enroot/Pyxis** from `s3:///scripts/install-enroot-pyxis.sh` (fetched with the instance role, so it works with a **private** bucket — no public S3 needed). Accepts an `s3://` URL (instance-role fetch) or an `http(s)://` URL (curl, public only, e.g. GitHub raw). Set to a single space to skip. Idempotent: a no-op if Enroot/Pyxis is already pre-baked into `AmiId` | | `PostInstallScriptArgs` | *(empty)* | Arguments passed to the post-install script | ## 4. On-Demand Compute Node Group (CPU) diff --git a/architectures/aws-pcs/tests/gpu-healthcheck-test.md b/architectures/aws-pcs/tests/gpu-healthcheck-test.md index 64d4c7eb8..f87d495f8 100644 --- a/architectures/aws-pcs/tests/gpu-healthcheck-test.md +++ b/architectures/aws-pcs/tests/gpu-healthcheck-test.md @@ -1,128 +1,60 @@ # Test: GPU Cluster Health Check -Validates GPU hardware, EFA networking, and NVLink interconnects using the -[GPU Cluster Health Check Suite](../../4.validation_and_observability/2.gpu-cluster-healthcheck) -from this repository. +Validates GPU hardware, EFA, and NVLink on a PCS GPU node group using the +repository's [GPU Cluster Health Check Suite](../../../4.validation_and_observability/2.gpu-cluster-healthcheck). -**Prerequisites**: -- Cluster with a GPU CNG deployed (P5, P5e, P5en, or P6-B200) -- At least 2 GPU nodes for multi-node NCCL check -- The health check scripts fetched to `/fsx` (shared storage) +**This page documents only the PCS-specific deltas.** For the suite itself — what +each check does, the severity classification (PASS / MONITOR / REBOOT / ISOLATE), +the lightweight vs intensive check levels, instance profiles, and the per-check +reference — see the suite's own +[README](../../../4.validation_and_observability/2.gpu-cluster-healthcheck/README.md). +Don't duplicate that here. ---- - -## Setup - -```bash -# Fetch the health check suite to shared /fsx -cd /fsx -git clone https://github.com/awslabs/awsome-distributed-ai.git --depth 1 --sparse -cd awsome-distributed-ai -git sparse-checkout set 4.validation_and_observability/2.gpu-cluster-healthcheck -cd 4.validation_and_observability/2.gpu-cluster-healthcheck -``` - -Or simply copy the directory from the repo checkout if already available. - ---- - -## Part A — Lightweight suite (single-node, ~15 min) - -Run on each GPU node to validate hardware is healthy before submitting -training jobs: - -```bash -export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH - -# Single node -srun -N 1 -n 1 -p gpu-p5 --exclusive \ - bash /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/gpu-healthcheck.sh \ - --suite lightweight -``` - -Or via sbatch for multiple nodes: -```bash -sbatch -N 2 -p gpu-p5 --exclusive \ - /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/slurm/sbatch-lightweight.sh -``` - -### Expected results (per node) - -| Check | What it validates | Pass criteria | -|---|---|---| -| 0: nvidia-smi | GPU count, driver, Xid errors | All GPUs detected, no Xid errors in dmesg | -| 1: DCGM L2 | PCIe BW, memory stress, SM stress | All DCGM sub-tests PASS | -| 2: EFA enumeration | EFA device count, RDMA, libfabric | EFA count matches instance profile (32 for p5, 16 for p5en, 8 for p6-b200) | -| 3: Topology | NVLink status, PCIe groups | All NVLinks UP, no uncorrectable errors | - -### Severity output - -Results are classified as: -- **PASS** — all checks green -- **MONITOR** — minor anomaly, non-blocking -- **REBOOT** — correctable with instance reboot -- **ISOLATE** — hardware fault, instance should be replaced - ---- - -## Part B — NCCL multi-node (check 5, ~10-20 min) - -Validates multi-node GPU communication over EFA. Overlaps with Test 6 (NCCL) -but adds per-instance bandwidth threshold validation: - -```bash -srun -N 2 -n 2 -p gpu-p5 --exclusive \ - bash /fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/gpu-healthcheck.sh \ - --check 5 -``` - -### Expected - -- EFA provider selected (`Selected Provider is efa`) -- Bus bandwidth meets instance-type threshold (e.g. p5: >700 GB/s peak for 2-node all_reduce at 128MB) -- No NCCL timeouts or EFA transport errors +**Prerequisites:** a deployed GPU CNG (P5/P5e/P5en/P6-B200/P6-B300) and the suite +available on shared `/fsx` (so every compute node sees the same scripts). --- -## Part C — Slurm prolog integration (production use) - -For production clusters, configure the lightweight checks as a Slurm prolog -so every GPU job validates hardware before running: - -```bash -# In slurm.conf (via PCS custom settings): -Prolog=/fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck/slurm/prolog-gpu-healthcheck.sh -``` - -A non-zero prolog exit drains the node and requeues the job. +## PCS-specific deltas + +1. **Stage the suite on `/fsx`** (shared) so all GPU nodes run the same copy: + ```bash + cd /fsx && git clone https://github.com/awslabs/awsome-distributed-ai.git --depth 1 + HC=/fsx/awsome-distributed-ai/4.validation_and_observability/2.gpu-cluster-healthcheck + ``` + +2. **Drive it through the PCS Slurm queue** — use `srun`/`sbatch` against your GPU + partition name (e.g. `gpu-p5`, `gpu-b200`), not the suite's bare host invocation: + ```bash + export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH + # lightweight suite (checks 0-3) on one GPU node + srun -p -N1 -n1 --exclusive bash $HC/gpu-healthcheck.sh --suite lightweight + # multi-node NCCL (check 5) — runs inside the Slurm allocation + srun -p -N2 -n2 --exclusive bash $HC/gpu-healthcheck.sh --check 5 + ``` + The suite's `slurm/` directory also ships `sbatch` wrappers and a + `prolog-gpu-healthcheck.sh` you can wire into PCS as a Slurm prolog (see the + suite README's Slurm section) — no PCS-specific change needed there. + +3. **When to run on a PCS cluster:** after any GPU CNG deploy or instance + replacement (lightweight), and before long multi-day training runs + (lightweight + check 5). Check 5 overlaps with [Test 6 NCCL](./compute-test.md) + but adds the suite's per-instance bandwidth thresholds. --- -## When to run - -| Scenario | Suite | Frequency | -|---|---|---| -| After initial GPU CNG deployment | Lightweight (full) | Once | -| Before multi-day training runs | Lightweight + check 5 | Per-run | -| Production steady-state | Prolog (checks 0, 2) | Every job | -| Node suspected faulty | Intensive (all checks) | On-demand | -| After instance replacement | Lightweight | Once | - ---- +## Verified on real hardware -## Verdict checklist +Lightweight suite (checks 0-3) on **p6-b200** (ap-south-1, via `srun -p gpu`): +**4/4 PASS** — | Check | Result | |---|---| -| Lightweight suite completes on all GPU nodes | ✅ | -| No ISOLATE/REBOOT severity results | ✅ | -| EFA device count matches instance profile | ✅ | -| NVLink all UP, no uncorrectable errors | ✅ | -| Multi-node NCCL bandwidth meets threshold | ✅ | - ---- - -## Reference - -- [GPU Cluster Health Check Suite README](../../4.validation_and_observability/2.gpu-cluster-healthcheck/README.md) -- Supported instance profiles: see `instance-profiles.conf` in the suite +| 0 nvidia-smi | PASS — 8 GPUs, no Xid/SXid, no ECC/retired-page errors | +| 1 DCGM L2 | PASS — all Level-2 diagnostics | +| 2 EFA enumeration | PASS — 8 EFA PCI / 10 RDMA devices (`fi_info` warning is non-blocking: libfabric lives in the container, not on the host) | +| 3 Topology | PASS — 8 GPUs, connectivity validated (a B200 "unsupported P2P path" warning is expected and non-blocking) | + +EFA device count matching the instance profile and the multi-node NCCL bandwidth +threshold are also covered by [compute-test.md](./compute-test.md) (NCCL +all_reduce hit 377 GB/s on 4× p6-b200). From d4678c9029c507a35e676417f817828bd2143ce5 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 08:17:37 +0000 Subject: [PATCH 59/89] docs: fix stale 'PostInstallScriptUrl empty = skip' wording + DcgmExporterImage anchor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After PostInstallScriptUrl's default flipped to empty=auto-install (s3://), several docs still said empty = skip / cleanest boot. Corrected to: empty = auto-install Enroot/Pyxis from the templates bucket (idempotent no-op on a pre-baked AMI); a single space = skip. Fixed: README §8.4 (was internally contradictory — default IS empty), OPERATIONS.md pre-baked-AMI note, tests/infra-test.md (4 spots incl. the "log absent" assertion that would have failed with empty). Also fixed PARAMETERS.md DcgmExporterImage anchor (single→double hyphen for the em-dash heading slug). Found via a full docs-consistency re-audit of the PostInstallScriptUrl/VPCName/EFA/MonitoringStack/IAM changes. --- architectures/aws-pcs/README.md | 14 ++++++++------ architectures/aws-pcs/docs/OPERATIONS.md | 8 +++++--- architectures/aws-pcs/docs/PARAMETERS.md | 2 +- architectures/aws-pcs/tests/infra-test.md | 10 +++++----- 4 files changed, 19 insertions(+), 15 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index c0973a268..63a4cb6ff 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -507,8 +507,9 @@ AMI_ID=$(aws cloudformation describe-stacks \ echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx ``` -**Step 3: Pass it to the cluster** as `AmiId` and clear `PostInstallScriptUrl` for -the cleanest boot: +**Step 3: Pass it to the cluster** as `AmiId`. Optionally skip the boot-time +Enroot/Pyxis install (it's already baked in) by setting `PostInstallScriptUrl` +to a single space: ```bash aws cloudformation create-stack \ @@ -517,13 +518,14 @@ aws cloudformation create-stack \ --parameters \ ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-1a \ ParameterKey=AmiId,ParameterValue=$AMI_ID \ - ParameterKey=PostInstallScriptUrl,ParameterValue= \ + ParameterKey=PostInstallScriptUrl,ParameterValue=' ' \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM ``` -The post-install hook is idempotent, so leaving `PostInstallScriptUrl` at its default -also works — the installer detects Enroot/Pyxis is already present and is a fast -no-op. Setting it empty just shaves a few seconds off boot. +Leaving `PostInstallScriptUrl` at its default (empty → auto-install Enroot/Pyxis +from the templates bucket) also works on a pre-baked AMI: the installer detects +Enroot/Pyxis is already present and is a fast idempotent no-op. Passing a single +space skips the download+check entirely, shaving a few seconds off boot. **Optional features of `pcs-ready-dlami-with-enroot-pyxis.yaml`** (defaults are off): - `BuildSchedule=Weekly`/`Monthly` for scheduled rebuilds against a moving base AMI diff --git a/architectures/aws-pcs/docs/OPERATIONS.md b/architectures/aws-pcs/docs/OPERATIONS.md index 06ea5a9a7..3ba90df9a 100644 --- a/architectures/aws-pcs/docs/OPERATIONS.md +++ b/architectures/aws-pcs/docs/OPERATIONS.md @@ -45,9 +45,11 @@ as a separate stack, then pass its output to the cluster. the default `PostInstallScriptUrl` delivers. - **Pre-baked AMI** — build the AMI separately (see the README's *Pre-baking Enroot/Pyxis into a custom AMI* section), then pass its `ami-xxx` as - the cluster's `AmiId`. Use `PostInstallScriptUrl=""` for the cleanest boot (the - installer is idempotent so leaving the default is a fast no-op, but skipping the - download saves a few seconds on every node launch). + the cluster's `AmiId`. Set `PostInstallScriptUrl=' '` (a single space) for the + cleanest boot — that skips the Enroot/Pyxis install entirely. (Leaving it at the + default — empty, which auto-installs from the templates bucket — also works on a + pre-baked AMI: the installer is idempotent and detects Enroot/Pyxis is already + present, a fast no-op; the single space just avoids the download+check.) ### 2.1 The AMI is single-Slurm-version, by design diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 0437d811e..0b6d5c011 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -107,4 +107,4 @@ See [Storage: FSx deployment types](../README.md#storage-fsx-deployment-types-re | `S3KeyPrefix` | `templates/` | S3 key prefix for the nested templates | | `MonitoringVersion` | `v2.9.1` | [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`). `v2.9.1` adds the `DCGM_EXPORTER_IMAGE` override (needed for B300 GPU metrics) and brings Grafana 13; `v2.6.4`+ carry the PCS `/opt` install + Docker-29.x DCGM fixes. Pin to a tag for stability. Migration notes: [OPERATIONS.md §3](./OPERATIONS.md#3-monitoring-monitoringversion) | | `MonitoringRepo` | `aws-samples/aws-parallelcluster-monitoring` | GitHub `owner/repo` for the monitoring stack; override with a fork + a branch in `MonitoringVersion` to test unreleased changes | -| `DcgmExporterImage` | DCGM 4.5.2 by digest | `dcgm-exporter` image used on GPU nodes. Defaults to a DCGM 4.5.2 build pinned by digest (`nvcr.io/nvidia/k8s/dcgm-exporter@sha256:a7ad6547...`) covering Hopper / B200 / B300. The digest pull bypasses the Docker-29.x OCI-index failure on newer NVCR tags. Override (any image reference, ideally also a digest) to pin to a different build — e.g. the monitoring stack's older default 4.2.0. No effect on CPU nodes. See [OPERATIONS.md §3.1](./OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it) | +| `DcgmExporterImage` | DCGM 4.5.2 by digest | `dcgm-exporter` image used on GPU nodes. Defaults to a DCGM 4.5.2 build pinned by digest (`nvcr.io/nvidia/k8s/dcgm-exporter@sha256:a7ad6547...`) covering Hopper / B200 / B300. The digest pull bypasses the Docker-29.x OCI-index failure on newer NVCR tags. Override (any image reference, ideally also a digest) to pin to a different build — e.g. the monitoring stack's older default 4.2.0. No effect on CPU nodes. See [OPERATIONS.md §3.1](./OPERATIONS.md#31-dcgmexporterimage--the-default-and-when-to-change-it) | diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md index c7f3f59e1..6ef26b270 100644 --- a/architectures/aws-pcs/tests/infra-test.md +++ b/architectures/aws-pcs/tests/infra-test.md @@ -60,7 +60,7 @@ The Test 1/6/7 container jobs are the functional proof that Pyxis works. > `pcs-ready-dlami-with-enroot-pyxis.yaml` carries its **own copy** of the Enroot/Pyxis > steps in its Image Builder UserData — editing `install-enroot-pyxis.sh` does **not** > change the AMI path until you rebuild. Build an AMI per supported `SlurmVersion`, -> deploy a cluster pinned to it (`AmiId=` + `PostInstallScriptUrl=""`), and +> deploy a cluster pinned to it (`AmiId=` + `PostInstallScriptUrl=' '`), and > run a container job. > - **On a clean first boot**, not a hand-patched node — post-install runs before > slurmd/profile.d/controller exist, and several bugs only appear there. @@ -92,7 +92,7 @@ Skip this test if you only touched the cluster templates. This is an **independent flow**, not a deploy-all parameter: build an AMI with the standalone template, then deploy a cluster pinned to that AMI ID with -`PostInstallScriptUrl=""` so nothing else runs at boot. +`PostInstallScriptUrl=' '` so nothing else runs at boot. The AMI is **single-Slurm-version by design** (Pyxis SPANK plugin ABI is version-locked) — so when you run this test, run it for **every supported @@ -132,7 +132,7 @@ aws cloudformation create-stack \ ParameterKey=PrimarySubnetAZ,ParameterValue=us-west-2a \ ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ ParameterKey=AmiId,ParameterValue=${AMI_ID} \ - ParameterKey=PostInstallScriptUrl,ParameterValue= \ + ParameterKey=PostInstallScriptUrl,ParameterValue=' ' \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ --profile claude --region us-west-2 ``` @@ -145,7 +145,7 @@ On any node from the new cluster: which enroot # /usr/bin/enroot (pre-baked) ls /opt/aws/pcs/scheduler/slurm-${SLURM_VERSION}/lib/slurm/spank_pyxis.so # built for matching Slurm cat /etc/aws/pcs/scheduler/slurm-${SLURM_VERSION}/plugstack.conf.d/pyxis.conf # references the .so -test ! -s /var/log/pcs-post-install.log && echo "post-install did not run (PostInstallScriptUrl='')" +test ! -s /var/log/pcs-post-install.log && echo "post-install did not run (PostInstallScriptUrl=' ')" ``` Then a container job through the login node (same form as Test 2): @@ -156,7 +156,7 @@ srun --partition=cpu1 --nodes=1 --ntasks=1 \ ``` **Expected:** `enroot` and the per-version Pyxis files exist **without** the -post-install hook running (because `PostInstallScriptUrl=""`); the container job +post-install hook running (because `PostInstallScriptUrl=' '`); the container job prints `PYXIS_FROM_AMI_OK`. Slurmd starts cleanly (no `Incompatible Slurm plugin version` in `journalctl -u slurmd`). From 41f5f35f39315712471b793431e1241f3797e90d Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 08:21:24 +0000 Subject: [PATCH 60/89] tests: add docs-consistency lint (tests/lint-docs.sh) as pre-merge Test 0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A runnable docs⇄template lint, the docs counterpart of template-lint — runs in seconds with no AWS. Catches the drift that bit us this PR (PostInstallScriptUrl semantics changed, 6 docs went stale): it fails on removed/renamed param names appearing as current (OnDemandEnableEfa, GrafanaPublicAccessCidr, the DeployMonitoring=true deploy-all usage, "S3 hosting not allowed", iam/ dir refs), on `PostInstallScriptUrl=""` shown as skip, on any deploy-all parameter missing a PARAMETERS.md row, and on broken README internal anchors. Allow-context regexes permit explicit "(renamed from…)" / history notes. Added as Test 0 in the pre-merge matrix. Passes clean on the current docs. --- architectures/aws-pcs/tests/README.md | 1 + architectures/aws-pcs/tests/lint-docs.sh | 79 ++++++++++++++++++++++++ 2 files changed, 80 insertions(+) create mode 100755 architectures/aws-pcs/tests/lint-docs.sh diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index caf8ce835..76d5cfe2f 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -14,6 +14,7 @@ Run this **complete set** before merging template or script changes. | # | Category | Tests | File | When to run | |---|---|---|---|---| +| 0 | **Docs lint** | `bash tests/lint-docs.sh` — no stale/renamed param refs in docs, every deploy-all param documented, README anchors resolve (the docs counterpart of template-lint; runs in seconds, no AWS) | [`lint-docs.sh`](./lint-docs.sh) | Every PR (esp. param renames) | | 1-3, 8 | **Infrastructure** | Monitoring stack, container runtime (first-boot + AMI build), template lint | [`infra-test.md`](./infra-test.md) | Every PR | | 4-6 | **Compute** | CPU queue, GPU families (G/P5/P6), NCCL multi-node EFA | [`compute-test.md`](./compute-test.md) | Every PR | | 7 | **Training** | FSDP Llama-2 7B distributed training | [`training-test.md`](./training-test.md) | GPU PRs | diff --git a/architectures/aws-pcs/tests/lint-docs.sh b/architectures/aws-pcs/tests/lint-docs.sh new file mode 100755 index 000000000..1adf2b74f --- /dev/null +++ b/architectures/aws-pcs/tests/lint-docs.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# Docs ⇄ template consistency lint for the AWS PCS reference architecture. +# +# Catches the most common drift after a parameter rename/removal: docs (and +# other templates) still referencing an old parameter name, a removed parameter +# presented as current, or a known-stale phrase. Run from anywhere; paths are +# resolved relative to this script's location (architectures/aws-pcs/). +# +# bash architectures/aws-pcs/tests/lint-docs.sh +# +# Exit code is non-zero if any check fails, so it can gate a PR in CI. +# +# When you intentionally rename/remove a parameter, update the BANNED list below +# in the same change — that is the point: the lint forces docs to keep up. + +set -uo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # architectures/aws-pcs +cd "$ROOT" + +# Files that describe the user-facing interface (must not mention removed params). +DOC_GLOBS=(README.md docs/*.md tests/*.md) + +fail=0 +report() { echo "FAIL: $1"; fail=1; } + +# 1. Removed / renamed parameters must not appear in docs as if current. +# Each entry is TAB-separated: \t\t. +# A hit is a failure UNLESS the line also matches the allowed-context regex +# (used to permit explicit "(renamed from X)" / "→" / "internally" history +# notes). Use NEVERMATCH as the allow-regex when no exception applies. +BANNED=( + $'OnDemandEnableEfa\tremoved\tOnDemandEnableEfa was replaced by OnDemandEfaInterfaceCount (0/1/2)' + $'GrafanaPublicAccessCidr\t[Rr]enamed|→ ?`?GrafanaAccessCidr\tGrafanaPublicAccessCidr was renamed to GrafanaAccessCidr' + $'DeployMonitoring=true\tMonitoringStack|internally|nested\tDeployMonitoring (bool) was replaced by MonitoringStack at the deploy-all layer' + $'S3 (public )?hosting is not allowed\tNEVERMATCH\tPostInstallScriptUrl now accepts s3:// URLs' + $'architectures/aws-pcs/iam/\tNEVERMATCH\tthe iam/ directory was removed; use docs/IAM.md + assets/cluster-*-iam.yaml' +) +for entry in "${BANNED[@]}"; do + IFS=$'\t' read -r pat allow msg <<<"$entry" + hits=$(grep -rnE "$pat" "${DOC_GLOBS[@]}" 2>/dev/null | grep -vE "$allow" || true) + if [ -n "$hits" ]; then + report "stale reference ($msg):" + echo "$hits" | sed 's/^/ /' + fi +done + +# 2. PostInstallScriptUrl: docs must not say empty = skip (empty now = auto-install; +# a single space is the skip sentinel). The literal `PostInstallScriptUrl=""` +# (empty string) shown as a skip is the wrong pattern; the correct "single +# space to skip" wording is fine. +hits=$(grep -rnE 'PostInstallScriptUrl=""' "${DOC_GLOBS[@]}" 2>/dev/null || true) +if [ -n "$hits" ]; then + report 'PostInstallScriptUrl="" (empty) shown as skip — empty now auto-installs; use a single space to skip:' + echo "$hits" | sed 's/^/ /' +fi + +# 3. Every parameter the deploy-all template declares should be documented in +# PARAMETERS.md (catches a new param added without a docs row). +params=$(awk '/^Parameters:/{p=1;next} /^[A-Za-z]/{p=0} p&&/^ [A-Z][A-Za-z0-9]+:/{gsub(/[: ]/,"");print}' assets/pcs-ml-cluster-deploy-all.yaml | sort -u) +for prm in $params; do + grep -q "\`$prm\`" docs/PARAMETERS.md || report "deploy-all parameter '$prm' is not documented in docs/PARAMETERS.md" +done + +# 4. Same-file Markdown anchor links in README.md resolve to a real heading. +# (Cross-file and external links are out of scope — kept simple on purpose.) +while IFS= read -r anchor; do + # build the set of heading slugs in README + slugs=$(grep -E '^#{1,6} ' README.md \ + | sed -E 's/^#{1,6} //; s/`//g' \ + | tr '[:upper:]' '[:lower:]' \ + | sed -E 's/[^a-z0-9 -]//g; s/ /-/g') + grep -qx "$anchor" <<<"$slugs" || report "README.md internal anchor '#$anchor' has no matching heading" +done < <(grep -oE '\]\(#[a-z0-9-]+\)' README.md | sed -E 's/\]\(#//; s/\)//' | sort -u) + +if [ "$fail" -eq 0 ]; then + echo "docs lint: PASS (no stale parameter references, no empty=skip wording, all deploy-all params documented, README anchors resolve)" +fi +exit $fail From 0c53c9f54825ce1b2f6a0190da88c0cb95bd73ea Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 08:26:29 +0000 Subject: [PATCH 61/89] tests: add NCCL busbw validity note (377 GB/s is reasonable, not peak-tuned) --- architectures/aws-pcs/tests/README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 76d5cfe2f..32864f293 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -130,6 +130,15 @@ NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl- | 4 GiB | 375 | | 16 GiB | **377** | +The curve is healthy (monotonic, latency-bound small → bandwidth-bound large, +asymptotes at 377). With p6-b200's 8×EFA (≈800 Gbps/node node-to-node) this is a +reasonable effective all_reduce busbw — EFA's `efa-direct` fabric is engaged on +all 8 NICs. It is **not a peak-tuned number**: NICs were auto-selected (no +`NCCL_ALGO`/topology-aware tuning) and the test ran from a general-purpose +nccl-tests image. The goal here was to confirm the templates establish +multi-node GPU communication over EFA, not to chase the hardware ceiling; for a +peak run use a B200-tuned NCCL build + the `topology-aware-nccl-tests` sbatch. + > Pre-merge gotcha hit here: the GPU nodes booted without Enroot/Pyxis (the > `PostInstallScriptUrl` 404, see caveat below). After installing > `install-enroot-pyxis.sh` manually on the **login node and all compute nodes** From 25305f5eaa1a00605656d32af9bb7d3ded79fcf6 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 08:29:23 +0000 Subject: [PATCH 62/89] tests: move p6-b200 NCCL/FSDP results out of README into their own test files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README is the index/matrix — detailed results belong with each test: - NCCL all_reduce busbw (p6-b200 ×4, 377 GB/s peak + validity note) → compute-test.md Test 6 - FSDP: README only kept distributed-stack-works + a long HF-429 troubleshooting saga; per review that triage detail isn't test-doc material. Replaced with a concise "HuggingFace rate limits" caveat in training-test.md Test 7 (use a higher-rate HF account / mirror / pre-tokenized /fsx data for large runs). - README region-coverage section now just points to those files. --- architectures/aws-pcs/tests/README.md | 65 ++------------------ architectures/aws-pcs/tests/compute-test.md | 11 ++++ architectures/aws-pcs/tests/training-test.md | 8 +++ 3 files changed, 25 insertions(+), 59 deletions(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 32864f293..1d80833ea 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -113,65 +113,12 @@ defaults (`PERSISTENT_2` / `SINGLE_AZ_HA_2`). | us-gov-east-1 (GovCloud East) | ⬜ not run | | us-gov-west-1 (GovCloud West) | ⬜ not run | -### p6-b200 ×4 NCCL all_reduce (ap-south-1, 32 GPU) - -NCCL `all_reduce_perf` over EFA (Pyxis container `public.ecr.aws/hpc-cloud/nccl-tests`, -4× p6-b200.48xlarge = 32 B200 GPUs, NCCL auto-selected NICs): - -- **NET/OFI provider: `efa`, fabric `efa-direct`, 8 NICs per node** (all EFA NICs in use) -- **Peak bus bandwidth: 377.4 GB/s** (16 GiB message, in/out-of-place) -- Avg bus bandwidth across the 8 B → 16 GiB sweep: 106 GB/s; 0 wrong / 0 out-of-bounds - -| Message size | busbw (GB/s) | -|---|---| -| 128 MiB | 295 | -| 256 MiB | 325 | -| 1 GiB | 367 | -| 4 GiB | 375 | -| 16 GiB | **377** | - -The curve is healthy (monotonic, latency-bound small → bandwidth-bound large, -asymptotes at 377). With p6-b200's 8×EFA (≈800 Gbps/node node-to-node) this is a -reasonable effective all_reduce busbw — EFA's `efa-direct` fabric is engaged on -all 8 NICs. It is **not a peak-tuned number**: NICs were auto-selected (no -`NCCL_ALGO`/topology-aware tuning) and the test ran from a general-purpose -nccl-tests image. The goal here was to confirm the templates establish -multi-node GPU communication over EFA, not to chase the hardware ceiling; for a -peak run use a B200-tuned NCCL build + the `topology-aware-nccl-tests` sbatch. - -> Pre-merge gotcha hit here: the GPU nodes booted without Enroot/Pyxis (the -> `PostInstallScriptUrl` 404, see caveat below). After installing -> `install-enroot-pyxis.sh` manually on the **login node and all compute nodes** -> and `systemctl restart slurmd` on the compute nodes (so the freshly-dropped -> `spank_pyxis.so` loads), `--container-image` jobs ran. A correct -> `PostInstallScriptUrl` at deploy time avoids all of this. - -### FSDP distributed training (ap-south-1, p6-b200) - -Ran the repo's canonical `3.test_cases/pytorch/FSDP/slurm/llama3_2_1b-training.sbatch` -unchanged (venv per `create_venv.sh`, `srun … torchrun --nproc_per_node=8`, -`--dataset=allenai/c4`). - -- **The distributed-training stack the architecture is responsible for works.** - Across 4× p6-b200 (32 GPUs): NCCL c10d rendezvous, FSDP2 wrap of the Llama-3 - 1B model (`1,154,549,760` params), and optimizer creation complete on every - rank. Each node exposes all 8 B200 GPUs to its `torchrun` (`nvidia-smi -L` = 8/node). -- **The training step loop could not be reached because of an external - HuggingFace rate-limit on `allenai/c4`**, not a cluster/template issue. Every - rank's dataloader does `load_dataset("allenai/c4", streaming=True)`, which - hits the c4 *tree* API (1024-shard listing); HF returns `429 Too Many Requests`. - Confirmed it is purely external by exhausting the mitigations: - - HF token (authenticated) — still 429 - - shared `HF_HOME=/fsx` cache + single-process prefetch of shards — still 429 - (streaming re-lists shards via the tree API regardless of cache) - - `HF_HUB_OFFLINE=1` — flips to `OfflineModeIsEnabled` (streaming needs the Hub) - - long `HF_HUB_ETAG_TIMEOUT`/`DOWNLOAD_TIMEOUT` + the loader's 20× retry — still 429 - - dropping to a single node (8 concurrent readers instead of 32) — still 429 -- **Conclusion:** the cluster is training-ready (NCCL/EFA/FSDP all proven; NCCL - all_reduce hit 377 GB/s above). Driving `allenai/c4` through the streaming - loader at scale needs an HF account with a higher rate limit, an HF mirror, or - pre-tokenized data staged on `/fsx` (non-streaming) — an operational/data - choice, independent of these templates. +The ap-south-1 (Mumbai) row was exercised most deeply — p6-b200 ×4 GPU. The +detailed results live with their tests, not here: NCCL all_reduce numbers in +[compute-test.md](./compute-test.md#test-6-nccl-multi-node-efa), distributed +training (and the HuggingFace rate-limit caveat) in +[training-test.md](./training-test.md#test-7-fsdp-sample-training), GPU health +check in [gpu-healthcheck-test.md](./gpu-healthcheck-test.md). Cross-region note: nested-stack `TemplateURL` and the in-instance `aws s3 cp` of boot scripts both work against an S3 bucket in a **different** diff --git a/architectures/aws-pcs/tests/compute-test.md b/architectures/aws-pcs/tests/compute-test.md index 7ca7f661e..50ba5bd6f 100644 --- a/architectures/aws-pcs/tests/compute-test.md +++ b/architectures/aws-pcs/tests/compute-test.md @@ -79,4 +79,15 @@ node. EFA itself is exercised by Test 6. > see the cards saturate. **Don't go to 128 GiB/256 GiB**: the all_reduce buffer exceeds > B300 GPU memory and the job is OOM-killed. For a higher peak, add nodes, not buffer size. +### Verified — p6-b200 ×4 (32 GPU, ap-south-1) + +`all_reduce_perf` over EFA (8 EFA NICs/node, `NET/OFI provider efa, fabric +efa-direct`, NICs auto-selected). busbw by message size: 128 MiB 295, 256 MiB +325, 1 GiB 367, 4 GiB 375, **16 GiB 377 GB/s** (peak); 0 wrong / 0 out-of-bounds. +The curve is healthy (latency-bound small → bandwidth-bound large, asymptotes at +377), a reasonable effective busbw for p6-b200's ≈800 Gbps/node node-to-node EFA. +This is **not a peak-tuned number** — NICs auto-selected (no `NCCL_ALGO` / +topology tuning), general-purpose nccl-tests image; for a higher peak use a +B200-tuned NCCL build + the `topology-aware-nccl-tests` sbatch. + --- diff --git a/architectures/aws-pcs/tests/training-test.md b/architectures/aws-pcs/tests/training-test.md index 1c62f69c4..f05ad7cc1 100644 --- a/architectures/aws-pcs/tests/training-test.md +++ b/architectures/aws-pcs/tests/training-test.md @@ -26,6 +26,14 @@ shared filesystems and the node count: file-locking on NFS throws `OSError: [Errno 116] Stale file handle`; export `HF_HOME=/fsx/.hf-cache` before submitting. + > **HuggingFace rate limits.** The test case streams its dataset from the Hub, so + > every rank pulls from HF at startup. Large datasets (e.g. `allenai/c4`, 1024 + > shards) can return `429 Too Many Requests` under many concurrent ranks even with + > an `HF_TOKEN` — the dataset is fetched, not the cluster, so this is an HF-account + > limit, not a cluster issue. For large multi-node runs, use an HF account with a + > higher rate limit, an HF mirror, or pre-tokenized data staged on `/fsx` (no + > streaming). + 3. **Submit 2 nodes** (the canonical sbatch defaults to 4) on your GPU queue. The venv must be on `PATH` for `torchrun` to resolve on every node — point `PATH` at the shared venv via `--export` (the canonical sbatch doesn't `activate` it): From 2ad22eedb177dc64cc1e1b6bb9887401cb2a1596 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 08:35:03 +0000 Subject: [PATCH 63/89] tests: rebuild region-coverage table + record s3:// PostInstall and Osaka OpenZFS findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Region coverage table reworked to the requested columns: Deploy / Mon / Storage (with the OpenZFS deployment type that worked) / Pyxis / GPU (CB·ODCR) / Verified date. 10 regions tested. - **Osaka (ap-northeast-3) does NOT support OpenZFSDeploymentType=SINGLE_AZ_HA_2** (default deploy fails: Invalid deploymentType) — documented the SINGLE_AZ_1 fallback. The other regions use the SINGLE_AZ_HA_2 default. - infra-test.md Test 2: noted the default PostInstallScriptUrl is now s3:// and verified it installs Enroot/Pyxis from a PRIVATE bucket via the instance role (no public S3 needed — dev-account friendly). --- architectures/aws-pcs/tests/README.md | 80 +++++++++++------------ architectures/aws-pcs/tests/infra-test.md | 8 +++ 2 files changed, 47 insertions(+), 41 deletions(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 1d80833ea..477abc4bc 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -71,47 +71,45 @@ MonitoringStack=Prometheus-LoginNode ## Region coverage The templates fetch nested stacks + boot scripts from a single S3 bucket -(`S3BucketName`) and resolve the PCS-Ready DLAMI from SSM per region. Validated -deploys (deploy-all: prereqs + cluster + login + CPU, monitoring on) across: - -| Region | Deploy | Login + monitoring | FSx Lustre `/fsx` + OpenZFS `/home` | Notes | -|---|---|---|---|---| -| **us-east-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ 1.2T Lustre + OpenZFS NFS | template bucket's home region | -| **us-east-2** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | primary test region; also EFA hpc6a/hpc8a + multi-user/accounting | -| **us-west-2** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | | -| **ap-northeast-1** | ✅ CREATE_COMPLETE | ✅ | ✅ | Tokyo | -| **ap-south-1** | ✅ CREATE_COMPLETE | ✅ 6 containers | ✅ | Mumbai; p6-b200 ×4 GPU (8 GPU/node, 8 EFA NICs), B200 driver 595.71.05 | - -### All PCS launch regions - -AWS PCS is available in the 18 regions below (per -`/aws/service/global-infrastructure/services/pcs/regions`). The 5 marked -**tested** above were validated end-to-end; the rest are expected to work (the -templates resolve the PCS-Ready DLAMI from SSM and FSx deployment types per -region) but have not been run. Confirm `LustreDeploymentType` / -`OpenZFSDeploymentType` availability in a new region before relying on the -defaults (`PERSISTENT_2` / `SINGLE_AZ_HA_2`). - -| Region | Status | -|---|---| -| us-east-1 (N. Virginia) | ✅ tested | -| us-east-2 (Ohio) | ✅ tested | -| us-west-2 (Oregon) | ✅ tested | -| ap-northeast-1 (Tokyo) | ✅ tested | -| ap-south-1 (Mumbai) | ✅ tested (GPU/B200) | -| ap-northeast-3 (Osaka) | ⬜ not run | -| ap-southeast-1 (Singapore) | ⬜ not run | -| ap-southeast-2 (Sydney) | ⬜ not run | -| eu-central-1 (Frankfurt) | ⬜ not run | -| eu-north-1 (Stockholm) | ⬜ not run | -| eu-south-1 (Milan) | ⬜ not run | -| eu-south-2 (Spain) | ⬜ not run | -| eu-west-1 (Ireland) | ⬜ not run | -| eu-west-2 (London) | ⬜ not run | -| eu-west-3 (Paris) | ⬜ not run | -| sa-east-1 (São Paulo) | ⬜ not run | -| us-gov-east-1 (GovCloud East) | ⬜ not run | -| us-gov-west-1 (GovCloud West) | ⬜ not run | +(`S3BucketName`) and resolve the PCS-Ready DLAMI from SSM per region. + +Columns: **Deploy** (deploy-all → CREATE_COMPLETE), **Mon** (6 monitoring +containers up on the login node), **Storage** (FSx Lustre `/fsx` + OpenZFS +`/home` created & mounted — with the OpenZFS deployment type that worked), +**Pyxis** (Enroot/Pyxis container job runs), **GPU** (multi-NIC GPU CNG tested, +incl. capacity reservation type), **Verified** (date, UTC). + +| Region | Deploy | Mon | Storage (OpenZFS type) | Pyxis | GPU (CB/ODCR) | Verified | +|---|---|---|---|---|---|---| +| **us-east-1** (N. Virginia) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **us-east-2** (Ohio) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | EFA hpc6a(1)/hpc8a(2) — CPU/EFA | 2026-06-17 | +| **us-west-2** (Oregon) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **ap-northeast-1** (Tokyo) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **ap-south-1** (Mumbai) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ **p6-b200 ×4** (Capacity Block) | 2026-06-17 | +| **ap-northeast-3** (Osaka) | ✅ | ✅ | ✅ `SINGLE_AZ_1` ¹ | ✅ | — | 2026-06-17 | +| **ap-southeast-1** (Singapore) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **ap-southeast-2** (Sydney) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **eu-central-1** (Frankfurt) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **eu-north-1** (Stockholm) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | + +¹ **Osaka (ap-northeast-3) does not support the default `OpenZFSDeploymentType=SINGLE_AZ_HA_2`** +— the default deploy fails at the OpenZFS filesystem with +`Invalid deploymentType (BadRequest)`. Deploy there with +`OpenZFSDeploymentType=SINGLE_AZ_1` (+ a valid `HomeThroughput`, e.g. 256). This +is the documented "not available in every Region" case (see the parameter's +description in `ml-cluster-prerequisites.yaml`); `SINGLE_AZ_1` is available in +all regions and is the safe fallback. + +### Remaining PCS launch regions (not yet run) + +AWS PCS is available in 18 regions total (per +`/aws/service/global-infrastructure/services/pcs/regions`). Beyond the 10 tested +above, these are expected to work but have not been run — confirm +`LustreDeploymentType` / `OpenZFSDeploymentType` availability before relying on +the defaults (`PERSISTENT_2` / `SINGLE_AZ_HA_2`), as Osaka above shows: +`eu-south-1` (Milan), `eu-south-2` (Spain), `eu-west-1` (Ireland), `eu-west-2` +(London), `eu-west-3` (Paris), `sa-east-1` (São Paulo), `us-gov-east-1`, +`us-gov-west-1` (GovCloud). The ap-south-1 (Mumbai) row was exercised most deeply — p6-b200 ×4 GPU. The detailed results live with their tests, not here: NCCL all_reduce numbers in diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md index 6ef26b270..5fd6d41f5 100644 --- a/architectures/aws-pcs/tests/infra-test.md +++ b/architectures/aws-pcs/tests/infra-test.md @@ -37,6 +37,14 @@ Deploy `pcs-ml-cluster-deploy-all.yaml` with both `AmiId` and `PostInstallScript left at their defaults (so SSM auto-resolves the latest PCS-Ready DLAMI and post-install runs the Enroot/Pyxis installer), then on any node: +> **The default `PostInstallScriptUrl` is now an `s3://` URL** (empty → +> `s3:///scripts/install-enroot-pyxis.sh`, fetched with +> the instance role). Verified end-to-end against a **private** test bucket +> (`s3://midaisuk-llm-dev/...`): the node's post-install log shows +> `Downloading post-install script from s3://…` and `pyxis.conf` is installed — +> so no public S3 is required (works in dev accounts). An `http(s)://` value still +> works too (curl), for public/GitHub-raw scripts. + ```bash which enroot # /usr/bin/enroot ls /opt/aws/pcs/scheduler/slurm-*/lib/slurm/spank_pyxis.so # per-version Pyxis SPANK plugin From 0412a3de9c47f6a53e398a2f19315e28e712eadf Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 11:39:34 +0000 Subject: [PATCH 64/89] tests: add Megatron-LM GPT-3 (Test 7b) + intensive GPU health check results - training-test.md: Test 7b Megatron-LM GPT-3 TP/PP/DP on p6-b200 x4 (~134 TFLOP/s/GPU, lm loss 10.91->10.46, EFA efa-direct 8 nics) - gpu-healthcheck-test.md: intensive suite EFA loopback PASS on p6-b200; caveat that check 5 (NCCL) needs the ECR '#' URI + in-image binary path, so validate multi-node NCCL via canonical Test 6 - README.md: matrix row 7,7b + Megatron canonical-asset row --- architectures/aws-pcs/tests/README.md | 3 +- .../aws-pcs/tests/gpu-healthcheck-test.md | 19 +++++++ architectures/aws-pcs/tests/training-test.md | 50 +++++++++++++++++++ 3 files changed, 71 insertions(+), 1 deletion(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 477abc4bc..f765f3a22 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -17,7 +17,7 @@ Run this **complete set** before merging template or script changes. | 0 | **Docs lint** | `bash tests/lint-docs.sh` — no stale/renamed param refs in docs, every deploy-all param documented, README anchors resolve (the docs counterpart of template-lint; runs in seconds, no AWS) | [`lint-docs.sh`](./lint-docs.sh) | Every PR (esp. param renames) | | 1-3, 8 | **Infrastructure** | Monitoring stack, container runtime (first-boot + AMI build), template lint | [`infra-test.md`](./infra-test.md) | Every PR | | 4-6 | **Compute** | CPU queue, GPU families (G/P5/P6), NCCL multi-node EFA | [`compute-test.md`](./compute-test.md) | Every PR | -| 7 | **Training** | FSDP Llama-2 7B distributed training | [`training-test.md`](./training-test.md) | GPU PRs | +| 7, 7b | **Training** | FSDP Llama-2 7B (HF-streamed) + Megatron-LM GPT-3 (TP/PP/DP, local data) | [`training-test.md`](./training-test.md) | GPU PRs | | 9 | **HPC EFA** | EFA on CPU instances (hpc6a/hpc7a/hpc8a), OSU benchmarks | [`hpc-efa-test.md`](./hpc-efa-test.md) | EFA wiring changes | | 10 | **Storage** | FSx health check + performance regression test (noatime benchmark) | [`storage-test.md`](./storage-test.md) | FSx / mount changes | | 11-12 | **Multi-user** | OpenLDAP directory + Slurm managed accounting | [`multi-user-test.md`](./multi-user-test.md) | Directory / accounting changes | @@ -140,6 +140,7 @@ every region tested. `OpenZFSDeploymentType=SINGLE_AZ_HA_2` and |---|---|---| | NCCL `all_reduce` | [`micro-benchmarks/nccl-tests/slurm/nccl-tests-container.sbatch`](../../../micro-benchmarks/nccl-tests) | Partition name, Enroot import on login node | | FSDP Llama-2 7B | [`3.test_cases/pytorch/FSDP`](../../../3.test_cases/pytorch/FSDP) | Cache on `/fsx`, 2 nodes | +| Megatron-LM GPT-3 (TP/PP/DP) | [`3.test_cases/megatron/megatron-lm`](../../../3.test_cases/megatron/megatron-lm) | Import `.sqsh` to `/fsx`, data under `/fsx/gpt2/`, 4 nodes | | GPU Health Check | [`4.validation_and_observability/2.gpu-cluster-healthcheck`](../../../4.validation_and_observability/2.gpu-cluster-healthcheck) | sbatch wrapper, partition name | --- diff --git a/architectures/aws-pcs/tests/gpu-healthcheck-test.md b/architectures/aws-pcs/tests/gpu-healthcheck-test.md index f87d495f8..ba41bc880 100644 --- a/architectures/aws-pcs/tests/gpu-healthcheck-test.md +++ b/architectures/aws-pcs/tests/gpu-healthcheck-test.md @@ -58,3 +58,22 @@ Lightweight suite (checks 0-3) on **p6-b200** (ap-south-1, via `srun -p gpu`): EFA device count matching the instance profile and the multi-node NCCL bandwidth threshold are also covered by [compute-test.md](./compute-test.md) (NCCL all_reduce hit 377 GB/s on 4× p6-b200). + +### Intensive suite (checks 4-6) on p6-b200 ×2 (ap-south-1) + +The intensive suite (`--suite intensive`) needs **exclusive nodes for 1-3 hr** (DCGM L4 +alone is 45 min – 2.25 hr/node), so it's a maintenance-window / pre-long-run check, not a +per-deploy gate. + +- **Check 6 — EFA loopback: PASS** on 2× p6-b200 (8 EFA domains tested per node, both nodes). +- **Check 4 — DCGM L4:** budget the full 45 min+/node; it disables MIG, stops concurrent + GPU telemetry (`dcgm-exporter`), and runs the EUD + pulse-power stress, so it cannot share + a node with other GPU work. + +> **⚠️ Validate multi-node NCCL via [compute-test.md Test 6](./compute-test.md#test-6-nccl-multi-node-efa), +> not intensive check 5.** The suite's `checks/5-nccl-allreduce.sh` defaults to an ECR image +> URI that enroot rejects (needs the `#` registry separator, +> `docker://public.ecr.aws#hpc-cloud/nccl-tests:`), and invokes `all_reduce_perf` by +> bare name when that image keeps the binaries under `/opt/nccl-tests/build/` (not on +> `PATH`). The canonical `nccl-tests-container.sbatch` in Test 6 handles both correctly and +> ran to 377 GB/s on this cluster. diff --git a/architectures/aws-pcs/tests/training-test.md b/architectures/aws-pcs/tests/training-test.md index f05ad7cc1..1d8867d6e 100644 --- a/architectures/aws-pcs/tests/training-test.md +++ b/architectures/aws-pcs/tests/training-test.md @@ -77,3 +77,53 @@ cd slurm && sbatch --nodes=2 --partition=gpu-p6b300 \ > subnet and pinning one breaks the cross-node NCCL bootstrap ring. --- + +## Test 7b: Megatron-LM GPT-3 (tensor + pipeline parallel) + +A second distributed-training path that exercises 3D parallelism (TP/PP/DP) rather than +FSDP, using the repo's canonical Megatron-LM case +[`3.test_cases/megatron/megatron-lm`](../../../3.test_cases/megatron/megatron-lm). Unlike +the FSDP case it does **not** stream from HuggingFace — data is tokenized once to local +`.bin`/`.idx` on `/fsx`, so it has no HF rate-limit exposure. Two PCS-specific deltas: + +1. **Build + import the image on the login node** (300 GiB root; the overlay can't live on + Lustre), writing the `.sqsh` to `/fsx`: + + ```bash + cd /fsx/awsome-distributed-ai/3.test_cases/megatron/megatron-lm + sudo docker build -t aws-megatron-lm -f 0.distributed-training.Dockerfile . + enroot import -o /fsx/aws-megatron-lm.sqsh dockerd://aws-megatron-lm:latest + ``` + +2. **Preprocess once, then train**, passing `IMAGE`/`DATA_PATH`/`FSX_MOUNT` as env (the + sbatch scripts read them). Preprocessing is CPU work but still needs a Pyxis-capable + node and the login `srun` client: + + ```bash + cd slurm/gpt3 + IMAGE=/fsx/aws-megatron-lm.sqsh DATA_PATH=/fsx FSX_MOUNT=/fsx:/fsx \ + sbatch -p gpu 1.data-preprocessing.sbatch # → /fsx/my-gpt2_text_document.{bin,idx} + # the training sbatch expects data under ${DATA_PATH}/gpt2/ — symlink it there + mkdir -p /fsx/gpt2 && ln -sf /fsx/my-gpt2_text_document.* /fsx/gpt2/ \ + && ln -sf /fsx/gpt2-vocab.json /fsx/gpt2-merges.txt /fsx/gpt2/ + IMAGE=/fsx/aws-megatron-lm.sqsh DATA_PATH=/fsx FSX_MOUNT=/fsx:/fsx \ + sbatch --nodes=4 -p gpu 2.distributed-training.sbatch + ``` + + For a short smoke run, add `--exit-interval 20` and `--log-throughput` to the megatron + args in `2.distributed-training.sbatch`. At `NODES≤4` the script picks `TP=4, PP=2, + GBS=288` automatically. + +**Expected:** NCCL over EFA (`NET/OFI Selected provider is efa, fabric is efa-direct (found +8 nics)` on p6-b200); `0` nan/skipped after warmup; `lm loss` begins descending once the +LR warmup kicks in (loss-scale auto-tuning skips the first ~16 steps with `lr=0` by design). + +### Verified — p6-b200 ×4 (32 GPU, ap-south-1) + +GPT-3 36-layer / hidden 4096 / 32 heads, seq 2048, TP=4 PP=2 (DP=4), GBS=288, fp16 + +activation recompute. Steady **~134 TFLOP/s/GPU** (iters 3–20, ~6.4 s/iter); `lm loss` +**10.91 → 10.46** over iters 17–20 as the cosine LR warmup ramps (grad norm 57→45); 0 nan, +0 skipped post-warmup. This is a general-purpose (un-tuned) container — not a peak MFU +number — but confirms TP+PP+DP 3D parallelism trains correctly over EFA on B200. + +--- From 28159191d6f019d767e94d1125bf77dd058113a7 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 15:40:43 +0000 Subject: [PATCH 65/89] docs(roadmap): add targeted ODCR support for GPU node groups Capture the follow-up to make CapacityReservationId work for targeted ODCRs (not just Capacity Blocks): a CapacityReservationType enum (none/capacity-block/targeted-odcr) that branches MarketType + placement group, with none/capacity-block staying backward compatible. Notes how to verify without GPU capacity (MinCount=0 launch-template assertion + cheap-type consumption check). --- architectures/aws-pcs/docs/ROADMAP.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/architectures/aws-pcs/docs/ROADMAP.md b/architectures/aws-pcs/docs/ROADMAP.md index d57e66ac8..b86022142 100644 --- a/architectures/aws-pcs/docs/ROADMAP.md +++ b/architectures/aws-pcs/docs/ROADMAP.md @@ -14,6 +14,21 @@ Priority: 🔴 high · 🟡 medium · 🟢 low NAT gateway). This unblocks `OpenZFSDeploymentType=MULTI_AZ` and higher-availability layouts. *(Note: OpenZFS MULTI_AZ wiring of the 2nd subnet into the FSx resource is a follow-up; the subnets + routing are in place.)* +- [ ] 🟡 **Targeted ODCR support for GPU node groups.** Today `CapacityReservationId` + on `add-cng-p5`/`add-cng-p6-b200`/`add-cng-p6-b300` is **Capacity Block for ML only** — + setting it forces `MarketType=capacity-block` and drops the placement group, so a + *targeted* On-Demand Capacity Reservation (ODCR) cannot be consumed (only "open" ODCRs, + via the empty/On-Demand path, work). Add a `CapacityReservationType` enum + (`none` | `capacity-block` | `targeted-odcr`) and branch the launch template: + `targeted-odcr` sets `CapacityReservationTarget` **without** `MarketType=capacity-block` + and **keeps** the placement group (On-Demand billing against the reservation). + `none`/`capacity-block` stay equivalent to today (backward compatible). Replaces the + current "do not put an ODCR ID here" caveat. Verification can be done **without GPU + capacity**: (1) static — create the GPU CNGs with `Min/MaxCount=0` and assert the + generated launch template's `CapacityReservationSpecification`/`InstanceMarketOptions`/ + `Placement` per type; (2) dynamic — the branch logic is instance-family-independent, so + exercise actual targeted-ODCR consumption (`InstanceLifecycle` empty = On-Demand, reserved + count decrements) on a cheap type (c6i/g5). - [ ] 🟢 **Trainium (Trn) validation.** Validate the templates on Trainium instances (e.g. trn1/trn2) — node group, EFA/networking, and a sample training run. - [ ] 🟡 **Graviton (arm64) CPU CNG support — `hpc7g` / `c7gn`.** EFA-capable arm64 From 231ed952cae94eb5bb0c7b8da60b376688c1f405 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:25:58 +0000 Subject: [PATCH 66/89] =?UTF-8?q?docs(readme):=20user-facing=20cleanup=20?= =?UTF-8?q?=E2=80=94=20surface=20new=20params,=20fix=20accuracy,=20slim=20?= =?UTF-8?q?advanced=20detail?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §4 Configuration: add DirectoryService / SSHAccessCidr / GrafanaAccessCidr / AdditionalSubnetAZ2-3 rows so the new user-facing features are discoverable - §4: fix HomeThroughput AllowedValues (SINGLE_AZ_HA_1 includes 64) - §1: correct capacity wording (open ODCR auto-consumed; targeted ODCR on roadmap) - §9: p6-b300 row shows 17 interfaces (16 EFA + 1 ENA), consistent with §4 - move EFA-on-CPU + FSx-Lustre-EFA into Advanced (§8.5/§8.6); consolidate the DCGM version detail into a note at the end of §8.1 Monitoring - §8.2: note the single-login-node / SPOF constraint for DirectoryService - §7: replace specific NCCL busbw numbers with a pointer to tests/compute-test.md - §4: document OnDemandInstanceType + PseriesInstanceType accepted values --- architectures/aws-pcs/README.md | 138 ++++++++++++++++++++------------ 1 file changed, 87 insertions(+), 51 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 63a4cb6ff..8c32a1b2f 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -10,7 +10,7 @@ This repository provides reference architectures and deployment templates for se - **Container runtime included**: Enroot/Pyxis is set up automatically, so `srun --container-image=...` works out of the box for containerized training. - **Monitoring built in**: Grafana + Prometheus run on the login node, with DCGM Exporter on GPU compute nodes feeding the pre-built GPU dashboards (`MonitoringStack=Prometheus-LoginNode`, on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR with `GrafanaAccessCidr`. - **GPU-ready, multi-NIC EFA**: dedicated launch templates for the P5 and P6 families, selected automatically by instance type, for high-bandwidth multi-node training. -- **Broad capacity-purchase support**: covers the full range of EC2 capacity options out of the box — On-Demand, On-Demand Capacity Reservations (ODCR), and Capacity Blocks for ML — selected per node group. +- **Flexible capacity options**: On-Demand, "open" On-Demand Capacity Reservations (ODCR, consumed automatically), and Capacity Blocks for ML — selected per node group. (Targeting a *specific* ODCR is on the [roadmap](./docs/ROADMAP.md).) - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. @@ -91,11 +91,16 @@ only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | | `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | -| `DeployOnDemandCNG` | `true` | Deploy the `cpu1` CPU queue (`OnDemandInstanceType`, default `c6i.4xlarge`) | +| `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | +| `OnDemandInstanceType` | `c6i.4xlarge` | Instance type for the On-Demand queue. Not limited to CPU — set any On-Demand type, e.g. a single-NIC GPU like `g6.12xlarge` (see [Example 1](#example-1-single-nic-gpu-queue-g6)) or an EFA-capable HPC type like `hpc7a.96xlarge`. (Multi-NIC P5/P6 GPUs use the P-series queue instead — see `PseriesInstanceType`.) Tune `OnDemandQueueName`/`OnDemandCngName`/`OnDemandMinCount`/`OnDemandMaxCount` alongside it | | `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandefainterfacecount) | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | -| `PseriesInstanceType` | `p5.48xlarge` | GPU instance type; auto-selects the multi-NIC template + EFA count | +| `PseriesInstanceType` | `p5.48xlarge` | Multi-NIC GPU instance type; auto-selects the matching template + EFA NIC count. Accepted values: `p5.48xlarge` / `p5e.48xlarge` (→ add-cng-p5, 32 NICs), `p5en.48xlarge` (→ add-cng-p5, 16 NICs), `p6-b200.48xlarge` (→ add-cng-p6-b200, 8 NICs), `p6-b300.48xlarge` (→ add-cng-p6-b300, 17 NICs). See [GPU compute](#gpu-compute-p5p6) | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | +| `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | +| `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | +| `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 on the login node to a trusted CIDR for direct Grafana access (default: SSM port-forward only). See [§8.1 Monitoring](#accessing-the-grafana-dashboards) | +| `AdditionalSubnetAZ2` / `AdditionalSubnetAZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2) for higher availability or AZ-matched capacity. NAT gateway stays in the primary AZ | **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the @@ -143,38 +148,6 @@ automatically. > group at `PseriesMinCount = PseriesMaxCount = ` so the reserved > nodes launch immediately, rather than scaling from 0. -### EFA on CPU HPC instances (`OnDemandEfaInterfaceCount`) - -For tightly-coupled HPC / MPI workloads on CPU-only HPC instances, set -`OnDemandEfaInterfaceCount` (deploy-all) or `EfaInterfaceCount` (modular -`add-cng.yaml`) to the instance type's EFA interface count. `0` (default) = no -EFA; `1` or `2` = enable EFA with that many interfaces — the CPU compute node -group then launches with EFA `NetworkInterfaces` and a cluster placement group -is auto-created. GPU CNGs (P5/P6) are unaffected — they use their own dedicated -multi-NIC EFA wiring per-family. - -| Instance type | EFA interfaces | Aggregate spec | Set the count to | -|---|---:|---:|---:| -| (any non-HPC type, e.g. `c6i.4xlarge`) | — | — | **0** (no EFA; default) | -| `hpc6a.48xlarge` | 1 | 100 Gbps | **1** | -| `hpc7a.96xlarge` (and 12/24/48) | 2 | 300 Gbps | **2** | -| `hpc6id.32xlarge` | 2 | 200 Gbps | **2** | -| `hpc8a.96xlarge` | 2 | 300 Gbps | **2** | -| `c7i.metal-48xl` etc. | 1 | varies | **1** | - -(Setting a count > 0 on a non-EFA type, or mismatching the count with the -instance type's actual `MaximumEfaInterfaces`, fails at launch.) - -**Placement group:** auto-created per-CNG by the template. Override with -`OnDemandPlacementGroupName=` to share one PG across multiple -CNGs (e.g. heterogeneous tightly-coupled jobs that span CPU + GPU). - -**Multi-NIC bandwidth needs multiple MPI pairs.** A single MPI pair uses one -libfabric endpoint and only one NIC. Use `osu_mbw_mr -np 32 -N 16` (or your -application's natural multi-pair pattern) to actually exercise both NICs on -hpc7a/hpc8a. See [tests/README.md Test 9](./tests/README.md#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) -for the full benchmark setup and validated bandwidth numbers. - ### Storage: FSx deployment types (Region availability) **FSx deployment types are not available in every Region.** Defaults match the most @@ -184,9 +157,8 @@ capable type; switch to a more widely available one if your Region needs it. |---|---|---|---|---| | Lustre (`/fsx`) | `LustreDeploymentType` | `PERSISTENT_2` | `PERSISTENT_1` | `PERSISTENT_2` (throughput 125/250/500/1000, metadata config) isn't in every Region; `PERSISTENT_1` (50/100/200) is in more Regions | | Lustre (`/fsx`) | `PerUnitStorageThroughput` | `250` | any valid number | Must be valid for the type: P2 = 125/250/500/1000, P1 = 50/100/200 | -| Lustre (`/fsx`) | `FSxLustreEnableEfa` | `false` | `true` | Enable EFA on the Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — see the Client-side Lustre-on-EFA + GDS item in [`docs/ROADMAP.md`](./docs/ROADMAP.md)). EFA-capable CPU CNGs (`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** (a CFN Rule fails the stack at create time on PERSISTENT_1). **Requires a much higher minimum `Capacity`**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly when enabling this | | OpenZFS (`/home`) | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | `SINGLE_AZ_HA_1`, `SINGLE_AZ_2`, `SINGLE_AZ_1` | `SINGLE_AZ_1` is in all Regions; HA/2 variants vary. `MULTI_AZ` excluded (needs a second subnet) | -| OpenZFS (`/home`) | `HomeThroughput` | `320` | any valid number | Throughput (MB/s). Valid values depend on the deployment type: `SINGLE_AZ_2`/`SINGLE_AZ_HA_2` = 160/320/640/1280/2560/3840/5120/7680/10240; `SINGLE_AZ_HA_1` = 128/256/512/1024/2048/3072/4096; `SINGLE_AZ_1` = 64/128/256/512/1024/2048/3072/4096 | +| OpenZFS (`/home`) | `HomeThroughput` | `320` | any valid number | Throughput (MB/s). Valid values depend on the deployment type: `SINGLE_AZ_2`/`SINGLE_AZ_HA_2` = 160/320/640/1280/2560/3840/5120/7680/10240; `SINGLE_AZ_HA_1`/`SINGLE_AZ_1` = 64/128/256/512/1024/2048/3072/4096 | Check support before deploying: [Lustre Regions](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-fsx-lustre.html) · @@ -194,6 +166,9 @@ Check support before deploying: If a deploy fails at the FSx resource with an "unsupported deployment type" error, switch these parameters to a type your Region supports. +> Enabling EFA / GPUDirect Storage on the Lustre filesystem (`FSxLustreEnableEfa`) is an +> advanced option — see [§8.6 FSx for Lustre over EFA (GPUDirect Storage)](#86-fsx-for-lustre-over-efa-gpudirect-storage). + --- ## 5. Usage Examples @@ -314,9 +289,9 @@ sbatch --partition=gpu-p6b300 nccl-tests-container.sbatch In `nccl-all_reduce_perf_.out`, EFA is active when you see `NET/OFI Selected provider is efa ... (found N nics)`; a healthy run ends with -`# Out of bounds values : 0 OK` and a busbw that scales with message size -(e.g. ~751 GB/s at 64 GiB on 2× p6-b300; raise `-e` past the 16 GiB default to -saturate larger fabrics). +`# Out of bounds values : 0 OK` and a busbw that scales with message size. For +expected bandwidth numbers per instance family and tuning notes, see +[tests/compute-test.md](./tests/compute-test.md#test-6-nccl-multi-node-efa). For a full training example, see the [PyTorch FSDP test case](../../3.test_cases/pytorch/FSDP); the full validation matrix is in [tests/README.md](./tests/README.md). @@ -341,23 +316,15 @@ node-local `/opt` (not the shared `/home`). Pre-built Grafana dashboards are provisioned automatically — see [Accessing the Grafana dashboards](#accessing-the-grafana-dashboards) for the list and a screenshot. -> **GPU metrics work out of the box across the supported GPU range** (Hopper / B200 / -> B300). `DcgmExporterImage` defaults to a DCGM 4.5.2 build pinned by digest, validated -> on 2× p6-b300 and on B200. The monitoring stack's own default (DCGM 4.2.0) tops out -> at B200 and can't pull newer NVCR tags on Docker 29.x — overriding via digest at the -> deploy-all level is what bridges that. Override `DcgmExporterImage` only if you need -> to pin to a different build; details: -> [OPERATIONS.md §3.1](./docs/OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it). - > **Prefer AWS-managed Prometheus/Grafana?** If you'd rather use Amazon Managed Service > for Prometheus + Amazon Managed Grafana instead of the self-hosted stack on the login > node, see [`4.validation_and_observability/4.prometheus-grafana`](../../4.validation_and_observability/4.prometheus-grafana). **Monitoring-related parameters:** - `MonitoringStack` (default `Prometheus-LoginNode`; `none` disables monitoring) -- `MonitoringVersion` — [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`; default `v2.9.1`). Pinned to a tag so upstream changes can't break deployments unexpectedly. `v2.9.1` adds the `DCGM_EXPORTER_IMAGE` override (lets `DcgmExporterImage` enable B300 GPU metrics); `v2.6.4`+ carry the PCS fixes (node-local `/opt` install + Docker-29.x DCGM tag). +- `MonitoringVersion` — [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`; default `v2.9.1`). Pinned to a tag so upstream changes can't break deployments unexpectedly. - `MonitoringRepo` — `owner/repo` to fetch from (default `aws-samples/aws-parallelcluster-monitoring`). Point at a fork + a branch in `MonitoringVersion` to test unreleased changes. -- `DcgmExporterImage` — dcgm-exporter image used on GPU nodes; defaults to a DCGM 4.5.2 build pinned by digest (covers Hopper/B200/B300). Override only if you need to pin to a different build (e.g. the older monitoring-default DCGM 4.2.0). +- `DcgmExporterImage` — dcgm-exporter image used on GPU nodes (default covers Hopper/B200/B300; see the DCGM note below). > Node type is identified by the `monitoring-role` tag (`login`/`compute`), not the EC2 > `Name` tag — the `Name` tag defaults to `PCS-` and is free for you to retag. @@ -444,6 +411,17 @@ node's model, instance type, utilization, temperature, power, and memory: For detailed validation steps and the full test matrix (monitoring, containers, CPU/GPU, NCCL, FSDP), see the [Test & Validation Guide](tests/README.md). +> **Note — DCGM exporter version (GPU metrics).** GPU metrics work out of the box across +> the supported range (Hopper / B200 / B300) without overriding `DcgmExporterImage`: +> it defaults to a **DCGM 4.5.2** build pinned by digest (validated on 2× p6-b300 and on +> B200). The monitoring stack's own default (DCGM 4.2.0) tops out at B200 and can't pull +> newer NVCR tags on Docker 29.x — pinning by digest at the deploy-all level is what +> bridges that, and `MonitoringVersion v2.9.1` is the first release carrying the +> `DCGM_EXPORTER_IMAGE` override that lets this through (`v2.6.4`+ carry the other PCS +> fixes: node-local `/opt` install + the Docker-29.x DCGM tag). Override `DcgmExporterImage` +> only to pin a different build; details: +> [OPERATIONS.md §3.1](./docs/OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it). + --- ### 8.2 User Management @@ -457,6 +435,11 @@ nodes — users you add are immediately visible cluster-wide. See the **[User Management Guide](./docs/USER-MANAGEMENT.md)** for step-by-step operations (adding/removing users, Slurm accounting, SSH access). +> **Constraint — single login node.** The OpenLDAP server runs on the (single) login +> node, so enabling `DirectoryService` means a one-login-node cluster. The user database +> lives on OpenZFS `/home`, so it survives a login-node replacement (data is recoverable), +> but the directory **service** is a single point of failure while the login node is down. + ### 8.3 IAM Permissions The cluster has two human roles, each with a ready-to-deploy IAM policy stack: @@ -534,6 +517,59 @@ space skips the download+check entirely, shaving a few seconds off boot. For production deploys that pin the AMI explicitly per cluster, none of these are needed. +### 8.5 CPU compute node group — advanced settings + +The On-Demand CPU queue is configured with `OnDemandInstanceType` (default +`c6i.4xlarge`) plus `OnDemandQueueName` / `OnDemandCngName` / `OnDemandMinCount` / +`OnDemandMaxCount`. The settings below cover tightly-coupled HPC / MPI workloads. + +#### EFA on CPU HPC instances (`OnDemandEfaInterfaceCount`) + +For tightly-coupled HPC / MPI workloads on CPU-only HPC instances, set +`OnDemandEfaInterfaceCount` (deploy-all) or `EfaInterfaceCount` (modular +`add-cng.yaml`) to the instance type's EFA interface count. `0` (default) = no +EFA; `1` or `2` = enable EFA with that many interfaces — the CPU compute node +group then launches with EFA `NetworkInterfaces` and a cluster placement group +is auto-created. GPU CNGs (P5/P6) are unaffected — they use their own dedicated +multi-NIC EFA wiring per-family. + +| Instance type | EFA interfaces | Aggregate spec | Set the count to | +|---|---:|---:|---:| +| (any non-HPC type, e.g. `c6i.4xlarge`) | — | — | **0** (no EFA; default) | +| `hpc6a.48xlarge` | 1 | 100 Gbps | **1** | +| `hpc7a.96xlarge` (and 12/24/48) | 2 | 300 Gbps | **2** | +| `hpc6id.32xlarge` | 2 | 200 Gbps | **2** | +| `hpc8a.96xlarge` | 2 | 300 Gbps | **2** | +| `c7i.metal-48xl` etc. | 1 | varies | **1** | + +(Setting a count > 0 on a non-EFA type, or mismatching the count with the +instance type's actual `MaximumEfaInterfaces`, fails at launch.) + +**Placement group:** auto-created per-CNG by the template. Override with +`OnDemandPlacementGroupName=` to share one PG across multiple +CNGs (e.g. heterogeneous tightly-coupled jobs that span CPU + GPU). + +**Multi-NIC bandwidth needs multiple MPI pairs.** A single MPI pair uses one +libfabric endpoint and only one NIC. Use `osu_mbw_mr -np 32 -N 16` (or your +application's natural multi-pair pattern) to actually exercise both NICs on +hpc7a/hpc8a. See [tests/README.md Test 9](./tests/README.md#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) +for the full benchmark setup and validated bandwidth numbers. + +### 8.6 FSx for Lustre over EFA (GPUDirect Storage) + +`FSxLustreEnableEfa` (default `false`) enables EFA on the `/fsx` Lustre filesystem. +**The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, +which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile +stack on the client — see the Client-side Lustre-on-EFA + GDS item in +[`docs/ROADMAP.md`](./docs/ROADMAP.md)). EFA-capable CPU CNGs +(`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary +benefit, useful when a single client is pushing past ~10 GBps. + +Constraints when enabling this: +- **PERSISTENT_2 SSD only** — a CFN Rule fails the stack at create time on `PERSISTENT_1`. +- **Much higher minimum `Capacity`** — at `PerUnitStorageThroughput=250` the minimum is + **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly. + --- ## 9. Templates @@ -551,7 +587,7 @@ parameter and default, see [PARAMETERS.md](./docs/PARAMETERS.md). | [`add-cng.yaml`](./assets/add-cng.yaml) | Compute node group — login nodes, CPU / single-NIC-GPU queues (C6i, G5, G6); switches to a multi-NIC EFA `NetworkInterfaces` block when `EfaInterfaceCount > 0` (HPC types: hpc6a/hpc7a/hpc6id/hpc8a …) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng.yaml&stackName=pcs-add-cng) | | [`add-cng-p5.yaml`](./assets/add-cng-p5.yaml) | P5/P5e/P5en nodes (16/32 EFA interfaces, by type) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p5.yaml&stackName=pcs-add-cng-p5) | | [`add-cng-p6-b200.yaml`](./assets/add-cng-p6-b200.yaml) | P6-B200 nodes (8 EFA interfaces) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b200.yaml&stackName=pcs-add-cng-p6-b200) | -| [`add-cng-p6-b300.yaml`](./assets/add-cng-p6-b300.yaml) | P6-B300 nodes (16 EFA interfaces) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b300.yaml&stackName=pcs-add-cng-p6-b300) | +| [`add-cng-p6-b300.yaml`](./assets/add-cng-p6-b300.yaml) | P6-B300 nodes (17 interfaces: 16 EFA + 1 ENA) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b300.yaml&stackName=pcs-add-cng-p6-b300) | | [`pcs-ready-dlami-with-enroot-pyxis.yaml`](./assets/pcs-ready-dlami-with-enroot-pyxis.yaml) | EC2 Image Builder: bake Enroot 3.5.0 + Pyxis 0.20.0 into the PCS-Ready DLAMI | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml&stackName=pcs-dlami) | `add-cng*` templates create a Slurm queue only when `QueueName` is set (leave it empty From 50993ee25b047784b9aa2e8ffad4a0f47c9d138f Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:33:00 +0000 Subject: [PATCH 67/89] docs: fix OpenZFS throughput values, note Slurm-version paths, drop stale diagram memo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - OPERATIONS.md: correct OpenZFS HomeThroughput allowed values (192/384/768 are not valid AWS values); SINGLE_AZ_2 groups with HA_2 (160..10240), SINGLE_AZ_HA_1/SINGLE_AZ_1 = 64..4096 — matches the template Rules - USER-MANAGEMENT.md: note that slurm-25.11 paths must become slurm-25.05 when deployed with SlurmVersion=25.05 - delete docs/architecture-components.md: an orphan diagram-drafting memo (not linked anywhere, the architecture image is already published) with stale content (32x EFA on all GPUs, single private subnet, optional Enroot/Pyxis) --- architectures/aws-pcs/docs/OPERATIONS.md | 4 +- architectures/aws-pcs/docs/USER-MANAGEMENT.md | 6 + .../aws-pcs/docs/architecture-components.md | 122 ------------------ 3 files changed, 8 insertions(+), 124 deletions(-) delete mode 100644 architectures/aws-pcs/docs/architecture-components.md diff --git a/architectures/aws-pcs/docs/OPERATIONS.md b/architectures/aws-pcs/docs/OPERATIONS.md index 3ba90df9a..bf55ab694 100644 --- a/architectures/aws-pcs/docs/OPERATIONS.md +++ b/architectures/aws-pcs/docs/OPERATIONS.md @@ -170,8 +170,8 @@ on the deployment type**: |---|---|---| | Lustre | `PERSISTENT_2` (default) | 125 / 250 / 500 / 1000 MB/s/TiB | | Lustre | `PERSISTENT_1` (older Regions) | 50 / 100 / 200 MB/s/TiB | -| OpenZFS | `SINGLE_AZ_HA_2` (default) | 160 / 320 / 640 / 1280 / 2560 / 3840 / 5120 / 7680 MB/s | -| OpenZFS | `SINGLE_AZ_HA_1` / `SINGLE_AZ_2` / `SINGLE_AZ_1` | 64 / 128 / 192 / 256 / 384 / 512 / 768 / 1024 MB/s | +| OpenZFS | `SINGLE_AZ_2` / `SINGLE_AZ_HA_2` (default) | 160 / 320 / 640 / 1280 / 2560 / 3840 / 5120 / 7680 / 10240 MB/s | +| OpenZFS | `SINGLE_AZ_HA_1` / `SINGLE_AZ_1` | 64 / 128 / 256 / 512 / 1024 / 2048 / 3072 / 4096 MB/s | The templates enforce the valid pair via CloudFormation `Rules` so a mismatch fails at stack-create time with a clear message instead of deep in the nested FSx stack. If you diff --git a/architectures/aws-pcs/docs/USER-MANAGEMENT.md b/architectures/aws-pcs/docs/USER-MANAGEMENT.md index e35252466..bd7850656 100644 --- a/architectures/aws-pcs/docs/USER-MANAGEMENT.md +++ b/architectures/aws-pcs/docs/USER-MANAGEMENT.md @@ -17,6 +17,12 @@ Run on the login node. `$ADMIN_PW` is the LDAP admin password from SSM (see vars must be passed **inline to `sudo`** (`sudo VAR=... cmd`) — `sudo -E` alone drops them under the default `env_reset`/`secure_path`. +> **Slurm path matches `SlurmVersion`.** The `sacctmgr` / `sbatch` paths below use +> `slurm-25.11` (the default). If you deployed with `SlurmVersion=25.05`, replace +> `slurm-25.11` with `slurm-25.05` in every path (or just run `export +> PATH=/opt/aws/pcs/scheduler/slurm-$(ls /opt/aws/pcs/scheduler | sed 's/slurm-//')/bin:$PATH` +> once and drop the absolute path). + | Task | Command (run on the login node) | |---|---| | Add a user | `sudo LDAP_ADMIN_PASSWORD="$ADMIN_PW" ldap-add-user.sh alice 10001 3000` | diff --git a/architectures/aws-pcs/docs/architecture-components.md b/architectures/aws-pcs/docs/architecture-components.md deleted file mode 100644 index 3ca5e8af0..000000000 --- a/architectures/aws-pcs/docs/architecture-components.md +++ /dev/null @@ -1,122 +0,0 @@ -# AWS PCS Architecture Components for Diagram - -## Main Components (Required for Diagram) - -### Network Layer -- **VPC** with dual CIDR blocks (10.0.0.0/16 + 10.1.0.0/16) -- **Public Subnet** (NAT Gateway placement) -- **Private Subnet** (compute nodes, FSx placement) -- **S3 VPC Endpoint** (gateway endpoint for template/data access) - -### Storage Layer -- **FSx for Lustre** (scratch storage, high-speed I/O for ML training) - - PERSISTENT_2 deployment type - - Configurable throughput (125-1000 MB/s/TiB) - - LZ4 compression -- **FSx for OpenZFS** (home directories, persistent user data) - - Single-AZ HA deployment - - NFS exports with no_root_squash - - Automatic daily backups - -### Compute Layer -- **AWS PCS Cluster** (Slurm scheduler) - - Head node (managed service, not visible) - - Slurm versions: 25.05, 25.11 -- **Login Node Group** (public subnet) - - SSH/SSM access point - - Container tooling (Enroot/Pyxis) - - Monitoring stack (Prometheus, Grafana) -- **On-Demand Compute Node Group** (private subnet) - - CPU-based workloads - - Auto-scaling with Slurm - - Optional Enroot/Pyxis via UserData -- **GPU Compute Node Group - P5/P6** (private subnet) - - 32x EFA network interfaces - - Multi-node ML training - - Optional Enroot/Pyxis via UserData - - Dynamic scaling (MinCount=0) - -### Optional Components -- **EC2 Image Builder** (custom AMI with Enroot/Pyxis pre-installed) - - Scheduled builds (manual, weekly, monthly) - - Published to SSM Parameter Store -- **Monitoring Stack** (on login node) - - Prometheus (metrics collection) - - Grafana (visualization) - - DCGM Exporter (GPU metrics) - - Slurm OpenMetrics - -## Key Architectural Features - -### Dual Deployment Modes -1. **Custom AMI Mode** (Production) - - ImageBuilder pre-installs Enroot/Pyxis (~20-30 min build) - - Fast node boot (~3 minutes) - - Best for production clusters - -2. **UserData Installation Mode** (Testing) - - Enroot/Pyxis installed on first boot (~8-12 minutes) - - No AMI build required - - Best for rapid iteration and testing - -### Prerequisites Stack Reuse -- Network and storage resources can be deployed once (pcs-shared-infra) -- Multiple clusters reference shared infrastructure via CloudFormation Exports/Imports -- Reduces deployment time from 20-30 minutes to 5-10 minutes for subsequent clusters - -### Dual Tagging Strategy -- **ClusterName tag**: User-friendly name (CloudFormation stack name) -- **pcs-cluster-id tag**: Actual PCS cluster ID (e.g., pcs_i3bddqwdrp) -- Both applied to all instances (login nodes, compute nodes) -- Used by monitoring stack for IMDS metadata and SSM parameter paths - -## Data Flow - -1. **User Access**: - - Users → SSH/SSM → Login Node (public subnet) - -2. **Job Submission**: - - Login Node → Slurm Scheduler (PCS managed) - - Scheduler → Compute Nodes (private subnet) - -3. **Storage Access**: - - Compute Nodes → FSx for Lustre (scratch, training data) - - Compute Nodes → FSx for OpenZFS (home directories) - - Compute Nodes → S3 (via VPC endpoint, checkpoints/results) - -4. **Container Workflow**: - - Users → Login Node (Enroot/Pyxis commands) - - Slurm jobs → Pull container images (Docker Hub, NVIDIA NGC) - - Compute Nodes → Run containerized training jobs - -5. **Monitoring**: - - Compute Nodes → Prometheus (metrics push) - - Users → Grafana dashboard (via Login Node) - - Slurm → OpenMetrics exporter → Prometheus - -## Security Features - -- **Network Isolation**: Compute nodes in private subnet, NAT Gateway for outbound only -- **Security Group**: All-to-all EFA communication within cluster -- **IAM Roles**: Integrated instance profiles for S3/SSM access -- **SSM Session Manager**: SSH alternative for login node access - -## Diagram Layout Recommendations - -- **Top Layer**: User access (SSH/SSM) → Login Node -- **Middle Layer**: Slurm scheduler (PCS managed, abstract as service icon) -- **Compute Layer**: On-Demand + GPU node groups (separate swim lanes) -- **Storage Layer**: FSx for Lustre + OpenZFS (bottom or side panel) -- **S3 VPC Endpoint**: Connection from private subnet to S3 service -- **Optional Panel**: ImageBuilder + Monitoring stack (dashed border) - -## Components to Exclude from Diagram - -- Internet Gateway (generic networking) -- NAT Gateway (generic networking) -- Route Tables (generic networking) -- Elastic IP (generic networking) -- Security Group rules (detail level) -- IAM roles/policies (non-visual) -- CloudFormation stacks (deployment mechanism) -- SSM parameters (configuration storage) From 90cab6bd141f025256609eb871c3daea547274e5 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:44:47 +0000 Subject: [PATCH 68/89] docs: align README config order with deploy-all, refresh features/architecture, slim test memos MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README: - §1 Key Features: add multi-user (OpenLDAP), access control (IAM + SSH/Grafana CIDR), and multi-AZ / broad Region coverage - §2 Architecture: subnets across up to 3 AZs, SSH/Grafana CIDR access, and the optional OpenLDAP / IAM add-ons - §4 Configuration: reorder the parameter table to match the deploy-all console parameter-group order (Network → PCS Cluster → On-Demand → GPU → Additional); point OnDemandEfaInterfaceCount at §8.5 tests/README: - drop the one-off 'GPU health check verified on B200' memo and the Mumbai 'exercised most deeply' paragraph (results live in the per-test files) - region-coverage GPU column is now a simple ran-on-reserved-capacity check (us-east-1/us-east-2/us-west-2 verified via Capacity Block for ML); drop the per-instance-type detail from the table --- architectures/aws-pcs/README.md | 24 +++++++++++++++++------- architectures/aws-pcs/tests/README.md | 26 +++++++------------------- 2 files changed, 24 insertions(+), 26 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 8c32a1b2f..e420d4ed3 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -12,6 +12,9 @@ This repository provides reference architectures and deployment templates for se - **GPU-ready, multi-NIC EFA**: dedicated launch templates for the P5 and P6 families, selected automatically by instance type, for high-bandwidth multi-node training. - **Flexible capacity options**: On-Demand, "open" On-Demand Capacity Reservations (ODCR, consumed automatically), and Capacity Blocks for ML — selected per node group. (Targeting a *specific* ODCR is on the [roadmap](./docs/ROADMAP.md).) - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). +- **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent POSIX users cluster-wide — pairs with Slurm accounting (`DirectoryService=OpenLDAP-LoginNode`). +- **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR (`SSHAccessCidr` / `GrafanaAccessCidr`). +- **Multi-AZ and broad Region coverage**: place private subnets across up to 3 AZs for availability or AZ-matched capacity; validated end-to-end across 10 Regions. - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. > Built on the AWS-managed **PCS-Ready DLAMI** (NVIDIA driver, CUDA, PCS agent, and @@ -26,14 +29,18 @@ This repository provides reference architectures and deployment templates for se ![AWS PCS diagram](./images/ml-pcs-architecture.png) A default deployment (`pcs-ml-cluster-deploy-all.yaml`) provisions: -- VPC with public/private subnets, NAT gateway, and S3 endpoint +- VPC with a public subnet and private subnets in up to 3 AZs, a NAT gateway (primary AZ), and an S3 endpoint - FSx for Lustre (`/fsx`, high-performance shared scratch) and FSx for OpenZFS (`/home`) - PCS cluster with the Slurm scheduler (25.05 or 25.11), on the PCS-Ready DLAMI -- Login node group (public subnet) with the monitoring stack (Prometheus + Grafana + Nginx) +- Login node group (public subnet) with the monitoring stack (Prometheus + Grafana + Nginx); SSH/Grafana can be opened to a trusted CIDR - CPU compute node group (private subnet); EFA can be enabled for HPC/MPI workloads - Optional GPU (P5/P6) node group with multi-NIC EFA, plus DCGM Exporter for the GPU dashboards - Enroot/Pyxis container runtime installed at first boot via `PostInstallScriptUrl` (or pre-baked into a custom AMI you build separately and pass as `AmiId`) +Optional add-ons (off by default): +- **Multi-user directory**: OpenLDAP on the login node + SSSD on every compute node (`DirectoryService`) +- **IAM policy stacks**: least-privilege cluster-admin / cluster-user policies you can deploy separately + --- ## 3. Quick Start @@ -85,22 +92,25 @@ Defaults give the most common production setup — Enroot/Pyxis installed at fir boot via `PostInstallScriptUrl` + `MonitoringStack=Prometheus-LoginNode` — so a default deploy only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: +The rows below follow the same order as the deploy-all console parameter groups +(Network → PCS Cluster → On-Demand CNG → GPU CNG → Additional Configuration). + | Parameter | Default | Purpose | |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | +| `AdditionalSubnetAZ2` / `AdditionalSubnetAZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2) for higher availability or AZ-matched capacity. NAT gateway stays in the primary AZ | | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | -| `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | +| `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | | `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | | `OnDemandInstanceType` | `c6i.4xlarge` | Instance type for the On-Demand queue. Not limited to CPU — set any On-Demand type, e.g. a single-NIC GPU like `g6.12xlarge` (see [Example 1](#example-1-single-nic-gpu-queue-g6)) or an EFA-capable HPC type like `hpc7a.96xlarge`. (Multi-NIC P5/P6 GPUs use the P-series queue instead — see `PseriesInstanceType`.) Tune `OnDemandQueueName`/`OnDemandCngName`/`OnDemandMinCount`/`OnDemandMaxCount` alongside it | -| `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [EFA on CPU HPC instances](#efa-on-cpu-hpc-instances-ondemandefainterfacecount) | +| `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [§8.5 CPU compute node group](#85-cpu-compute-node-group--advanced-settings) | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | | `PseriesInstanceType` | `p5.48xlarge` | Multi-NIC GPU instance type; auto-selects the matching template + EFA NIC count. Accepted values: `p5.48xlarge` / `p5e.48xlarge` (→ add-cng-p5, 32 NICs), `p5en.48xlarge` (→ add-cng-p5, 16 NICs), `p6-b200.48xlarge` (→ add-cng-p6-b200, 8 NICs), `p6-b300.48xlarge` (→ add-cng-p6-b300, 17 NICs). See [GPU compute](#gpu-compute-p5p6) | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -| `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | -| `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | +| `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 on the login node to a trusted CIDR for direct Grafana access (default: SSM port-forward only). See [§8.1 Monitoring](#accessing-the-grafana-dashboards) | -| `AdditionalSubnetAZ2` / `AdditionalSubnetAZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2) for higher availability or AZ-matched capacity. NAT gateway stays in the primary AZ | +| `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index f765f3a22..51559beab 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -23,11 +23,6 @@ Run this **complete set** before merging template or script changes. | 11-12 | **Multi-user** | OpenLDAP directory + Slurm managed accounting | [`multi-user-test.md`](./multi-user-test.md) | Directory / accounting changes | | 13 | **GPU health** | GPU Cluster Health Check suite (DCGM, EFA, NVLink, NCCL thresholds) | [`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md) | GPU CNG deploys | -> **GPU health check verified on B200** (ap-south-1 p6-b200): lightweight suite -> **4/4 PASS** — nvidia-smi (8 GPUs, no Xid/ECC), DCGM L2 diagnostics, EFA -> enumeration (8 PCI / 10 RDMA devices), topology (8 GPUs, connectivity validated; -> a B200-specific "unsupported P2P path" warning is non-blocking). - --- ## Quick-start: single-cluster shortcut @@ -76,16 +71,16 @@ The templates fetch nested stacks + boot scripts from a single S3 bucket Columns: **Deploy** (deploy-all → CREATE_COMPLETE), **Mon** (6 monitoring containers up on the login node), **Storage** (FSx Lustre `/fsx` + OpenZFS `/home` created & mounted — with the OpenZFS deployment type that worked), -**Pyxis** (Enroot/Pyxis container job runs), **GPU** (multi-NIC GPU CNG tested, -incl. capacity reservation type), **Verified** (date, UTC). +**Pyxis** (Enroot/Pyxis container job runs), **GPU** (a GPU CNG launched and ran +on reserved capacity — Capacity Block or ODCR), **Verified** (date, UTC). -| Region | Deploy | Mon | Storage (OpenZFS type) | Pyxis | GPU (CB/ODCR) | Verified | +| Region | Deploy | Mon | Storage (OpenZFS type) | Pyxis | GPU | Verified | |---|---|---|---|---|---|---| -| **us-east-1** (N. Virginia) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | -| **us-east-2** (Ohio) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | EFA hpc6a(1)/hpc8a(2) — CPU/EFA | 2026-06-17 | -| **us-west-2** (Oregon) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | +| **us-east-1** (N. Virginia) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ | 2026-06-17 | +| **us-east-2** (Ohio) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ | 2026-06-17 | +| **us-west-2** (Oregon) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ | 2026-06-17 | | **ap-northeast-1** (Tokyo) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | -| **ap-south-1** (Mumbai) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ **p6-b200 ×4** (Capacity Block) | 2026-06-17 | +| **ap-south-1** (Mumbai) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | ✅ | 2026-06-17 | | **ap-northeast-3** (Osaka) | ✅ | ✅ | ✅ `SINGLE_AZ_1` ¹ | ✅ | — | 2026-06-17 | | **ap-southeast-1** (Singapore) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | | **ap-southeast-2** (Sydney) | ✅ | ✅ | ✅ `SINGLE_AZ_HA_2` | ✅ | — | 2026-06-17 | @@ -111,13 +106,6 @@ the defaults (`PERSISTENT_2` / `SINGLE_AZ_HA_2`), as Osaka above shows: (London), `eu-west-3` (Paris), `sa-east-1` (São Paulo), `us-gov-east-1`, `us-gov-west-1` (GovCloud). -The ap-south-1 (Mumbai) row was exercised most deeply — p6-b200 ×4 GPU. The -detailed results live with their tests, not here: NCCL all_reduce numbers in -[compute-test.md](./compute-test.md#test-6-nccl-multi-node-efa), distributed -training (and the HuggingFace rate-limit caveat) in -[training-test.md](./training-test.md#test-7-fsdp-sample-training), GPU health -check in [gpu-healthcheck-test.md](./gpu-healthcheck-test.md). - Cross-region note: nested-stack `TemplateURL` and the in-instance `aws s3 cp` of boot scripts both work against an S3 bucket in a **different** region (S3 global namespace; no `--region` needed) — verified ap-south-1 → From 621cb9ea7b8ec23d60acd6b5fb635bfc5a071334 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:47:21 +0000 Subject: [PATCH 69/89] docs: drop multi-AZ/Region key-feature bullet, note GPU column is from earlier rounds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README §1: remove the 'Multi-AZ and broad Region coverage' Key Features bullet (multi-AZ stays documented in §4 / Architecture; not a headline feature) - tests/README region coverage: clarify the GPU column ✅ reflects earlier GPU validation rounds, not necessarily the row's deploy/monitoring verification date --- architectures/aws-pcs/README.md | 1 - architectures/aws-pcs/tests/README.md | 4 +++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index e420d4ed3..634897af3 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -14,7 +14,6 @@ This repository provides reference architectures and deployment templates for se - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). - **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent POSIX users cluster-wide — pairs with Slurm accounting (`DirectoryService=OpenLDAP-LoginNode`). - **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR (`SSHAccessCidr` / `GrafanaAccessCidr`). -- **Multi-AZ and broad Region coverage**: place private subnets across up to 3 AZs for availability or AZ-matched capacity; validated end-to-end across 10 Regions. - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. > Built on the AWS-managed **PCS-Ready DLAMI** (NVIDIA driver, CUDA, PCS agent, and diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 51559beab..a9a19434e 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -72,7 +72,9 @@ Columns: **Deploy** (deploy-all → CREATE_COMPLETE), **Mon** (6 monitoring containers up on the login node), **Storage** (FSx Lustre `/fsx` + OpenZFS `/home` created & mounted — with the OpenZFS deployment type that worked), **Pyxis** (Enroot/Pyxis container job runs), **GPU** (a GPU CNG launched and ran -on reserved capacity — Capacity Block or ODCR), **Verified** (date, UTC). +on reserved capacity — Capacity Block or ODCR; ✅ marks regions where this was +exercised in **earlier** GPU validation rounds, not necessarily on the row's +deploy/monitoring/storage verification date), **Verified** (date, UTC). | Region | Deploy | Mon | Storage (OpenZFS type) | Pyxis | GPU | Verified | |---|---|---|---|---|---|---| From e91c9e12fa1f2c04f87bcd0f6d0c64f87f6059d4 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:48:32 +0000 Subject: [PATCH 70/89] =?UTF-8?q?docs(readme):=20keep=20Key=20Features=20c?= =?UTF-8?q?oncise=20=E2=80=94=20drop=20param=20names,=20link=20to=20detail?= =?UTF-8?q?=20sections?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Key Features bullets describe capability + link to the relevant section instead of embedding parameter names (MonitoringStack/GrafanaAccessCidr/DirectoryService/ SSHAccessCidr). Detailed config stays in §4 and §8. --- architectures/aws-pcs/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 634897af3..ba0247cfd 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -8,12 +8,12 @@ This repository provides reference architectures and deployment templates for se - **One click to an ML-training-ready cluster**: a single CloudFormation stack gives you a complete, ready-to-train environment — Slurm scheduler, GPU compute with EFA, shared FSx storage, the Enroot/Pyxis container runtime, and monitoring — with only the Availability Zone to choose. Submit distributed training jobs minutes after launch. - **Container runtime included**: Enroot/Pyxis is set up automatically, so `srun --container-image=...` works out of the box for containerized training. -- **Monitoring built in**: Grafana + Prometheus run on the login node, with DCGM Exporter on GPU compute nodes feeding the pre-built GPU dashboards (`MonitoringStack=Prometheus-LoginNode`, on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR with `GrafanaAccessCidr`. +- **Monitoring built in**: Grafana + Prometheus on the login node, with DCGM Exporter on GPU nodes feeding pre-built GPU dashboards (on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR. See [§8.1 Monitoring](#81-monitoring). - **GPU-ready, multi-NIC EFA**: dedicated launch templates for the P5 and P6 families, selected automatically by instance type, for high-bandwidth multi-node training. -- **Flexible capacity options**: On-Demand, "open" On-Demand Capacity Reservations (ODCR, consumed automatically), and Capacity Blocks for ML — selected per node group. (Targeting a *specific* ODCR is on the [roadmap](./docs/ROADMAP.md).) +- **Flexible capacity options**: On-Demand, "open" On-Demand Capacity Reservations (consumed automatically), and Capacity Blocks for ML — selected per node group. (Targeting a *specific* ODCR is on the [roadmap](./docs/ROADMAP.md).) - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). -- **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent POSIX users cluster-wide — pairs with Slurm accounting (`DirectoryService=OpenLDAP-LoginNode`). -- **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR (`SSHAccessCidr` / `GrafanaAccessCidr`). +- **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent users — pairs with Slurm accounting. See [§8.2 User Management](#82-user-management). +- **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR. See [§8.3 IAM Permissions](#83-iam-permissions). - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. > Built on the AWS-managed **PCS-Ready DLAMI** (NVIDIA driver, CUDA, PCS agent, and From ff66f10eeba6b8ee43a96f29e3781de85d97162e Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:53:14 +0000 Subject: [PATCH 71/89] =?UTF-8?q?docs(readme):=20tidy=20=C2=A74=20intro=20?= =?UTF-8?q?=E2=80=94=20single=20concise=20lead-in,=20group=20separators=20?= =?UTF-8?q?in=20the=20table?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - replace the two overlapping intro paragraphs with one: only PrimarySubnetAZ is required; the table is the most-used subset grouped by the console's parameter groups (separator rows), with storage in its own table and the full reference in PARAMETERS.md - keep the in-table group separator rows for discoverability --- architectures/aws-pcs/README.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index ba0247cfd..a828c4ebd 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -87,26 +87,30 @@ are deleted with the stack. ## 4. Configuration -Defaults give the most common production setup — Enroot/Pyxis installed at first -boot via `PostInstallScriptUrl` + `MonitoringStack=Prometheus-LoginNode` — so a default deploy -only needs the Availability Zone (`PrimarySubnetAZ`). The most-used parameters: - -The rows below follow the same order as the deploy-all console parameter groups -(Network → PCS Cluster → On-Demand CNG → GPU CNG → Additional Configuration). +The defaults give a working ML-training cluster, so the only required parameter is +the Availability Zone (`PrimarySubnetAZ`); everything else is optional. The table +below is the most-used subset, grouped by the deploy-all console's parameter groups +(separator rows show each group). Storage parameters have their [own table](#storage-fsx-deployment-types-region-availability) +below; for the complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | Parameter | Default | Purpose | |---|---|---| +| **— 1. Network Configuration —** | | | | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `AdditionalSubnetAZ2` / `AdditionalSubnetAZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2) for higher availability or AZ-matched capacity. NAT gateway stays in the primary AZ | +| **— 2. PCS Cluster Configuration —** | | | | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | | `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | +| **— 4. On-Demand Compute Node Group —** | | | | `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | | `OnDemandInstanceType` | `c6i.4xlarge` | Instance type for the On-Demand queue. Not limited to CPU — set any On-Demand type, e.g. a single-NIC GPU like `g6.12xlarge` (see [Example 1](#example-1-single-nic-gpu-queue-g6)) or an EFA-capable HPC type like `hpc7a.96xlarge`. (Multi-NIC P5/P6 GPUs use the P-series queue instead — see `PseriesInstanceType`.) Tune `OnDemandQueueName`/`OnDemandCngName`/`OnDemandMinCount`/`OnDemandMaxCount` alongside it | | `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [§8.5 CPU compute node group](#85-cpu-compute-node-group--advanced-settings) | +| **— 5. GPU Compute Node Group (P5/P6, optional) —** | | | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | | `PseriesInstanceType` | `p5.48xlarge` | Multi-NIC GPU instance type; auto-selects the matching template + EFA NIC count. Accepted values: `p5.48xlarge` / `p5e.48xlarge` (→ add-cng-p5, 32 NICs), `p5en.48xlarge` (→ add-cng-p5, 16 NICs), `p6-b200.48xlarge` (→ add-cng-p6-b200, 8 NICs), `p6-b300.48xlarge` (→ add-cng-p6-b300, 17 NICs). See [GPU compute](#gpu-compute-p5p6) | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | +| **— 7. Additional Cluster Configuration (Monitoring, Multi-User) —** | | | | `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 on the login node to a trusted CIDR for direct Grafana access (default: SSM port-forward only). See [§8.1 Monitoring](#accessing-the-grafana-dashboards) | | `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | From d59fce81063baad69ccec052cf1db748e6cb46b9 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 16:59:25 +0000 Subject: [PATCH 72/89] =?UTF-8?q?deploy-all:=20regroup=20params=20?= =?UTF-8?q?=E2=80=94=20fold=20Container=20Runtime=20into=20Additional,=20m?= =?UTF-8?q?ove=20FSx=20after=20it?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Container Runtime (PostInstallScriptUrl/Args) is rarely touched directly, so it no longer gets its own console group: folded into '5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)'. FSx Storage moves after Additional (now group 6). New order: Network, PCS Cluster, On-Demand, GPU, Additional, FSx, Developer. README §4 separators and PARAMETERS.md §2b updated to match. --- architectures/aws-pcs/README.md | 6 ++-- .../assets/pcs-ml-cluster-deploy-all.yaml | 33 +++++++++---------- architectures/aws-pcs/docs/PARAMETERS.md | 9 ++--- 3 files changed, 20 insertions(+), 28 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index a828c4ebd..1c437d89b 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -102,15 +102,15 @@ below; for the complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | | `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | -| **— 4. On-Demand Compute Node Group —** | | | +| **— 3. On-Demand Compute Node Group —** | | | | `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | | `OnDemandInstanceType` | `c6i.4xlarge` | Instance type for the On-Demand queue. Not limited to CPU — set any On-Demand type, e.g. a single-NIC GPU like `g6.12xlarge` (see [Example 1](#example-1-single-nic-gpu-queue-g6)) or an EFA-capable HPC type like `hpc7a.96xlarge`. (Multi-NIC P5/P6 GPUs use the P-series queue instead — see `PseriesInstanceType`.) Tune `OnDemandQueueName`/`OnDemandCngName`/`OnDemandMinCount`/`OnDemandMaxCount` alongside it | | `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [§8.5 CPU compute node group](#85-cpu-compute-node-group--advanced-settings) | -| **— 5. GPU Compute Node Group (P5/P6, optional) —** | | | +| **— 4. GPU Compute Node Group (P5/P6, optional) —** | | | | `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | | `PseriesInstanceType` | `p5.48xlarge` | Multi-NIC GPU instance type; auto-selects the matching template + EFA NIC count. Accepted values: `p5.48xlarge` / `p5e.48xlarge` (→ add-cng-p5, 32 NICs), `p5en.48xlarge` (→ add-cng-p5, 16 NICs), `p6-b200.48xlarge` (→ add-cng-p6-b200, 8 NICs), `p6-b300.48xlarge` (→ add-cng-p6-b300, 17 NICs). See [GPU compute](#gpu-compute-p5p6) | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -| **— 7. Additional Cluster Configuration (Monitoring, Multi-User) —** | | | +| **— 5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime) —** | | | | `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 on the login node to a trusted CIDR for direct Grafana access (default: SSM port-forward only). See [§8.1 Monitoring](#accessing-the-grafana-dashboards) | | `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | diff --git a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml index 09d5b1c05..a8e54e6e4 100644 --- a/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml +++ b/architectures/aws-pcs/assets/pcs-ml-cluster-deploy-all.yaml @@ -30,12 +30,7 @@ Metadata: - ManagedAccounting - AccountingPolicyEnforcement - Label: - default: "3. Container Runtime (Post-install Script)" - Parameters: - - PostInstallScriptUrl - - PostInstallScriptArgs - - Label: - default: "4. On-Demand Compute Node Group" + default: "3. On-Demand Compute Node Group" Parameters: - DeployOnDemandCNG - OnDemandInstanceType @@ -46,7 +41,7 @@ Metadata: - OnDemandEfaInterfaceCount - OnDemandPlacementGroupName - Label: - default: "5. GPU Compute Node Group - P5/P6 Series (Optional)" + default: "4. GPU Compute Node Group - P5/P6 Series (Optional)" Parameters: - DeployPseriesCNG - PseriesInstanceType @@ -55,6 +50,18 @@ Metadata: - CapacityReservationId - PseriesCngName - PseriesQueueName + - Label: + default: "5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)" + Parameters: + - MonitoringStack + - GrafanaAccessCidr + - MonitoringRepo + - MonitoringVersion + - DcgmExporterImage + - DirectoryService + - DirectoryDomainSuffix + - PostInstallScriptUrl + - PostInstallScriptArgs - Label: default: "6. FSx Storage (/fsx and /home)" Parameters: @@ -68,17 +75,7 @@ Metadata: - HomeThroughput - OpenZFSDeploymentType - Label: - default: "7. Additional Cluster Configuration (Monitoring, Multi-User)" - Parameters: - - MonitoringStack - - GrafanaAccessCidr - - MonitoringRepo - - MonitoringVersion - - DcgmExporterImage - - DirectoryService - - DirectoryDomainSuffix - - Label: - default: "8. Developer / Advanced (do not change for normal use)" + default: "7. Developer / Advanced (do not change for normal use)" Parameters: - S3BucketName - S3KeyPrefix diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 0b6d5c011..4081d95f8 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -40,7 +40,7 @@ it directly. | `ManagedAccounting` | `disabled` | Enable Slurm managed accounting (requires Slurm 24.11+) | | `AccountingPolicyEnforcement` | `none` | Slurm accounting policy enforcement (`none` or `associations,limits,safe`) | -## 2b. Additional Cluster Configuration (Monitoring, Multi-User) +## 2b. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime) | Parameter | Default | Purpose | |---|---|---| @@ -48,13 +48,8 @@ it directly. | `GrafanaAccessCidr` | *(empty)* | When set to a CIDR, opens HTTPS/443 (Grafana) on the login node to that CIDR via the login-only security group. Empty = SSM port-forward only. **443 also exposes the unauthenticated `/prometheus/`, `/pushgateway/`, `/slurmexporter/` proxy paths**, not just the password-gated Grafana. Use the tightest CIDR you can. (Renamed from `GrafanaPublicAccessCidr`) | | `DirectoryService` | `none` | Multi-user directory. `none` = single `ubuntu` user. `OpenLDAP-LoginNode` = slapd on the login node (DB on shared `/home/ldap-db`) + SSSD on all compute nodes. **Single login node only** — keep the login node group at 1 instance while enabled. See [USER-MANAGEMENT.md](./USER-MANAGEMENT.md) | | `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP domain suffix. Only used when `DirectoryService != none` | - -## 3. Container Runtime (Post-install Script) - -| Parameter | Default | Purpose | -|---|---|---| | `PostInstallScriptUrl` | *(empty → auto)* | Script run on every node at first boot (PCS equivalent of ParallelCluster `OnNodeConfigured`). **Empty (default) auto-installs Enroot/Pyxis** from `s3:///scripts/install-enroot-pyxis.sh` (fetched with the instance role, so it works with a **private** bucket — no public S3 needed). Accepts an `s3://` URL (instance-role fetch) or an `http(s)://` URL (curl, public only, e.g. GitHub raw). Set to a single space to skip. Idempotent: a no-op if Enroot/Pyxis is already pre-baked into `AmiId` | -| `PostInstallScriptArgs` | *(empty)* | Arguments passed to the post-install script | +| `PostInstallScriptArgs` | *(empty)* | Arguments passed to the post-install script. Normally left empty — most users never touch the container-runtime parameters | ## 4. On-Demand Compute Node Group (CPU) From 0cd9c1baf9879804569db25e29facf9090c1ae25 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:06:28 +0000 Subject: [PATCH 73/89] =?UTF-8?q?docs(readme):=20tighten=20Architecture/Mo?= =?UTF-8?q?nitoring,=20link=20GPU=20health=20check=20from=20=C2=A77?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - move the PCS-Ready DLAMI 'no AMI build needed' note from Key Features into §2 Architecture, pointing to §4 'AMI and container runtime' for the detail (dedup) - §7: link the GPU Cluster Health Check suite as a pre-long-run check - §8.1 Monitoring: replace the MonitoringRepo/Version/DcgmExporterImage bullets with a one-line pointer to PARAMETERS.md (users rarely change these); move the monitoring-role vs Name-tag note to the end of §8.1 --- architectures/aws-pcs/README.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 1c437d89b..78bf746ab 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -16,13 +16,6 @@ This repository provides reference architectures and deployment templates for se - **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR. See [§8.3 IAM Permissions](#83-iam-permissions). - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. -> Built on the AWS-managed **PCS-Ready DLAMI** (NVIDIA driver, CUDA, PCS agent, and -> Slurm 25.05/25.11 pre-installed), so no custom AMI build is required by default — -> the cluster comes up without an Image Builder step. For frequent scaling, you can -> pre-bake Enroot/Pyxis into a custom DLAMI with the standalone -> [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) -> template and pass the result as `AmiId`. - ## 2. Architecture ![AWS PCS diagram](./images/ml-pcs-architecture.png) @@ -40,6 +33,12 @@ Optional add-ons (off by default): - **Multi-user directory**: OpenLDAP on the login node + SSSD on every compute node (`DirectoryService`) - **IAM policy stacks**: least-privilege cluster-admin / cluster-user policies you can deploy separately +Every node runs on the AWS-managed **PCS-Ready DLAMI** (NVIDIA driver, CUDA, PCS agent, +and Slurm 25.05/25.11 pre-installed), so **no custom AMI build is required** — the cluster +comes up without an Image Builder step, and Enroot/Pyxis is layered on at first boot. +See [AMI and container runtime](#ami-and-container-runtime) for pinning the AMI and the +optional pre-bake path for faster scaling. + --- ## 3. Quick Start @@ -306,6 +305,10 @@ In `nccl-all_reduce_perf_.out`, EFA is active when you see expected bandwidth numbers per instance family and tuning notes, see [tests/compute-test.md](./tests/compute-test.md#test-6-nccl-multi-node-efa). +Before a long run, it's also worth checking GPU/EFA/NVLink health with the +[GPU Cluster Health Check suite](./tests/gpu-healthcheck-test.md) (nvidia-smi, DCGM +diagnostics, EFA enumeration, NCCL thresholds). + For a full training example, see the [PyTorch FSDP test case](../../3.test_cases/pytorch/FSDP); the full validation matrix is in [tests/README.md](./tests/README.md). @@ -333,14 +336,10 @@ and a screenshot. > for Prometheus + Amazon Managed Grafana instead of the self-hosted stack on the login > node, see [`4.validation_and_observability/4.prometheus-grafana`](../../4.validation_and_observability/4.prometheus-grafana). -**Monitoring-related parameters:** -- `MonitoringStack` (default `Prometheus-LoginNode`; `none` disables monitoring) -- `MonitoringVersion` — [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`; default `v2.9.1`). Pinned to a tag so upstream changes can't break deployments unexpectedly. -- `MonitoringRepo` — `owner/repo` to fetch from (default `aws-samples/aws-parallelcluster-monitoring`). Point at a fork + a branch in `MonitoringVersion` to test unreleased changes. -- `DcgmExporterImage` — dcgm-exporter image used on GPU nodes (default covers Hopper/B200/B300; see the DCGM note below). - -> Node type is identified by the `monitoring-role` tag (`login`/`compute`), not the EC2 -> `Name` tag — the `Name` tag defaults to `PCS-` and is free for you to retag. +`MonitoringStack` toggles the stack (`Prometheus-LoginNode` / `none`). The defaults work +out of the box; the source repo/version (`MonitoringRepo` / `MonitoringVersion`) and the +DCGM exporter image (`DcgmExporterImage`) are pinned and rarely need changing — see +[PARAMETERS.md](./docs/PARAMETERS.md) if you do. #### Accessing the Grafana dashboards @@ -435,6 +434,10 @@ NCCL, FSDP), see the [Test & Validation Guide](tests/README.md). > only to pin a different build; details: > [OPERATIONS.md §3.1](./docs/OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it). +> **Note — node-type tagging.** The monitoring stack identifies login vs compute nodes by +> the `monitoring-role` tag (`login`/`compute`), **not** the EC2 `Name` tag — so the `Name` +> tag (default `PCS-`) is free for you to retag without breaking dashboards. + --- ### 8.2 User Management From 1c13036b292c10c7d06580a42712714ed4ad406a Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:08:44 +0000 Subject: [PATCH 74/89] =?UTF-8?q?docs(readme):=20split=20=C2=A74=20config?= =?UTF-8?q?=20table=20into=20small=20per-group=20tables?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the single long table (with in-table separator rows and verbose Purpose cells) with one small table per console parameter group (Network / PCS cluster / On-Demand / GPU / Additional). Trim each Purpose to one line and move the long PseriesInstanceType accepted-values list to the GPU compute subsection. Easier to scan; group structure is obvious from the subheadings. --- architectures/aws-pcs/README.md | 57 +++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 20 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 78bf746ab..3e0398a19 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -87,32 +87,49 @@ are deleted with the stack. ## 4. Configuration The defaults give a working ML-training cluster, so the only required parameter is -the Availability Zone (`PrimarySubnetAZ`); everything else is optional. The table -below is the most-used subset, grouped by the deploy-all console's parameter groups -(separator rows show each group). Storage parameters have their [own table](#storage-fsx-deployment-types-region-availability) -below; for the complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). +the Availability Zone (`PrimarySubnetAZ`); everything else is optional. The most-used +parameters are grouped below to match the deploy-all console's parameter groups. Storage +has its [own table](#storage-fsx-deployment-types-region-availability) below; for the +complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). + +**Network** | Parameter | Default | Purpose | |---|---|---| -| **— 1. Network Configuration —** | | | | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | -| `AdditionalSubnetAZ2` / `AdditionalSubnetAZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2) for higher availability or AZ-matched capacity. NAT gateway stays in the primary AZ | -| **— 2. PCS Cluster Configuration —** | | | -| `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`); 25.11 is needed for the Slurm OpenMetrics dashboards. Drives Pyxis build version too. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | -| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves to the latest **PCS-Ready Deep Learning AMI** (Ubuntu 24.04) from SSM. Pin to a specific `ami-xxx` for production, or pass an AMI built by [`pcs-ready-dlami-with-enroot-pyxis.yaml`](#84-pre-baking-enrootpyxis-into-a-custom-ami) | -| `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR for direct SSH (default: SSM only, no public access). See [§6 Accessing the Cluster](#6-accessing-the-cluster) | -| **— 3. On-Demand Compute Node Group —** | | | +| `AdditionalSubnetAZ2` / `…AZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2). NAT gateway stays in the primary AZ | + +**PCS cluster** + +| Parameter | Default | Purpose | +|---|---|---| +| `SlurmVersion` | `25.11` | `25.05` or `25.11`; 25.11 is needed for the Slurm OpenMetrics dashboards. See [OPERATIONS.md §1](./docs/OPERATIONS.md#1-slurm-version-selection) | +| `AmiId` | *(empty → SSM auto)* | Empty auto-resolves the latest PCS-Ready DLAMI. See [AMI and container runtime](#ami-and-container-runtime) | +| `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR (default: SSM only). See [§6](#6-accessing-the-cluster) | + +**On-Demand compute queue** + +| Parameter | Default | Purpose | +|---|---|---| | `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | -| `OnDemandInstanceType` | `c6i.4xlarge` | Instance type for the On-Demand queue. Not limited to CPU — set any On-Demand type, e.g. a single-NIC GPU like `g6.12xlarge` (see [Example 1](#example-1-single-nic-gpu-queue-g6)) or an EFA-capable HPC type like `hpc7a.96xlarge`. (Multi-NIC P5/P6 GPUs use the P-series queue instead — see `PseriesInstanceType`.) Tune `OnDemandQueueName`/`OnDemandCngName`/`OnDemandMinCount`/`OnDemandMaxCount` alongside it | -| `OnDemandEfaInterfaceCount` | `0` | `0` = no EFA (default). Set `1`/`2` for HPC/MPI workloads on EFA-capable CPU types (hpc6a=1, hpc7a/hpc8a/hpc6id=2). Enables EFA `NetworkInterfaces` + auto-creates a cluster placement group. See [§8.5 CPU compute node group](#85-cpu-compute-node-group--advanced-settings) | -| **— 4. GPU Compute Node Group (P5/P6, optional) —** | | | -| `DeployPseriesCNG` | `false` | Deploy a GPU (P5/P6) queue — see [GPU compute](#gpu-compute-p5p6) | -| `PseriesInstanceType` | `p5.48xlarge` | Multi-NIC GPU instance type; auto-selects the matching template + EFA NIC count. Accepted values: `p5.48xlarge` / `p5e.48xlarge` (→ add-cng-p5, 32 NICs), `p5en.48xlarge` (→ add-cng-p5, 16 NICs), `p6-b200.48xlarge` (→ add-cng-p6-b200, 8 NICs), `p6-b300.48xlarge` (→ add-cng-p6-b300, 17 NICs). See [GPU compute](#gpu-compute-p5p6) | +| `OnDemandInstanceType` | `c6i.4xlarge` | Any On-Demand type — CPU, a single-NIC GPU (`g6.12xlarge`, [Example 1](#example-1-single-nic-gpu-queue-g6)), or an HPC type. Multi-NIC P5/P6 use the GPU queue instead | +| `OnDemandEfaInterfaceCount` | `0` | `1`/`2` enables EFA for HPC/MPI on EFA-capable CPU types. See [§8.5](#85-cpu-compute-node-group--advanced-settings) | + +**GPU compute queue (P5/P6, optional)** + +| Parameter | Default | Purpose | +|---|---|---| +| `DeployPseriesCNG` | `false` | Deploy a multi-NIC GPU (P5/P6) queue | +| `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -| **— 5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime) —** | | | -| `MonitoringStack` | `Prometheus-LoginNode` | Deploy Prometheus + Grafana on the login node, plus DCGM Exporter on GPU compute nodes. Set to `none` to disable monitoring | -| `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 on the login node to a trusted CIDR for direct Grafana access (default: SSM port-forward only). See [§8.1 Monitoring](#accessing-the-grafana-dashboards) | -| `DirectoryService` | `none` | Set to `OpenLDAP-LoginNode` for a multi-user cluster (shared LDAP users across all nodes). See [§8.2 User Management](#82-user-management) | + +**Additional (monitoring, multi-user, container runtime)** + +| Parameter | Default | Purpose | +|---|---|---| +| `MonitoringStack` | `Prometheus-LoginNode` | Prometheus + Grafana on the login node, DCGM Exporter on GPU nodes. `none` disables it. See [§8.1](#81-monitoring) | +| `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 (Grafana) on the login node to a trusted CIDR (default: SSM port-forward only) | +| `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.2](#82-user-management) | **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the From 6ae6ad69042d023292e612b4a2e2879ed3c5e96c Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:16:30 +0000 Subject: [PATCH 75/89] =?UTF-8?q?docs(readme):=20number=20=C2=A74=20config?= =?UTF-8?q?=20groups,=20move=20Storage=20to=20Advanced=20=C2=A78.1=20with?= =?UTF-8?q?=20deploy-small=20tip?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §4 config sub-tables now carry the console group numbers (1 Network … 5 Additional) - move the FSx deployment-types/sizing content out of §4 into a new §8.1 Storage at the top of Advanced Features; renumber the rest (Monitoring 8.2 … FSx-EFA 8.7) and fix all §8.x anchor references in README + PARAMETERS.md - add a 'deploy small, expand after' tip: FSx Lustre/OpenZFS can be grown after create, and a smaller filesystem deploys faster — start near minimum Capacity/HomeCapacity and expand once the stack is CREATE_COMPLETE --- architectures/aws-pcs/README.md | 92 ++++++++++++++---------- architectures/aws-pcs/docs/PARAMETERS.md | 4 +- 2 files changed, 55 insertions(+), 41 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 3e0398a19..c55c1b891 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -8,12 +8,12 @@ This repository provides reference architectures and deployment templates for se - **One click to an ML-training-ready cluster**: a single CloudFormation stack gives you a complete, ready-to-train environment — Slurm scheduler, GPU compute with EFA, shared FSx storage, the Enroot/Pyxis container runtime, and monitoring — with only the Availability Zone to choose. Submit distributed training jobs minutes after launch. - **Container runtime included**: Enroot/Pyxis is set up automatically, so `srun --container-image=...` works out of the box for containerized training. -- **Monitoring built in**: Grafana + Prometheus on the login node, with DCGM Exporter on GPU nodes feeding pre-built GPU dashboards (on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR. See [§8.1 Monitoring](#81-monitoring). +- **Monitoring built in**: Grafana + Prometheus on the login node, with DCGM Exporter on GPU nodes feeding pre-built GPU dashboards (on by default). Reach Grafana privately via SSM port-forward, or open it to a trusted CIDR. See [§8.2 Monitoring](#82-monitoring). - **GPU-ready, multi-NIC EFA**: dedicated launch templates for the P5 and P6 families, selected automatically by instance type, for high-bandwidth multi-node training. - **Flexible capacity options**: On-Demand, "open" On-Demand Capacity Reservations (consumed automatically), and Capacity Blocks for ML — selected per node group. (Targeting a *specific* ODCR is on the [roadmap](./docs/ROADMAP.md).) - **High-performance storage**: FSx for Lustre (shared scratch, `/fsx`) and FSx for OpenZFS (home directories, `/home`). -- **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent users — pairs with Slurm accounting. See [§8.2 User Management](#82-user-management). -- **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR. See [§8.3 IAM Permissions](#83-iam-permissions). +- **Multi-user ready**: opt-in OpenLDAP directory on the login node with SSSD on every compute node, so a team shares one cluster with consistent users — pairs with Slurm accounting. See [§8.3 User Management](#83-user-management). +- **Access control built in**: ready-to-deploy least-privilege IAM policy stacks for cluster admins and users, and login-node SSH / Grafana access gated to a trusted CIDR. See [§8.4 IAM Permissions](#84-iam-permissions). - **Modular components**: compose individual stacks (network/storage prerequisites, cluster scheduler, per-family compute node groups) instead of the all-in-one nested stack when you want to reuse infrastructure across clusters or iterate on one piece at a time. ## 2. Architecture @@ -89,17 +89,17 @@ are deleted with the stack. The defaults give a working ML-training cluster, so the only required parameter is the Availability Zone (`PrimarySubnetAZ`); everything else is optional. The most-used parameters are grouped below to match the deploy-all console's parameter groups. Storage -has its [own table](#storage-fsx-deployment-types-region-availability) below; for the +parameters are covered in [§8.1 Storage](#81-storage-fsx-deployment-types--sizing); for the complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). -**Network** +**1. Network** | Parameter | Default | Purpose | |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `AdditionalSubnetAZ2` / `…AZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2). NAT gateway stays in the primary AZ | -**PCS cluster** +**2. PCS cluster** | Parameter | Default | Purpose | |---|---|---| @@ -107,15 +107,15 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves the latest PCS-Ready DLAMI. See [AMI and container runtime](#ami-and-container-runtime) | | `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR (default: SSM only). See [§6](#6-accessing-the-cluster) | -**On-Demand compute queue** +**3. On-Demand compute queue** | Parameter | Default | Purpose | |---|---|---| | `DeployOnDemandCNG` | `true` | Deploy the On-Demand compute queue (the `cpu1` queue by default) | | `OnDemandInstanceType` | `c6i.4xlarge` | Any On-Demand type — CPU, a single-NIC GPU (`g6.12xlarge`, [Example 1](#example-1-single-nic-gpu-queue-g6)), or an HPC type. Multi-NIC P5/P6 use the GPU queue instead | -| `OnDemandEfaInterfaceCount` | `0` | `1`/`2` enables EFA for HPC/MPI on EFA-capable CPU types. See [§8.5](#85-cpu-compute-node-group--advanced-settings) | +| `OnDemandEfaInterfaceCount` | `0` | `1`/`2` enables EFA for HPC/MPI on EFA-capable CPU types. See [§8.6](#86-cpu-compute-node-group--advanced-settings) | -**GPU compute queue (P5/P6, optional)** +**4. GPU compute queue (P5/P6, optional)** | Parameter | Default | Purpose | |---|---|---| @@ -123,13 +123,13 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -**Additional (monitoring, multi-user, container runtime)** +**5. Additional (monitoring, multi-user, container runtime)** | Parameter | Default | Purpose | |---|---|---| -| `MonitoringStack` | `Prometheus-LoginNode` | Prometheus + Grafana on the login node, DCGM Exporter on GPU nodes. `none` disables it. See [§8.1](#81-monitoring) | +| `MonitoringStack` | `Prometheus-LoginNode` | Prometheus + Grafana on the login node, DCGM Exporter on GPU nodes. `none` disables it. See [§8.2](#82-monitoring) | | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 (Grafana) on the login node to a trusted CIDR (default: SSM port-forward only) | -| `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.2](#82-user-management) | +| `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.3](#83-user-management) | **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the @@ -143,7 +143,7 @@ choices that need the most thought. needed. Enroot 3.5.0 + Pyxis 0.20.0 are layered on at first boot via [`assets/scripts/install-enroot-pyxis.sh`](./assets/scripts/install-enroot-pyxis.sh) (~8–12 min boot). For **frequent scaling**, pre-bake Enroot/Pyxis into a custom DLAMI -once with [§8.4](#84-pre-baking-enrootpyxis-into-a-custom-ami) and pass that +once with [§8.5](#85-pre-baking-enrootpyxis-into-a-custom-ami) and pass that `ami-xxx` as `AmiId` (~3 min boot, deterministic state). The post-install hook is idempotent — it no-ops on a pre-baked AMI. @@ -177,26 +177,8 @@ automatically. > group at `PseriesMinCount = PseriesMaxCount = ` so the reserved > nodes launch immediately, rather than scaling from 0. -### Storage: FSx deployment types (Region availability) - -**FSx deployment types are not available in every Region.** Defaults match the most -capable type; switch to a more widely available one if your Region needs it. - -| Filesystem | Parameter | Default | Other values | Notes | -|---|---|---|---|---| -| Lustre (`/fsx`) | `LustreDeploymentType` | `PERSISTENT_2` | `PERSISTENT_1` | `PERSISTENT_2` (throughput 125/250/500/1000, metadata config) isn't in every Region; `PERSISTENT_1` (50/100/200) is in more Regions | -| Lustre (`/fsx`) | `PerUnitStorageThroughput` | `250` | any valid number | Must be valid for the type: P2 = 125/250/500/1000, P1 = 50/100/200 | -| OpenZFS (`/home`) | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | `SINGLE_AZ_HA_1`, `SINGLE_AZ_2`, `SINGLE_AZ_1` | `SINGLE_AZ_1` is in all Regions; HA/2 variants vary. `MULTI_AZ` excluded (needs a second subnet) | -| OpenZFS (`/home`) | `HomeThroughput` | `320` | any valid number | Throughput (MB/s). Valid values depend on the deployment type: `SINGLE_AZ_2`/`SINGLE_AZ_HA_2` = 160/320/640/1280/2560/3840/5120/7680/10240; `SINGLE_AZ_HA_1`/`SINGLE_AZ_1` = 64/128/256/512/1024/2048/3072/4096 | - -Check support before deploying: -[Lustre Regions](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-fsx-lustre.html) · -[OpenZFS Regions](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/available-aws-regions.html). -If a deploy fails at the FSx resource with an "unsupported deployment type" error, -switch these parameters to a type your Region supports. - -> Enabling EFA / GPUDirect Storage on the Lustre filesystem (`FSxLustreEnableEfa`) is an -> advanced option — see [§8.6 FSx for Lustre over EFA (GPUDirect Storage)](#86-fsx-for-lustre-over-efa-gpudirect-storage). +Storage (FSx for Lustre `/fsx` + OpenZFS `/home`) has sensible defaults; deployment +types, throughput, and capacity are covered in [§8.1 Storage](#81-storage-fsx-deployment-types--sizing). --- @@ -333,7 +315,39 @@ the full validation matrix is in [tests/README.md](./tests/README.md). ## 8. Advanced Features -### 8.1 Monitoring +### 8.1 Storage: FSx deployment types & sizing + +The `/fsx` (Lustre) and `/home` (OpenZFS) filesystems have working defaults. Two things +are worth knowing before changing them: **deployment types are not available in every +Region**, and **starting small then expanding is usually faster to deploy**. + +> **Tip — deploy small, expand after.** Both FSx for Lustre and OpenZFS can be **grown +> after creation** (increase storage capacity, and for Lustre throughput, via an FSx +> update — no data migration). A large filesystem also takes longer to create, which +> lengthens the whole stack deploy. So for the first deploy, leave `Capacity` / +> `HomeCapacity` at (or near) the minimum to get the cluster up quickly, then expand the +> filesystem to the size you need once the stack is `CREATE_COMPLETE`. + +**FSx deployment types are not available in every Region.** Defaults match the most +capable type; switch to a more widely available one if your Region needs it. + +| Filesystem | Parameter | Default | Other values | Notes | +|---|---|---|---|---| +| Lustre (`/fsx`) | `LustreDeploymentType` | `PERSISTENT_2` | `PERSISTENT_1` | `PERSISTENT_2` (throughput 125/250/500/1000, metadata config) isn't in every Region; `PERSISTENT_1` (50/100/200) is in more Regions | +| Lustre (`/fsx`) | `PerUnitStorageThroughput` | `250` | any valid number | Must be valid for the type: P2 = 125/250/500/1000, P1 = 50/100/200 | +| OpenZFS (`/home`) | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | `SINGLE_AZ_HA_1`, `SINGLE_AZ_2`, `SINGLE_AZ_1` | `SINGLE_AZ_1` is in all Regions; HA/2 variants vary. `MULTI_AZ` excluded (needs a second subnet) | +| OpenZFS (`/home`) | `HomeThroughput` | `320` | any valid number | Throughput (MB/s). Valid values depend on the deployment type: `SINGLE_AZ_2`/`SINGLE_AZ_HA_2` = 160/320/640/1280/2560/3840/5120/7680/10240; `SINGLE_AZ_HA_1`/`SINGLE_AZ_1` = 64/128/256/512/1024/2048/3072/4096 | + +Check support before deploying: +[Lustre Regions](https://docs.aws.amazon.com/fsx/latest/LustreGuide/using-fsx-lustre.html) · +[OpenZFS Regions](https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/available-aws-regions.html). +If a deploy fails at the FSx resource with an "unsupported deployment type" error, +switch these parameters to a type your Region supports. + +> Enabling EFA / GPUDirect Storage on the Lustre filesystem (`FSxLustreEnableEfa`) is an +> advanced option — see [§8.7 FSx for Lustre over EFA (GPUDirect Storage)](#87-fsx-for-lustre-over-efa-gpudirect-storage). + +### 8.2 Monitoring With `MonitoringStack=Prometheus-LoginNode` (default), an integrated monitoring stack based on [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) @@ -457,7 +471,7 @@ NCCL, FSDP), see the [Test & Validation Guide](tests/README.md). --- -### 8.2 User Management +### 8.3 User Management By default, the cluster runs as a single `ubuntu` user. For **multi-user clusters** (per-user Slurm accounting, isolated home directories, team-based @@ -473,7 +487,7 @@ operations (adding/removing users, Slurm accounting, SSH access). > lives on OpenZFS `/home`, so it survives a login-node replacement (data is recoverable), > but the directory **service** is a single point of failure while the login node is down. -### 8.3 IAM Permissions +### 8.4 IAM Permissions The cluster has two human roles, each with a ready-to-deploy IAM policy stack: @@ -487,7 +501,7 @@ Each template creates the customer-managed policies and an IAM group, and can attach existing users at deploy time. See the **[IAM Permissions Guide](./docs/IAM.md)** for roles, deploy instructions, security considerations, and the verification matrix. -### 8.4 Pre-baking Enroot/Pyxis into a custom AMI +### 8.5 Pre-baking Enroot/Pyxis into a custom AMI The all-in-one template installs Enroot/Pyxis at **first boot** via `PostInstallScriptUrl`, which is fast to deploy and avoids an Image Builder step. @@ -550,7 +564,7 @@ space skips the download+check entirely, shaving a few seconds off boot. For production deploys that pin the AMI explicitly per cluster, none of these are needed. -### 8.5 CPU compute node group — advanced settings +### 8.6 CPU compute node group — advanced settings The On-Demand CPU queue is configured with `OnDemandInstanceType` (default `c6i.4xlarge`) plus `OnDemandQueueName` / `OnDemandCngName` / `OnDemandMinCount` / @@ -588,7 +602,7 @@ application's natural multi-pair pattern) to actually exercise both NICs on hpc7a/hpc8a. See [tests/README.md Test 9](./tests/README.md#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) for the full benchmark setup and validated bandwidth numbers. -### 8.6 FSx for Lustre over EFA (GPUDirect Storage) +### 8.7 FSx for Lustre over EFA (GPUDirect Storage) `FSxLustreEnableEfa` (default `false`) enables EFA on the `/fsx` Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index 4081d95f8..aac5ca11e 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -7,7 +7,7 @@ the latest PCS-Ready Deep Learning AMI auto-resolved from SSM, Enroot/Pyxis inst at first boot via `PostInstallScriptUrl`, monitoring enabled — so a default deploy only needs the Availability Zone (`PrimarySubnetAZ`). To pre-bake Enroot/Pyxis into a custom AMI for faster boots, build it separately with -[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#84-pre-baking-enrootpyxis-into-a-custom-ami) +[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#85-pre-baking-enrootpyxis-into-a-custom-ami) and pass its output as `AmiId`. For conceptual guidance (GPU instance/EFA selection, FSx Region availability, container @@ -35,7 +35,7 @@ it directly. | `SlurmVersion` | `25.11` | Slurm version (`25.05` or `25.11`). Drives which monitoring you get (Slurm OpenMetrics is 25.11+ only) and is also threaded into the CNG UserData so the right-version Pyxis is installed; see [OPERATIONS.md §1](./OPERATIONS.md#1-slurm-version-selection) | | `LoginNodeInstanceType` | `m6i.4xlarge` | Login node instance type | | `RootVolumeSize` | `300` | Root EBS volume size (GiB) on every node (login + compute); 300 leaves room for large container images (Megatron `.sqsh` ~20 GB) | -| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#84-pre-baking-enrootpyxis-into-a-custom-ami)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | +| `AmiId` | *(empty → SSM auto-resolve)* | AMI ID for every node group. **Empty (default) auto-resolves to the latest PCS-Ready Deep Learning AMI** (Ubuntu 24.04, x86_64) from SSM (`/aws/service/pcs/ami/dlami-base-ubuntu2404/x86_64/latest/ami-id`). For production, **pin to a specific `ami-xxx`** so a later scale-out cannot drift onto a newer base. Use a custom AMI built off the PCS-Ready DLAMI base (e.g. via [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#85-pre-baking-enrootpyxis-into-a-custom-ami)) when you want Enroot/Pyxis pre-baked or other customizations. See [OPERATIONS.md §4](./OPERATIONS.md#4-ami-selection-amiid--pin-in-production) | | `SSHAccessCidr` | *(empty)* | When set to a CIDR, opens SSH/22 on the login node to that CIDR via a login-only security group (attached to the login node only, never compute). Empty (default) = SSH over SSM only. Set to your office/VPN range for direct `ssh`/`scp`/VS Code Remote (common for multi-user clusters) | | `ManagedAccounting` | `disabled` | Enable Slurm managed accounting (requires Slurm 24.11+) | | `AccountingPolicyEnforcement` | `none` | Slurm accounting policy enforcement (`none` or `associations,limits,safe`) | From b692f7ef602b9b75d10ecd157830073fc1ad17a9 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:27:35 +0000 Subject: [PATCH 76/89] docs: drop test-profile flags, align PARAMETERS with deploy-all groups, refocus DEPLOY-TESTING, move IAM verification to tests - remove --profile claude (project test setting) from DEPLOY-TESTING.md and tests/infra-test.md; generalize fixed bucket/region to placeholders - DEPLOY-TESTING.md rewritten for the real audience: a third party deploying not-yet-published templates by hosting them in their own S3 bucket and pointing S3BucketName/S3KeyPrefix at it - PARAMETERS.md sections + order now match the deploy-all console parameter groups exactly (1 Network, 2 PCS Cluster, 3 On-Demand, 4 GPU, 5 Additional incl. monitoring repo/version/dcgm + container runtime, 6 FSx, 7 Developer); fix stale anchor links - move the IAM verification results out of docs/IAM.md into a reproducible tests/iam-test.md (representative two-role use case); add the row to tests/README --- architectures/aws-pcs/docs/DEPLOY-TESTING.md | 229 +++++++++---------- architectures/aws-pcs/docs/IAM.md | 21 +- architectures/aws-pcs/docs/PARAMETERS.md | 55 ++--- architectures/aws-pcs/tests/README.md | 1 + architectures/aws-pcs/tests/iam-test.md | 106 +++++++++ architectures/aws-pcs/tests/infra-test.md | 12 +- 6 files changed, 252 insertions(+), 172 deletions(-) create mode 100644 architectures/aws-pcs/tests/iam-test.md diff --git a/architectures/aws-pcs/docs/DEPLOY-TESTING.md b/architectures/aws-pcs/docs/DEPLOY-TESTING.md index 7ea660083..4c495b498 100644 --- a/architectures/aws-pcs/docs/DEPLOY-TESTING.md +++ b/architectures/aws-pcs/docs/DEPLOY-TESTING.md @@ -1,193 +1,176 @@ -# Development & Testing Deploy Procedures +# Deploying updated templates before they are published -## Prerequisites +The one-click Quick Start in the [README](../README.md#3-quick-start) deploys from the +**public production bucket** (`awsome-distributed-ai`), which only has the templates that +have already been merged and published. This guide is for the other case: **you have +template/script changes that are not yet in the public bucket** (e.g. a fork, a feature +branch, or a PR under review) and you want to deploy and test them. -- AWS CLI configured (`--profile claude` in this project) -- Test S3 bucket: `midaisuk-llm-dev` (us-east-1) -- Production S3 bucket: `awsome-distributed-ai` (us-east-1) -- Test region: `us-east-2` +The approach is the same in every case: **host the templates + boot scripts in an S3 +bucket you control, then point the deploy at that bucket** via the `S3BucketName` / +`S3KeyPrefix` parameters. The nested stacks and the first-boot scripts are all fetched +from `s3:///...`, so overriding those two parameters redirects +the entire deploy to your copy. ---- +## 1. Prerequisites -## Upload templates + scripts to test bucket +- AWS CLI configured with credentials for your account. +- An S3 bucket you own, in any Region (the bucket is reached by its global name, so it + does not need to be in the deploy Region). It can be **private** — the templates are + fetched by your CLI/CloudFormation, and the boot scripts are fetched by the node + instance role, so no public access is required. +- A local checkout of the branch/fork with the changes you want to test. + +This guide uses these placeholders — substitute your own: ```bash -cd architectures/aws-pcs -aws s3 sync assets/ s3://midaisuk-llm-dev/templates/ \ - --exclude "*" --include "*.yaml" --include "*.sh" \ - --profile claude +BUCKET=my-pcs-templates # an S3 bucket you control +PREFIX=templates/ # key prefix (keep the trailing slash) +REGION=us-east-1 # the Region to deploy the cluster into +AZ=us-east-1a # an Availability Zone in $REGION ``` -This uploads both CloudFormation templates and boot scripts in one command -(scripts live under `assets/scripts/`, synced to `s3://bucket/templates/scripts/`). +## 2. Upload the templates + scripts to your bucket + +Run from the repo's `architectures/aws-pcs` directory (so `assets/` is the source): + +```bash +aws s3 sync assets/ "s3://${BUCKET}/${PREFIX}" \ + --exclude "*" --include "*.yaml" --include "*.sh" +``` ---- +This uploads both the CloudFormation templates (`*.yaml`) and the boot scripts (`*.sh`) +in one command. The scripts live under `assets/scripts/`, so they land at +`s3://${BUCKET}/${PREFIX}scripts/` — which is exactly where the default +`PostInstallScriptUrl` looks (`s3:///scripts/install-enroot-pyxis.sh`). +Re-run this sync after every change you want to test. -## Deploy a test cluster +## 3. Deploy pointing at your bucket -### Minimal (single ubuntu user, default) +Pass `S3BucketName` (and `S3KeyPrefix` if you changed it from the `templates/` default). +That single override is what makes the nested stacks **and** the first-boot Enroot/Pyxis +installer come from your copy instead of the public bucket: ```bash aws cloudformation create-stack \ --stack-name pcs-test \ - --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --template-url "https://${BUCKET}.s3.amazonaws.com/${PREFIX}pcs-ml-cluster-deploy-all.yaml" \ --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ - ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ + ParameterKey=PrimarySubnetAZ,ParameterValue=${AZ} \ + ParameterKey=S3BucketName,ParameterValue=${BUCKET} \ + ParameterKey=S3KeyPrefix,ParameterValue=${PREFIX} \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --region us-east-2 --profile claude + --region ${REGION} ``` -### With multi-user (OpenLDAP on login node) +> **Why this matters.** If you deploy the updated top-level template but leave +> `S3BucketName` at its default, the nested stacks and boot scripts are still pulled from +> the **public** bucket — so you'd be testing your top-level change against the old +> published nested templates/scripts. Always override `S3BucketName` to your bucket when +> testing unpublished changes. + +Add any parameters you're testing on top — for example multi-user: ```bash -aws cloudformation create-stack \ - --stack-name pcs-multiuser \ - --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ - ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ ParameterKey=DirectoryService,ParameterValue=OpenLDAP-LoginNode \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --region us-east-2 --profile claude ``` -### With EFA on HPC instances +or an EFA-capable CPU queue: ```bash -aws cloudformation create-stack \ - --stack-name pcs-hpc \ - --template-url https://midaisuk-llm-dev.s3.us-east-1.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-2b \ - ParameterKey=S3BucketName,ParameterValue=midaisuk-llm-dev \ ParameterKey=OnDemandInstanceType,ParameterValue=hpc8a.96xlarge \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --region us-east-2 --profile claude + ParameterKey=OnDemandEfaInterfaceCount,ParameterValue=2 \ ``` ---- +See [PARAMETERS.md](./PARAMETERS.md) for the full list. -## Monitor deploy progress +## 4. Monitor progress ```bash -# Quick status -aws cloudformation describe-stacks --stack-name --region us-east-2 --profile claude \ +aws cloudformation describe-stacks --stack-name pcs-test --region ${REGION} \ --query 'Stacks[0].StackStatus' --output text -# Nested stack progress -aws cloudformation describe-stack-events --stack-name --region us-east-2 --profile claude \ +aws cloudformation describe-stack-events --stack-name pcs-test --region ${REGION} \ --query 'StackEvents[?ResourceStatus!=`CREATE_IN_PROGRESS`].[Timestamp,LogicalResourceId,ResourceStatus]' \ --output text | head -20 ``` -Typical timeline (~25-30 min): -1. PrerequisitesStack (VPC + FSx) — ~10 min -2. ClusterStack (PCS cluster) — ~5 min -3. LoginNodeGroupStack + OnDemandCNGStack (parallel) — ~5 min -4. Node boot + cloud-init (post-install, monitoring, directory) — ~3-8 min +Typical timeline (~25-30 min): Prerequisites (VPC + FSx) ~10 min → Cluster ~5 min → +Login + compute node groups (parallel) ~5 min → node boot + cloud-init (post-install, +monitoring, directory) ~3-8 min. ---- +## 5. Connect and verify -## Connect to the login node +Connect to the login node over SSM (no public IP or SSH key needed): ```bash -# Find login node -CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name --region us-east-2 --profile claude \ +CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name pcs-test --region ${REGION} \ --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text) -LOGIN_ID=$(aws ec2 describe-instances --region us-east-2 --profile claude \ +LOGIN_ID=$(aws ec2 describe-instances --region ${REGION} \ --filters "Name=tag:pcs-cluster-id,Values=$CLUSTER_ID" \ "Name=tag:monitoring-role,Values=login" \ "Name=instance-state-name,Values=running" \ --query 'Reservations[0].Instances[0].InstanceId' --output text) - -# SSH config (one-time) -cat >> ~/.ssh/config << EOF -Host pcs-login - HostName $LOGIN_ID - User ubuntu - ProxyCommand sh -c "aws ssm start-session --profile claude --region us-east-2 --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'" - StrictHostKeyChecking no - UserKnownHostsFile /dev/null -EOF - -# Inject SSH key (one-time) -aws ssm send-command --instance-ids $LOGIN_ID --region us-east-2 --profile claude \ - --document-name "AWS-RunShellScript" \ - --parameters "commands=[\"mkdir -p /home/ubuntu/.ssh; echo '$(cat ~/.ssh/id_rsa.pub)' >> /home/ubuntu/.ssh/authorized_keys; chown -R ubuntu:ubuntu /home/ubuntu/.ssh; chmod 700 /home/ubuntu/.ssh; chmod 600 /home/ubuntu/.ssh/authorized_keys\"]" - -# Connect -ssh pcs-login +aws ssm start-session --target $LOGIN_ID --region ${REGION} ``` ---- - -## Verification checks +Then `sudo su - ubuntu` and run the checks relevant to your change. A few quick ones: -### Basic cluster health ```bash -ssh pcs-login 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; sinfo -N; squeue' -``` +# Slurm sees the queues / nodes (adjust the version if you set SlurmVersion=25.05) +export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; sinfo -N; squeue -### Monitoring -```bash -ssh pcs-login 'sudo docker ps --format "table {{.Names}}\t{{.Status}}"' +# Container runtime came from your bucket (the log shows the s3:// it fetched) +grep s3:// /var/log/pcs-post-install.log + +# Monitoring containers (when MonitoringStack=Prometheus-LoginNode) +sudo docker ps --format "table {{.Names}}\t{{.Status}}" ``` -### Multi-user (when DirectoryService=OpenLDAP-LoginNode) -```bash -# On login node -ssh pcs-login 'systemctl status slapd | head -3' -ssh pcs-login 'getent passwd testuser1' # after creating a user -ssh pcs-login 'cat /var/log/directory-setup.log | tail -10' +For the full reproducible test matrix (monitoring, container runtime, CPU/GPU, NCCL, +FSDP, multi-user, GPU health), see [tests/README.md](../tests/README.md). -# On compute node (via srun) -ssh pcs-login 'export PATH=/opt/aws/pcs/scheduler/slurm-25.11/bin:$PATH; srun -N 1 -n 1 -p cpu1 bash -c "getent passwd testuser1"' -``` +## 6. Iterate -### LDAP admin password -```bash -# From SSM (preferred) -aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/ldap/admin-password" \ - --with-decryption --query 'Parameter.Value' --output text --region us-east-2 --profile claude +After editing a template or script, re-run the **sync** (step 2), then +`update-stack` (same parameters as `create-stack`) — or delete and recreate if the change +can't be applied in place (e.g. a subnet CIDR change): -# Fallback (if SSM put failed at boot) -ssh pcs-login 'sudo cat /home/ldap-db/.admin-password' +```bash +aws cloudformation update-stack --stack-name pcs-test \ + --template-url "https://${BUCKET}.s3.amazonaws.com/${PREFIX}pcs-ml-cluster-deploy-all.yaml" \ + --parameters ParameterKey=PrimarySubnetAZ,UsePreviousValue=true \ + ParameterKey=S3BucketName,UsePreviousValue=true \ + ... \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region ${REGION} ``` ---- +> A stack update only re-runs first-boot scripts on **newly launched** nodes; existing +> nodes keep what they booted with. To re-test a boot-script change, let the node group +> scale a fresh node (or replace the affected nodes). -## Cleanup +## 7. Cleanup ```bash -aws cloudformation delete-stack --stack-name --region us-east-2 --profile claude +aws cloudformation delete-stack --stack-name pcs-test --region ${REGION} ``` -Nested stacks are deleted automatically. If DELETE_FAILED on CNG stacks -(PCS timing dependency), delete PCS CNGs first: +Nested stacks (and FSx — back up data first) are deleted automatically. If a CNG stack +hits `DELETE_FAILED` (a PCS timing dependency), delete the PCS compute node groups first, +then retry: + ```bash -CLUSTER_ID= -for cng in $(aws pcs list-compute-node-groups --cluster-identifier $CLUSTER_ID --region us-east-2 --profile claude --query 'computeNodeGroups[].id' --output text); do - aws pcs delete-compute-node-group --cluster-identifier $CLUSTER_ID --compute-node-group-identifier $cng --region us-east-2 --profile claude +for cng in $(aws pcs list-compute-node-groups --cluster-identifier $CLUSTER_ID --region ${REGION} \ + --query 'computeNodeGroups[].id' --output text); do + aws pcs delete-compute-node-group --cluster-identifier $CLUSTER_ID \ + --compute-node-group-identifier $cng --region ${REGION} done sleep 60 -aws cloudformation delete-stack --stack-name --region us-east-2 --profile claude +aws cloudformation delete-stack --stack-name pcs-test --region ${REGION} ``` ---- +## After the change is published -## Production deploy (post-merge) - -After PR merge, maintainer syncs to prod bucket: -```bash -aws s3 sync assets/ s3://awsome-distributed-ai/templates/ -``` - -Users deploy from prod bucket (no S3BucketName override needed — it's the default): -```bash -aws cloudformation create-stack \ - --stack-name pcs-ml-cluster \ - --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-1a \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM -``` +Once the change is merged and the maintainer has synced `assets/` to the public bucket +(`awsome-distributed-ai`), no override is needed — the defaults already point there, and +the [README Quick Start](../README.md#3-quick-start) works as written. diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md index 499672475..dbfe48cfc 100644 --- a/architectures/aws-pcs/docs/IAM.md +++ b/architectures/aws-pcs/docs/IAM.md @@ -147,22 +147,11 @@ The same approach works for the user policy — exercise the user workflows --- -## Verification - -Re-validated end-to-end on a real account (us-east-2) against the major-update -templates — a dedicated role attached to **only** the admin (or user) managed -policies, nothing else: - -| Check | Result | -|---|---| -| Both IAM CFN stacks `CREATE_COMPLETE` | ✅ | -| **A role with only the admin policies runs `pcs-ml-cluster-deploy-all.yaml`** (DirectoryService=OpenLDAP-LoginNode + SSHAccessCidr + MonitoringStack=Prometheus-LoginNode) — every nested stack `CREATE_COMPLETE`, no `AccessDenied` | ✅ | -| Same admin-only role tears it down: `delete-stack` → `DELETE_COMPLETE` with no `AccessDenied` (delete perms cover the full teardown) | ✅ | -| Admin role needs **no `iam:CreatePolicy`** — the instance role's perms are inline (`PutRolePolicy`), so deploy-all works with the admin policy as-is (`iam:CreatePolicy` simulates `implicitDeny`, and deploy-all still completes) | ✅ | -| `simulate-principal-policy` (with proper resource ARNs / `iam:PassedToService` context): `cfn:CreateStack`, `ec2:RunInstances/CreateSubnet/CreateSecurityGroup`, `fsx:CreateFileSystem`, `pcs:CreateCluster`, `iam:CreateRole/PutRolePolicy/CreateInstanceProfile/PassRole(ec2,pcs)`, `ssm:Put/GetParameter` on `/pcs/*/ldap/*`, `kms:Decrypt` → all `allowed` | ✅ | -| User role: `pcs:GetCluster`/`ListComputeNodeGroups`, `cloudformation:DescribeStacks`, `ec2:DescribeInstances` → `allowed`; `cfn:CreateStack`, `pcs:CreateCluster`, `ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole` → `implicitDeny` | ✅ | -| User `ssm:StartSession` on login node (`Name=PCS-login`) `allowed`; on compute node (`Name=PCS-cpu1`) `implicitDeny` | ✅ | -| **Security: user reads `/pcs/*/grafana/*` (`allowed`) but NOT `/pcs/*/ldap/*`** (the OpenLDAP admin password) → `implicitDeny` | ✅ | +## Verifying the policies + +To confirm the admin policy can deploy a cluster end-to-end and the user policy is +correctly constrained (login-only SSM, no LDAP-password access), see the reproducible +procedure in [tests/iam-test.md](../tests/iam-test.md). > **Note on the template-source bucket.** The admin policy grants no `s3:GetObject`, > because the production templates live in the public `awsome-distributed-ai` diff --git a/architectures/aws-pcs/docs/PARAMETERS.md b/architectures/aws-pcs/docs/PARAMETERS.md index aac5ca11e..fea9bc97d 100644 --- a/architectures/aws-pcs/docs/PARAMETERS.md +++ b/architectures/aws-pcs/docs/PARAMETERS.md @@ -1,11 +1,11 @@ # Parameter Reference — `pcs-ml-cluster-deploy-all.yaml` -Full parameter list for the all-in-one deployment template, grouped to match the -sections shown in the CloudFormation console (the console also shows friendly labels via -`AWS::CloudFormation::Interface`). Defaults give the most common production setup — -the latest PCS-Ready Deep Learning AMI auto-resolved from SSM, Enroot/Pyxis installed -at first boot via `PostInstallScriptUrl`, monitoring enabled — so a default deploy -only needs the Availability Zone (`PrimarySubnetAZ`). To pre-bake Enroot/Pyxis into +Full parameter list for the all-in-one deployment template. The sections and their order +match the CloudFormation console's parameter groups exactly (the console shows friendly +labels via `AWS::CloudFormation::Interface`). Defaults give the most common production +setup — the latest PCS-Ready Deep Learning AMI auto-resolved from SSM, Enroot/Pyxis +installed at first boot via `PostInstallScriptUrl`, monitoring enabled — so a default +deploy only needs the Availability Zone (`PrimarySubnetAZ`). To pre-bake Enroot/Pyxis into a custom AMI for faster boots, build it separately with [`pcs-ready-dlami-with-enroot-pyxis.yaml`](../README.md#85-pre-baking-enrootpyxis-into-a-custom-ami) and pass its output as `AmiId`. @@ -40,18 +40,7 @@ it directly. | `ManagedAccounting` | `disabled` | Enable Slurm managed accounting (requires Slurm 24.11+) | | `AccountingPolicyEnforcement` | `none` | Slurm accounting policy enforcement (`none` or `associations,limits,safe`) | -## 2b. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime) - -| Parameter | Default | Purpose | -|---|---|---| -| `MonitoringStack` | `Prometheus-LoginNode` | Monitoring stack to deploy. `Prometheus-LoginNode` = self-hosted Prometheus + Grafana + DCGM Exporter on the login node. `none` = no monitoring. (Renamed from the old boolean `DeployMonitoring`; `-` enum, extensible to future `AMP-AMG`/`CloudWatch`) | -| `GrafanaAccessCidr` | *(empty)* | When set to a CIDR, opens HTTPS/443 (Grafana) on the login node to that CIDR via the login-only security group. Empty = SSM port-forward only. **443 also exposes the unauthenticated `/prometheus/`, `/pushgateway/`, `/slurmexporter/` proxy paths**, not just the password-gated Grafana. Use the tightest CIDR you can. (Renamed from `GrafanaPublicAccessCidr`) | -| `DirectoryService` | `none` | Multi-user directory. `none` = single `ubuntu` user. `OpenLDAP-LoginNode` = slapd on the login node (DB on shared `/home/ldap-db`) + SSSD on all compute nodes. **Single login node only** — keep the login node group at 1 instance while enabled. See [USER-MANAGEMENT.md](./USER-MANAGEMENT.md) | -| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP domain suffix. Only used when `DirectoryService != none` | -| `PostInstallScriptUrl` | *(empty → auto)* | Script run on every node at first boot (PCS equivalent of ParallelCluster `OnNodeConfigured`). **Empty (default) auto-installs Enroot/Pyxis** from `s3:///scripts/install-enroot-pyxis.sh` (fetched with the instance role, so it works with a **private** bucket — no public S3 needed). Accepts an `s3://` URL (instance-role fetch) or an `http(s)://` URL (curl, public only, e.g. GitHub raw). Set to a single space to skip. Idempotent: a no-op if Enroot/Pyxis is already pre-baked into `AmiId` | -| `PostInstallScriptArgs` | *(empty)* | Arguments passed to the post-install script. Normally left empty — most users never touch the container-runtime parameters | - -## 4. On-Demand Compute Node Group (CPU) +## 3. On-Demand Compute Node Group (CPU) | Parameter | Default | Purpose | |---|---|---| @@ -61,10 +50,10 @@ it directly. | `OnDemandMaxCount` | `4` | CPU queue maximum nodes | | `OnDemandCngName` | `cpu1` | CPU node-group name | | `OnDemandQueueName` | `cpu1` | CPU Slurm queue name | -| `OnDemandEfaInterfaceCount` | `0` | EFA interfaces on the CPU CNG. **`0` (default) = no EFA** (standard ENA). `1` or `2` = enable EFA with that many interfaces (switches the LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` + a cluster placement group). Set the count to the instance type's `MaximumEfaInterfaces`: `hpc8a.96xlarge`/`hpc7a.*`/`hpc6id.32xlarge`=2; `hpc6a.48xlarge`/`c7i.metal`=1. **EFA needs an EFA-capable type** — a non-EFA type (e.g. the default `c6i.4xlarge`) fails to launch with count > 0. No effect on the GPU CNG. See [README §EFA on CPU HPC instances](../README.md#efa-on-cpu-hpc-instances-ondemandefainterfacecount) | +| `OnDemandEfaInterfaceCount` | `0` | EFA interfaces on the CPU CNG. **`0` (default) = no EFA** (standard ENA). `1` or `2` = enable EFA with that many interfaces (switches the LaunchTemplate to a `NetworkInterfaces` block with `InterfaceType=efa` + a cluster placement group). Set the count to the instance type's `MaximumEfaInterfaces`: `hpc8a.96xlarge`/`hpc7a.*`/`hpc6id.32xlarge`=2; `hpc6a.48xlarge`/`c7i.metal`=1. **EFA needs an EFA-capable type** — a non-EFA type (e.g. the default `c6i.4xlarge`) fails to launch with count > 0. No effect on the GPU CNG. See [README §8.6 CPU compute node group](../README.md#86-cpu-compute-node-group--advanced-settings) | | `OnDemandPlacementGroupName` | *(empty)* | Existing cluster placement group name to launch nodes into. Empty + `OnDemandEfaInterfaceCount > 0` auto-creates a per-CNG cluster placement group; supplying a name reuses an existing one (e.g. shared across CPU + GPU CNGs for heterogeneous tightly-coupled jobs). Ignored when `OnDemandEfaInterfaceCount = 0` | -## 5. GPU Compute Node Group — P5/P6 (Optional) +## 4. GPU Compute Node Group — P5/P6 (Optional) See [GPU compute](../README.md#gpu-compute-p5p6) for instance/EFA/capacity guidance. @@ -78,19 +67,34 @@ See [GPU compute](../README.md#gpu-compute-p5p6) for instance/EFA/capacity guida | `PseriesCngName` | `gpu-p5` | GPU node-group name | | `PseriesQueueName` | `gpu-p5` | GPU Slurm queue name | -## 6. FSx for Lustre (`/fsx`) + FSx for OpenZFS (`/home`) (Advanced) +## 5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime) + +| Parameter | Default | Purpose | +|---|---|---| +| `MonitoringStack` | `Prometheus-LoginNode` | Monitoring stack to deploy. `Prometheus-LoginNode` = self-hosted Prometheus + Grafana + DCGM Exporter on the login node. `none` = no monitoring. (Renamed from the old boolean `DeployMonitoring`; `-` enum, extensible to future `AMP-AMG`/`CloudWatch`) | +| `GrafanaAccessCidr` | *(empty)* | When set to a CIDR, opens HTTPS/443 (Grafana) on the login node to that CIDR via the login-only security group. Empty = SSM port-forward only. **443 also exposes the unauthenticated `/prometheus/`, `/pushgateway/`, `/slurmexporter/` proxy paths**, not just the password-gated Grafana. Use the tightest CIDR you can. (Renamed from `GrafanaPublicAccessCidr`) | +| `MonitoringRepo` | `aws-samples/aws-parallelcluster-monitoring` | GitHub `owner/repo` for the monitoring stack; override with a fork + a branch in `MonitoringVersion` to test unreleased changes | +| `MonitoringVersion` | `v2.9.1` | [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`). `v2.9.1` adds the `DCGM_EXPORTER_IMAGE` override (needed for B300 GPU metrics) and brings Grafana 13; `v2.6.4`+ carry the PCS `/opt` install + Docker-29.x DCGM fixes. Pin to a tag for stability. Migration notes: [OPERATIONS.md §3](./OPERATIONS.md#3-monitoring-monitoringversion) | +| `DcgmExporterImage` | DCGM 4.5.2 by digest | `dcgm-exporter` image used on GPU nodes. Defaults to a DCGM 4.5.2 build pinned by digest (`nvcr.io/nvidia/k8s/dcgm-exporter@sha256:a7ad6547...`) covering Hopper / B200 / B300. The digest pull bypasses the Docker-29.x OCI-index failure on newer NVCR tags. Override (any image reference, ideally also a digest) to pin to a different build — e.g. the monitoring stack's older default 4.2.0. No effect on CPU nodes. See [OPERATIONS.md §3.1](./OPERATIONS.md#31-dcgmexporterimage-the-default-and-when-to-change-it) | +| `DirectoryService` | `none` | Multi-user directory. `none` = single `ubuntu` user. `OpenLDAP-LoginNode` = slapd on the login node (DB on shared `/home/ldap-db`) + SSSD on all compute nodes. **Single login node only** — keep the login node group at 1 instance while enabled. See [USER-MANAGEMENT.md](./USER-MANAGEMENT.md) | +| `DirectoryDomainSuffix` | `dc=cluster,dc=internal` | LDAP domain suffix. Only used when `DirectoryService != none` | +| `PostInstallScriptUrl` | *(empty → auto)* | Script run on every node at first boot (PCS equivalent of ParallelCluster `OnNodeConfigured`). **Empty (default) auto-installs Enroot/Pyxis** from `s3:///scripts/install-enroot-pyxis.sh` (fetched with the instance role, so it works with a **private** bucket — no public S3 needed). Accepts an `s3://` URL (instance-role fetch) or an `http(s)://` URL (curl, public only, e.g. GitHub raw). Set to a single space to skip. Idempotent: a no-op if Enroot/Pyxis is already pre-baked into `AmiId` | +| `PostInstallScriptArgs` | *(empty)* | Arguments passed to the post-install script. Normally left empty — most users never touch the container-runtime parameters | + +## 6. FSx Storage (`/fsx` and `/home`) -See [Storage: FSx deployment types](../README.md#storage-fsx-deployment-types-region-availability) for Region availability. +See [README §8.1 Storage](../README.md#81-storage-fsx-deployment-types--sizing) for Region +availability and the "deploy small, expand after" tip. | Parameter | Default | Purpose | |---|---|---| -| `Capacity` | `1200` | FSx for Lustre (`/fsx`) capacity (GiB; 1200 or increments of 2400) | +| `Capacity` | `1200` | FSx for Lustre (`/fsx`) capacity (GiB; 1200 or increments of 2400). Can be increased after creation, so start small for a faster first deploy | | `LustreDeploymentType` | `PERSISTENT_2` | FSx for Lustre (`/fsx`) deployment type (`PERSISTENT_2` / `PERSISTENT_1`) — Region-dependent | | `PerUnitStorageThroughput` | `250` | FSx for Lustre (`/fsx`) throughput (MB/s/TiB); valid values depend on the deployment type | | `Compression` | `LZ4` | FSx for Lustre (`/fsx`) data compression (`LZ4` / `NONE`) | | `LustreVersion` | `2.15` | FSx for Lustre (`/fsx`) software version (`2.15` / `2.12`) | | `FSxLustreEnableEfa` | `false` | Enable EFA on the FSx for Lustre filesystem. **The headline feature is GPUDirect Storage (GDS) for P5/P5e/P5en/P6-B200 GPU clients**, which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile stack on the client — tracked as a follow-up in [docs/ROADMAP.md](./ROADMAP.md)). EFA-capable CPU CNGs (`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary benefit, useful when a single client is pushing past ~10 GBps. **PERSISTENT_2 SSD only** — a CFN Rule on the prerequisites template fails the stack at create time when combined with PERSISTENT_1 (rather than silently ignoring the opt-in). **Requires a much larger `Capacity` than non-EFA**: at `PerUnitStorageThroughput=250` the minimum is **19200 GiB** (16× the 1200 GiB non-EFA default). The full minimum-capacity matrix per throughput tier is in the [FSx for Lustre User Guide](https://docs.aws.amazon.com/fsx/latest/LustreGuide/efa.html). The FSx side rejects undersized capacity at stack-create time with a clear error | -| `HomeCapacity` | `512` | FSx for OpenZFS (`/home`) capacity (GiB) | +| `HomeCapacity` | `512` | FSx for OpenZFS (`/home`) capacity (GiB). Can be increased after creation | | `HomeThroughput` | `320` | FSx for OpenZFS (`/home`) throughput (MB/s) | | `OpenZFSDeploymentType` | `SINGLE_AZ_HA_2` | FSx for OpenZFS (`/home`) deployment type (`SINGLE_AZ_HA_2` / `SINGLE_AZ_HA_1` / `SINGLE_AZ_2` / `SINGLE_AZ_1`) — Region-dependent | @@ -100,6 +104,3 @@ See [Storage: FSx deployment types](../README.md#storage-fsx-deployment-types-re |---|---|---| | `S3BucketName` | `awsome-distributed-ai` | S3 bucket the nested templates are fetched from | | `S3KeyPrefix` | `templates/` | S3 key prefix for the nested templates | -| `MonitoringVersion` | `v2.9.1` | [aws-parallelcluster-monitoring](https://github.com/aws-samples/aws-parallelcluster-monitoring) git ref (release tag, branch, or `latest`). `v2.9.1` adds the `DCGM_EXPORTER_IMAGE` override (needed for B300 GPU metrics) and brings Grafana 13; `v2.6.4`+ carry the PCS `/opt` install + Docker-29.x DCGM fixes. Pin to a tag for stability. Migration notes: [OPERATIONS.md §3](./OPERATIONS.md#3-monitoring-monitoringversion) | -| `MonitoringRepo` | `aws-samples/aws-parallelcluster-monitoring` | GitHub `owner/repo` for the monitoring stack; override with a fork + a branch in `MonitoringVersion` to test unreleased changes | -| `DcgmExporterImage` | DCGM 4.5.2 by digest | `dcgm-exporter` image used on GPU nodes. Defaults to a DCGM 4.5.2 build pinned by digest (`nvcr.io/nvidia/k8s/dcgm-exporter@sha256:a7ad6547...`) covering Hopper / B200 / B300. The digest pull bypasses the Docker-29.x OCI-index failure on newer NVCR tags. Override (any image reference, ideally also a digest) to pin to a different build — e.g. the monitoring stack's older default 4.2.0. No effect on CPU nodes. See [OPERATIONS.md §3.1](./OPERATIONS.md#31-dcgmexporterimage--the-default-and-when-to-change-it) | diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index a9a19434e..3f9582aac 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -22,6 +22,7 @@ Run this **complete set** before merging template or script changes. | 10 | **Storage** | FSx health check + performance regression test (noatime benchmark) | [`storage-test.md`](./storage-test.md) | FSx / mount changes | | 11-12 | **Multi-user** | OpenLDAP directory + Slurm managed accounting | [`multi-user-test.md`](./multi-user-test.md) | Directory / accounting changes | | 13 | **GPU health** | GPU Cluster Health Check suite (DCGM, EFA, NVLink, NCCL thresholds) | [`gpu-healthcheck-test.md`](./gpu-healthcheck-test.md) | GPU CNG deploys | +| 14 | **IAM** | cluster-admin deploys+deletes (no `iam:CreatePolicy`); cluster-user is SSM-login-only and can't read the LDAP password | [`iam-test.md`](./iam-test.md) | IAM policy changes | --- diff --git a/architectures/aws-pcs/tests/iam-test.md b/architectures/aws-pcs/tests/iam-test.md new file mode 100644 index 000000000..c3d4fcc9b --- /dev/null +++ b/architectures/aws-pcs/tests/iam-test.md @@ -0,0 +1,106 @@ +# IAM Test: cluster-admin / cluster-user policies + +Validates the two IAM policy stacks ([`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) +and [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml), documented in +[docs/IAM.md](../docs/IAM.md)): + +1. A principal with **only** the cluster-admin policy can deploy and tear down a full + cluster — without `iam:CreatePolicy`. +2. A principal with **only** the cluster-user policy is constrained to SSM access on the + login node and cannot read the OpenLDAP admin password. + +This is the representative "two-role" use case: an admin who owns cluster lifecycle, and a +user who only runs jobs / views dashboards. + +--- + +## Setup + +Deploy both policy stacks, then create a throwaway role attached to **only** the admin +(or only the user) policy — nothing else — so the test reflects exactly what each policy +grants. + +```bash +# Deploy the policy + group stacks (see docs/IAM.md for the parameters) +aws cloudformation create-stack --stack-name pcs-iam-admin \ + --template-url https://.s3.amazonaws.com/cluster-admin-iam.yaml \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region +aws cloudformation create-stack --stack-name pcs-iam-user \ + --template-url https://.s3.amazonaws.com/cluster-user-iam.yaml \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region +``` + +Attach the resulting managed policy (admin or user) to a dedicated test role you can +`assume-role` into, or use `aws iam simulate-principal-policy` against the policy ARN for +a no-deploy check. + +--- + +## Test A — admin policy deploys and deletes a cluster + +Assume the admin-only role, then run a representative deploy (multi-user + SSH + monitoring +all on, to exercise the IAM, SSM, and KMS permissions): + +```bash +aws cloudformation create-stack --stack-name pcs-iam-test \ + --template-url https://.s3.amazonaws.com/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue= \ + ParameterKey=DirectoryService,ParameterValue=OpenLDAP-LoginNode \ + ParameterKey=SSHAccessCidr,ParameterValue=/32 \ + ParameterKey=MonitoringStack,ParameterValue=Prometheus-LoginNode \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region +# ... wait for CREATE_COMPLETE, then: +aws cloudformation delete-stack --stack-name pcs-iam-test --region +``` + +**Expected:** +- Every nested stack reaches `CREATE_COMPLETE` with **no `AccessDenied`**. +- The same admin-only role tears the stack down to `DELETE_COMPLETE` with no `AccessDenied`. +- It works **without `iam:CreatePolicy`** — the instance role's permissions are attached + inline (`PutRolePolicy`), not as a managed policy, so deploy-all needs no policy-creation + right. + +No-deploy equivalent with `simulate-principal-policy` (pass proper resource ARNs and the +`iam:PassedToService` context): `cloudformation:CreateStack`, +`ec2:RunInstances`/`CreateSubnet`/`CreateSecurityGroup`, `fsx:CreateFileSystem`, +`pcs:CreateCluster`, `iam:CreateRole`/`PutRolePolicy`/`CreateInstanceProfile`/`PassRole` +(to `ec2` and `pcs`), `ssm:PutParameter`/`GetParameter` on `/pcs/*/ldap/*`, `kms:Decrypt` +→ all `allowed`; `iam:CreatePolicy` → `implicitDeny` (expected, and not needed). + +--- + +## Test B — user policy is constrained + +With the user-only role / policy: + +**Expected (allowed):** `pcs:GetCluster`/`ListComputeNodeGroups`, +`cloudformation:DescribeStacks`, `ec2:DescribeInstances`, `ssm:StartSession` **on the login +node**, and reading `/pcs/*/grafana/*` (the Grafana password). + +**Expected (denied — `implicitDeny`):** `cloudformation:CreateStack`, `pcs:CreateCluster`, +`ec2:RunInstances`, `fsx:CreateFileSystem`, `iam:CreateRole`; `ssm:StartSession` on a +**compute** node; and — critically for security — reading `/pcs/*/ldap/*` (the OpenLDAP +admin password). + +```bash +# Example simulate checks (repeat per action/resource): +aws iam simulate-principal-policy --policy-source-arn \ + --action-names ssm:StartSession \ + --resource-arns arn:aws:ec2:::instance/ +# → allowed for the login node, implicitDeny for a compute node +``` + +--- + +## Verified + +Run end-to-end on a real account (us-east-2) against the major-update templates, with a +dedicated role attached to only the admin (or user) policy: + +- Both IAM stacks `CREATE_COMPLETE`. +- Admin-only role: deploy-all (DirectoryService + SSHAccessCidr + monitoring) → + `CREATE_COMPLETE`, `delete-stack` → `DELETE_COMPLETE`, both with no `AccessDenied`, and + with `iam:CreatePolicy` simulating `implicitDeny` (confirming it isn't needed). +- User-only role: status/describe/login-SSM allowed; create/compute-SSM denied; can read + `/pcs/*/grafana/*` but **not** `/pcs/*/ldap/*`. diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md index 5fd6d41f5..b6c7417eb 100644 --- a/architectures/aws-pcs/tests/infra-test.md +++ b/architectures/aws-pcs/tests/infra-test.md @@ -117,16 +117,16 @@ aws cloudformation create-stack \ --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ --parameters ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --profile claude --region us-west-2 + --region us-west-2 aws cloudformation wait stack-create-complete \ --stack-name pcs-dlami-${SLURM_VERSION/./} \ - --profile claude --region us-west-2 + --region us-west-2 AMI_ID=$(aws cloudformation describe-stacks \ --stack-name pcs-dlami-${SLURM_VERSION/./} \ --query 'Stacks[0].Outputs[?OutputKey==`DLAMIforPCSAmiId`].OutputValue' \ - --output text --profile claude --region us-west-2) + --output text --region us-west-2) echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx ``` @@ -142,7 +142,7 @@ aws cloudformation create-stack \ ParameterKey=AmiId,ParameterValue=${AMI_ID} \ ParameterKey=PostInstallScriptUrl,ParameterValue=' ' \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ - --profile claude --region us-west-2 + --region us-west-2 ``` ### Step 3 — verify the bake landed and a container job runs @@ -171,8 +171,8 @@ version` in `journalctl -u slurmd`). ### Step 4 — clean up ```bash -aws cloudformation delete-stack --stack-name pcs-amitest-${SLURM_VERSION/./} --profile claude --region us-west-2 -aws cloudformation delete-stack --stack-name pcs-dlami-${SLURM_VERSION/./} --profile claude --region us-west-2 +aws cloudformation delete-stack --stack-name pcs-amitest-${SLURM_VERSION/./} --region us-west-2 +aws cloudformation delete-stack --stack-name pcs-dlami-${SLURM_VERSION/./} --region us-west-2 ``` The DLAMI stack's AMI itself is **not** automatically deregistered when the stack is From 14727df7a7f65593ac3eab1349390bc376650919 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:32:11 +0000 Subject: [PATCH 77/89] =?UTF-8?q?docs(readme):=20fold=20FSx-over-EFA=20(GD?= =?UTF-8?q?S)=20into=20=C2=A78.1=20Storage=20as=20a=20subsection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The standalone §8.7 'FSx for Lustre over EFA (GPUDirect Storage)' is storage content, so it's now a #### subsection at the end of §8.1 Storage instead of a separate top-level Advanced feature. Removes the cross-reference note. --- architectures/aws-pcs/README.md | 31 ++++++++++++++----------------- 1 file changed, 14 insertions(+), 17 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index c55c1b891..002019c03 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -344,8 +344,20 @@ Check support before deploying: If a deploy fails at the FSx resource with an "unsupported deployment type" error, switch these parameters to a type your Region supports. -> Enabling EFA / GPUDirect Storage on the Lustre filesystem (`FSxLustreEnableEfa`) is an -> advanced option — see [§8.7 FSx for Lustre over EFA (GPUDirect Storage)](#87-fsx-for-lustre-over-efa-gpudirect-storage). +#### FSx for Lustre over EFA (GPUDirect Storage) + +`FSxLustreEnableEfa` (default `false`) enables EFA on the `/fsx` Lustre filesystem. +**The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, +which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile +stack on the client — see the Client-side Lustre-on-EFA + GDS item in +[`docs/ROADMAP.md`](./docs/ROADMAP.md)). EFA-capable CPU CNGs +(`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary +benefit, useful when a single client is pushing past ~10 GBps. + +Constraints when enabling this: +- **PERSISTENT_2 SSD only** — a CFN Rule fails the stack at create time on `PERSISTENT_1`. +- **Much higher minimum `Capacity`** — at `PerUnitStorageThroughput=250` the minimum is + **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly. ### 8.2 Monitoring @@ -602,21 +614,6 @@ application's natural multi-pair pattern) to actually exercise both NICs on hpc7a/hpc8a. See [tests/README.md Test 9](./tests/README.md#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) for the full benchmark setup and validated bandwidth numbers. -### 8.7 FSx for Lustre over EFA (GPUDirect Storage) - -`FSxLustreEnableEfa` (default `false`) enables EFA on the `/fsx` Lustre filesystem. -**The headline feature is GPUDirect Storage (GDS) for P5 / P5e / P5en / P6-B200 GPU CNGs**, -which DMAs file data straight into GPU memory (requires the NVIDIA `nvidia-fs` / cuFile -stack on the client — see the Client-side Lustre-on-EFA + GDS item in -[`docs/ROADMAP.md`](./docs/ROADMAP.md)). EFA-capable CPU CNGs -(`OnDemandEfaInterfaceCount > 0`) get the EFA *transport* path to storage as a secondary -benefit, useful when a single client is pushing past ~10 GBps. - -Constraints when enabling this: -- **PERSISTENT_2 SSD only** — a CFN Rule fails the stack at create time on `PERSISTENT_1`. -- **Much higher minimum `Capacity`** — at `PerUnitStorageThroughput=250` the minimum is - **19200 GiB** (16× the 1200 GiB default). Set `Capacity` accordingly. - --- ## 9. Templates From 5a96257bb33ee89c6f7443c763c47a863d910f77 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:34:05 +0000 Subject: [PATCH 78/89] =?UTF-8?q?docs(readme):=20add=20=C2=A78.7=20pointin?= =?UTF-8?q?g=20to=20DEPLOY-TESTING.md=20for=20unpublished-template=20deplo?= =?UTF-8?q?ys?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an Advanced Features subsection that links docs/DEPLOY-TESTING.md (deploying fork/ branch/PR templates from your own S3 bucket via S3BucketName/S3KeyPrefix). This also gives DEPLOY-TESTING.md its README entry point — every docs/ file is now linked from the README. --- architectures/aws-pcs/README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 002019c03..55192ccd0 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -614,6 +614,15 @@ application's natural multi-pair pattern) to actually exercise both NICs on hpc7a/hpc8a. See [tests/README.md Test 9](./tests/README.md#test-9-efa-on-cpu-hpc-instances-hpc6a--hpc7a--hpc8a) for the full benchmark setup and validated bandwidth numbers. +### 8.7 Deploying updated templates before they are published + +The Quick Start deploys from the public production bucket (`awsome-distributed-ai`), which +only holds already-published templates. If you need to test **template or script changes +that aren't published yet** — a fork, a feature branch, or a PR under review — host the +templates + boot scripts in an S3 bucket you control and point the deploy at it via +`S3BucketName` / `S3KeyPrefix`. The step-by-step procedure (upload, deploy, iterate, +clean up) is in [docs/DEPLOY-TESTING.md](./docs/DEPLOY-TESTING.md). + --- ## 9. Templates From 1efef89aec0a77c7151f5ddfd2b5ce1d42e8667d Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:35:53 +0000 Subject: [PATCH 79/89] =?UTF-8?q?docs(readme):=20fix=20=C2=A74=20group=205?= =?UTF-8?q?=20heading/content=20mismatch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The '5. Additional' heading said 'container runtime' but the table only listed the common params (MonitoringStack/GrafanaAccessCidr/DirectoryService). Retitle to '(monitoring, multi-user)' and add a one-line note that the rarely-changed monitoring-source + container-runtime params live in the same console group (see PARAMETERS.md). --- architectures/aws-pcs/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 55192ccd0..f77ebb825 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -123,7 +123,7 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -**5. Additional (monitoring, multi-user, container runtime)** +**5. Additional (monitoring, multi-user)** | Parameter | Default | Purpose | |---|---|---| @@ -131,6 +131,10 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 (Grafana) on the login node to a trusted CIDR (default: SSM port-forward only) | | `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.3](#83-user-management) | +(The console's group 5 also holds the monitoring-source and container-runtime +parameters — `MonitoringRepo` / `MonitoringVersion` / `DcgmExporterImage` / +`PostInstallScriptUrl` — which rarely need changing; see [PARAMETERS.md](./docs/PARAMETERS.md).) + **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the choices that need the most thought. From 147f6aa4ba273e51ddff4556e87917d60f27c1a7 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:37:12 +0000 Subject: [PATCH 80/89] =?UTF-8?q?docs(readme):=20drop=20the=20redundant=20?= =?UTF-8?q?note=20under=20=C2=A74=20group=205?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 'console's group 5 also holds…' line was redundant with the PARAMETERS.md pointer right below it. Group 5 is just the heading + the 3 common params. --- architectures/aws-pcs/README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index f77ebb825..8226daf7a 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -131,10 +131,6 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 (Grafana) on the login node to a trusted CIDR (default: SSM port-forward only) | | `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.3](#83-user-management) | -(The console's group 5 also holds the monitoring-source and container-runtime -parameters — `MonitoringRepo` / `MonitoringVersion` / `DcgmExporterImage` / -`PostInstallScriptUrl` — which rarely need changing; see [PARAMETERS.md](./docs/PARAMETERS.md).) - **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the choices that need the most thought. From 338ee1e39244ab3d7306768255e7e21f62eafe52 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:38:34 +0000 Subject: [PATCH 81/89] =?UTF-8?q?docs(readme):=20match=20=C2=A74=20group?= =?UTF-8?q?=20headings=20to=20the=20deploy-all=20ParameterGroup=20names?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §4 sub-table headings now use the exact console parameter-group labels (1. Network Configuration … 5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)), and group 5 lists PostInstallScriptUrl so the heading and its rows agree. --- architectures/aws-pcs/README.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 8226daf7a..1c6721c97 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -92,14 +92,14 @@ parameters are grouped below to match the deploy-all console's parameter groups. parameters are covered in [§8.1 Storage](#81-storage-fsx-deployment-types--sizing); for the complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). -**1. Network** +**1. Network Configuration** | Parameter | Default | Purpose | |---|---|---| | `PrimarySubnetAZ` | *(required)* | Availability Zone to deploy into — the one required parameter | | `AdditionalSubnetAZ2` / `…AZ3` | *(empty)* | Add private subnets in up to 2 more AZs (primary + 2). NAT gateway stays in the primary AZ | -**2. PCS cluster** +**2. PCS Cluster Configuration** | Parameter | Default | Purpose | |---|---|---| @@ -107,7 +107,7 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `AmiId` | *(empty → SSM auto)* | Empty auto-resolves the latest PCS-Ready DLAMI. See [AMI and container runtime](#ami-and-container-runtime) | | `SSHAccessCidr` | *(empty)* | Open SSH/22 on the login node to a trusted CIDR (default: SSM only). See [§6](#6-accessing-the-cluster) | -**3. On-Demand compute queue** +**3. On-Demand Compute Node Group** | Parameter | Default | Purpose | |---|---|---| @@ -115,7 +115,7 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `OnDemandInstanceType` | `c6i.4xlarge` | Any On-Demand type — CPU, a single-NIC GPU (`g6.12xlarge`, [Example 1](#example-1-single-nic-gpu-queue-g6)), or an HPC type. Multi-NIC P5/P6 use the GPU queue instead | | `OnDemandEfaInterfaceCount` | `0` | `1`/`2` enables EFA for HPC/MPI on EFA-capable CPU types. See [§8.6](#86-cpu-compute-node-group--advanced-settings) | -**4. GPU compute queue (P5/P6, optional)** +**4. GPU Compute Node Group - P5/P6 Series (Optional)** | Parameter | Default | Purpose | |---|---|---| @@ -123,13 +123,16 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). | `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | | `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | -**5. Additional (monitoring, multi-user)** +**5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)** | Parameter | Default | Purpose | |---|---|---| | `MonitoringStack` | `Prometheus-LoginNode` | Prometheus + Grafana on the login node, DCGM Exporter on GPU nodes. `none` disables it. See [§8.2](#82-monitoring) | | `GrafanaAccessCidr` | *(empty)* | Open HTTPS/443 (Grafana) on the login node to a trusted CIDR (default: SSM port-forward only) | | `DirectoryService` | `none` | `OpenLDAP-LoginNode` for a multi-user cluster. See [§8.3](#83-user-management) | +| `PostInstallScriptUrl` | *(empty → auto)* | First-boot script; empty auto-installs Enroot/Pyxis from your templates bucket. Rarely changed. See [PARAMETERS.md](./docs/PARAMETERS.md) | + +(Group 5 also has `MonitoringRepo` / `MonitoringVersion` / `DcgmExporterImage` — pinned defaults, rarely changed; see [PARAMETERS.md](./docs/PARAMETERS.md).) **See [PARAMETERS.md](./docs/PARAMETERS.md) for the complete parameter reference** (all 7 console parameter groups, with every default). The concept guides below cover the From 0ade772bfaeb3b3569f89035179970e110df0681 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:40:55 +0000 Subject: [PATCH 82/89] docs(tests): fix stale PostInstallScriptUrl caveat MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The default is no longer the awslabs/main GitHub-raw URL — it's empty, which resolves to s3:///scripts/install-enroot-pyxis.sh. Rewrite the caveat: when testing unpublished templates, point S3BucketName at your own bucket and sync the scripts there, or first-boot Enroot/Pyxis fetch fails (cluster still CREATE_COMPLETE). Link DEPLOY-TESTING.md. --- architectures/aws-pcs/tests/README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/architectures/aws-pcs/tests/README.md b/architectures/aws-pcs/tests/README.md index 3f9582aac..1bbc375c5 100644 --- a/architectures/aws-pcs/tests/README.md +++ b/architectures/aws-pcs/tests/README.md @@ -117,11 +117,15 @@ us-east-1 bucket. The PCS-Ready DLAMI SSM parameter every region tested. `OpenZFSDeploymentType=SINGLE_AZ_HA_2` and `LustreDeploymentType=PERSISTENT_2` (the defaults) were available in all five. -> **Pre-merge caveat:** `PostInstallScriptUrl` defaults to the `awslabs/main` -> GitHub-raw URL, which 404s until PR #1120 merges — so pre-merge test deploys -> must override it (point at the test bucket / fork) or compute nodes boot -> without Enroot/Pyxis (`/var/log/pcs-post-install.log` shows exit 127). The -> cluster still reaches CREATE_COMPLETE; only the container runtime is missing. +> **Testing unpublished templates:** an empty `PostInstallScriptUrl` (the default) +> resolves to `s3:///scripts/install-enroot-pyxis.sh` — i.e. +> the boot script comes from the **same bucket as the nested templates**. So when +> testing changes that aren't in the public bucket yet, point `S3BucketName` at your own +> bucket **and** sync the scripts there (the `aws s3 sync assets/ …` step covers both); +> otherwise the first-boot fetch fails and nodes come up without Enroot/Pyxis +> (`/var/log/pcs-post-install.log` shows the failed `aws s3 cp`). The cluster still +> reaches CREATE_COMPLETE; only the container runtime is missing. See +> [docs/DEPLOY-TESTING.md](../docs/DEPLOY-TESTING.md). --- From 56bf60cd854befc9c0a434379726ff24e1bd87e2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:48:22 +0000 Subject: [PATCH 83/89] docs(tests): remove private test-bucket name from infra-test.md Generalize the s3:// verification note and Test 8 build/deploy commands to use / placeholders instead of the project's private midaisuk-llm-dev bucket, matching DEPLOY-TESTING.md. --- architectures/aws-pcs/tests/infra-test.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/architectures/aws-pcs/tests/infra-test.md b/architectures/aws-pcs/tests/infra-test.md index b6c7417eb..ba1d9c6c6 100644 --- a/architectures/aws-pcs/tests/infra-test.md +++ b/architectures/aws-pcs/tests/infra-test.md @@ -39,11 +39,10 @@ post-install runs the Enroot/Pyxis installer), then on any node: > **The default `PostInstallScriptUrl` is now an `s3://` URL** (empty → > `s3:///scripts/install-enroot-pyxis.sh`, fetched with -> the instance role). Verified end-to-end against a **private** test bucket -> (`s3://midaisuk-llm-dev/...`): the node's post-install log shows -> `Downloading post-install script from s3://…` and `pyxis.conf` is installed — -> so no public S3 is required (works in dev accounts). An `http(s)://` value still -> works too (curl), for public/GitHub-raw scripts. +> the instance role). Verified end-to-end against a **private** test bucket: the +> node's post-install log shows `Downloading post-install script from s3://…` and +> `pyxis.conf` is installed — so no public S3 is required (works in dev accounts). An +> `http(s)://` value still works too (curl), for public/GitHub-raw scripts. ```bash which enroot # /usr/bin/enroot @@ -114,7 +113,7 @@ SLURM_VERSION=25.11 # repeat for 25.05 if relevant aws cloudformation create-stack \ --stack-name pcs-dlami-${SLURM_VERSION/./} \ - --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ + --template-url https://.s3.amazonaws.com/pcs-ready-dlami-with-enroot-pyxis.yaml \ --parameters ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \ --region us-west-2 @@ -135,7 +134,7 @@ echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx ```bash aws cloudformation create-stack \ --stack-name pcs-amitest-${SLURM_VERSION/./} \ - --template-url https://midaisuk-llm-dev.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --template-url https://.s3.amazonaws.com/pcs-ml-cluster-deploy-all.yaml \ --parameters \ ParameterKey=PrimarySubnetAZ,ParameterValue=us-west-2a \ ParameterKey=SlurmVersion,ParameterValue=${SLURM_VERSION} \ From 41fb46fd9b614eefba7f4d7d937c6a499e19b5c2 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:55:23 +0000 Subject: [PATCH 84/89] docs: add Deploy buttons to IAM role table; move custom-AMI detail to docs/CUSTOM-AMI.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - IAM.md: add 1-click Deploy buttons to the two-role summary table (matching the deploy table further down) - move the §8.5 step-by-step pre-bake procedure into docs/CUSTOM-AMI.md; README §8.5 keeps a concise summary + Launch button + link (heading text unchanged so the #85-... anchor referenced elsewhere stays valid) --- architectures/aws-pcs/README.md | 63 +++------------------ architectures/aws-pcs/docs/CUSTOM-AMI.md | 70 ++++++++++++++++++++++++ architectures/aws-pcs/docs/IAM.md | 8 +-- 3 files changed, 81 insertions(+), 60 deletions(-) create mode 100644 architectures/aws-pcs/docs/CUSTOM-AMI.md diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 1c6721c97..99f8bef95 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -519,65 +519,16 @@ for roles, deploy instructions, security considerations, and the verification ma ### 8.5 Pre-baking Enroot/Pyxis into a custom AMI The all-in-one template installs Enroot/Pyxis at **first boot** via -`PostInstallScriptUrl`, which is fast to deploy and avoids an Image Builder step. -For **frequent scaling** in production, pre-baking Enroot/Pyxis into a custom AMI -drops node boot time from ~8–12 min to ~3 min and pins every node to a deterministic -state. This is a separate, standalone path: build the AMI once with -[`pcs-ready-dlami-with-enroot-pyxis.yaml`](./assets/pcs-ready-dlami-with-enroot-pyxis.yaml), -then pass the resulting `ami-xxx` as `AmiId` to the cluster. - -**Step 1: Build the AMI** (~30 min one-time, separate stack) +`PostInstallScriptUrl` (no Image Builder step). For **frequent scaling** in production, +pre-baking Enroot/Pyxis into a custom AMI drops node boot from ~8–12 min to ~3 min and +pins every node to a deterministic state. It's a standalone path: build the AMI once with +[`pcs-ready-dlami-with-enroot-pyxis.yaml`](./assets/pcs-ready-dlami-with-enroot-pyxis.yaml) +(single-Slurm-version by design), then pass the resulting `ami-xxx` as `AmiId`. [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml&stackName=pcs-dlami) -```bash -aws cloudformation create-stack \ - --stack-name pcs-dlami \ - --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ - --parameters ParameterKey=SlurmVersion,ParameterValue=25.11 \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM -``` - -The AMI is **single-Slurm-version by design**: Pyxis is a SPANK plugin whose ABI is -locked to its compile-time Slurm version, so pass the same `SlurmVersion` you'll use -on the cluster. - -**Step 2: Read the resulting AMI ID** from the stack output `DLAMIforPCSAmiId`: - -```bash -AMI_ID=$(aws cloudformation describe-stacks \ - --stack-name pcs-dlami \ - --query 'Stacks[0].Outputs[?OutputKey==`DLAMIforPCSAmiId`].OutputValue' \ - --output text) -echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx -``` - -**Step 3: Pass it to the cluster** as `AmiId`. Optionally skip the boot-time -Enroot/Pyxis install (it's already baked in) by setting `PostInstallScriptUrl` -to a single space: - -```bash -aws cloudformation create-stack \ - --stack-name pcs-ml-cluster \ - --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ - --parameters \ - ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-1a \ - ParameterKey=AmiId,ParameterValue=$AMI_ID \ - ParameterKey=PostInstallScriptUrl,ParameterValue=' ' \ - --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM -``` - -Leaving `PostInstallScriptUrl` at its default (empty → auto-install Enroot/Pyxis -from the templates bucket) also works on a pre-baked AMI: the installer detects -Enroot/Pyxis is already present and is a fast idempotent no-op. Passing a single -space skips the download+check entirely, shaving a few seconds off boot. - -**Optional features of `pcs-ready-dlami-with-enroot-pyxis.yaml`** (defaults are off): -- `BuildSchedule=Weekly`/`Monthly` for scheduled rebuilds against a moving base AMI -- `EnableLifecyclePolicy=true` to deprecate older AMIs after `LifecycleDeprecateAfterWeeks` -- `PublishToSsm=true` to publish the latest AMI ID to an SSM parameter for downstream stacks - -For production deploys that pin the AMI explicitly per cluster, none of these are needed. +See **[docs/CUSTOM-AMI.md](./docs/CUSTOM-AMI.md)** for the full build → read AMI ID → +deploy procedure and the optional scheduled-rebuild / lifecycle / SSM-publish features. ### 8.6 CPU compute node group — advanced settings diff --git a/architectures/aws-pcs/docs/CUSTOM-AMI.md b/architectures/aws-pcs/docs/CUSTOM-AMI.md new file mode 100644 index 000000000..1f9c53da8 --- /dev/null +++ b/architectures/aws-pcs/docs/CUSTOM-AMI.md @@ -0,0 +1,70 @@ +# Pre-baking Enroot/Pyxis into a custom AMI + +The all-in-one template installs Enroot/Pyxis at **first boot** via +`PostInstallScriptUrl`, which is fast to deploy and avoids an Image Builder step. For +**frequent scaling** in production, pre-baking Enroot/Pyxis into a custom AMI drops node +boot time from ~8–12 min to ~3 min and pins every node to a deterministic state. + +This is a separate, standalone path: build the AMI once with +[`pcs-ready-dlami-with-enroot-pyxis.yaml`](../assets/pcs-ready-dlami-with-enroot-pyxis.yaml), +then pass the resulting `ami-xxx` as `AmiId` to the cluster. + +## Step 1: Build the AMI (~30 min one-time, separate stack) + +[![Launch](../images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml&stackName=pcs-dlami) + +```bash +aws cloudformation create-stack \ + --stack-name pcs-dlami \ + --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml \ + --parameters ParameterKey=SlurmVersion,ParameterValue=25.11 \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` + +The AMI is **single-Slurm-version by design**: Pyxis is a SPANK plugin whose ABI is +locked to its compile-time Slurm version, so pass the same `SlurmVersion` you'll use on +the cluster. + +## Step 2: Read the resulting AMI ID + +From the stack output `DLAMIforPCSAmiId`: + +```bash +AMI_ID=$(aws cloudformation describe-stacks \ + --stack-name pcs-dlami \ + --query 'Stacks[0].Outputs[?OutputKey==`DLAMIforPCSAmiId`].OutputValue' \ + --output text) +echo "$AMI_ID" # ami-0xxxxxxxxxxxxxxxx +``` + +## Step 3: Pass it to the cluster as `AmiId` + +Optionally skip the boot-time Enroot/Pyxis install (it's already baked in) by setting +`PostInstallScriptUrl` to a single space: + +```bash +aws cloudformation create-stack \ + --stack-name pcs-ml-cluster \ + --template-url https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml \ + --parameters \ + ParameterKey=PrimarySubnetAZ,ParameterValue=us-east-1a \ + ParameterKey=AmiId,ParameterValue=$AMI_ID \ + ParameterKey=PostInstallScriptUrl,ParameterValue=' ' \ + --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM +``` + +Leaving `PostInstallScriptUrl` at its default (empty → auto-install Enroot/Pyxis from the +templates bucket) also works on a pre-baked AMI: the installer detects Enroot/Pyxis is +already present and is a fast idempotent no-op. Passing a single space skips the +download+check entirely, shaving a few seconds off boot. + +## Optional features of `pcs-ready-dlami-with-enroot-pyxis.yaml` + +Defaults are off: +- `BuildSchedule=Weekly`/`Monthly` for scheduled rebuilds against a moving base AMI +- `EnableLifecyclePolicy=true` to deprecate older AMIs after `LifecycleDeprecateAfterWeeks` +- `PublishToSsm=true` to publish the latest AMI ID to an SSM parameter for downstream stacks + +For production deploys that pin the AMI explicitly per cluster, none of these are needed. + +> The build path is validated by [tests/infra-test.md Test 8](../tests/infra-test.md#test-8-pre-baked-ami-build-standalone-dlami-template). diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md index dbfe48cfc..ff92dd857 100644 --- a/architectures/aws-pcs/docs/IAM.md +++ b/architectures/aws-pcs/docs/IAM.md @@ -3,10 +3,10 @@ The cluster distinguishes **two human roles** with very different responsibilities, and ships a ready-to-deploy IAM policy stack for each: -| Role | Who | What they can do | Template | -|---|---|---|---| -| **Cluster admin** | The person who deploys/updates/deletes the cluster | Full CRUD on the infrastructure: CloudFormation, PCS, EC2 (VPC/SG/launch templates/placement groups/NAT/EIP), FSx, scoped IAM, SSM Parameter Store, KMS, Secrets Manager, and (optionally) Image Builder | [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | -| **Cluster user** | Engineers who just run jobs on an existing cluster | SSM session **to the login node only**, port-forward Grafana, read the Grafana password, read PCS cluster/queue status. **Cannot create, modify, or delete anything**, and cannot open shells on compute nodes | [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | +| Role | Who | What they can do | Template | Deploy | +|---|---|---|---|---| +| **Cluster admin** | The person who deploys/updates/deletes the cluster | Full CRUD on the infrastructure: CloudFormation, PCS, EC2 (VPC/SG/launch templates/placement groups/NAT/EIP), FSx, scoped IAM, SSM Parameter Store, KMS, Secrets Manager, and (optionally) Image Builder | [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** | Engineers who just run jobs on an existing cluster | SSM session **to the login node only**, port-forward Grafana, read the Grafana password, read PCS cluster/queue status. **Cannot create, modify, or delete anything**, and cannot open shells on compute nodes | [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | Splitting the roles means the deployer's broad permissions never have to be handed to every engineer who just wants to `srun`, and the user role can be From b077f57908dde3c2405bb1833e2e4fef8bea74a1 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:57:48 +0000 Subject: [PATCH 85/89] docs(iam): use the Launch-stack image for the policy Deploy buttons, single location Match the custom-AMI style: the cluster-admin / cluster-user Deploy buttons now use the launch-stack.svg image, kept in one place (the 'Deploying the policies' table); drop the duplicate kbd buttons from the role summary table. --- architectures/aws-pcs/docs/IAM.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/architectures/aws-pcs/docs/IAM.md b/architectures/aws-pcs/docs/IAM.md index ff92dd857..68aa924cb 100644 --- a/architectures/aws-pcs/docs/IAM.md +++ b/architectures/aws-pcs/docs/IAM.md @@ -3,10 +3,10 @@ The cluster distinguishes **two human roles** with very different responsibilities, and ships a ready-to-deploy IAM policy stack for each: -| Role | Who | What they can do | Template | Deploy | -|---|---|---|---|---| -| **Cluster admin** | The person who deploys/updates/deletes the cluster | Full CRUD on the infrastructure: CloudFormation, PCS, EC2 (VPC/SG/launch templates/placement groups/NAT/EIP), FSx, scoped IAM, SSM Parameter Store, KMS, Secrets Manager, and (optionally) Image Builder | [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | -| **Cluster user** | Engineers who just run jobs on an existing cluster | SSM session **to the login node only**, port-forward Grafana, read the Grafana password, read PCS cluster/queue status. **Cannot create, modify, or delete anything**, and cannot open shells on compute nodes | [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | +| Role | Who | What they can do | Template | +|---|---|---|---| +| **Cluster admin** | The person who deploys/updates/deletes the cluster | Full CRUD on the infrastructure: CloudFormation, PCS, EC2 (VPC/SG/launch templates/placement groups/NAT/EIP), FSx, scoped IAM, SSM Parameter Store, KMS, Secrets Manager, and (optionally) Image Builder | [`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) | +| **Cluster user** | Engineers who just run jobs on an existing cluster | SSM session **to the login node only**, port-forward Grafana, read the Grafana password, read PCS cluster/queue status. **Cannot create, modify, or delete anything**, and cannot open shells on compute nodes | [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml) | Splitting the roles means the deployer's broad permissions never have to be handed to every engineer who just wants to `srun`, and the user role can be @@ -21,10 +21,10 @@ Each template creates the customer-managed IAM policies, an IAM group with them attached, and (optionally) adds existing IAM users to that group. Deploy both as CloudFormation stacks: -| Stack | Creates | 1-click | +| Stack | Creates | Deploy | |---|---|---| -| **Cluster admin** | `-PCSClusterAdmin-core` (always) + `-PCSClusterAdmin-imagebuilder` (if opted in) managed policies + an IAM group | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | -| **Cluster user** | `-PCSClusterUser` managed policy + an IAM group | [🚀 Deploy](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | +| **Cluster admin** | `-PCSClusterAdmin-core` (always) + `-PCSClusterAdmin-imagebuilder` (if opted in) managed policies + an IAM group | [![Launch](../images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** | `-PCSClusterUser` managed policy + an IAM group | [![Launch](../images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | Or from the CLI (use `--template-body` against a local checkout for a pre-merge sandbox test): From 9c69bc47a6112aa9ee4f32c4880bf8a09d82f160 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 17:59:34 +0000 Subject: [PATCH 86/89] =?UTF-8?q?docs(readme):=20use=20the=20Launch-stack?= =?UTF-8?q?=20image=20for=20all=20=C2=A79=20template=20Deploy=20buttons?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the kbd 🚀 buttons in the Templates table with the launch-stack.svg image, consistent with the Quick Start, §8.5 custom-AMI, and IAM policy Deploy buttons. --- architectures/aws-pcs/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 99f8bef95..ff5b83cf3 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -588,14 +588,14 @@ parameter and default, see [PARAMETERS.md](./docs/PARAMETERS.md). | Template | Purpose | Deploy | |---|---|---| -| [`pcs-ml-cluster-deploy-all.yaml`](./assets/pcs-ml-cluster-deploy-all.yaml) | All-in-one: Prerequisites + (optional AMI) + Cluster + login/CPU/GPU CNGs | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml&stackName=pcs-ml-cluster) | -| [`ml-cluster-prerequisites.yaml`](./assets/ml-cluster-prerequisites.yaml) | VPC, subnets, security groups, FSx for Lustre + OpenZFS | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/ml-cluster-prerequisites.yaml&stackName=pcs-prerequisites) | -| [`cluster.yaml`](./assets/cluster.yaml) | PCS cluster core (Slurm scheduler only, no nodes) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster.yaml&stackName=pcs-cluster) | -| [`add-cng.yaml`](./assets/add-cng.yaml) | Compute node group — login nodes, CPU / single-NIC-GPU queues (C6i, G5, G6); switches to a multi-NIC EFA `NetworkInterfaces` block when `EfaInterfaceCount > 0` (HPC types: hpc6a/hpc7a/hpc6id/hpc8a …) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng.yaml&stackName=pcs-add-cng) | -| [`add-cng-p5.yaml`](./assets/add-cng-p5.yaml) | P5/P5e/P5en nodes (16/32 EFA interfaces, by type) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p5.yaml&stackName=pcs-add-cng-p5) | -| [`add-cng-p6-b200.yaml`](./assets/add-cng-p6-b200.yaml) | P6-B200 nodes (8 EFA interfaces) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b200.yaml&stackName=pcs-add-cng-p6-b200) | -| [`add-cng-p6-b300.yaml`](./assets/add-cng-p6-b300.yaml) | P6-B300 nodes (17 interfaces: 16 EFA + 1 ENA) | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b300.yaml&stackName=pcs-add-cng-p6-b300) | -| [`pcs-ready-dlami-with-enroot-pyxis.yaml`](./assets/pcs-ready-dlami-with-enroot-pyxis.yaml) | EC2 Image Builder: bake Enroot 3.5.0 + Pyxis 0.20.0 into the PCS-Ready DLAMI | [🚀](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml&stackName=pcs-dlami) | +| [`pcs-ml-cluster-deploy-all.yaml`](./assets/pcs-ml-cluster-deploy-all.yaml) | All-in-one: Prerequisites + (optional AMI) + Cluster + login/CPU/GPU CNGs | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ml-cluster-deploy-all.yaml&stackName=pcs-ml-cluster) | +| [`ml-cluster-prerequisites.yaml`](./assets/ml-cluster-prerequisites.yaml) | VPC, subnets, security groups, FSx for Lustre + OpenZFS | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/ml-cluster-prerequisites.yaml&stackName=pcs-prerequisites) | +| [`cluster.yaml`](./assets/cluster.yaml) | PCS cluster core (Slurm scheduler only, no nodes) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster.yaml&stackName=pcs-cluster) | +| [`add-cng.yaml`](./assets/add-cng.yaml) | Compute node group — login nodes, CPU / single-NIC-GPU queues (C6i, G5, G6); switches to a multi-NIC EFA `NetworkInterfaces` block when `EfaInterfaceCount > 0` (HPC types: hpc6a/hpc7a/hpc6id/hpc8a …) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng.yaml&stackName=pcs-add-cng) | +| [`add-cng-p5.yaml`](./assets/add-cng-p5.yaml) | P5/P5e/P5en nodes (16/32 EFA interfaces, by type) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p5.yaml&stackName=pcs-add-cng-p5) | +| [`add-cng-p6-b200.yaml`](./assets/add-cng-p6-b200.yaml) | P6-B200 nodes (8 EFA interfaces) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b200.yaml&stackName=pcs-add-cng-p6-b200) | +| [`add-cng-p6-b300.yaml`](./assets/add-cng-p6-b300.yaml) | P6-B300 nodes (17 interfaces: 16 EFA + 1 ENA) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/add-cng-p6-b300.yaml&stackName=pcs-add-cng-p6-b300) | +| [`pcs-ready-dlami-with-enroot-pyxis.yaml`](./assets/pcs-ready-dlami-with-enroot-pyxis.yaml) | EC2 Image Builder: bake Enroot 3.5.0 + Pyxis 0.20.0 into the PCS-Ready DLAMI | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/pcs-ready-dlami-with-enroot-pyxis.yaml&stackName=pcs-dlami) | `add-cng*` templates create a Slurm queue only when `QueueName` is set (leave it empty for login nodes). The P-series templates need a `CapacityReservationId` when using a From 1510bc79b035713cccff3299ecb05d067fb26f70 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 18:02:50 +0000 Subject: [PATCH 87/89] =?UTF-8?q?docs(readme):=20add=20Launch-stack=20Depl?= =?UTF-8?q?oy=20buttons=20to=20=C2=A78.4=20IAM=20Permissions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turn the two-role bullet list into a table with per-role Launch-stack Deploy buttons, consistent with §9 Templates, §8.5, and docs/IAM.md. --- architectures/aws-pcs/README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index ff5b83cf3..6c1700c59 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -506,11 +506,10 @@ operations (adding/removing users, Slurm accounting, SSH access). The cluster has two human roles, each with a ready-to-deploy IAM policy stack: -- **Cluster admin** ([`cluster-admin-iam.yaml`](./assets/cluster-admin-iam.yaml)) — - deploy / update / delete the cluster (CloudFormation, PCS, EC2, FSx, scoped IAM). -- **Cluster user** ([`cluster-user-iam.yaml`](./assets/cluster-user-iam.yaml)) — - SSM session to the **login node only** + read-only status; cannot create, - modify, or delete anything. +| Role | What they can do | Deploy | +|---|---|---| +| **Cluster admin** ([`cluster-admin-iam.yaml`](./assets/cluster-admin-iam.yaml)) | deploy / update / delete the cluster (CloudFormation, PCS, EC2, FSx, scoped IAM) | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-admin-iam.yaml&stackName=pcs-cluster-admins) | +| **Cluster user** ([`cluster-user-iam.yaml`](./assets/cluster-user-iam.yaml)) | SSM session to the **login node only** + read-only status; cannot create, modify, or delete anything | [![Launch](images/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateUrl=https://awsome-distributed-ai.s3.amazonaws.com/templates/cluster-user-iam.yaml&stackName=pcs-cluster-users) | Each template creates the customer-managed policies and an IAM group, and can attach existing users at deploy time. See the **[IAM Permissions Guide](./docs/IAM.md)** From e582e65200c162aec9237e144e7db661bd7453c1 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 18:13:30 +0000 Subject: [PATCH 88/89] =?UTF-8?q?docs:=20pre-PR=20polish=20=E2=80=94=20num?= =?UTF-8?q?ber=20test=20headings=20to=20match=20the=20matrix,=20clarify=20?= =?UTF-8?q?NIC=20wording,=20flag=20CB=20billing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - tests: number the file headings (Tests 11-12 / Test 13 / Test 14) so they line up with the test matrix in tests/README.md - compute-test: '16 cards' → 'the 16 EFA NICs' to avoid confusion with GPU count - README §4: add a CB-billing ⚠️ to the CapacityReservationId row (links to GPU compute) --- architectures/aws-pcs/README.md | 2 +- architectures/aws-pcs/tests/compute-test.md | 2 +- architectures/aws-pcs/tests/gpu-healthcheck-test.md | 2 +- architectures/aws-pcs/tests/iam-test.md | 2 +- architectures/aws-pcs/tests/multi-user-test.md | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 6c1700c59..86839a092 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -121,7 +121,7 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). |---|---|---| | `DeployPseriesCNG` | `false` | Deploy a multi-NIC GPU (P5/P6) queue | | `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | -| `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | +| `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR. ⚠️ A Capacity Block bills for its whole reserved window — see [GPU compute](#gpu-compute-p5p6) | **5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)** diff --git a/architectures/aws-pcs/tests/compute-test.md b/architectures/aws-pcs/tests/compute-test.md index 50ba5bd6f..ab39ee9ec 100644 --- a/architectures/aws-pcs/tests/compute-test.md +++ b/architectures/aws-pcs/tests/compute-test.md @@ -72,7 +72,7 @@ node. EFA itself is exercised by Test 6. (N = EFA interface count: 32 for p5/p5e, 16 for p5en, 8 for p6-b200, 16 for p6-b300). - Correctness: `# Out of bounds values : 0 OK`, every size `#wrong: 0`. - `busbw` rises with message size — on 2× p6-b300, ~654 GB/s at 16 GiB but **~751 GB/s at - 64 GiB** (16 cards aren't saturated at 16 GiB); ~480 GB/s on 2× p5. + 64 GiB** (the 16 EFA NICs aren't saturated at 16 GiB); ~480 GB/s on 2× p5. > **Sizing the sweep on B300.** The canonical script sweeps to `-e 16G`. On p6-b300 that > under-reports peak bandwidth — raise it to `-e 64G` (edit the `all_reduce_perf` line) to diff --git a/architectures/aws-pcs/tests/gpu-healthcheck-test.md b/architectures/aws-pcs/tests/gpu-healthcheck-test.md index ba41bc880..ffe4e628f 100644 --- a/architectures/aws-pcs/tests/gpu-healthcheck-test.md +++ b/architectures/aws-pcs/tests/gpu-healthcheck-test.md @@ -1,4 +1,4 @@ -# Test: GPU Cluster Health Check +# Test 13: GPU Cluster Health Check Validates GPU hardware, EFA, and NVLink on a PCS GPU node group using the repository's [GPU Cluster Health Check Suite](../../../4.validation_and_observability/2.gpu-cluster-healthcheck). diff --git a/architectures/aws-pcs/tests/iam-test.md b/architectures/aws-pcs/tests/iam-test.md index c3d4fcc9b..2b5a0691a 100644 --- a/architectures/aws-pcs/tests/iam-test.md +++ b/architectures/aws-pcs/tests/iam-test.md @@ -1,4 +1,4 @@ -# IAM Test: cluster-admin / cluster-user policies +# Test 14: IAM Policies (cluster-admin / cluster-user) Validates the two IAM policy stacks ([`cluster-admin-iam.yaml`](../assets/cluster-admin-iam.yaml) and [`cluster-user-iam.yaml`](../assets/cluster-user-iam.yaml), documented in diff --git a/architectures/aws-pcs/tests/multi-user-test.md b/architectures/aws-pcs/tests/multi-user-test.md index 5af01beef..3e5d1302c 100644 --- a/architectures/aws-pcs/tests/multi-user-test.md +++ b/architectures/aws-pcs/tests/multi-user-test.md @@ -1,4 +1,4 @@ -# Test: Multi-User Directory (OpenLDAP) +# Tests 11-12: Multi-User Directory (OpenLDAP) + Slurm Accounting Run this test when `DirectoryService=OpenLDAP-LoginNode` is enabled. Validates that the OpenLDAP directory on the login node is functional, that LDAP users are From d4da5f97e415a865b6a651ca960cb725915bb2a3 Mon Sep 17 00:00:00 2001 From: Daisuke Miyamoto Date: Wed, 17 Jun 2026 18:15:57 +0000 Subject: [PATCH 89/89] docs(readme): drop the Capacity Block billing note (cost is out of scope for this doc) --- architectures/aws-pcs/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/architectures/aws-pcs/README.md b/architectures/aws-pcs/README.md index 86839a092..6c1700c59 100644 --- a/architectures/aws-pcs/README.md +++ b/architectures/aws-pcs/README.md @@ -121,7 +121,7 @@ complete reference see [PARAMETERS.md](./docs/PARAMETERS.md). |---|---|---| | `DeployPseriesCNG` | `false` | Deploy a multi-NIC GPU (P5/P6) queue | | `PseriesInstanceType` | `p5.48xlarge` | Picks the matching template + EFA NIC count automatically. See [GPU compute](#gpu-compute-p5p6) for the accepted types | -| `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR. ⚠️ A Capacity Block bills for its whole reserved window — see [GPU compute](#gpu-compute-p5p6) | +| `CapacityReservationId` | *(empty)* | Capacity **Block** ID for the GPU queue; empty for On-Demand/ODCR | **5. Additional Cluster Configuration (Monitoring, Multi-User, Container Runtime)**