Skip to content

feat(aws-pcs): cluster-scoped Name tag + PCS-API login lookup in docs #1183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DaisukeMiyamoto merged 6 commits into from
Jul 14, 2026
Merged

feat(aws-pcs): cluster-scoped Name tag + PCS-API login lookup in docs #1183

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e9323c2
feat(aws-pcs): cluster-scoped Name tag + PCS-API login lookup + READM…
DaisukeMiyamoto Jul 11, 2026
b9ee204
fix(aws-pcs): CngName tag holds the bare CNG name, not <cluster>-<cng>
DaisukeMiyamoto Jul 12, 2026
9ecd426
fix(aws-pcs): split login-lookup snippet + guard empty ClusterId + dr…
DaisukeMiyamoto Jul 12, 2026
4b52e3e
fix(aws-pcs): scope cluster-user IAM to a single deploy-all stack
DaisukeMiyamoto Jul 12, 2026
43a54fc
Revert JUPYTER.md scope from 9ecd426d — out of scope, untested
DaisukeMiyamoto Jul 12, 2026
e50b72c
Merge branch 'main' into feat/cluster-scoped-name-and-lookup
DaisukeMiyamoto Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 47 additions & 18 deletions architectures/aws-pcs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,16 +287,24 @@ want direct SSH, set `SSHAccessCidr` to a trusted CIDR at deploy time to open SS
on the login node to that CIDR.

**Console:** [EC2 Console](https://console.aws.amazon.com/ec2/home#Instances:) → filter
by `aws:pcs:compute-node-group-name = login` → **Connect** → **Session Manager**.
by `Name` = `<your-stack-name>-login` → **Connect** → **Session Manager**.

**CLI** (AWS CloudShell has the required permissions):
**CLI** (AWS CloudShell has the required permissions). Set `STACK_NAME` /
`AWS_REGION` once, then a single command resolves the login node's
instance ID — it works no matter what your CNG is named because the

@KeitaW KeitaW Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

§6 rationale is slightly self-contradictory — the resilience is against the Name tag, not the CNG name

Small wording thing: the §6 intro says the lookup "works no matter what your CNG is named because the lookup goes through the PCS API." But the lookup hardcodes computeNodeGroups[?name==\login`], so it *does* depend on the login CNG being named login(which deploy-all guarantees). What the PCS-API path actually buys you is independence from the mutable EC2Name` tag — that's the real win worth stating.

Suggested change
instance ID — it works no matter what your CNG is named because the
instance ID — it works no matter how instances are Name-tagged because the

lookup goes through the PCS API, not the human-facing tag:

```bash
INSTANCE_ID=$(aws ec2 describe-instances \
--filters "Name=tag:aws:pcs:compute-node-group-name,Values=login" \
"Name=instance-state-name,Values=running" \
--query 'Reservations[0].Instances[0].InstanceId' --output text)
aws ssm start-session --target $INSTANCE_ID
STACK_NAME=pcs-ml-cluster # your CloudFormation stack name
AWS_REGION=us-east-1 # your region

CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region "$AWS_REGION" --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text)
[ -n "$CLUSTER_ID" ] && [ "$CLUSTER_ID" != "None" ] || { echo "No ClusterId — check STACK_NAME/AWS_REGION"; return 1; }

LOGIN_CNG_ID=$(aws pcs list-compute-node-groups --cluster-identifier "$CLUSTER_ID" --region "$AWS_REGION" --query 'computeNodeGroups[?name==`login`].id' --output text)
LOGIN_INSTANCE_ID=$(aws ec2 describe-instances --region "$AWS_REGION" --filters "Name=tag:aws:pcs:compute-node-group-id,Values=$LOGIN_CNG_ID" "Name=instance-state-name,Values=running" --query 'Reservations[0].Instances[0].InstanceId' --output text)

@KeitaW KeitaW Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The split login-lookup guards only the first of three hops — a renamed login CNG still lands on --target None

The split + CLUSTER_ID guard is a solid fix for the round-1 silent-None. The residual: the guard covers CLUSTER_ID, but LOGIN_CNG_ID and LOGIN_INSTANCE_ID are unguarded. If list-compute-node-groups finds no CNG named login (renamed, or not yet running), LOGIN_CNG_ID comes back empty → the describe-instances filter becomes Values= → LOGIN_INSTANCE_ID=None → aws ssm start-session --target None fails with an opaque SSM error — the round-1 class, one hop down. A single guard right before the session covers all three hops:

[ -n "$LOGIN_INSTANCE_ID" ] && [ "$LOGIN_INSTANCE_ID" != "None" ] || { echo "No running login node for cluster $CLUSTER_ID"; return 1; }

Two asides: the existing CLUSTER_ID guard uses return 1, which only halts inside a function or a sourced script — pasted at an interactive prompt (which the README instructs) return prints an error and execution continues anyway, so the guard's "stop here" intent isn't enforced interactively. And credit due — the new Test 15 gate (echo $LOGIN_INSTANCE_ID must print an i-…) already catches this before a customer would, so this is polish, not a hole. docs/IAM.md's equivalent block has the same shape and would take the same one-line guard.


aws ssm start-session --target "$LOGIN_INSTANCE_ID" --region "$AWS_REGION"
```

Then `sudo su - ubuntu` and use `sinfo` / `squeue` / `scontrol show nodes`.
Expand Down Expand Up @@ -435,28 +443,41 @@ DCGM exporter image (`DcgmExporterImage`) are pinned and rarely need changing
#### Accessing the Grafana dashboards

Log in to Grafana as **`admin`**; the password is generated per cluster and stored in
SSM Parameter Store. Retrieve it (with `CLUSTER_ID` from the stack's `ClusterId` output):
SSM Parameter Store. Retrieve it — the same `STACK_NAME` / `AWS_REGION` used in §6
resolve the cluster's `ClusterId` from the stack Outputs:

```bash
aws ssm get-parameter --name "/pcs/${CLUSTER_ID}/grafana/admin-password" \
--with-decryption --query 'Parameter.Value' --output text
STACK_NAME=pcs-ml-cluster # your CloudFormation stack name
AWS_REGION=us-east-1 # your region

aws ssm get-parameter --region "$AWS_REGION" --with-decryption --query 'Parameter.Value' --output text \
--name "/pcs/$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region "$AWS_REGION" --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text)/grafana/admin-password"
```

There are two ways to reach the UI.

##### Option A — SSM port forwarding (default, private)

No public access required; works even when the login node has no inbound rules.
The monitoring stack runs on the instance tagged `monitoring-role=login`;
scope by the cluster's `aws:pcs:cluster-id` so multi-cluster VPCs don't
cross the streams.

```bash
# Login node instance ID
INSTANCE_ID=$(aws ec2 describe-instances \
--filters "Name=tag:aws:pcs:compute-node-group-name,Values=login" \
STACK_NAME=pcs-ml-cluster # your CloudFormation stack name
AWS_REGION=us-east-1 # your region

CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region "$AWS_REGION" --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text)
[ -n "$CLUSTER_ID" ] && [ "$CLUSTER_ID" != "None" ] || { echo "No ClusterId — check STACK_NAME/AWS_REGION"; return 1; }

LOGIN_INSTANCE_ID=$(aws ec2 describe-instances --region "$AWS_REGION" \
--filters "Name=tag:aws:pcs:cluster-id,Values=$CLUSTER_ID" \
"Name=tag:monitoring-role,Values=login" \
"Name=instance-state-name,Values=running" \
--query 'Reservations[0].Instances[0].InstanceId' --output text)

# Port-forward remote 443 -> local 8443 (needs the Session Manager plugin)
aws ssm start-session --target $INSTANCE_ID \
aws ssm start-session --target "$LOGIN_INSTANCE_ID" --region "$AWS_REGION" \
--document-name AWS-StartPortForwardingSession \
--parameters '{"portNumber":["443"],"localPortNumber":["8443"]}'
```
Expand All @@ -474,11 +495,19 @@ attaches it to the login node, so you can open:
https://<login-node-public-ip>/grafana/
```

Get the login node's public IP from the EC2 console, or:
Get the login node's public IP from the EC2 console, or (again scoping by
`monitoring-role=login` + `aws:pcs:cluster-id` for the monitoring instance):

```bash
aws ec2 describe-instances \
--filters "Name=tag:aws:pcs:compute-node-group-name,Values=login" \
STACK_NAME=pcs-ml-cluster # your CloudFormation stack name
AWS_REGION=us-east-1 # your region

CLUSTER_ID=$(aws cloudformation describe-stacks --stack-name "$STACK_NAME" --region "$AWS_REGION" --query 'Stacks[0].Outputs[?OutputKey==`ClusterId`].OutputValue' --output text)
[ -n "$CLUSTER_ID" ] && [ "$CLUSTER_ID" != "None" ] || { echo "No ClusterId — check STACK_NAME/AWS_REGION"; return 1; }

aws ec2 describe-instances --region "$AWS_REGION" \
--filters "Name=tag:aws:pcs:cluster-id,Values=$CLUSTER_ID" \
"Name=tag:monitoring-role,Values=login" \
"Name=instance-state-name,Values=running" \
--query 'Reservations[0].Instances[0].PublicIpAddress' --output text
```
Expand Down Expand Up @@ -528,7 +557,7 @@ NCCL, FSDP), see the [Test & Validation Guide](tests/README.md).

> **Note — node-type tagging.** The monitoring stack identifies login vs compute nodes by
> the `monitoring-role` tag (`login`/`compute`), **not** the EC2 `Name` tag — so the `Name`
> tag (default `PCS-<cngname>`) is free for you to retag without breaking dashboards.
> tag (default `<ClusterName>-<cngname>`) is free for you to retag without breaking dashboards.

> **Note — monitoring across login-node replacement.** Prometheus + Grafana run on the
> (single) login node. Metric collection, the Grafana password, and the built-in
Expand Down
14 changes: 7 additions & 7 deletions architectures/aws-pcs/assets/add-cng-p5.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -345,15 +345,15 @@ Resources:
Tags:
- Key: HPCRecipes
Value: "true"
# Name is free for arbitrary operator use. The monitoring stack
# identifies login vs compute nodes via the monitoring-role tag
# (below), not Name, so dashboards work regardless of this value.
# Default to PCS-<cngname> (mirrors CngName) for a recognizable
# console name; retag freely afterward.
# Name defaults to <ClusterName>-<CngName> so the EC2 console
# can tell instances of different clusters apart at a glance;
# retag freely, nothing operational reads it (monitoring keys
# off monitoring-role below, docs look up instances via the
# PCS API + aws:pcs:* tags).
- Key: Name
Value: !Sub 'PCS-${CngName}'
Value: !Sub '${ClusterName}-${CngName}'
- Key: CngName
Value: !Sub 'PCS-${CngName}'
Value: !Ref CngName
- Key: ClusterName
Value: !Ref ClusterName
- Key: pcs-cluster-id
Expand Down
14 changes: 7 additions & 7 deletions architectures/aws-pcs/assets/add-cng-p6-b200.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -332,15 +332,15 @@ Resources:
Tags:
- Key: HPCRecipes
Value: "true"
# Name is free for arbitrary operator use. The monitoring stack
# identifies login vs compute nodes via the monitoring-role tag
# (below), not Name, so dashboards work regardless of this value.
# Default to PCS-<cngname> (mirrors CngName) for a recognizable
# console name; retag freely afterward.
# Name defaults to <ClusterName>-<CngName> so the EC2 console
# can tell instances of different clusters apart at a glance;
# retag freely, nothing operational reads it (monitoring keys
# off monitoring-role below, docs look up instances via the
# PCS API + aws:pcs:* tags).
- Key: Name
Value: !Sub 'PCS-${CngName}'
Value: !Sub '${ClusterName}-${CngName}'
- Key: CngName
Value: !Sub 'PCS-${CngName}'
Value: !Ref CngName
- Key: ClusterName
Value: !Ref ClusterName
- Key: pcs-cluster-id
Expand Down
14 changes: 7 additions & 7 deletions architectures/aws-pcs/assets/add-cng-p6-b300.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -335,15 +335,15 @@ Resources:
Tags:
- Key: HPCRecipes
Value: "true"
# Name is free for arbitrary operator use. The monitoring stack
# identifies login vs compute nodes via the monitoring-role tag
# (below), not Name, so dashboards work regardless of this value.
# Default to PCS-<cngname> (mirrors CngName) for a recognizable
# console name; retag freely afterward.
# Name defaults to <ClusterName>-<CngName> so the EC2 console
# can tell instances of different clusters apart at a glance;
# retag freely, nothing operational reads it (monitoring keys
# off monitoring-role below, docs look up instances via the
# PCS API + aws:pcs:* tags).
- Key: Name
Value: !Sub 'PCS-${CngName}'
Value: !Sub '${ClusterName}-${CngName}'
- Key: CngName
Value: !Sub 'PCS-${CngName}'
Value: !Ref CngName
- Key: ClusterName
Value: !Ref ClusterName
- Key: pcs-cluster-id
Expand Down
14 changes: 7 additions & 7 deletions architectures/aws-pcs/assets/add-cng.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -356,15 +356,15 @@ Resources:
Tags:
- Key: HPCRecipes
Value: "true"
# Name is free for arbitrary operator use. The monitoring stack
# identifies login vs compute nodes via the monitoring-role tag
# (below), not Name, so dashboards work regardless of this value.
# Default to PCS-<cngname> (mirrors CngName) for a recognizable
# console name; retag freely afterward.
# Name defaults to <ClusterName>-<CngName> so the EC2 console
# can tell instances of different clusters apart at a glance;
# retag freely, nothing operational reads it (monitoring keys
# off monitoring-role below, docs look up instances via the
# PCS API + aws:pcs:* tags).
- Key: Name
Value: !Sub 'PCS-${CngName}'
Value: !Sub '${ClusterName}-${CngName}'
- Key: CngName
Value: !Sub 'PCS-${CngName}'
Value: !Ref CngName
- Key: ClusterName
Value: !Ref ClusterName
- Key: pcs-cluster-id
Expand Down
Loading