fix(aws-pcs): scope cluster-user ssm:StartSession documents separately from the login-node tag - #4
Closed
KeitaW wants to merge 1 commit into
Conversation
…y from the login-node tag The PCSClusterUser policy gated the SSM session-document ARNs (AWS-StartSSHSession, AWS-StartPortForwardingSession*, SSM-SessionManagerRunShell) under the same ssm:resourceTag/Name=PCS-login* condition as the instance ARN. Documents carry no Name tag, so StringLike fails closed on the document sub-authorization and ssm:StartSession with a custom document is denied — breaking SSH-over-SSM and port-forwarding for cluster users (a plain Session Manager shell still works, so a smoke test passes). Split into a tag-gated instance statement (SSMSessionToLoginNodeInstance) + an untagged document statement (SSMSessionDocuments), matching AWS's Session Manager tag-restriction pattern. Login/compute isolation is unchanged — the instance statement still scopes to PCS-login*. Verified with iam simulate-custom-policy (before -> after): SSH document: implicitDeny -> allowed login instance: allowed -> allowed compute instance: deny -> deny Claude-Session: https://claude.ai/code/session_014tqpywkVSmnUMJGNYfogmF
DaisukeMiyamoto
added a commit
that referenced
this pull request
Aug 24, 2026
…slabs#1236 #4) Both feed the raw.githubusercontent.com URL that install-monitoring.sh fetches and bash-runs. These are admin-only params, so the AllowedPattern is input hygiene (catch typos at CFN validation, match the AmiId/FSx-id convention) rather than an anti-exploit measure. MonitoringRepo: ^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$ (exactly owner/repo). MonitoringVersion: ^[A-Za-z0-9._/-]+$ -- deliberately allows '/' so the documented fork+branch testing workflow (feat/... branches) keeps working; rejects whitespace and shell metacharacters. Applied across all 4 add-cng templates.
DaisukeMiyamoto
added a commit
that referenced
this pull request
Aug 28, 2026
… lifecycle actions (awslabs#1236) * feat(aws-pcs): extract first-boot logic to scripts, harden all boot fetches Phase 1 of the node-lifecycle-actions migration: move the needrestart guard and the FSx OpenZFS//home + Lustre//fsx mounts out of inline cloud-init runcmd into standalone bash scripts under assets/scripts/, fetched from the templates bucket like the existing post-install and directory scripts. All five boot-script fetches now go through a pcs-fetch helper: - 3 attempts 15s apart, every attempt logged to /var/log/pcs-boot-fetch.log, grep-able 'ERROR: failed to fetch' on final failure (a silent directory fetch failure on a replacement login node cost a support round-trip) - pins the AWS CLI to the classic transfer client via a scoped AWS_CONFIG_FILE: on P5/P6 instance types 'auto' resolves to the CRT client, which does not follow S3 region redirects, so cross-region fetches fail (the root cause of post-install exit 127 on GPU CNGs) - re-probes the bucket region each attempt; an empty probe falls back to the classic client's redirect-following instead of failing HTTPS fetches (post-install http(s), monitoring GitHub raw) get curl --retry 3 plus the same ERROR logging. Verified e2e on us-east-2 deploy-all (login + compute): all fetches logged, mounts up, Enroot/Pyxis exit 0, OpenLDAP server+client working, ldap-add-user resolves on both nodes, srun + Pyxis container jobs pass. Failure path verified: nonexistent key exits 1 after 3 logged attempts. * feat(aws-pcs): replace CNG UserData with PCS node lifecycle actions Move ALL first-boot logic from cloud-init UserData to NodeLifecycleActions on the ComputeNodeGroup resource (requires PCS agent >= 1.5.0-1, in PCS-Ready DLAMI builds since 2026-07-20 — see docs/PCS-READY-DLAMI.md). The four CNG templates no longer carry a UserData block at all. nodeBootstrapped (in order, before slurmd starts): 1. needrestart-guard FIRST_BOOT_ONLY / CONTINUE 2. mount-openzfs-home EVERY_BOOT / TERMINATE 3. mount-lustre-fsx EVERY_BOOT / TERMINATE (only when Lustre is set) 4. setup-directory FIRST_BOOT_ONLY / CONTINUE (only when enabled) 5. post-install FIRST_BOOT_ONLY / CONTINUE (only when set) nodeReady: 6. install-monitoring FIRST_BOOT_ONLY / CONTINUE (only when enabled) Why all-at-once instead of piecemeal: nodeBootstrapped runs only after cloud-init user-data completes (pcs_bootstrap_finalize), so any UserData step that depends on a lifecycle-mounted filesystem deadlocks — verified on a real deploy where directory setup waited 10 minutes for a /home mount that could not happen until cloud-init exited. Dependencies must live in one execution domain; within a stage, ordering is guaranteed. Script interface changes (lifecycle actions pass positional args, not env): - setup-directory.sh: role/domain-suffix/cluster-id/bucket/prefix as args 1-5 (env interface kept); generates the admin password itself (SSM reuse logic unchanged); hard-fails unless /home is a mountpoint. - install-enroot-pyxis.sh: accepts the Slurm version as arg 1 (PCS_SLURM_VERSION env kept for the custom-AMI build path). - install-monitoring.sh (new): monitoring installer wrapper — GitHub fetch with curl retry, apt dpkg-lock drop-in, 3-attempt install loop. The agent replaces the hand-rolled fetch plumbing: per-script retries and logs (/var/log/amazon/pcs/lifecycle/actions/<stage>/<name>.log), and its downloader is unaffected by the AWS CLI CRT region-redirect bug that motivated pcs-fetch. Lifecycle config changes via UpdateComputeNodeGroup now reach existing CNGs with DRAIN semantics instead of requiring CNG recreation (the LaunchTemplate Version pin problem). lint-docs.sh: the four-template lock-step check now covers the whole NodeLifecycleActions block, and every referenced lifecycle script must exist in assets/scripts/. Verified e2e on us-east-2 deploy-all (login + compute, directory + monitoring + Enroot/Pyxis enabled, cross-region templates bucket): all six scripts exit 0 in order, /home //fsx mounted, slapd + SSSD up, ldap-add-user resolves on both nodes, Grafana/Prometheus containers running, srun + Pyxis container jobs pass. * docs+fix(aws-pcs): align docs with lifecycle actions; keep space-skip working Template fixes surfaced by the user-impact audit: - PostInstallScriptUrl skip sentinel: deploy-all passes a single space through to add-cng, where the lifecycle ScriptLocation pattern would reject it and fail CNG creation. HasPostInstall/HasPostInstallArgs now treat empty AND single-space as 'no post-install' (verified: a CNG with the skip value creates cleanly with no post-install entry). - AmiId descriptions (4x add-cng + deploy-all + PARAMETERS.md) now state the PCS agent >= 1.5.0-1 floor for pinned/custom AMIs. - PostInstallScriptUrl/Args descriptions document the new contract: https:// only (no plain http), SlurmVersion as first argument, PostInstallScriptArgs as a single unsplit argument. Docs updated for the new mechanics: log paths moved to /var/log/amazon/pcs/lifecycle/actions/<stage>/<name>.log (README diagram, OPERATIONS 2.3/4.1/4.2/4.4/6, PARAMETERS, DEPLOY-TESTING, USER-MANAGEMENT 5.2, tests/infra + storage + readme-walkthrough), mount failures now TERMINATE and replace the node (storage-test troubleshooting), Lustre tuning hooks reference lifecycle actions instead of UserData. Pre-PR e2e on a fresh us-east-2 deploy-all (directory + monitoring + Enroot/Pyxis): all six lifecycle scripts exit 0 on login + compute, no UserData leftovers on nodes, docs commands run as written (executor grep, sinfo, docker ps, enroot/pyxis paths, Prometheus targets up, Grafana 200), srun + Pyxis container job pass, ldap-add-user resolves on both nodes, skip-sentinel CNG attach validated. * feat(aws-pcs)!: replace PostInstallScriptUrl/Args with InstallEnrootPyxis The generic post-install hook existed because PCS had no native way to run a custom script at first boot. Node lifecycle actions ARE that native way now, and the only thing this repo ever shipped through the hook is the Enroot/Pyxis installer — so name the feature for what it does: - InstallEnrootPyxis ('true'/'false') replaces PostInstallScriptUrl + PostInstallScriptArgs on deploy-all and all four add-cng templates. The lifecycle entry is named install-enroot-pyxis (its log follows: .../nodeBootstrapped/install-enroot-pyxis.log). - The script location is fixed to s3://<S3BucketName>/<S3KeyPrefix>scripts/install-enroot-pyxis.sh — dev overrides ride the existing S3BucketName/S3KeyPrefix redirection, which removes the last reason pre-merge deploys had to set a URL. - Defaults preserve prior behavior at both layers: deploy-all 'true' (was: empty auto-installs), modular add-cng 'false' (was: empty skips). Users with custom post-install scripts add them to the compute node group's NodeLifecycleActions directly (documented in the param descriptions, README and PARAMETERS.md). - lint-docs: PostInstallScriptUrl/Args are now BANNED names in docs; the empty-vs-space skip-wording check is gone with the sentinel. BREAKING CHANGE: PostInstallScriptUrl / PostInstallScriptArgs no longer exist. Set InstallEnrootPyxis=false instead of the single-space sentinel; attach custom first-boot scripts as node lifecycle actions. * chore(aws-pcs): publish new lifecycle scripts; lint manifest coverage The publish manifest is an explicit allowlist — without these entries the lifecycle-migration templates would reach the production bucket while their scripts did not, and every post-merge deploy would fail at the agent's script download (the mount scripts TERMINATE, so nodes would enter a replace loop). Add the four new scripts: needrestart-guard.sh, mount-openzfs-home.sh, mount-lustre-fsx.sh, install-monitoring.sh. lint-docs.sh now cross-checks every template-referenced lifecycle script against the manifest so this class of gap fails the publish workflow's lint step instead of production deploys (verified: removing an entry makes the lint FAIL). * docs(aws-pcs): migration notes for the lifecycle-actions changes; final sweep Add OPERATIONS.md §8 documenting what changes for users coming from the UserData-based templates (InstallEnrootPyxis replaces PostInstallScriptUrl/ Args, agent >= 1.5.0-1 floor, new log locations, mount failures now replace the node, installer argument contract) — the repo had no in-tree record of these behavior changes. Sweep of README/docs/tests for remaining UserData-era wording: README architecture diagram now lists all six lifecycle scripts, ROADMAP's FSx-EFA item targets a lifecycle-action script, OPERATIONS 4.2/6.1 and infra/storage-test phrasing updated. * fix(aws-pcs): harden IMDS region lookup in the TERMINATE mount scripts Under set -euo pipefail, REGION=$(curl ...) took curl's exit status, so a connection failure killed the script on the assignment line and skipped the ${REGION:?} diagnostic one line below — on an OnError:TERMINATE action the instance is then replaced before anyone can read the (truncated, error-less) log. And curl -s (no -f) exits 0 on a 4xx/5xx and captures the error body, so under HttpTokens=required a throttled token PUT or a 401 GET landed an HTML/401 body in REGION and passed the guard, producing a bogus FSx DNS name and a mount failure -> terminate. Add -f (4xx/5xx -> empty), --retry/--connect-timeout/--max-time (ride out a throttled/slow IMDS), and || true (reach the :? diagnostic instead of dying on the assignment). Both mount scripts, all three call sites. Verified: a connection failure now exits 1 with the message instead of a silent exit 7. Addresses review batch 1/6 (IMDS guard). * Retry mount in TERMINATE lifecycle scripts (PR awslabs#1236 #2) mount-openzfs-home.sh and mount-lustre-fsx.sh run as OnError:TERMINATE lifecycle actions with a single-shot mount. The common first-boot NFS DNS settle race (mount.nfs: Failed to resolve server) fails instantly, which under TERMINATE replaces the node into the same window. Wrap the mount in a bounded retry loop (6 attempts, 10s backoff) verified with mountpoint(8), so TERMINATE fires only on a persistent failure. * Constrain FSx filesystem ID params with AllowedPattern (PR awslabs#1236 #3) FSxOpenZFSFilesystemId feeds the unconditional mount-openzfs-home action (OnError:TERMINATE) with no AllowedPattern/Default. An empty or malformed value passes stack create, then the mount script fails at boot and the node is TERMINATEd and replaced into the same bad value -- an endless terminate/replace loop. Add a mandatory '^fs-[0-9a-f]{8,17}$' pattern so CFN rejects it at create time. Add the sibling '^$|^fs-...$' pattern to FSxLustreFilesystemId, whose empty value is the documented 'no /fsx' opt-out (HasLustre). Applied identically across all 4 add-cng templates. * Constrain MonitoringRepo/MonitoringVersion with AllowedPattern (PR awslabs#1236 #4) Both feed the raw.githubusercontent.com URL that install-monitoring.sh fetches and bash-runs. These are admin-only params, so the AllowedPattern is input hygiene (catch typos at CFN validation, match the AmiId/FSx-id convention) rather than an anti-exploit measure. MonitoringRepo: ^[A-Za-z0-9._-]+/[A-Za-z0-9._-]+$ (exactly owner/repo). MonitoringVersion: ^[A-Za-z0-9._/-]+$ -- deliberately allows '/' so the documented fork+branch testing workflow (feat/... branches) keeps working; rejects whitespace and shell metacharacters. Applied across all 4 add-cng templates. * Set only the slurmd needrestart override, not the whole hash (PR awslabs#1236 awslabs#5) $nrconf{override_rc} = { qr(^slurmd) => 0 } reassigns the entire override_rc hash, discarding needrestart's shipped defaults and leaving slurmd as the only entry. Mutate the single key instead: $nrconf{override_rc}{qr(^slurmd)} = 0; so the shipped defaults are preserved and only slurmd is added. Pre-existing since awslabs#1165 (ac66712); this PR carries the one-line fix in the extracted script rather than deferring it to a separate PR. * Update OPERATIONS §6.2 needrestart snippet to match the awslabs#5 fix The doc code sample still showed the whole-hash reassignment; align it with the one-key form now written by needrestart-guard.sh. * Document the stack-update upgrade path in OPERATIONS §8 (PR awslabs#1236 M1) Two gaps the review flagged: - §8 said existing stacks keep working as-is but never said the update itself DRAINs and replaces the whole fleet (launch-template version bump + NodeLifecycleActions change). Added a fleet-cycle note. - A custom-bucket stack has none of the four scripts this revision adds; updating it fails mount-openzfs-home (TERMINATE) on every replacement = replace loop to an empty cluster. Added a sync-before-update prerequisite linking DEPLOY-TESTING §2. Also: fixed the mount-row debug advice (a fleet-wide failure leaves no surviving node; point to OnError:STOP_SEQUENCE / configure-cloudwatch-logs.sh), noted the load-bearing S3 read in the PARAMETERS S3BucketName/S3KeyPrefix rows, and framed the DEPLOY-TESTING sync as an update prerequisite. * Remove the unused S3BucketName=local template sentinel (PR awslabs#1236 M2) UseLocalTemplates (S3BucketName='local') switched nested-stack TemplateURLs to relative paths for an aws cloudformation package local-dev flow. It is undocumented, has no reference in docs/tests, and dates to the first reference-cluster commit -- never exercised by this project's workflows, which sync to a real bucket. It is also now a trap: the sentinel is forwarded verbatim to the CNG child stacks as their script bucket, so boot scripts resolve to a literal s3://local/... and fail; with the mounts now load-bearing (OnError:TERMINATE) that terminates the node instead of degrading gracefully. cfn package rewrites nested TemplateURLs but not the runtime script fetch, so 'local' cannot supply boot scripts by design. Delete the condition and collapse all 7 TemplateURLs to the S3 URL form. * Verify LDAP admin bind before writing SSM in setup-directory (PR awslabs#1236 M3) debconf-set-selections seeds slapd's olcRootPW only when apt-get actually installs slapd. If slapd is already present, the install is a no-op and slapd keeps its previous admin password, but the script still overwrote SSM with the newly configured password and swallowed the failing OU-creation binds (2>/dev/null || true) -- a false success leaving a stored credential that does not bind. Add an ldapwhoami check after slapd is up: on bind failure, log loudly and return before creating OUs or overwriting SSM, so a working stored credential is never clobbered by a non-binding one. Directory action is OnError:CONTINUE, so this logs and the node continues. * fix(pcs): match whole fstab line and drop stray mount arg in mount-openzfs-home grep -qF is a substring match: a commented-out /home fstab line (added by an operator debugging a hung mount) satisfies the guard, so the active entry is never re-added. mount -a then no-ops, the script reports success, and the stash is restored onto — then deleted from — a local /home that the next boot silently shadows. Use grep -qxF (whole-line match) on both guards so a commented line no longer counts as present. Also drop the stray 'defaults' positional from 'mount -a -t nfs defaults' (introduced with the mount retry loop); mount -a takes no such argument. * fix(pcs): don't let a failed /fsx chmod terminate a healthy node mount-lustre-fsx.sh writes no fstab entry, so it re-mounts and re-chmods on every boot (EVERY_BOOT). Under set -e a failing 'chmod 1777 /fsx' exits non-zero even when the mount at the previous line succeeded, so OnError: TERMINATE destroys a node whose /fsx is fine. A chmod failure here is a shared-side condition (root_squash / read-only FSx / MDS hiccup), never node-local — terminating just replaces the node into the same failure (launch/drain/replace loop). Decouple the chmod from the exit status and warn loudly instead, keeping the healthy node. Leaves the every-boot chmod itself in place (a safe first-boot-only guard on shared Lustre is non-trivial given .lustre/lost+found); tracked in review reply. * docs(pcs): drop removed empty/single-space InstallEnrootPyxis semantics InstallEnrootPyxis is now AllowedValues ['true','false']; empty and single-space are no longer legal. Rewrote the three passages that still described them (OPERATIONS.md 2.1, CUSTOM-AMI.md, PARAMETERS.md 5.3 preamble), naming the per-template default (deploy-all 'true', add-cng* 'false'). Renamed the PARAMETERS.md 5.3 heading and the README console label to the actual console group 'Container Runtime (Enroot/Pyxis)', restoring the mirror-the-console invariant. The OPERATIONS.md 8 migration row keeps its single-space mention (correct: it explains the old behavior when migrating off PostInstallScript*). * feat(pcs): default InstallEnrootPyxis=true in the standalone add-cng templates The four add-cng*.yaml defaulted InstallEnrootPyxis to 'false' while deploy-all defaults 'true', so the documented one-click path for adding a queue to an existing cluster (README Launch Stack buttons) produced nodes with no container runtime — srun --container-image would fail on the new queue only, while the README says the runtime is on by default. Rather than paper over the split with per-template scoping in the docs, make the default consistent: the container runtime is a headline behavior of this ML reference architecture, and the installer is idempotent (a fast no-op when pre-baked into AmiId). Deploy-all already passes the value down explicitly, so this only changes the standalone add-cng* path. Set 'false' to opt out. Simplifies the docs that had to name the per-template default (OPERATIONS 2.1, CUSTOM-AMI); README's unscoped 'on by default' is now accurate everywhere. All four templates validate; docs lint passes. * docs(pcs): update in-file comments crediting UserData for the moved interface Five comments still described the pre-PR UserData env interface, one made false by this PR's own change (setup-directory.sh:30 said LDAP_ADMIN_PASSWORD is 'auto-generated by UserData' though the script now generates it itself). Swept all five to describe the current interface: the lifecycle action passes values positionally (env vars remain for manual / custom-AMI runs), and SlurmVersion arrives as $1 with PCS_SLURM_VERSION as the fallback. * test(pcs): widen manifest cross-check to scripts fetched by other scripts The manifest guard grepped assets/add-cng*.yaml only, so ldap-add-user.sh — fetched at boot by setup-directory.sh, not via a template ScriptLocation — was invisible to it. Dropping its manifest entry left both lint-docs.sh and the staging script green while the object never reached the production bucket, so the login node's aws s3 cp fails and the cluster comes up without the ldap-add-user.sh helper USER-MANAGEMENT.md documents (silent degrade). Scan assets/*.yaml and assets/scripts/*.sh so boot-time fetches from other scripts are covered too. Verified: normal run still PASS (7/7 present and in the manifest, no false positives); dropping the ldap-add-user.sh entry now FAILs as intended. * fix(pcs): don't silently rotate the LDAP admin password on an SSM read error setup-directory.sh read the stored admin password with 'aws ssm get-parameter ... 2>/dev/null || echo ""', collapsing every failure (ParameterNotFound, AccessDenied, throttling) into "nothing stored yet". A transient read failure during a login-node replacement therefore regenerated the password, reconfigured slapd, and overwrote SSM with --overwrite while the user DB on /home/ldap-db persisted — the exact silent rotation the reuse block exists to prevent. Capture the exit status (set -e-safe if/else) and branch: reuse a returned value; keep the freshly generated password only on rc==0-empty or a genuine ParameterNotFound; on any other error, log loudly and return 1 rather than regenerate against the persistent DB. The directory action is OnError:CONTINUE, so the node continues (log-and-continue), consistent with the ldapwhoami bind check added earlier for the debconf-only-on-fresh-install case. * docs(pcs): drop non-working 'latest' from MonitoringVersion guidance raw.githubusercontent.com resolves only real tag/branch refs, so a MonitoringVersion of 'latest' 404s the post-install.sh fetch and (under OnError:CONTINUE) leaves monitoring silently uninstalled. Remove 'latest' from the parameter Description / ConstraintDescription across all five templates and from PARAMETERS.md; the default stays a real tag (v2.10.2). * fix(pcs): harden mount + monitoring boot scripts, fix doc anchors mount-openzfs-home.sh: keep exactly one active /home fstab entry via ensure_home_fstab_entry (mountpoint-field match, rewrite in place), tolerate rsync exit 23/24 via safe_rsync so a benign ACL/vanished-file code can't trip set -e into TERMINATE, and mount /home specifically instead of mount -a so an unrelated custom-AMI NFS entry can't fail the action. install-monitoring.sh: fetch post-install.sh into a mktemp -d workdir (no fixed /tmp path a dispatched job could race), reconcile the role header (login=server, compute=exporters), note curl --retry doesn't retry HTTP 4xx, scope the apt lock-timeout drop-in comment, and chmod 600 the upstream install log to close the Grafana-password exposure on the multi-user login node. docs: fix the 3.1 cross-file anchor double-hyphen (README, PARAMETERS) and note that under CACHE_ONCE a re-sync needs an instance replacement to reach a running node (DEPLOY-TESTING). * fix(pcs): preserve existing /home dir modes on first-boot restore The first-boot /home restore rsynced the node-local snapshot back over the freshly mounted shared export with `-aA --ignore-existing`. For a directory that already exists on the shared side (e.g. an admin-tightened /home/ubuntu), rsync still applied the snapshot's mode/times to it, silently loosening the shared directory's permissions on the next new node's first boot. Add `--no-perms --omit-dir-times` so the restore only fills in missing files and never rewrites the mode/times of a directory that already exists on the export. `--ignore-existing` still protects existing shared files. (PR awslabs#1236 review #3) Also: - docs(OPERATIONS §8): note the standalone add-cng*.yaml default flip to InstallEnrootPyxis=true (deploy-all already defaulted true) so migrating stacks that relied on the old `false` default aren't surprised. - docs(README): fix the Test 9 cross-file link (heading lives in tests/hpc-efa-test.md, not tests/README.md).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Splits the
PCSClusterUserpolicy's singleSSMSessionToLoginNodestatement into atag-gated instance statement and an untagged document statement, so cluster users
can actually open SSH-over-SSM and port-forwarding sessions to the login node.
Why
The original statement gated the instance ARN and the SSM session-document ARNs under
one
StringLike ssm:resourceTag/Name: PCS-login*condition. SSM documents have noNametag, so for the document sub-authorization the key is absent and
StringLike(without...IfExists) fails closed.ssm:StartSessionwith a custom document authorizes againstboth the instance and the document — so SSH-over-SSM / port-forward are denied. A plain
Session Manager shell still works (instance-only authz), which is why a quick check passes.
This is the reason AWS's own tag-restricted Session Manager examples
keep the tag-gated instance statement separate from the untagged document statement.
Fix
SSMSessionToLoginNodeInstance—ssm:StartSessiononarn:aws:ec2:*:*:instance/*, gated byssm:resourceTag/Name: PCS-login*(unchanged scoping).SSMSessionDocuments—ssm:StartSessionon the four session documents, no condition.Login/compute isolation is unchanged: the instance statement still only matches
PCS-login*, so compute nodes remain unreachable.Verification (
iam simulate-custom-policy, deployed both before & after)ssm:StartSessionondocument/AWS-StartSSHSessiondocument/AWS-StartPortForwardingSessionName=PCS-login)Name=PCS-compute)cfn-lintclean. Verified end-to-end against a livecluster-user-iam.yamldeployment.Found while reviewing awslabs#1137; opening against
feat/multi-userso it can fold into the PR.