Skip to content

fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs #1165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
KeitaW merged 7 commits into from
Jul 7, 2026
Merged

fix(aws-pcs): stop needrestart from restarting slurmd and killing running jobs #1165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1f3489d
fix(aws-pcs): stop needrestart from restarting slurmd and killing run…
DaisukeMiyamoto Jul 6, 2026
fc69fd0
fix(aws-pcs): use ASCII dash in needrestart guard comment
DaisukeMiyamoto Jul 6, 2026
04401f6
fix(aws-pcs): apply needrestart slurmd guard to GPU CNG templates too
DaisukeMiyamoto Jul 6, 2026
5af4edc
docs(aws-pcs): document needrestart/slurmd job-kill issue and fix (OP…
DaisukeMiyamoto Jul 6, 2026
3a71d07
docs(aws-pcs): reframe OPERATIONS 6.2 as user-facing guidance
DaisukeMiyamoto Jul 6, 2026
e0c495e
fix(aws-pcs): narrow needrestart guard to slurmd only
DaisukeMiyamoto Jul 6, 2026
052e716
docs(aws-pcs): tighten OPERATIONS 6.2
DaisukeMiyamoto Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions architectures/aws-pcs/assets/add-cng-p5.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,6 +393,25 @@ Resources:
MIME-Version: 1.0

runcmd:
# --- Protect running jobs from unattended-upgrades / needrestart ---
# apt-daily-upgrade updates base libraries (e.g. glibc). needrestart then
# auto-restarts every service linked against them. slurmd links libc, so it
# gets restarted mid-upgrade - which KILLS the jobs running under it (the
# step is torn down and the job requeues from scratch). Exclude slurmd
# from needrestart's automatic restart so a security upgrade can
# never take down a running job. Security packages still install; only the
# slurmd auto-restart is suppressed. (PCS runs the controller managed-side;
# slurmd is the only Slurm systemd service on login/compute nodes. qr(^slurmd)
# also covers the versioned units, e.g. slurmd-25.11.)
# runcmd runs under /bin/sh (dash) - keep this POSIX-clean, no bashisms.
- |
mkdir -p /etc/needrestart/conf.d
cat > /etc/needrestart/conf.d/90-pcs-slurm.conf <<'NRCONF'
# AWS PCS: never auto-restart slurmd - restarting it kills
# the jobs running under it. Managed by add-cng UserData.
$nrconf{override_rc} = { qr(^slurmd) => 0 };
NRCONF
echo "needrestart: slurmd excluded from auto-restart" | tee /var/log/pcs-needrestart-guard.log
# Post-install script (optional) - generic OnNodeConfigured-style hook.
# Downloads and runs a user-supplied script (e.g. Enroot/Pyxis install).
- |
Expand Down
19 changes: 19 additions & 0 deletions architectures/aws-pcs/assets/add-cng-p6-b200.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,25 @@ Resources:
MIME-Version: 1.0

runcmd:
# --- Protect running jobs from unattended-upgrades / needrestart ---
# apt-daily-upgrade updates base libraries (e.g. glibc). needrestart then
# auto-restarts every service linked against them. slurmd links libc, so it
# gets restarted mid-upgrade - which KILLS the jobs running under it (the
# step is torn down and the job requeues from scratch). Exclude slurmd
# from needrestart's automatic restart so a security upgrade can
# never take down a running job. Security packages still install; only the
# slurmd auto-restart is suppressed. (PCS runs the controller managed-side;
# slurmd is the only Slurm systemd service on login/compute nodes. qr(^slurmd)
# also covers the versioned units, e.g. slurmd-25.11.)
# runcmd runs under /bin/sh (dash) - keep this POSIX-clean, no bashisms.
- |
mkdir -p /etc/needrestart/conf.d
cat > /etc/needrestart/conf.d/90-pcs-slurm.conf <<'NRCONF'
# AWS PCS: never auto-restart slurmd - restarting it kills
# the jobs running under it. Managed by add-cng UserData.
$nrconf{override_rc} = { qr(^slurmd) => 0 };
NRCONF
echo "needrestart: slurmd excluded from auto-restart" | tee /var/log/pcs-needrestart-guard.log
# Post-install script (optional) - generic OnNodeConfigured-style hook.
# Downloads and runs a user-supplied script (e.g. Enroot/Pyxis install).
- |
Expand Down
19 changes: 19 additions & 0 deletions architectures/aws-pcs/assets/add-cng-p6-b300.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -383,6 +383,25 @@ Resources:
MIME-Version: 1.0

runcmd:
# --- Protect running jobs from unattended-upgrades / needrestart ---
# apt-daily-upgrade updates base libraries (e.g. glibc). needrestart then
# auto-restarts every service linked against them. slurmd links libc, so it
# gets restarted mid-upgrade - which KILLS the jobs running under it (the
# step is torn down and the job requeues from scratch). Exclude slurmd
# from needrestart's automatic restart so a security upgrade can
# never take down a running job. Security packages still install; only the
# slurmd auto-restart is suppressed. (PCS runs the controller managed-side;
# slurmd is the only Slurm systemd service on login/compute nodes. qr(^slurmd)
# also covers the versioned units, e.g. slurmd-25.11.)
# runcmd runs under /bin/sh (dash) - keep this POSIX-clean, no bashisms.
- |
mkdir -p /etc/needrestart/conf.d
cat > /etc/needrestart/conf.d/90-pcs-slurm.conf <<'NRCONF'
# AWS PCS: never auto-restart slurmd - restarting it kills
# the jobs running under it. Managed by add-cng UserData.
$nrconf{override_rc} = { qr(^slurmd) => 0 };
NRCONF
echo "needrestart: slurmd excluded from auto-restart" | tee /var/log/pcs-needrestart-guard.log
# Post-install script (optional) - generic OnNodeConfigured-style hook.
# Downloads and runs a user-supplied script (e.g. Enroot/Pyxis install).
- |
Expand Down
19 changes: 19 additions & 0 deletions architectures/aws-pcs/assets/add-cng.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,25 @@ Resources:
MIME-Version: 1.0

runcmd:
# --- Protect running jobs from unattended-upgrades / needrestart ---
# apt-daily-upgrade updates base libraries (e.g. glibc). needrestart then
# auto-restarts every service linked against them. slurmd links libc, so it
# gets restarted mid-upgrade - which KILLS the jobs running under it (the
# step is torn down and the job requeues from scratch). Exclude slurmd
# from needrestart's automatic restart so a security upgrade can
# never take down a running job. Security packages still install; only the
# slurmd auto-restart is suppressed. (PCS runs the controller managed-side;
# slurmd is the only Slurm systemd service on login/compute nodes. qr(^slurmd)
# also covers the versioned units, e.g. slurmd-25.11.)
# runcmd runs under /bin/sh (dash) — keep this POSIX-clean, no bashisms.
- |
mkdir -p /etc/needrestart/conf.d
cat > /etc/needrestart/conf.d/90-pcs-slurm.conf <<'NRCONF'
# AWS PCS: never auto-restart slurmd - restarting it kills
# the jobs running under it. Managed by add-cng.yaml UserData.
$nrconf{override_rc} = { qr(^slurmd) => 0 };

@KeitaW KeitaW Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qr(^slurmd) also prefix-matches slurmdbd — no change needed here (FYI)

qr(^slurmd) is an unanchored-prefix regex, so it also matches slurmdbd.service in addition to slurmd and the versioned slurmd-25.11. On PCS login/compute nodes this has no practical effect — as the comment right above correctly notes, slurmd is the only Slurm unit here (the controller and slurmdbd are managed PCS-side). I'd leave it exactly as written: tightening it to qr(^slurmd($|-|\.)) would only guard a unit that isn't present on these nodes. Flagging purely so the prefix-match is a known, deliberate property. It's the same qr(^name) => 0 idiom needrestart ships for dbus/display-managers.

NRCONF
echo "needrestart: slurmd excluded from auto-restart" | tee /var/log/pcs-needrestart-guard.log

@KeitaW KeitaW Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tee writes the success line unconditionally (nit)

The echo … | tee runs regardless of whether the preceding mkdir -p / cat > actually succeeded (no set -e, no && chain), so in the near-impossible failure case (e.g. a read-only /etc at first boot) the log would still read needrestart: slurmd excluded from auto-restart when it wasn't. Cosmetic only — the failure it would misreport isn't a realistic first-boot condition, so I wouldn't hold the PR for it. Mentioning it in case you'd prefer the log line to be strictly truthful (chain the write with && before the echo).

# Post-install script (optional) - generic OnNodeConfigured-style hook.
# Downloads and runs a user-supplied script (e.g. Enroot/Pyxis install).
# Accepts BOTH an s3:// URL (fetched with `aws s3 cp` using the instance
Expand Down
21 changes: 21 additions & 0 deletions architectures/aws-pcs/docs/OPERATIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -466,6 +466,27 @@ cleanest mitigation is upstream — recording it here so users seeing it know th
workaround and the next contributor doesn't waste time looking for a bug in this PR's
scripts.

### 6.2 `needrestart` restarting `slurmd` would stop running jobs — already handled

When an unattended security upgrade updates a base library `slurmd` links (e.g. glibc),
`needrestart` restarts `slurmd`, and that restart tears down the `slurmstepd` steps under
it — **stopping every job on the node**. It is reproducible, not random, and not a reboot
or a Slurm-package upgrade.

The compute-node-group templates already guard against this: each `add-cng*` UserData
writes a `needrestart` drop-in so `slurmd` is never auto-restarted (security updates still
install; `needrestart` only defers the `slurmd` restart):

```perl
# /etc/needrestart/conf.d/90-pcs-slurm.conf (written by add-cng* UserData)
$nrconf{override_rc} = { qr(^slurmd) => 0 };
```

`slurmd` is the only Slurm systemd service on these nodes (the controller is managed by
PCS); `qr(^slurmd)` also matches the versioned units (e.g. `slurmd-25.11`). The drop-in is
a standalone `conf.d/*.conf` naming only `slurmd`, so if a later DLAMI or Slurm unit
handles this differently it neither conflicts nor errors — at worst it becomes redundant.

## 7. Recommendations recap

For a new production deploy:
Expand Down