refactor(ci): isolate SPUR runner sessions by amd-ivaganev · Pull Request #138 · ROCm/rocm-aic

amd-ivaganev · 2026-08-13T19:19:35Z

Summary

scope SPUR runner worktrees by GitHub run and attempt while retaining SHA-scoped image artifacts between stages
make terminal cleanup best-effort so NFS cleanup failures cannot replace a successful hardware-test result
share interruption handling across the dist-build, smoke-test, and tiny-test runner scripts
record the exact active Slurm job and isolated remote process group, then cancel both when a wrapper is interrupted
add a numeric cleanup-only workflow dispatch for recovering older orphaned jobs

Root cause

A canceled GitHub job terminated its local SSH client but left the remote poller and Slurm job alive. A later attempt reused the same SHA-only worktree. NFS preserved the open script as a .nfs* file, so tiny-test passed but terminal rm -rf failed and changed the job result to failure.

Scope

This PR contains the self-hosted runner lifecycle refactor and its Slurm job-marker hook. It is intentionally separate from #137, which only fixes GPU reservation and device propagation for SPUR hardware tests.

Validation

bash -n for the changed runner and Slurm scripts
workflow YAML parse
git diff --check
git range-diff confirms the split commits are patch-identical to the changes removed from fix(ci): reserve GPUs for SPUR hardware tests #137
ShellCheck source annotations use repository-relative paths while runtime sourcing remains location-independent

Scope remote worktrees by GitHub run and attempt, while retaining the SHA-scoped image tarball used to hand artifacts between hardware-test stages. Make worktree and tarball cleanup best-effort so NFS .nfs files cannot replace a successful test result with a cleanup failure. Record each active Slurm job and isolated remote process group in per-stage control files. On interruption, reconnect as the CI user to cancel the exact job and terminate its poller; use a cancellation sentinel to cover startup races. Add a numeric cleanup-only workflow dispatch for recovering jobs orphaned before automatic cancellation tracking was available.

Point ShellCheck's source annotations at the repository-relative path for spur-ci-common.sh in all three SPUR test wrappers. Keep runtime sourcing based on SCRIPT_DIR so installed CI scripts remain location-independent.

sbates130272

@amd-ivaganev I would like to move any scripts that need to be installed on the github self-hosted runner in ./github/scripts/runners and keep other scripts used by workflows in .github/scripts/workflows please!

amd-ivaganev · 2026-08-13T20:02:27Z

@amd-ivaganev I would like to move any scripts that need to be installed on the github self-hosted runner in ./github/scripts/runners and keep other scripts used by workflows in .github/scripts/workflows please!

@sbates130272 does this look reasonable:

  Planned organization:

   Current file                    Destination                   Reason
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   spur-ci-common.sh               .github/scripts/runners/      Installed dependency of runner wrappers
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-dist-build.sh              .github/scripts/runners/      Executed from /usr/local/lib/aic-ci
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-smoke-test.sh              .github/scripts/runners/      Executed from /usr/local/lib/aic-ci
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-tiny-test.sh               .github/scripts/runners/      Executed from /usr/local/lib/aic-ci
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-cliff.sh                   .github/scripts/runners/      Executed from /usr/local/lib/aic-ci
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-cliff-harvest.sh           .github/scripts/workflows/    Executed from the checked-out repository
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   spur-monitoring-cpu-smoke.sh    .github/scripts/workflows/    Executed from the checked-out repository
  ──────────────────────────────  ────────────────────────────  ──────────────────────────────────────────
   render-release-notes.sh         .github/scripts/workflows/    Executed from the checked-out repository

Move scripts installed under /usr/local/lib/aic-ci into .github/scripts/runners and limit install-ci-scripts to that directory. Move checkout-invoked helpers into .github/scripts/workflows, update their workflow call sites, and keep ShellCheck source annotations aligned with the new runner path.

amd-ivaganev · 2026-08-13T20:59:04Z

/run-ci-fast

github-actions · 2026-08-13T21:01:06Z

❌ dist-build-fast failed for 0027f51. View logs.

github-actions · 2026-08-13T21:11:38Z

❌ dist-build-fast failed for 0027f51. View logs.

sbates130272 · 2026-08-13T22:30:19Z

@amd-ivaganev csn you work with @john00003 on this? Thanks!

amd-ivaganev · 2026-08-17T21:52:11Z

@john00003 are you working on this or an alternative? Let me know if I can help.

john00003 · 2026-08-17T22:16:29Z

@amd-ivaganev Not right now. My docker isolation work is separate from the work you are doing in this PR. Your changes look really good.

I know Stephen tagged me on this, was there anything you wanted me to finish or add onto this? If not, I can just give this a more thorough review.

amd-ivaganev · 2026-08-17T22:48:33Z

@amd-ivaganev Not right now. My docker isolation work is separate from the work you are doing in this PR. Your changes look really good.

I know Stephen tagged me on this, was there anything you wanted me to finish or add onto this? If not, I can just give this a more thorough review.

Nothing to add, just making sure it won't conflict with your plans. Have a look-see whenever you have time and if there's no changes needed I can merge it. This was all originally driven by issues with re-running failed CI runs that I experienced with an unrelated PR. Stephen asked to decouple them so this became a separate thing.

amd-ivaganev · 2026-08-17T22:48:40Z

/run-ci-fast

github-actions · 2026-08-17T22:50:28Z

❌ dist-build-fast failed for 0027f51. View logs.

github-actions · 2026-08-17T23:07:02Z

❌ dist-build-fast failed for 0027f51. View logs.

github-actions · 2026-08-18T05:53:01Z

❌ Tiny-test (fast) failed for 0027f51. View logs.

john00003 · 2026-08-18T15:21:21Z

@amd-ivaganev Still need to review the changes, but I see the failures look along the lines of:

[vllm] aic-vllm-gpu0  | (EngineCore pid=521) ValueError: Free memory on device cuda:0 (49.41/287.98 GiB) on startup is less than desired GPU memory utilization (0.3, 86.4 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
  [vllm] aic-vllm-gpu0  | [rank0]:[W817 23:43:29.129797205 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

I believe this is being caused by the Docker container using GPUs that it is not supposed to (the reserved GPUs in SPUR mess up when being passed to a Docker container). This is exactly the issue I am trying to solve.

I have a hacky workaround ready, I will submit that PR soon.

github-actions · 2026-08-18T17:10:15Z

❌ Tiny-test (fast) failed for 0027f51. View logs.

amd-ivaganev · 2026-08-18T19:46:10Z

I believe this is being caused by the Docker container using GPUs that it is not supposed to (the reserved GPUs in SPUR mess up when being passed to a Docker container). This is exactly the issue I am trying to solve.

I have a hacky workaround ready, I will submit that PR soon.

Yes, I've seen this happen before, but eventually lucked into a node with enough free VRAM to pass. Are batch jobs also not getting {ROCR,HIP}_VISIBLE_DEVICES set in them like the interactive ones? I'd check but the cluster appears to be down...

john00003 · 2026-08-18T19:53:54Z

@amd-ivaganev Ah, is this the Tuesday maintenance period?

I was not aware of {ROCR,HIP}_VISIBLE_DEVICES potentially missing, I will similarly check once the cluster is back up. I can file a ticket if that is the case.

amd-ivaganev · 2026-08-18T20:43:53Z

@amd-ivaganev Ah, is this the Tuesday maintenance period?

I was not aware of {ROCR,HIP}_VISIBLE_DEVICES potentially missing, I will similarly check once the cluster is back up. I can file a ticket if that is the case.

I thought that's why you had posted the export output when you engaged the spur folks about interactive jobs. At any rate, if they had been set the software wouldn't have tried to use GPU 0 which was already in use.

I had codex whip me up a python script to decode & spit out the assigned GPUs from the spur job id. We can add that in the workflows on the runner so it sets them by itself, but I wanted to double check first.

john00003 · 2026-08-18T20:49:47Z

@amd-ivaganev Sorry, poor wording on my end. I meant I wasn't aware that it could be missing from batch jobs, I knew it was missing in --pty jobs.

I checked this again, ROCR_VISIBLE_DEVICES is set in batch jobs, but is missing in --pty jobs. HIP_VISIBLE_DEVICES is missing in both.

john00003 · 2026-08-18T20:53:06Z

@amd-ivaganev I can install that on my runner once it is ready. My work was doing something similar but I can rework it on top of your changes.

Or I'll ping you if I finish my changes soon enough.

github-actions · 2026-08-18T22:27:20Z

❌ Tiny-test (fast) failed for 0027f51. View logs.

github-actions · 2026-08-18T23:03:27Z

❌ Tiny-test (fast) failed for 0027f51. View logs.

amd-ivaganev · 2026-08-18T23:13:03Z

/run-ci-fast

github-actions · 2026-08-18T23:37:28Z

✅ Fast hardware CI passed for 0027f51 (dist-build-fast + smoke-test-fast + tiny-test). This is a single-arch dev check — run /run-ci for the full gate.

Done.

…sync Upstream ROCm#138 landed the same runner/workflow script split under .github/scripts/, so drop this branch's competing top-level scripts/ move and keep only the README version check. The sync script and its tests move to .github/scripts/workflows/ to match.

amd-ivaganev added 2 commits August 13, 2026 12:18

fix(ci): resolve SPUR helper during shellcheck

3cba750

Point ShellCheck's source annotations at the repository-relative path for spur-ci-common.sh in all three SPUR test wrappers. Keep runtime sourcing based on SCRIPT_DIR so installed CI scripts remain location-independent.

amd-ivaganev requested review from gaoikawa, sbates130272 and vcave as code owners August 13, 2026 19:19

amd-ivaganev mentioned this pull request Aug 13, 2026

fix(ci): reserve GPUs for SPUR hardware tests #137

Merged

sbates130272 previously requested changes Aug 13, 2026

View reviewed changes

amd-ivaganev merged commit 211814c into main Aug 18, 2026
9 checks passed

2imi9 mentioned this pull request Aug 19, 2026

fix(ci): update README sync for source-built stack #131

Merged

1 task

Uh oh!

Conversation

amd-ivaganev commented Aug 13, 2026

Summary

Root cause

Scope

Validation

Uh oh!

sbates130272 left a comment

Choose a reason for hiding this comment

Uh oh!

amd-ivaganev commented Aug 13, 2026

Uh oh!

amd-ivaganev commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

sbates130272 commented Aug 13, 2026

Uh oh!

amd-ivaganev commented Aug 17, 2026

Uh oh!

john00003 commented Aug 17, 2026

Uh oh!

amd-ivaganev commented Aug 17, 2026

Uh oh!

amd-ivaganev commented Aug 17, 2026

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

github-actions Bot commented Aug 17, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

john00003 commented Aug 18, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

amd-ivaganev commented Aug 18, 2026

Uh oh!

john00003 commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amd-ivaganev commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

john00003 commented Aug 18, 2026

Uh oh!

john00003 commented Aug 18, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

amd-ivaganev commented Aug 18, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

john00003 commented Aug 18, 2026 •

edited

Loading

amd-ivaganev commented Aug 18, 2026 •

edited

Loading