Repository navigation
fix(ci): use NVMe storage for H100 runner workspaces - #1627
Conversation
📝 WalkthroughWalkthroughThree H100 runner Helm value templates switch ephemeral volumes to hostPath under /raid/actions-runner, inject POD_NAME into runner and dind containers, and scope mounts with subPathExpr so each pod uses isolated subdirectories; an initContainer creates/chowns per-pod dirs and dind removes them on preStop. ChangesPer-pod Storage Isolation for Runner Scale Sets
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Kubernetes runner configuration files to use host-path volumes with pod-specific subdirectories instead of empty directories, and adds an init container to prepare these directories. The reviewer identified two critical issues across all modified files: first, using hostPath with pod-specific subdirectories poses a high risk of disk space exhaustion on the host NVMe because Kubernetes does not automatically clean up these directories when ephemeral pods are deleted; second, using a heavy Docker-in-Docker image for the init container to perform basic directory setup introduces unnecessary startup latency, which can be optimized by using a lightweight image like alpine or busybox.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| - name: work | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate |
There was a problem hiding this comment.
⚠️ Disk Space Exhaustion Risk on Host NVMe
Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.
Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.
Recommended Solutions:
-
Option A (Self-contained Pod Cleanup - Recommended): Mount the parent
runner-nvmevolume into thedindcontainer and use apreStoplifecycle hook to delete/runner-nvme/${POD_NAME}. Sincedindruns as root, it can clean up all files (including those created by Docker).# In the dind container definition: lifecycle: preStop: exec: command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
-
Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in
/raid/actions-runner/that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).
| - name: work | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate |
There was a problem hiding this comment.
⚠️ Disk Space Exhaustion Risk on Host NVMe
Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.
Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.
Recommended Solutions:
-
Option A (Self-contained Pod Cleanup - Recommended): Mount the parent
runner-nvmevolume into thedindcontainer and use apreStoplifecycle hook to delete/runner-nvme/${POD_NAME}. Sincedindruns as root, it can clean up all files (including those created by Docker).# In the dind container definition: lifecycle: preStop: exec: command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
-
Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in
/raid/actions-runner/that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).
| - name: work | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate |
There was a problem hiding this comment.
⚠️ Disk Space Exhaustion Risk on Host NVMe
Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.
Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.
Recommended Solutions:
-
Option A (Self-contained Pod Cleanup - Recommended): Mount the parent
runner-nvmevolume into thedindcontainer and use apreStoplifecycle hook to delete/runner-nvme/${POD_NAME}. Sincedindruns as root, it can clean up all files (including those created by Docker).# In the dind container definition: lifecycle: preStop: exec: command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
-
Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in
/raid/actions-runner/that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).
|
|
||
| initContainers: | ||
| - name: prepare-runner-nvme | ||
| image: fra.ocir.io/idqj093njucb/docker:dind |
There was a problem hiding this comment.
⚡ Use a Lightweight Image for the Init Container
The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.
Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.
image: alpine|
|
||
| initContainers: | ||
| - name: prepare-runner-nvme | ||
| image: fra.ocir.io/idqj093njucb/docker:dind |
There was a problem hiding this comment.
⚡ Use a Lightweight Image for the Init Container
The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.
Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.
image: alpine|
|
||
| initContainers: | ||
| - name: prepare-runner-nvme | ||
| image: fra.ocir.io/idqj093njucb/docker:dind |
There was a problem hiding this comment.
⚡ Use a Lightweight Image for the Init Container
The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.
Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.
image: alpine
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml`:
- Around line 99-110: The per-POD subPath directories under /raid/actions-runner
are never pruned; update the initContainer prepare-runner-nvme (or add a cluster
CronJob) to remove stale directories under /raid/actions-runner while preserving
the current $(POD_NAME) directory: implement a safe retention policy (e.g.,
delete dirs older than N days or keep last M) using a targeted find/remove
operation that excludes the current $(POD_NAME) and verifies
ownership/permissions before deletion; ensure this cleanup runs with the same
volume mounts and sufficient privileges as prepare-runner-nvme and
document/parameterize the retention window.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5343b708-bf24-4701-b6f0-bd2d4d3fa82d
📒 Files selected for processing (3)
scripts/k8s-runner-resources/runner-values-1-gpu-h100.yamlscripts/k8s-runner-resources/runner-values-2-gpu-h100.yamlscripts/k8s-runner-resources/runner-values-4-gpu-h100.yaml
Signed-off-by: key4ng <rukeyang@gmail.com>
c66ca66 to
58df753
Compare
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| command: | ||
| - sh | ||
| - -c | ||
| - mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks |
There was a problem hiding this comment.
🟡 Nit: The preStop hook correctly uses ${POD_NAME:?} to bail out if the variable is empty, but this init container command uses bare ${POD_NAME} throughout. If POD_NAME were ever unset, chown -R 1001:1001 /runner-nvme/ would recursively chown the entire NVMe runner directory (all pods' data), and the orphaned directories created under /runner-nvme/ would never be cleaned up by any pod's preStop.
metadata.name is always populated by Kubernetes so this is unlikely in practice, but worth guarding consistently — same pattern applies in the 2-gpu and 4-gpu files.
| - mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks | |
| - mkdir -p /runner-nvme/${POD_NAME:?}/_work /runner-nvme/${POD_NAME:?}/.cache /runner-nvme/${POD_NAME:?}/tmp /runner-nvme/${POD_NAME:?}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME:?} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME:?}/tmp /models/.locks /models/hub/.locks |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml (1)
156-173:⚠️ Potential issue | 🟠 Major | ⚡ Quick winThe cleanup mount breaks pod isolation across all H100 runner variants.
Each
dindsidecar mounts the shared/raid/actions-runnerroot at/runner-nvmeso it can delete${POD_NAME}on shutdown. Because workflows control that daemon through Docker, they can also bind-mount/runner-nvmeand access sibling runners’ workspaces, caches, and Docker state on the same node. Scope that mount withsubPathExpr: $(POD_NAME)and havepreStopdelete the mounted directory contents instead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml` around lines 156 - 173, The current lifecycle preStop exec removes /runner-nvme/${POD_NAME} while the volumeMount mounts the entire shared root at mountPath /runner-nvme (name: runner-nvme), which breaks pod isolation; change the volumeMount to scope the mount with subPathExpr: $(POD_NAME) so each pod only gets its own subdirectory, and update the preStop (lifecycle.preStop.exec.command) to remove only the contents of the mounted directory (e.g., delete files under /runner-nvme) rather than attempting to remove sibling subdirectories; keep references to POD_NAME for subPathExpr and ensure the startupProbe and other fields remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml`:
- Around line 156-173: The current lifecycle preStop exec removes
/runner-nvme/${POD_NAME} while the volumeMount mounts the entire shared root at
mountPath /runner-nvme (name: runner-nvme), which breaks pod isolation; change
the volumeMount to scope the mount with subPathExpr: $(POD_NAME) so each pod
only gets its own subdirectory, and update the preStop
(lifecycle.preStop.exec.command) to remove only the contents of the mounted
directory (e.g., delete files under /runner-nvme) rather than attempting to
remove sibling subdirectories; keep references to POD_NAME for subPathExpr and
ensure the startupProbe and other fields remain unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 559784b5-f93d-47bc-b415-e855938f8a09
📒 Files selected for processing (3)
scripts/k8s-runner-resources/runner-values-1-gpu-h100.yamlscripts/k8s-runner-resources/runner-values-2-gpu-h100.yamlscripts/k8s-runner-resources/runner-values-4-gpu-h100.yaml
Signed-off-by: key4ng <rukeyang@gmail.com>
github-actions
Bot
added
the
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7e61d96643
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| volumeMounts: | ||
| - mountPath: /runner-nvme | ||
| name: runner-nvme |
There was a problem hiding this comment.
On H100 runners, workflow steps talk to this dind daemon via DOCKER_HOST, so any job that can run Docker can now bind-mount /runner-nvme from the daemon container (for example with -v /runner-nvme:/host) and read or delete sibling pods' workspaces, caches, and Docker data on the same node. That bypasses the per-pod subPathExpr isolation this change is trying to add; the same parent mount is repeated in the 2- and 4-GPU H100 values.
Useful? React with 👍 / 👎.
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate |
There was a problem hiding this comment.
If an H100 node is missing the /raid NVMe mount or it is not mounted yet, DirectoryOrCreate lets kubelet create /raid/actions-runner on whatever filesystem backs that path, which can silently put workspaces, caches, and Docker layers back on the boot disk instead of failing fast. That reintroduces the eviction/root-disk exhaustion scenario this change is meant to avoid; the same DirectoryOrCreate hostPath is repeated in the other H100 values.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/k8s-runner-resources/README.md`:
- Line 126: The docs incorrectly call the dind container an "init container";
update the sentence to say the H100 values mount the parent NVMe path into the
dind sidecar (not an init container) and clarify that the actual init container
is prepare-runner-nvme; keep the mention of the preStop hook that removes
/runner-nvme/${POD_NAME:?} and the hostPath behavior, but replace "dind
sidecar-style init container" with something like "dind sidecar (the init
container is prepare-runner-nvme)" to accurately reference the containers and
hooks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 57571d8f-1f33-4d02-8839-061b2c3d0897
📒 Files selected for processing (1)
scripts/k8s-runner-resources/README.md
|
|
||
| The H100 runner scale-set values mount `/raid/actions-runner` from the host into each runner pod. Workspace, cache, temp, and Docker storage paths use `subPathExpr: $(POD_NAME)/...` so concurrent pods on the same node do not share mutable state. | ||
|
|
||
| Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees. |
There was a problem hiding this comment.
Incorrect container terminology.
The documentation refers to "the dind sidecar-style init container," but the dind container is a regular sidecar container (with a preStop hook), not an init container. The actual init container is prepare-runner-nvme.
📝 Suggested correction
-Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.
+Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees. | |
| Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/k8s-runner-resources/README.md` at line 126, The docs incorrectly
call the dind container an "init container"; update the sentence to say the H100
values mount the parent NVMe path into the dind sidecar (not an init container)
and clarify that the actual init container is prepare-runner-nvme; keep the
mention of the preStop hook that removes /runner-nvme/${POD_NAME:?} and the
hostPath behavior, but replace "dind sidecar-style init container" with
something like "dind sidecar (the init container is prepare-runner-nvme)" to
accurately reference the containers and hooks.
This reverts commit 63b532a. Signed-off-by: Keyang Ru <rukeyang@gmail.com>
2 participants
Description
Problem
H100 GitHub Actions runners can evict CI jobs when model downloads, workspaces, temp files, or Docker layer data consume the node boot volume. The live Moirai runners were fixed by moving heavy runner paths onto the local NVMe-backed
/raid/actions-runnerpath and by preparing writable model cache lock directories.Solution
Persist the same H100-only storage layout in the ARC runner scale-set values. The H100 runner pods now use per-pod
subPathExprdirectories under/raid/actions-runnerfor workspace, cache, temp, and Docker storage, and an init container prepares ownership and model cache lock directories before the runner starts. The dind sidecar-style init container also gets apreStophook that removes/raid/actions-runner/<pod-name>through the mounted parent path to avoid orphaning heavy hostPath directories after normal pod termination.Changes
runner-values-1-gpu-h100.yaml,runner-values-2-gpu-h100.yaml, andrunner-values-4-gpu-h100.yaml./home/runner/_work,/home/runner/.cache,/tmp, and/var/lib/dockeronto/raid/actions-runnerwith per-pod subdirectories.POD_NAMEenv wiring forsubPathExprin the runner and dind containers.prepare-runner-nvmeinit container to create/chown runner NVMe directories and prepare/models/.locks,/models/hub, and/models/hub/.locks.preStopcleanup hook for/runner-nvme/${POD_NAME:?}./raid/actions-runnerpath during live validation.Test Plan
Live validation in the Moirai dev cluster:
/models/.lockspermission failure./runner/_workor/home/runner/_work,/home/runner/.cache,/tmp, and/modelson the 28T NVMe volume./models/.locksand/models/hub/.locksare writable by therunneruser.arc-runner-2-gpu-h100andarc-runner-4-gpu-h100live pods started with the same NVMe mount pattern.preStophooks do not run for hard node failure or force deletion, so host-level pruning can still be added later as defense in depth.Local validation:
Reviewed
scripts/k8s-runner-resources/README.mdto confirm the H100 NVMe cleanup note documents the hostPath garbage-collection risk and the Summerwind controller caveat.ruby -ryaml -e 'ARGV.each { |f| data = YAML.load_file(f); spec = data.fetch("template").fetch("spec"); dind = spec.fetch("initContainers").find { |c| c["name"] == "dind" }; mounts = dind.fetch("volumeMounts").map { |m| [m["mountPath"], m["name"]] }.to_h; raise "#{f}: dind missing /runner-nvme mount" unless mounts["/runner-nvme"] == "runner-nvme"; cmd = dind.dig("lifecycle", "preStop", "exec", "command"); raise "#{f}: missing preStop cleanup" unless cmd&.join(" ")&.include?("/runner-nvme/${POD_NAME:?"); puts "OK #{File.basename(f)}" }' scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml scripts/k8s-runner-resources/runner-values-2-gpu-h100.yaml scripts/k8s-runner-resources/runner-values-4-gpu-h100.yamlRendered all three H100 values files with the ARC chart:
Checklist
cargo +nightly fmtpassescargo clippy --all-targets --all-features -- -D warningspassesSummary by CodeRabbit