Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions scripts/k8s-runner-resources/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,16 @@ kubectl get pods -n actions-runner-system

Each runner set should have a listener pod in `Running` state. Runner pods will be created on-demand when workflows target the corresponding `runs-on` label (matching `runnerScaleSetName` in each values file).

## H100 Runner NVMe Storage

The H100 runner scale-set values mount `/raid/actions-runner` from the host into each runner pod. Workspace, cache, temp, and Docker storage paths use `subPathExpr: $(POD_NAME)/...` so concurrent pods on the same node do not share mutable state.

Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Incorrect container terminology.

The documentation refers to "the dind sidecar-style init container," but the dind container is a regular sidecar container (with a preStop hook), not an init container. The actual init container is prepare-runner-nvme.

📝 Suggested correction
-Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.
+Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar-style init container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.
Because Kubernetes does not garbage collect `hostPath` contents when a pod is deleted, the H100 values also mount the parent NVMe path into the dind sidecar container and use a `preStop` hook to remove `/runner-nvme/${POD_NAME:?}` during normal pod termination. This cleanup is best-effort; force deletion or node failure can still leave orphaned directories. Add host-level pruning if the cluster needs stronger cleanup guarantees.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/k8s-runner-resources/README.md` at line 126, The docs incorrectly
call the dind container an "init container"; update the sentence to say the H100
values mount the parent NVMe path into the dind sidecar (not an init container)
and clarify that the actual init container is prepare-runner-nvme; keep the
mention of the preStop hook that removes /runner-nvme/${POD_NAME:?} and the
hostPath behavior, but replace "dind sidecar-style init container" with
something like "dind sidecar (the init container is prepare-runner-nvme)" to
accurately reference the containers and hooks.


Do not copy this exact `/raid/actions-runner` setup to non-H100 runner values unless the target node pool has the same NVMe layout and enough free space. During Moirai validation, A10 nodes did not safely support this path.

For the legacy `actions.summerwind.dev` `RunnerDeployment` controller, the generated dind lifecycle is controller-managed, so dind `preStop` hooks from the template may not survive into pods. Use the official GitHub ARC scale-set values above for this configuration, or validate any Summerwind-specific cleanup separately.

## Upgrading

To update a runner set (e.g. after changing values):
Expand Down
55 changes: 53 additions & 2 deletions scripts/k8s-runner-resources/runner-values-1-gpu-h100.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,16 @@ template:

volumes:
- name: work
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
Comment on lines 48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

⚠️ Disk Space Exhaustion Risk on Host NVMe

Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.

Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.

Recommended Solutions:

  • Option A (Self-contained Pod Cleanup - Recommended): Mount the parent runner-nvme volume into the dind container and use a preStop lifecycle hook to delete /runner-nvme/${POD_NAME}. Since dind runs as root, it can clean up all files (including those created by Docker).

    # In the dind container definition:
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
  • Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in /raid/actions-runner/ that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).

Comment on lines +49 to +51

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fail when the NVMe mount is missing

If an H100 node is missing the /raid NVMe mount or it is not mounted yet, DirectoryOrCreate lets kubelet create /raid/actions-runner on whatever filesystem backs that path, which can silently put workspaces, caches, and Docker layers back on the boot disk instead of failing fast. That reintroduces the eviction/root-disk exhaustion scenario this change is meant to avoid; the same DirectoryOrCreate hostPath is repeated in the other H100 values.

Useful? React with 👍 / 👎.

- name: model-cache
persistentVolumeClaim:
claimName: model-cache
- name: runner-nvme
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
- name: dshm
emptyDir:
medium: Memory
Expand All @@ -59,7 +65,9 @@ template:
- name: dind-externals
emptyDir: {}
- name: docker-storage
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate

containers:
- name: runner
Expand All @@ -71,6 +79,10 @@ template:
value: unix:///var/run/docker.sock
- name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
value: '120'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: "8"
Expand All @@ -84,12 +96,36 @@ template:
mountPath: /dev/shm
- name: work
mountPath: /home/runner/_work
subPathExpr: $(POD_NAME)/_work
- name: runner-nvme
mountPath: /home/runner/.cache
subPathExpr: $(POD_NAME)/.cache
- name: runner-nvme
mountPath: /tmp
subPathExpr: $(POD_NAME)/tmp
- name: dind-sock
mountPath: /var/run
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker
Comment thread
coderabbitai[bot] marked this conversation as resolved.

initContainers:
- name: prepare-runner-nvme
image: fra.ocir.io/idqj093njucb/docker:dind

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

⚡ Use a Lightweight Image for the Init Container

The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.

Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.

        image: alpine

command:
- sh
- -c
- mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: The preStop hook correctly uses ${POD_NAME:?} to bail out if the variable is empty, but this init container command uses bare ${POD_NAME} throughout. If POD_NAME were ever unset, chown -R 1001:1001 /runner-nvme/ would recursively chown the entire NVMe runner directory (all pods' data), and the orphaned directories created under /runner-nvme/ would never be cleaned up by any pod's preStop.

metadata.name is always populated by Kubernetes so this is unlikely in practice, but worth guarding consistently — same pattern applies in the 2-gpu and 4-gpu files.

Suggested change
- mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks
- mkdir -p /runner-nvme/${POD_NAME:?}/_work /runner-nvme/${POD_NAME:?}/.cache /runner-nvme/${POD_NAME:?}/tmp /runner-nvme/${POD_NAME:?}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME:?} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME:?}/tmp /models/.locks /models/hub/.locks

env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
- mountPath: /models
name: model-cache
- name: init-dind-externals
args:
- '-r'
Expand All @@ -109,10 +145,21 @@ template:
env:
- name: DOCKER_GROUP_GID
value: '123'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: fra.ocir.io/idqj093njucb/docker:dind
restartPolicy: Always
securityContext:
privileged: true
lifecycle:
preStop:
exec:
command:
- sh
- -c
- rm -rf -- "/runner-nvme/${POD_NAME:?}"
startupProbe:
exec:
command:
Expand All @@ -122,11 +169,15 @@ template:
initialDelaySeconds: 0
periodSeconds: 5
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
Comment on lines 171 to +173

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not expose the NVMe parent to job-controlled Docker

On H100 runners, workflow steps talk to this dind daemon via DOCKER_HOST, so any job that can run Docker can now bind-mount /runner-nvme from the daemon container (for example with -v /runner-nvme:/host) and read or delete sibling pods' workspaces, caches, and Docker data on the same node. That bypasses the per-pod subPathExpr isolation this change is trying to add; the same parent mount is repeated in the 2- and 4-GPU H100 values.

Useful? React with 👍 / 👎.

- mountPath: /home/runner/_work
name: work
subPathExpr: $(POD_NAME)/_work
- mountPath: /var/run
name: dind-sock
- mountPath: /home/runner/externals
name: dind-externals
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker
55 changes: 53 additions & 2 deletions scripts/k8s-runner-resources/runner-values-2-gpu-h100.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,16 @@ template:

volumes:
- name: work
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
Comment on lines 48 to +51

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

⚠️ Disk Space Exhaustion Risk on Host NVMe

Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.

Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.

Recommended Solutions:

  • Option A (Self-contained Pod Cleanup - Recommended): Mount the parent runner-nvme volume into the dind container and use a preStop lifecycle hook to delete /runner-nvme/${POD_NAME}. Since dind runs as root, it can clean up all files (including those created by Docker).

    # In the dind container definition:
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
  • Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in /raid/actions-runner/ that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).

- name: model-cache
persistentVolumeClaim:
claimName: model-cache
- name: runner-nvme
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
- name: dshm
emptyDir:
medium: Memory
Expand All @@ -59,7 +65,9 @@ template:
- name: dind-externals
emptyDir: {}
- name: docker-storage
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate

containers:
- name: runner
Expand All @@ -71,6 +79,10 @@ template:
value: unix:///var/run/docker.sock
- name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
value: '120'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: "16"
Expand All @@ -84,12 +96,36 @@ template:
mountPath: /dev/shm
- name: work
mountPath: /home/runner/_work
subPathExpr: $(POD_NAME)/_work
- name: runner-nvme
mountPath: /home/runner/.cache
subPathExpr: $(POD_NAME)/.cache
- name: runner-nvme
mountPath: /tmp
subPathExpr: $(POD_NAME)/tmp
- name: dind-sock
mountPath: /var/run
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker

initContainers:
- name: prepare-runner-nvme
image: fra.ocir.io/idqj093njucb/docker:dind

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

⚡ Use a Lightweight Image for the Init Container

The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.

Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.

        image: alpine

command:
- sh
- -c
- mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
- mountPath: /models
name: model-cache
- name: init-dind-externals
args:
- '-r'
Expand All @@ -109,10 +145,21 @@ template:
env:
- name: DOCKER_GROUP_GID
value: '123'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: fra.ocir.io/idqj093njucb/docker:dind
restartPolicy: Always
securityContext:
privileged: true
lifecycle:
preStop:
exec:
command:
- sh
- -c
- rm -rf -- "/runner-nvme/${POD_NAME:?}"
startupProbe:
exec:
command:
Expand All @@ -122,11 +169,15 @@ template:
initialDelaySeconds: 0
periodSeconds: 5
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
- mountPath: /home/runner/_work
name: work
subPathExpr: $(POD_NAME)/_work
- mountPath: /var/run
name: dind-sock
- mountPath: /home/runner/externals
name: dind-externals
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker
55 changes: 53 additions & 2 deletions scripts/k8s-runner-resources/runner-values-4-gpu-h100.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,16 @@ template:

volumes:
- name: work
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
Comment on lines 47 to +50

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

⚠️ Disk Space Exhaustion Risk on Host NVMe

Since these runner pods are ephemeral, using hostPath with subPathExpr: $(POD_NAME)/... creates a new directory on the host's NVMe (/raid/actions-runner/<pod-name>) for every single runner pod. Kubernetes does not automatically clean up hostPath directories when a pod is deleted.

Over time, this will accumulate orphaned directories (containing heavy workspaces, caches, and docker layers) and eventually exhaust the NVMe disk space, leading to node disk pressure and job evictions.

Recommended Solutions:

  • Option A (Self-contained Pod Cleanup - Recommended): Mount the parent runner-nvme volume into the dind container and use a preStop lifecycle hook to delete /runner-nvme/${POD_NAME}. Since dind runs as root, it can clean up all files (including those created by Docker).

    # In the dind container definition:
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "rm -rf /runner-nvme/${POD_NAME}"]
  • Option B (Host-level Cleanup): Implement a cronjob or systemd timer on the host nodes to periodically prune directories in /raid/actions-runner/ that do not correspond to active pods or are older than a certain threshold (e.g., 24 hours).

- name: model-cache
persistentVolumeClaim:
claimName: model-cache
- name: runner-nvme
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate
- name: dshm
emptyDir:
medium: Memory
Expand All @@ -58,7 +64,9 @@ template:
- name: dind-externals
emptyDir: {}
- name: docker-storage
emptyDir: {}
hostPath:
path: /raid/actions-runner
type: DirectoryOrCreate

containers:
- name: runner
Expand All @@ -70,6 +78,10 @@ template:
value: unix:///var/run/docker.sock
- name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
value: '120'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: "32"
Expand All @@ -83,12 +95,36 @@ template:
mountPath: /dev/shm
- name: work
mountPath: /home/runner/_work
subPathExpr: $(POD_NAME)/_work
- name: runner-nvme
mountPath: /home/runner/.cache
subPathExpr: $(POD_NAME)/.cache
- name: runner-nvme
mountPath: /tmp
subPathExpr: $(POD_NAME)/tmp
- name: dind-sock
mountPath: /var/run
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker

initContainers:
- name: prepare-runner-nvme
image: fra.ocir.io/idqj093njucb/docker:dind

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

⚡ Use a Lightweight Image for the Init Container

The prepare-runner-nvme init container uses fra.ocir.io/idqj093njucb/docker:dind just to run a simple shell script (mkdir, chown, chmod). This image is very large and increases pod startup latency due to image pull overhead.

Using a lightweight image like alpine or busybox (~5MB) will significantly speed up pod initialization.

        image: alpine

command:
- sh
- -c
- mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
- mountPath: /models
name: model-cache
- name: init-dind-externals
args:
- '-r'
Expand All @@ -108,10 +144,21 @@ template:
env:
- name: DOCKER_GROUP_GID
value: '123'
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: fra.ocir.io/idqj093njucb/docker:dind
restartPolicy: Always
securityContext:
privileged: true
lifecycle:
preStop:
exec:
command:
- sh
- -c
- rm -rf -- "/runner-nvme/${POD_NAME:?}"
startupProbe:
exec:
command:
Expand All @@ -121,11 +168,15 @@ template:
initialDelaySeconds: 0
periodSeconds: 5
volumeMounts:
- mountPath: /runner-nvme
name: runner-nvme
- mountPath: /home/runner/_work
name: work
subPathExpr: $(POD_NAME)/_work
- mountPath: /var/run
name: dind-sock
- mountPath: /home/runner/externals
name: dind-externals
- name: docker-storage
mountPath: /var/lib/docker
subPathExpr: $(POD_NAME)/docker