Repository navigation
fix(ci): use NVMe storage for H100 runner workspaces #1627
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -46,10 +46,16 @@ template: | |||||
|
|
||||||
| volumes: | ||||||
| - name: work | ||||||
| emptyDir: {} | ||||||
| hostPath: | ||||||
| path: /raid/actions-runner | ||||||
| type: DirectoryOrCreate | ||||||
|
Comment on lines
48
to
+51
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||
| - name: model-cache | ||||||
| persistentVolumeClaim: | ||||||
| claimName: model-cache | ||||||
| - name: runner-nvme | ||||||
| hostPath: | ||||||
| path: /raid/actions-runner | ||||||
| type: DirectoryOrCreate | ||||||
| - name: dshm | ||||||
| emptyDir: | ||||||
| medium: Memory | ||||||
|
|
@@ -59,7 +65,9 @@ template: | |||||
| - name: dind-externals | ||||||
| emptyDir: {} | ||||||
| - name: docker-storage | ||||||
| emptyDir: {} | ||||||
| hostPath: | ||||||
| path: /raid/actions-runner | ||||||
| type: DirectoryOrCreate | ||||||
|
|
||||||
| containers: | ||||||
| - name: runner | ||||||
|
|
@@ -71,6 +79,10 @@ template: | |||||
| value: unix:///var/run/docker.sock | ||||||
| - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS | ||||||
| value: '120' | ||||||
| - name: POD_NAME | ||||||
| valueFrom: | ||||||
| fieldRef: | ||||||
| fieldPath: metadata.name | ||||||
| resources: | ||||||
| requests: | ||||||
| cpu: "8" | ||||||
|
|
@@ -84,12 +96,36 @@ template: | |||||
| mountPath: /dev/shm | ||||||
| - name: work | ||||||
| mountPath: /home/runner/_work | ||||||
| subPathExpr: $(POD_NAME)/_work | ||||||
| - name: runner-nvme | ||||||
| mountPath: /home/runner/.cache | ||||||
| subPathExpr: $(POD_NAME)/.cache | ||||||
| - name: runner-nvme | ||||||
| mountPath: /tmp | ||||||
| subPathExpr: $(POD_NAME)/tmp | ||||||
| - name: dind-sock | ||||||
| mountPath: /var/run | ||||||
| - name: docker-storage | ||||||
| mountPath: /var/lib/docker | ||||||
| subPathExpr: $(POD_NAME)/docker | ||||||
|
coderabbitai[bot] marked this conversation as resolved.
|
||||||
|
|
||||||
| initContainers: | ||||||
| - name: prepare-runner-nvme | ||||||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚡ Use a Lightweight Image for the Init ContainerThe Using a lightweight image like image: alpine |
||||||
| command: | ||||||
| - sh | ||||||
| - -c | ||||||
| - mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 Nit: The
Suggested change
|
||||||
| env: | ||||||
| - name: POD_NAME | ||||||
| valueFrom: | ||||||
| fieldRef: | ||||||
| fieldPath: metadata.name | ||||||
| volumeMounts: | ||||||
| - mountPath: /runner-nvme | ||||||
| name: runner-nvme | ||||||
| - mountPath: /models | ||||||
| name: model-cache | ||||||
| - name: init-dind-externals | ||||||
| args: | ||||||
| - '-r' | ||||||
|
|
@@ -109,10 +145,21 @@ template: | |||||
| env: | ||||||
| - name: DOCKER_GROUP_GID | ||||||
| value: '123' | ||||||
| - name: POD_NAME | ||||||
| valueFrom: | ||||||
| fieldRef: | ||||||
| fieldPath: metadata.name | ||||||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||||||
| restartPolicy: Always | ||||||
| securityContext: | ||||||
| privileged: true | ||||||
| lifecycle: | ||||||
| preStop: | ||||||
| exec: | ||||||
| command: | ||||||
| - sh | ||||||
| - -c | ||||||
| - rm -rf -- "/runner-nvme/${POD_NAME:?}" | ||||||
| startupProbe: | ||||||
| exec: | ||||||
| command: | ||||||
|
|
@@ -122,11 +169,15 @@ template: | |||||
| initialDelaySeconds: 0 | ||||||
| periodSeconds: 5 | ||||||
| volumeMounts: | ||||||
| - mountPath: /runner-nvme | ||||||
| name: runner-nvme | ||||||
|
Comment on lines
171
to
+173
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
On H100 runners, workflow steps talk to this dind daemon via Useful? React with 👍 / 👎. |
||||||
| - mountPath: /home/runner/_work | ||||||
| name: work | ||||||
| subPathExpr: $(POD_NAME)/_work | ||||||
| - mountPath: /var/run | ||||||
| name: dind-sock | ||||||
| - mountPath: /home/runner/externals | ||||||
| name: dind-externals | ||||||
| - name: docker-storage | ||||||
| mountPath: /var/lib/docker | ||||||
| subPathExpr: $(POD_NAME)/docker | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -46,10 +46,16 @@ template: | |
|
|
||
| volumes: | ||
| - name: work | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
|
Comment on lines
48
to
+51
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| - name: model-cache | ||
| persistentVolumeClaim: | ||
| claimName: model-cache | ||
| - name: runner-nvme | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
| - name: dshm | ||
| emptyDir: | ||
| medium: Memory | ||
|
|
@@ -59,7 +65,9 @@ template: | |
| - name: dind-externals | ||
| emptyDir: {} | ||
| - name: docker-storage | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
|
|
||
| containers: | ||
| - name: runner | ||
|
|
@@ -71,6 +79,10 @@ template: | |
| value: unix:///var/run/docker.sock | ||
| - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS | ||
| value: '120' | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| resources: | ||
| requests: | ||
| cpu: "16" | ||
|
|
@@ -84,12 +96,36 @@ template: | |
| mountPath: /dev/shm | ||
| - name: work | ||
| mountPath: /home/runner/_work | ||
| subPathExpr: $(POD_NAME)/_work | ||
| - name: runner-nvme | ||
| mountPath: /home/runner/.cache | ||
| subPathExpr: $(POD_NAME)/.cache | ||
| - name: runner-nvme | ||
| mountPath: /tmp | ||
| subPathExpr: $(POD_NAME)/tmp | ||
| - name: dind-sock | ||
| mountPath: /var/run | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| subPathExpr: $(POD_NAME)/docker | ||
|
|
||
| initContainers: | ||
| - name: prepare-runner-nvme | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚡ Use a Lightweight Image for the Init ContainerThe Using a lightweight image like image: alpine |
||
| command: | ||
| - sh | ||
| - -c | ||
| - mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| volumeMounts: | ||
| - mountPath: /runner-nvme | ||
| name: runner-nvme | ||
| - mountPath: /models | ||
| name: model-cache | ||
| - name: init-dind-externals | ||
| args: | ||
| - '-r' | ||
|
|
@@ -109,10 +145,21 @@ template: | |
| env: | ||
| - name: DOCKER_GROUP_GID | ||
| value: '123' | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
| restartPolicy: Always | ||
| securityContext: | ||
| privileged: true | ||
| lifecycle: | ||
| preStop: | ||
| exec: | ||
| command: | ||
| - sh | ||
| - -c | ||
| - rm -rf -- "/runner-nvme/${POD_NAME:?}" | ||
| startupProbe: | ||
| exec: | ||
| command: | ||
|
|
@@ -122,11 +169,15 @@ template: | |
| initialDelaySeconds: 0 | ||
| periodSeconds: 5 | ||
| volumeMounts: | ||
| - mountPath: /runner-nvme | ||
| name: runner-nvme | ||
| - mountPath: /home/runner/_work | ||
| name: work | ||
| subPathExpr: $(POD_NAME)/_work | ||
| - mountPath: /var/run | ||
| name: dind-sock | ||
| - mountPath: /home/runner/externals | ||
| name: dind-externals | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| subPathExpr: $(POD_NAME)/docker | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -45,10 +45,16 @@ template: | |
|
|
||
| volumes: | ||
| - name: work | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
|
Comment on lines
47
to
+50
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| - name: model-cache | ||
| persistentVolumeClaim: | ||
| claimName: model-cache | ||
| - name: runner-nvme | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
| - name: dshm | ||
| emptyDir: | ||
| medium: Memory | ||
|
|
@@ -58,7 +64,9 @@ template: | |
| - name: dind-externals | ||
| emptyDir: {} | ||
| - name: docker-storage | ||
| emptyDir: {} | ||
| hostPath: | ||
| path: /raid/actions-runner | ||
| type: DirectoryOrCreate | ||
|
|
||
| containers: | ||
| - name: runner | ||
|
|
@@ -70,6 +78,10 @@ template: | |
| value: unix:///var/run/docker.sock | ||
| - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS | ||
| value: '120' | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| resources: | ||
| requests: | ||
| cpu: "32" | ||
|
|
@@ -83,12 +95,36 @@ template: | |
| mountPath: /dev/shm | ||
| - name: work | ||
| mountPath: /home/runner/_work | ||
| subPathExpr: $(POD_NAME)/_work | ||
| - name: runner-nvme | ||
| mountPath: /home/runner/.cache | ||
| subPathExpr: $(POD_NAME)/.cache | ||
| - name: runner-nvme | ||
| mountPath: /tmp | ||
| subPathExpr: $(POD_NAME)/tmp | ||
| - name: dind-sock | ||
| mountPath: /var/run | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| subPathExpr: $(POD_NAME)/docker | ||
|
|
||
| initContainers: | ||
| - name: prepare-runner-nvme | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚡ Use a Lightweight Image for the Init ContainerThe Using a lightweight image like image: alpine |
||
| command: | ||
| - sh | ||
| - -c | ||
| - mkdir -p /runner-nvme/${POD_NAME}/_work /runner-nvme/${POD_NAME}/.cache /runner-nvme/${POD_NAME}/tmp /runner-nvme/${POD_NAME}/docker /models/.locks /models/hub /models/hub/.locks && chown -R 1001:1001 /runner-nvme/${POD_NAME} && chown 1001:1001 /models/.locks /models/hub /models/hub/.locks && chmod 1777 /runner-nvme/${POD_NAME}/tmp /models/.locks /models/hub/.locks | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| volumeMounts: | ||
| - mountPath: /runner-nvme | ||
| name: runner-nvme | ||
| - mountPath: /models | ||
| name: model-cache | ||
| - name: init-dind-externals | ||
| args: | ||
| - '-r' | ||
|
|
@@ -108,10 +144,21 @@ template: | |
| env: | ||
| - name: DOCKER_GROUP_GID | ||
| value: '123' | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: metadata.name | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
| restartPolicy: Always | ||
| securityContext: | ||
| privileged: true | ||
| lifecycle: | ||
| preStop: | ||
| exec: | ||
| command: | ||
| - sh | ||
| - -c | ||
| - rm -rf -- "/runner-nvme/${POD_NAME:?}" | ||
| startupProbe: | ||
| exec: | ||
| command: | ||
|
|
@@ -121,11 +168,15 @@ template: | |
| initialDelaySeconds: 0 | ||
| periodSeconds: 5 | ||
| volumeMounts: | ||
| - mountPath: /runner-nvme | ||
| name: runner-nvme | ||
| - mountPath: /home/runner/_work | ||
| name: work | ||
| subPathExpr: $(POD_NAME)/_work | ||
| - mountPath: /var/run | ||
| name: dind-sock | ||
| - mountPath: /home/runner/externals | ||
| name: dind-externals | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| subPathExpr: $(POD_NAME)/docker | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incorrect container terminology.
The documentation refers to "the dind sidecar-style init container," but the dind container is a regular sidecar container (with a
preStophook), not an init container. The actual init container isprepare-runner-nvme.📝 Suggested correction
📝 Committable suggestion
🤖 Prompt for AI Agents