Repository navigation
fix(ci): Use H100 runner for Specific Tests #357
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| apiVersion: actions.summerwind.dev/v1alpha1 | ||
| kind: HorizontalRunnerAutoscaler | ||
| metadata: | ||
| name: arc-runner-h100-autoscaler | ||
| namespace: actions-runner-system | ||
| spec: | ||
| scaleTargetRef: | ||
| kind: RunnerDeployment | ||
| name: arc-runner-gpu-h100 | ||
|
|
||
| minReplicas: 10 | ||
| maxReplicas: 20 | ||
|
|
||
| metrics: | ||
| - type: PercentageRunnersBusy | ||
| scaleUpThreshold: "0.95" | ||
| scaleDownThreshold: "0.25" | ||
| scaleUpFactor: "0.5" | ||
| scaleDownFactor: "0.5" | ||
| --- | ||
| apiVersion: actions.summerwind.dev/v1alpha1 | ||
| kind: HorizontalRunnerAutoscaler | ||
| metadata: | ||
| name: arc-cpu-runner-autoscaler | ||
| namespace: actions-runner-system | ||
| spec: | ||
| scaleTargetRef: | ||
| kind: RunnerDeployment | ||
| name: arc-runner-cpu | ||
|
|
||
| minReplicas: 4 | ||
| maxReplicas: 8 | ||
|
|
||
| metrics: | ||
| - type: PercentageRunnersBusy | ||
| scaleUpThreshold: "0.95" | ||
| scaleDownThreshold: "0.25" | ||
| scaleUpFactor: "0.5" | ||
| scaleDownFactor: "0.5" |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| apiVersion: actions.summerwind.dev/v1alpha1 | ||
| kind: RunnerDeployment | ||
| metadata: | ||
| name: arc-runner-cpu | ||
| namespace: actions-runner-system | ||
| spec: | ||
| replicas: 4 | ||
| template: | ||
| spec: | ||
| repository: lightseekorg/smg | ||
| labels: | ||
| - k8s-runner-cpu | ||
| serviceAccountName: arc-runner-sa | ||
|
|
||
| containers: | ||
| - name: runner | ||
| image: fra.ocir.io/idqj093njucb/action-runner:v0.0.1 | ||
| resources: | ||
| requests: | ||
| cpu: "8" | ||
| memory: "16Gi" | ||
| limits: | ||
| cpu: "8" | ||
| memory: "16Gi" | ||
| - name: docker | ||
| image: fra.ocir.io/idqj093njucb/docker:dind |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| apiVersion: actions.summerwind.dev/v1alpha1 | ||
| kind: RunnerDeployment | ||
| metadata: | ||
| name: arc-runner-gpu-h100 | ||
| namespace: actions-runner-system | ||
| spec: | ||
| replicas: 10 | ||
| template: | ||
| spec: | ||
| repository: lightseekorg/smg | ||
| labels: | ||
| - 4-gpu-h100 | ||
| - k8s-runner-gpu | ||
| serviceAccountName: arc-runner-sa | ||
|
|
||
| nodeSelector: | ||
| nvidia.com/gpu: "true" | ||
| beta.kubernetes.io/instance-type: BM.GPU.H100.8 | ||
|
|
||
|
Comment on lines
+16
to
+19
There was a problem hiding this comment. 🧩 Analysis chain🌐 Web query:
💡 Result:
What to change
Notes
Source
Replace deprecated instance-type label to avoid scheduling failures on newer clusters.
🔧 Suggested update nodeSelector:
nvidia.com/gpu: "true"
- beta.kubernetes.io/instance-type: BM.GPU.H100.8
+ node.kubernetes.io/instance-type: BM.GPU.H100.8 nodeSelector:
nvidia.com/gpu: "true"
- beta.kubernetes.io/instance-type: BM.GPU.A10.4
+ node.kubernetes.io/instance-type: BM.GPU.A10.4Also applies to: 96-99 🤖 Prompt for AI Agents |
||
| tolerations: | ||
| - key: "nvidia.com/gpu" | ||
| operator: "Equal" | ||
| value: "true" | ||
| effect: "NoSchedule" | ||
|
|
||
| affinity: | ||
| podAffinity: | ||
| preferredDuringSchedulingIgnoredDuringExecution: | ||
| - weight: 100 | ||
| podAffinityTerm: | ||
| labelSelector: | ||
| matchExpressions: | ||
| - key: runner-deployment-name | ||
| operator: In | ||
| values: | ||
| - arc-runner-gpu-h100 | ||
| topologyKey: kubernetes.io/hostname | ||
|
|
||
| volumes: | ||
| - name: model-cache | ||
| persistentVolumeClaim: | ||
| claimName: model-cache | ||
| - name: docker-sock | ||
| emptyDir: {} | ||
| - name: docker-storage | ||
| emptyDir: {} | ||
| - name: dshm | ||
| emptyDir: | ||
| medium: Memory | ||
| sizeLimit: 16Gi | ||
|
|
||
| containers: | ||
| - name: runner | ||
| image: fra.ocir.io/idqj093njucb/action-runner:v0.0.1 | ||
| resources: | ||
| limits: | ||
| nvidia.com/gpu: 4 | ||
| volumeMounts: | ||
| - name: model-cache | ||
| mountPath: /models | ||
| - name: docker-sock | ||
| mountPath: /var/run | ||
| - name: dshm | ||
| mountPath: /dev/shm | ||
| env: | ||
| - name: DOCKER_HOST | ||
| value: unix:///var/run/docker.sock | ||
| - name: docker | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
| securityContext: | ||
|
XinyueZhang369 marked this conversation as resolved.
|
||
| privileged: true # Required for DinD | ||
| env: | ||
| - name: DOCKER_TLS_CERTDIR | ||
| value: "" # Disables TLS for shared socket use | ||
|
Contributor
There was a problem hiding this comment. Disabling TLS for the Docker socket (
Collaborator
Author
There was a problem hiding this comment. Need this for docker in docker
Comment on lines
+69
to
+74
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: # First, let's look at the file to understand the context
wc -l scripts/k8s-runner-resources/arc-runner-gpu.yamlRepository: lightseekorg/smg Length of output: 112 🏁 Script executed: # Read the relevant sections - lines 69-74 and 149-154
sed -n '65,80p' scripts/k8s-runner-resources/arc-runner-gpu.yaml
echo "---"
sed -n '145,160p' scripts/k8s-runner-resources/arc-runner-gpu.yamlRepository: lightseekorg/smg Length of output: 1131 🌐 Web query:
💡 Result: DinD on Kubernetes (2025→2026) security best practices (privileged, rootless, TLS)1) Prefer not running DinD at all (best risk reduction)
2) If you must run DinD, treat it as high risk
3) Rootless: helpful, but don’t assume it removes the need for privilege
4) Use Kubernetes User Namespaces where available (additional containment)
5) Never expose an unauthenticated Docker API; use TLS or SSH
6) If privileged DinD is unavoidable, reduce blast radius aggressively
Key takeaway: In Kubernetes, “DinD” usually pushes you toward privileged (or “privileged-like”) operation; the safest pattern is to avoid DinD and use rootless BuildKit/Buildx-on-Kubernetes where possible. [1][2][5] Sources: Docker docs on Buildx Kubernetes rootless mode [1], Kubernetes on privileged container impacts [2], Kubernetes Pod Security Standards [3], Docker rootless mode [4], Docker rootless DinD tip showing Privileged DinD with TLS disabled is a security risk for CI runners. A privileged Docker daemon plus This applies to both locations: lines 69-74 and 149-154. 🤖 Prompt for AI Agents |
||
| volumeMounts: | ||
| - name: docker-sock | ||
| mountPath: /var/run | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| --- | ||
| apiVersion: actions.summerwind.dev/v1alpha1 | ||
| kind: RunnerDeployment | ||
| metadata: | ||
| name: arc-runner-gpu-a10 | ||
| namespace: actions-runner-system | ||
| spec: | ||
| replicas: 2 | ||
|
Member
There was a problem hiding this comment. I thought we have 4 A10, do we only have 2? I also didn't see hpa for a10.
Collaborator
Author
There was a problem hiding this comment. we probably don't need to provision all 4 A20 all the time, so here is set to 2 for now. For HPA, today the auto scaler somehow kept creating cpu and a10 runner pods that cannot register to the repo regardless the max number, so I deleted all the old auto scalers for cpu, a10 and h100, this one is a new configuration, I want to bake it for some times, since most resources are h100, I only create for h100 for now for baking, once the scaling strategy works stably, I'll create the same for a10 and update this file |
||
| template: | ||
| spec: | ||
| repository: lightseekorg/smg | ||
| labels: | ||
| - 4-gpu-a10 | ||
| - k8s-runner-gpu | ||
|
Comment on lines
+92
to
+93
Contributor
There was a problem hiding this comment. Similar to the H100 runner, the labels:
- 4-gpu-a10
- k8s-runner-gpu
- runner-deployment-name: arc-runner-gpu-a10 |
||
| serviceAccountName: arc-runner-sa | ||
|
|
||
| nodeSelector: | ||
| nvidia.com/gpu: "true" | ||
| beta.kubernetes.io/instance-type: BM.GPU.A10.4 | ||
|
|
||
| tolerations: | ||
| - key: "nvidia.com/gpu" | ||
| operator: "Equal" | ||
| value: "true" | ||
| effect: "NoSchedule" | ||
|
|
||
| affinity: | ||
| podAffinity: | ||
| preferredDuringSchedulingIgnoredDuringExecution: | ||
| - weight: 100 | ||
| podAffinityTerm: | ||
| labelSelector: | ||
| matchExpressions: | ||
| - key: runner-deployment-name | ||
| operator: In | ||
| values: | ||
| - arc-runner-gpu-a10 | ||
| topologyKey: kubernetes.io/hostname | ||
|
|
||
| volumes: | ||
| - name: model-cache | ||
| persistentVolumeClaim: | ||
| claimName: model-cache | ||
| - name: docker-sock | ||
| emptyDir: {} | ||
| - name: docker-storage | ||
| emptyDir: {} | ||
| - name: dshm | ||
| emptyDir: | ||
| medium: Memory | ||
| sizeLimit: 16Gi | ||
|
|
||
| containers: | ||
| - name: runner | ||
| image: fra.ocir.io/idqj093njucb/action-runner:v0.0.1 | ||
| resources: | ||
| limits: | ||
| nvidia.com/gpu: 4 | ||
| volumeMounts: | ||
| - name: model-cache | ||
| mountPath: /models | ||
| - name: docker-sock | ||
| mountPath: /var/run | ||
| - name: dshm | ||
| mountPath: /dev/shm | ||
| env: | ||
| - name: DOCKER_HOST | ||
| value: unix:///var/run/docker.sock | ||
| - name: docker | ||
| image: fra.ocir.io/idqj093njucb/docker:dind | ||
| securityContext: | ||
| privileged: true # Required for DinD | ||
| env: | ||
| - name: DOCKER_TLS_CERTDIR | ||
| value: "" # Disables TLS for shared socket use | ||
| volumeMounts: | ||
| - name: docker-sock | ||
| mountPath: /var/run | ||
| - name: docker-storage | ||
| mountPath: /var/lib/docker | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| apiVersion: v1 | ||
| kind: ServiceAccount | ||
| metadata: | ||
| name: arc-runner-sa | ||
| namespace: actions-runner-system | ||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: Role | ||
| metadata: | ||
| name: arc-runner | ||
| namespace: actions-runner-system | ||
| rules: | ||
| # Argo Workflows | ||
| - apiGroups: [""] | ||
| resources: | ||
| - secrets | ||
| verbs: | ||
| - get | ||
| - list | ||
| - watch | ||
|
|
||
| # Pods | ||
| - apiGroups: [""] | ||
| resources: | ||
| - pods | ||
| - pods/log | ||
| - pods/exec | ||
| verbs: | ||
|
XinyueZhang369 marked this conversation as resolved.
|
||
| - get | ||
| - list | ||
| - watch | ||
|
XinyueZhang369 marked this conversation as resolved.
|
||
| --- | ||
| apiVersion: rbac.authorization.k8s.io/v1 | ||
| kind: RoleBinding | ||
| metadata: | ||
| name: arc-runner-rb | ||
| namespace: actions-runner-system | ||
| roleRef: | ||
| apiGroup: rbac.authorization.k8s.io | ||
| kind: Role | ||
| name: arc-runner | ||
| subjects: | ||
| - kind: ServiceAccount | ||
| name: arc-runner-sa | ||
| namespace: actions-runner-system | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
podAffinityTermforarc-runner-gpu-h100usesrunner-deployment-namein itslabelSelector. For clarity and explicit control, it would be beneficial to explicitly add this label to thelabelssection of the runner pod template. This ensures that the affinity rule correctly targets pods belonging to this specific runner deployment.