Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions scripts/k8s-runner-resources/arc-runner-autoscaler.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,8 @@ spec:
kind: RunnerDeployment
name: arc-runner-2-gpu-h100

minReplicas: 2
maxReplicas: 10
minReplicas: 0
maxReplicas: 0
Comment on lines +83 to +84

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since the 2-gpu-h100 runner tier has been migrated to the new gha-runner-scale-set (using runner-values-2-gpu-h100.yaml), the legacy summerwind RunnerDeployment (arc-runner-2-gpu-h100 in arc-runner-gpu.yaml) and this HorizontalRunnerAutoscaler are now obsolete.

Instead of just scaling them to 0, consider completely removing the arc-runner-2-gpu-h100 deployment and autoscaler blocks from the codebase to clean up dead configuration and prevent any accidental scale-ups in the future.


metrics:
- type: TotalNumberOfQueuedAndInProgressWorkflowRuns
Expand Down
71 changes: 63 additions & 8 deletions scripts/k8s-runner-resources/runner-values-2-gpu-h100.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ githubConfigUrl: "https://github.com/lightseekorg/smg"
githubConfigSecret: github-arc-secret

## Runner scaling configuration
minRunners: 0
maxRunners: 6
minRunners: 2
maxRunners: 8

## name of the runner scale set to create. Defaults to the helm release name
runnerScaleSetName: "2-gpu-h100"
Expand All @@ -20,6 +20,14 @@ template:
spec:
terminationGracePeriodSeconds: 600

# hostNetwork puts the runner in the host netns so PD disaggregation KV
# transfer (NIXL/Mooncake) sees the host's populated RoCE GID tables. An
# overlay-CNI pod sees the mlx5 NICs as PORT_ACTIVE but with an empty GID
# table, so RDMA init dies at worker startup. ClusterFirstWithHostNet keeps
# cluster DNS working while on the host network.
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

nodeSelector:
nvidia.com/gpu: "true"
node.kubernetes.io/instance-type: BM.GPU.H100.8
Expand All @@ -30,8 +38,11 @@ template:
value: "true"
effect: "NoSchedule"

# With hostNetwork the pod shares the host port space, so spread 2-gpu PD
# runners one-per-node to avoid host-port collisions between concurrent PD
# jobs (replaces the previous GPU bin-packing podAffinity).
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
Comment on lines 44 to 46

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: preferredDuringSchedulingIgnoredDuringExecution is a soft constraint — the scheduler can still co-locate two 2-gpu runners on the same node when eligible nodes are scarce. With hostNetwork: true and maxRunners: 8, if the BM.GPU.H100.8 pool has fewer than 8 nodes available, co-scheduling will happen and the host-port collisions the comment warns about become real.

Consider using requiredDuringSchedulingIgnoredDuringExecution instead to make the one-runner-per-node guarantee hard, so the scheduler fails loudly (pod stays Pending) rather than silently causing port conflicts at runtime.

Comment on lines +45 to 46

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Enforce one host-network runner per node

This new anti-affinity is only a scheduler preference, so when the scale set grows beyond the number of readily available H100 nodes (or the scheduler prioritizes satisfying the 2-GPU request), Kubernetes may still place multiple hostNetwork 2-GPU runners on the same host. In that case their DinD/PD jobs share the node network namespace and can hit the host-port collisions this block is intended to prevent; use required anti-affinity or otherwise partition the host ports if co-location must be impossible.

Useful? React with 👍 / 👎.

- weight: 100
podAffinityTerm:
Expand All @@ -40,7 +51,6 @@ template:
- key: actions.github.com/scale-set-name
operator: In
values:
- 1-gpu-h100
- 2-gpu-h100
topologyKey: kubernetes.io/hostname

Expand All @@ -63,18 +73,51 @@ template:

containers:
- name: runner
image: fra.ocir.io/idqj093njucb/action-runner:v0.0.2
image: fra.ocir.io/idqj093njucb/action-runner:v0.0.3
command:
- /home/runner/run.sh
securityContext:
# The runner runs as non-root (uid 1001), so capabilities.add:
# [IPC_LOCK] only reaches the bounding set (not effective) and the
# default 64KiB RLIMIT_MEMLOCK blocks RDMA completion-queue
# allocation (ibv_create_cq -> ENOMEM, e.g. sglang/Mooncake).
# privileged makes IPC_LOCK effective and lifts the memlock cap.
privileged: true
Comment on lines +79 to +85

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

Avoid making the runner container itself privileged.

This is broader than the existing privileged dind sidecar: every workflow step scheduled onto this scale set now runs inside a privileged, host-network container, which is effectively node-level access. If the real requirement is memlock/RDMA setup, keep that privilege in the narrowest possible helper/runtime path or otherwise treat this label as trusted-only; leaving the main runner privileged materially weakens the runner isolation model.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/k8s-runner-resources/runner-values-2-gpu-h100.yaml` around lines 79 -
85, The runner container is marked privileged (securityContext.privileged: true)
which grants node-level access; remove privileged: true from the runner's
securityContext and instead confine elevated privileges to a narrow helper: add
a privileged initContainer or privileged sidecar (e.g., the existing dind
helper) that performs the RDMA/memlock setup, or grant only the IPC_LOCK
capability via securityContext.capabilities.add on that helper container and
raise RLIMIT_MEMLOCK there; update references to the runner pod spec
(securityContext) and any init/sidecar container definitions so only the helper
has privileged or IPC_LOCK rights while the main runner remains non-privileged.

env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
- name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
value: '120'
- name: HF_HOME
value: /models
- name: HF_TOKEN # required: ci_download_model.sh runs `hf download` for the gated Llama model
valueFrom:
secretKeyRef:
key: HUGGINGFACE_API_KEY
name: huggingface-secret
- name: OPENAI_API_KEY
valueFrom:
secretKeyRef:
key: OPENAI_API_KEY
name: openai-api-key
- name: ANTHROPIC_API_KEY
valueFrom:
secretKeyRef:
key: ANTHROPIC_API_KEY
name: anthropic-api-key
- name: XAI_API_KEY
valueFrom:
secretKeyRef:
key: XAI_API_KEY
name: xai-api-key
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
resources:
requests:
cpu: "16"
memory: "64Gi"
cpu: '32'
memory: 128Gi
limits:
nvidia.com/gpu: 2
volumeMounts:
Expand All @@ -97,7 +140,7 @@ template:
- /home/runner/tmpDir/
command:
- cp
image: fra.ocir.io/idqj093njucb/action-runner:v0.0.2
image: fra.ocir.io/idqj093njucb/action-runner:v0.0.3
volumeMounts:
- mountPath: /home/runner/tmpDir
name: dind-externals
Expand All @@ -109,10 +152,22 @@ template:
env:
- name: DOCKER_GROUP_GID
value: '123'
# Suppress dind's default TLS TCP listener (:2376). dockerd only needs
# the unix socket here; under hostNetwork a TCP listener would bind a
# node port unnecessarily.
- name: DOCKER_TLS_CERTDIR
value: ""
image: fra.ocir.io/idqj093njucb/docker:dind
restartPolicy: Always
securityContext:
privileged: true
resources:
limits:
cpu: '2'
memory: 4Gi
requests:
cpu: '1'
memory: 2Gi
Comment on lines +164 to +170

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The dind container is configured with a memory limit of 4Gi and CPU limit of 2.

In a GitHub Actions runner using Docker-in-Docker (DinD), any workflow steps that run inside Docker containers (e.g., docker run ..., docker build ..., or action steps using docker images) are executed by the dockerd daemon inside this dind container.

Because of this, those steps will be constrained by the dind container's resource limits (2 CPUs and 4Gi memory), rather than the runner container's larger allocation (32 CPUs and 128Gi memory). If your CI workflows run heavy ML workloads or large docker builds inside Docker, they may experience severe CPU throttling or OOM kills.

Consider increasing these limits or removing them to allow dind to utilize the pod's full resource capacity when needed.

Additionally, when using a Docker-in-Docker (DinD) setup, it is necessary to disable TLS for the Docker socket by setting the DOCKER_TLS_CERTDIR environment variable to an empty string.

References
  1. When using a Docker-in-Docker (DinD) setup, it is necessary to disable TLS for the Docker socket by setting the DOCKER_TLS_CERTDIR environment variable to an empty string.

startupProbe:
exec:
command:
Expand Down