fix(ci): enable ephemeral runners to prevent dead runner accumulation

[slin1237]

Summary

Set ephemeral: true on all ARC RunnerDeployment specs to fix dead runner accumulation in GitHub.

What changed

• scripts/k8s-runner-resources/arc-runner-cpu.yaml — added ephemeral: true
• scripts/k8s-runner-resources/arc-runner-gpu.yaml — added ephemeral: true to both H100 and A10 RunnerDeployments

Why

Without ephemeral mode, runner pods that get killed, evicted, or scaled down never deregister from GitHub, leaving 120+ ghost "offline" runner entries. Each CI job spawns a new pod, but the old registration persists.

How

With ephemeral: true, each runner picks up exactly one job, self-deregisters from GitHub on completion, then the pod terminates. The ARC controller replaces it to maintain minReplicas. This is the recommended mode for ARC with autoscaling.

Test plan

• Manually cleaned up 120 dead offline runners via gh api
• Applied updated configs to k8s cluster (kubectl apply)
• Verify new runner pods register, pick up a job, and deregister cleanly
• Verify no new ghost runners accumulate after CI runs

Summary by CodeRabbit

• Chores
  • Configured CPU and GPU runners to operate as ephemeral, enabling automatic cleanup after execution.

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a crucial change to the CI/CD infrastructure by configuring GitHub Actions runners to operate in ephemeral mode. This modification ensures that each runner pod processes a single job and then gracefully deregisters itself, preventing the accumulation of stale and inactive runner entries within GitHub. The primary impact is a cleaner and more efficient runner management system, reducing operational overhead and improving the reliability of CI workflows.

Highlights

• Ephemeral Runners Enabled: The ephemeral: true setting has been added to all ARC RunnerDeployment specifications for both CPU and GPU runners.
• Prevents Dead Runner Accumulation: This change addresses the issue of ghost 'offline' runner entries accumulating in GitHub by ensuring runners deregister after completing a single job.

Changelog

• scripts/k8s-runner-resources/arc-runner-cpu.yaml
  • Added ephemeral: true to the runner deployment specification.
• scripts/k8s-runner-resources/arc-runner-gpu.yaml
  • Added ephemeral: true to the H100 GPU runner deployment specification.
  • Added ephemeral: true to the A10 GPU runner deployment specification.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

---

📝 Walkthrough

Walkthrough

This pull request adds the ephemeral: true flag to RunnerDeployment specifications in two Kubernetes manifest files. The changes mark runners as ephemeral in both CPU and GPU runner configurations, affecting three deployment instances across the files.

Changes

Cohort / File(s)	Summary
Kubernetes Runner Manifests scripts/k8s-runner-resources/arc-runner-cpu.yaml, scripts/k8s-runner-resources/arc-runner-gpu.yaml	Added ephemeral: true flag to spec.template in RunnerDeployment configurations. CPU runner has 1 addition; GPU runners (h100 and a10) have 2 additions.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Possibly related issues

• ci: migrate from summerwind ARC to GitHub-maintained ARC v2 #432: Addresses ARC ephemeral runner handling by marking runners as ephemeral in the deployment configurations.

Possibly related PRs

• fix(ci): Use H100 runner for Specific Tests #357: Previously modified the same RunnerDeployment manifests under scripts/k8s-runner-resources.

Suggested labels

ci

Suggested reviewers

• CatherineSue
• XinyueZhang369

Poem

🐰 Ephemeral runners now take flight,
With flags set true, they're trimmed just right,
CPU and GPU pods so fleeting,
In Kubernetes gardens, quite a neat greeting! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: enabling ephemeral runners to fix dead runner accumulation. It is specific, concise, and directly reflects the purpose of the configuration updates.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch slin/benchmark-fix

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

The pull request correctly enables ephemeral: true for all RunnerDeployment resources (arc-runner-cpu, arc-runner-gpu-h100, and arc-runner-gpu-a10). This is the recommended configuration for the Actions Runner Controller (ARC) when using autoscaling or to prevent the accumulation of "offline" ghost runners in GitHub, as it ensures each runner pod is cleaned up and deregistered after completing a single job. The changes are consistent across all configuration files and align with the existing autoscaler settings.