feat(runners): add hotfix runner lane with subprocess timeout - #835
Conversation
Add a dedicated hotfix lane (gha-runner-hotfix) with 2 CPU / 4GB memory defaults for fast-turnaround CI jobs. Also adds timeout parameter to run_cmd() (default 120s, 15s for docker info) to prevent hangs on Docker/gh CLI failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThis change adds a new "hotfix" runner lane to the local certificate runner configuration system, introduces timeout parameter support to command execution with a 120-second default, and extends resource definitions for the new lane with corresponding CPU, memory, and GPU settings. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@pmoves/tools/local_cert_runners.py`:
- Around line 49-57: run_cmd currently lets subprocess.TimeoutExpired bubble up
(breaking callers that rely on check=False to get a CompletedProcess); wrap the
subprocess.run call in a try/except for subprocess.TimeoutExpired, and if a
timeout occurs: re-raise the exception only when check is True, otherwise return
a subprocess.CompletedProcess instance representing a failed run (non-zero
returncode, empty or exception-derived stdout/stderr) so callers like
_runner_log_args and docker_rm can gracefully fall back.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: aaf6ba9a-e1bf-46ae-8387-d6ca05072080
📒 Files selected for processing (1)
pmoves/tools/local_cert_runners.py
subprocess.TimeoutExpired bypasses check=False — catch it explicitly and return a synthetic CompletedProcess(returncode=-1) so callers like docker_rm and _runner_log_args degrade gracefully instead of crashing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- AB-9 (runner queue starvation) RESOLVED: 3/4 runners online, CI queue healthy, CodeQL completing in ~4min - PRs #834/#835 merge tracking added - Dependabot: 0 open (medium alert resolved) - Trivy failure triage: agent-zero timeout (infra), archon/deepresearch upstream dep pins needed, pmoves-yt urllib3 quick fix - Docker Bench Security unblocked by AB-9 resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update date from 2026-02-26 to 2026-03-09 - Re-prioritize 14 open items into 3 tiers: 4 production-blocking, 6 tracked improvements, 5 cosmetic - Add "Blocks Production?" column with rationale per item - Confirm no P2s fixed by PRs #827-#835 (CI/docs/build-gate only) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…836) * docs(audit): resolve AB-9, update dashboard with Mar 9 findings - AB-9 (runner queue starvation) RESOLVED: 3/4 runners online, CI queue healthy, CodeQL completing in ~4min - PRs #834/#835 merge tracking added - Dependabot: 0 open (medium alert resolved) - Trivy failure triage: agent-zero timeout (infra), archon/deepresearch upstream dep pins needed, pmoves-yt urllib3 quick fix - Docker Bench Security unblocked by AB-9 resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs(security): refresh P2 tracker with tiered prioritization - Update date from 2026-02-26 to 2026-03-09 - Re-prioritize 14 open items into 3 tiers: 4 production-blocking, 6 tracked improvements, 5 cosmetic - Add "Blocks Production?" column with rationale per item - Confirm no P2s fixed by PRs #827-#835 (CI/docs/build-gate only) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(docs): correct P2 open-item count from 14 to 15 Items #1-#14 and #16 are open (#15 is closed), totaling 15. Fixes count in both P2 tracker and dashboard. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Shaela Bello <slbello@uncg.edu> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: self-hosted runners were installed as bare-metal services (WSL2 systemd, Windows svc.cmd) that stopped and had no auto-recovery. The local-certification phase policy was always designed for "Both runners on local Docker containers" but was never implemented. Resolution: - Started Docker-based runners via existing local_cert_runners.py (make ci-runners-local-cert-up) - Updated lane_hosts.json to reflect containerized topology - Updated runner_phase_policy.json to match actual workflow label sets (self-hosted,Linux,X64,ai-lab,gpu instead of self-hosted,ai-lab,gpu) - Dashboard updated: AB-9 RESOLVED, CI queue HEALTHY (3/4 online) Timeline of missed fixes: - PRs #832/#834/#835: Added CI throttle timeouts without addressing root - PR #842: Captured 0/4 runners, noted AB-9 REGRESSED - This fix: Discovered local_cert_runners.py already had full Docker runner management — just needed to be executed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Summary
gha-runner-hotfix) with 2 CPU / 4GB memory defaults for fast-turnaround CI jobstimeoutparameter torun_cmd()(default 120s) to prevent hangs on Docker/gh CLI failurestimeout=15fordocker infocalls specificallyTest plan
py -3 -c "import ast; ast.parse(open('pmoves/tools/local_cert_runners.py').read())"— syntax OKpy -3 pmoves/tools/local_cert_runners.py status— verify hotfix lane appears in outputpy -3 pmoves/tools/local_cert_runners.py up --lane hotfix— verify hotfix container starts🤖 Generated with Claude Code
Summary by CodeRabbit