Skip to content

feat(runners): add hotfix runner lane with subprocess timeout - #835

Merged
POWERFULMOVES merged 2 commits into
mainfrom
feat/hotfix-runner-lane
Mar 9, 2026
Merged

feat(runners): add hotfix runner lane with subprocess timeout#835
POWERFULMOVES merged 2 commits into
mainfrom
feat/hotfix-runner-lane

Conversation

@POWERFULMOVES

@POWERFULMOVES POWERFULMOVES commented Mar 9, 2026

Copy link
Copy Markdown
Owner

Summary

  • Adds dedicated hotfix runner lane (gha-runner-hotfix) with 2 CPU / 4GB memory defaults for fast-turnaround CI jobs
  • Adds timeout parameter to run_cmd() (default 120s) to prevent hangs on Docker/gh CLI failures
  • Uses timeout=15 for docker info calls specifically

Test plan

  • py -3 -c "import ast; ast.parse(open('pmoves/tools/local_cert_runners.py').read())" — syntax OK
  • py -3 pmoves/tools/local_cert_runners.py status — verify hotfix lane appears in output
  • py -3 pmoves/tools/local_cert_runners.py up --lane hotfix — verify hotfix container starts

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a new "hotfix" runner lane with dedicated container and resource configuration.
    • Implemented timeout support for command execution with configurable timeout values.

Add a dedicated hotfix lane (gha-runner-hotfix) with 2 CPU / 4GB memory
defaults for fast-turnaround CI jobs.

Also adds timeout parameter to run_cmd() (default 120s, 15s for
docker info) to prevent hangs on Docker/gh CLI failures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Warning

Rate limit exceeded

@POWERFULMOVES has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 19 minutes and 4 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b6faba98-f7ae-4be8-9ca9-4e161457062f

📥 Commits

Reviewing files that changed from the base of the PR and between 3b0ebcf and 5e08232.

📒 Files selected for processing (1)
  • pmoves/tools/local_cert_runners.py
📝 Walkthrough

Walkthrough

This change adds a new "hotfix" runner lane to the local certificate runner configuration system, introduces timeout parameter support to command execution with a 120-second default, and extends resource definitions for the new lane with corresponding CPU, memory, and GPU settings.

Changes

Cohort / File(s) Summary
Hotfix Lane & Timeout Support
pmoves/tools/local_cert_runners.py
Added "hotfix" runner lane to LANES configuration with Docker container and runner name; introduced timeout parameter (default 120s) to run_cmd function; applied 15s timeout to Docker info query in _runner_log_args; extended LANE_RESOURCES with "hotfix" CPU/memory/GPU settings; updated docstring to reflect new lane.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 A hotfix lane now hops along,
With timeouts set to keep us strong,
Fifteen seconds for Docker's call,
One-twenty seconds to cover all,
Swift as rabbits, code runs clean! ✨

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description includes a summary of changes and test plan, but is missing critical required sections: Testing command output/results, CHIT Contract Check acknowledgment, and documentation updates acknowledgment. Complete the Testing section with actual command outputs, check the required checkboxes (CHIT Contract Check, contracts/schemas update, and documentation), and add any reviewer notes if needed.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the two main changes: adding a hotfix runner lane and introducing subprocess timeout support.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/hotfix-runner-lane

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pmoves/tools/local_cert_runners.py`:
- Around line 49-57: run_cmd currently lets subprocess.TimeoutExpired bubble up
(breaking callers that rely on check=False to get a CompletedProcess); wrap the
subprocess.run call in a try/except for subprocess.TimeoutExpired, and if a
timeout occurs: re-raise the exception only when check is True, otherwise return
a subprocess.CompletedProcess instance representing a failed run (non-zero
returncode, empty or exception-derived stdout/stderr) so callers like
_runner_log_args and docker_rm can gracefully fall back.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: aaf6ba9a-e1bf-46ae-8387-d6ca05072080

📥 Commits

Reviewing files that changed from the base of the PR and between 11687a4 and 3b0ebcf.

📒 Files selected for processing (1)
  • pmoves/tools/local_cert_runners.py

Comment thread pmoves/tools/local_cert_runners.py Outdated
subprocess.TimeoutExpired bypasses check=False — catch it explicitly
and return a synthetic CompletedProcess(returncode=-1) so callers
like docker_rm and _runner_log_args degrade gracefully instead of
crashing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@POWERFULMOVES
POWERFULMOVES merged commit ff9f5da into main Mar 9, 2026
6 checks passed
POWERFULMOVES pushed a commit that referenced this pull request Mar 9, 2026
- AB-9 (runner queue starvation) RESOLVED: 3/4 runners online,
  CI queue healthy, CodeQL completing in ~4min
- PRs #834/#835 merge tracking added
- Dependabot: 0 open (medium alert resolved)
- Trivy failure triage: agent-zero timeout (infra), archon/deepresearch
  upstream dep pins needed, pmoves-yt urllib3 quick fix
- Docker Bench Security unblocked by AB-9 resolution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POWERFULMOVES pushed a commit that referenced this pull request Mar 9, 2026
- Update date from 2026-02-26 to 2026-03-09
- Re-prioritize 14 open items into 3 tiers:
  4 production-blocking, 6 tracked improvements, 5 cosmetic
- Add "Blocks Production?" column with rationale per item
- Confirm no P2s fixed by PRs #827-#835 (CI/docs/build-gate only)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
POWERFULMOVES added a commit that referenced this pull request Mar 9, 2026
…836)

* docs(audit): resolve AB-9, update dashboard with Mar 9 findings

- AB-9 (runner queue starvation) RESOLVED: 3/4 runners online,
  CI queue healthy, CodeQL completing in ~4min
- PRs #834/#835 merge tracking added
- Dependabot: 0 open (medium alert resolved)
- Trivy failure triage: agent-zero timeout (infra), archon/deepresearch
  upstream dep pins needed, pmoves-yt urllib3 quick fix
- Docker Bench Security unblocked by AB-9 resolution

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(security): refresh P2 tracker with tiered prioritization

- Update date from 2026-02-26 to 2026-03-09
- Re-prioritize 14 open items into 3 tiers:
  4 production-blocking, 6 tracked improvements, 5 cosmetic
- Add "Blocks Production?" column with rationale per item
- Confirm no P2s fixed by PRs #827-#835 (CI/docs/build-gate only)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docs): correct P2 open-item count from 14 to 15

Items #1-#14 and #16 are open (#15 is closed), totaling 15.
Fixes count in both P2 tracker and dashboard.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Shaela Bello <slbello@uncg.edu>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@POWERFULMOVES
POWERFULMOVES deleted the feat/hotfix-runner-lane branch March 9, 2026 20:57
POWERFULMOVES pushed a commit that referenced this pull request Mar 10, 2026
Root cause: self-hosted runners were installed as bare-metal services
(WSL2 systemd, Windows svc.cmd) that stopped and had no auto-recovery.
The local-certification phase policy was always designed for "Both
runners on local Docker containers" but was never implemented.

Resolution:
- Started Docker-based runners via existing local_cert_runners.py
  (make ci-runners-local-cert-up)
- Updated lane_hosts.json to reflect containerized topology
- Updated runner_phase_policy.json to match actual workflow label sets
  (self-hosted,Linux,X64,ai-lab,gpu instead of self-hosted,ai-lab,gpu)
- Dashboard updated: AB-9 RESOLVED, CI queue HEALTHY (3/4 online)

Timeline of missed fixes:
- PRs #832/#834/#835: Added CI throttle timeouts without addressing root
- PR #842: Captured 0/4 runners, noted AB-9 REGRESSED
- This fix: Discovered local_cert_runners.py already had full Docker
  runner management — just needed to be executed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants