Skip to content

ci: spread verify and Gitleaks across two runner instances - #206

Closed
nish3451 wants to merge 9 commits into
mainfrom
ci/runner-scaleout-tinystudio-io-lane1-20260814-230537
Closed

ci: spread verify and Gitleaks across two runner instances#206
nish3451 wants to merge 9 commits into
mainfrom
ci/runner-scaleout-tinystudio-io-lane1-20260814-230537

Conversation

@nish3451

@nish3451 nish3451 commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Problem

The repo's two required checks (verify, Gitleaks) run on a single self-hosted runner (netcup-rs2000-tinystudio-verify1). Every PR and main push serializes ~70s verify + ~10s Gitleaks on one executor; the documented backlog reached ~60 queued runs at peak. The VPS host is 8-core and already runs the identical hardened runner pattern per repo (0509 runs three instances), so a second instance is the established fleet convention.

Change

  1. Runner scale-out: runs-on → 2-element matrix over per-instance labels tinystudio-io-verify1 / tinystudio-io-verify2 in both workflows. Job names (verify, Gitleaks) unchanged → required-check contexts preserved. Second instance netcup-rs2000-tinystudio-verify2 is already provisioned, online, and executing jobs (verified: Gitleaks runs land on it and complete in ~16s vs ~2min on verify1).
  2. Gitleaks allowlist fix: main's .gitleaks.toml used a top-level [[allowlists]], which gitleaks ≥ 8.19 ignores for rules with their own per-rule allowlists (generic-api-key). The public Cloudflare Web Analytics beacon site tag quoted in the evidence doc was therefore flagged in history on every Secret Scan, leaving the required Gitleaks check red. Move the exemption to a per-rule allowlist (rule redeclared by id only, so upstream regex/entropy stays verbatim). Verified with gitleaks 8.24.3 (action's pinned version): full history clean; same tag shape in any other file still flagged.

Follow-up (out of band, documented in lane report)

Keep the second runner's systemd instance enabled; it already auto-restarts.

Co-authored-by: CommandCodeBot noreply@commandcode.ai

…choke-point)

The repo runs both required checks on a single self-hosted runner
(netcup-rs2000-tinystudio-verify1), so bursts of PRs (~60 queued
verify/Gitleaks runs at peak) drain strictly serially. Route both jobs
through a 2-element runs-on matrix over the per-instance labels
tinystudio-io-verify1/verify2; job names are unchanged so the required
check contexts stay intact. The second runner instance itself is
provisioned out-of-band on the VPS (documented in the lane report).

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@nish3451, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 51 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Free

Run ID: b6db6949-ef2c-4631-94f7-b72eeb7f64ea

📥 Commits

Reviewing files that changed from the base of the PR and between e0ee160 and 2ae144a.

📒 Files selected for processing (4)
  • .github/workflows/ci.yml
  • .github/workflows/secret-scan.yml
  • .gitleaks.toml
  • .lane/reports/ci-runner-scaleout-tinystudio-io-lane1-20260814-230537.md

Note

🎁 Summarized by CodeRabbit Free

Your organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login.

Comment @coderabbitai help to get the list of available commands.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

The top-level [[allowlists]] entry never applied: gitleaks >= 8.19 ignores
top-level allowlists for rules that ship their own per-rule allowlists, and
generic-api-key is one of them. The public Cloudflare Web Analytics beacon
site tag quoted in docs/evidence/web-analytics-beacon-404-2026-08-09.md
(commits 2df7aad/bc6c40f history) was therefore flagged on every Secret
Scan run since those commits landed, leaving the required Gitleaks check
red on main. Redeclare the rule by id only (the [extend] merge keeps the
upstream regex/entropy/keywords verbatim) and attach the exemption as a
per-rule allowlist with the same tight AND scope: one evidence file, one
beacon-tag shape. Verified with gitleaks 8.24.3 (the action's pinned
version): full history clean, and the same tag shape in any other file is
still flagged.

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@nish3451

Copy link
Copy Markdown
Collaborator Author

Superseded by PR #214 (lane-1 re-starvation closeout). This branch's per-instance matrix routes verify/Gitleaks legs to the label 'tinystudio-io-verify1', which no runner registers (verify1's real per-instance label is 'tinystudio-io'; verify2's is 'tinystudio-io-verify2'). Every push to this branch spawned 2 matrix legs that can never be scheduled — 7 of the 9 queued runs (18:00-21:01Z) are this branch, which is exactly the re-starvation the queue reported. Main's shared 'vps-verify' label already engages both runner instances (verified on runs 31846412323/31846416272), so the intended scale-out is live; this matrix is unmergeable by design. Closing as superseded.

@nish3451 nish3451 closed this Aug 14, 2026
nish3451 added a commit that referenced this pull request Aug 14, 2026
…gistered runner label is the queue source (#214)

Main's shared vps-verify label already routes verify/Gitleaks across both
runner instances (netcup-rs2000-tinystudio-verify1/2, live-verified). The
re-starvation backing item 1126's closeout is PR #206's per-instance
matrix: its tinystudio-io-verify1 leg matches no registered runner and
queues forever on every push to that branch (9 queued runs, oldest 2h03m).

Co-authored-by: nish3451 <nish3451@users.noreply.github.com>
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
nish3451 added a commit that referenced this pull request Aug 15, 2026
…ved by PR #214 close-out (#217)

Live queue check (2026-08-14 23:50Z): 11 queued runs all seconds old on
normal PR branches, zero runs on PR #206's branch, both vps-verify runners
online and engaged (main push on verify1, PR #112 on verify2). The item's
2h03m / 9-run observation was PR #206's unregistered tinystudio-io-verify1
matrix label, closed out by PR #214; the queue has not re-starved since.

Co-authored-by: nish3451 <nish3451@users.noreply.github.com>
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
nish3451 added a commit that referenced this pull request Aug 15, 2026
…ved (queue empty at 03:45Z) (#224)

* docs(evidence): re-verify CI queue re-starvation item — already resolved, queue empty at 03:45Z

Live check (2026-08-15 03:45Z): zero queued and zero in-progress runs, both
vps-verify runners online and engaged (recent main push drained in ~16s, the
03:01Z 8-PR batch in <=9m across both runners). The item's 9-run / 2h03m
observation was PR #206's unregistered tinystudio-io-verify1 matrix label,
closed out by PR #214 and re-verified by PR #217; PR #206 is CLOSED, no open
PR touches .github/workflows/*. No repo change required.

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>

* docs(evidence): record 04:01Z wave observation — queue drained, PR #224 CI green

Observed the 04:01Z concurrent 8-run wave live: 4 queued at peak, oldest wait
~14m, fully drained by 04:15:28Z. PR #224's own verify+Gitleaks passed
(verify on netcup-rs2000-tinystudio-verify1, Gitleaks picked up at 04:10Z).
Max wait orders of magnitude below the item's 2h03m; no re-starvation.

Co-authored-by: CommandCodeBot <noreply@commandcode.ai>

---------

Co-authored-by: nish3451 <nish3451@users.noreply.github.com>
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant