Skip to content

ci: split the release-green sweep so a hosted runner can finish it - #87

Merged
LMPrado-DZ23 merged 4 commits into
release/v3.8.55from
ci/shard-release-green
Sep 20, 2026
Merged

LMPrado-DZ23 merged 4 commits into
release/v3.8.55from
ci/shard-release-green

Conversation

@LMPrado-DZ23

Copy link
Copy Markdown
Owner

Four consecutive full sweeps died at exit 143, and no full-CI verdict exists for release/v3.8.55. #76 blamed a missing timeout-minutes; #86 records the measurement that disproved that. This is the fix.

What is actually wrong

07:17:26  job starts
07:23:23  ✅ every static and drift gate is green
          ▶ the four serial suites start
08:17:37  ##[error] Process completed with exit code 143   ← 60m11s

A runner on this repository stops at ~60 minutes — run 35468579833 said so in words: "The runner has received a shutdown signal". The suites' own ceilings are 80 + 15 + 40 + 20 = 155 minutes serial. One job cannot carry that, and no timeout-minutes value changes it.

The shape

Job Does
resolve resolves the release branch once and outputs the exact SHA
slow-suite (×7) unit ×4, integration ×2, vitest — each ≤55min
release-green static + drift + full-ci + package artifact + boot smoke, then merges the shard reports and owns the verdict

Every job checks out the SHA resolve produced, so the merged report belongs to one commit instead of to whatever each job happened to fetch at its own start time.

The shards measure; they do not judge. A red suite still exits 0 so its report reaches the aggregator — and the aggregator runs on !cancelled(), not success(), so a shard that died still gets its verdict pronounced.

The part with teeth

Splitting a verdict across seven jobs creates exactly one new way to be wrong: a job reporting green while measuring nothing. That is the same defect class I have already fixed three times on this branch, so it is guarded up front, not afterwards.

  • --expect-slow names every shard id the matrix produces — not just the suite names. With a bare unit, one surviving shard would satisfy the expectation and the other three could vanish. A suite with no report is recorded as a HARD failure reading "it did not run, so it is NOT green."
  • --slow-gates=<typo> throws instead of selecting nothing.
  • A run that records zero gates is a HARD failure for that reason alone.
  • A test derives the expected ids from the matrix and compares the two. It fails closed: a stale list goes red rather than quiet.

Verification

Check Result
tests/unit/validate-release-green.test.ts 47/47
red-first: drop 2 unit shards from the expect list test 46 fails, as designed
--no-static --slow-gates=none ❌ ran zero gates — nothing was measured, so nothing is green
--slow-gates=unit-tests refuses to start: unknown suite 'unit-tests'
--merge-slow without --expect-slow refuses to start
merge smoke: 2 unit shards green, integration absent ❌ NOT release-green
YAML parses, all budgets < 60 ✓
prettier --check clean

I am dispatching the sweep against this branch to produce the verdict itself; I will post the result here rather than claim it in advance.

Honestly still open

main-green carries the same 155 minutes in one job and will keep dying at 60 minutes. It validates the released state rather than this release candidate, so it does not block this line — but it is the same fix and it is not done.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Sep 20, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 2972d5fa-7b91-4d00-85e9-8776053c3caf


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

LMPrado-DZ23 pushed a commit that referenced this pull request Sep 20, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
zodyprado-web and others added 2 commits September 20, 2026 05:48
Four consecutive full sweeps died at exit 143 and none of them produced a
verdict for release/v3.8.55. #76 blamed a missing `timeout-minutes`; #86
records the measurement that disproved it. The real shape:

  07:17:26  job starts
  07:19:01  ✅ Typecheck   07:22:03 ✅ ESLint   07:22:39 ✅ complexity
  07:23:23  ✅ Docs sync + fabricated-docs — every static gate green
            ▶ four serial suites start
  08:17:37  ##[error] exit 143        ← 60m11s

A runner on this repository stops at ~60 minutes ("The runner has received a
shutdown signal", run 35468579833). The suites' own ceilings are 80 + 15 + 40
+ 20 = 155 minutes serial. No single job can carry that, and no value of
`timeout-minutes` changes it. So the sweep stops being one job.

  resolve      → resolves the branch ONCE and outputs the exact SHA
  slow-suite   → 7 parallel jobs: unit ×4, integration ×2, vitest
  release-green → static + drift + full-ci + pack + boot, then MERGES their
                  reports and owns the verdict

Every job checks out the SHA `resolve` produced, so the merged report belongs
to one commit rather than to whatever each job happened to fetch.

The shards measure; they do not judge. A red suite still exits 0 so its report
reaches the aggregator — and the aggregator runs on `!cancelled()`, not
`success()`, so a shard that DIED still gets its verdict pronounced.

That last part is the whole risk of this shape, so it is the part with teeth:

  --expect-slow names every shard id the matrix produces. A suite with no
  report is recorded as a HARD failure reading "it did not run, so it is NOT
  green". Not a warning, not an absence.
  --slow-gates=<typo> throws instead of selecting nothing (a job that runs
  zero tests must not report green).
  A run that records zero gates is a HARD failure for that reason alone.
  A test derives the expected ids FROM the matrix and compares; it fails
  closed, so a stale list goes red rather than quiet.

This is the same defect class I have now fixed three times in this branch —
a gate reporting green while measuring nothing — so it is guarded before it
can happen rather than after.

  47/47 tests/unit/validate-release-green.test.ts
  red-first: dropping two unit shards from the expect list fails test 46
  --no-static --slow-gates=none → ❌ "ran zero gates"
  --slow-gates=unit-tests       → refuses to start
  --merge-slow without --expect-slow → refuses to start
  merge smoke: 2 unit shards green + integration absent → ❌ NOT release-green
  YAML parses; prettier clean

Still open, and not touched here: `main-green` carries the same 155 minutes in
one job and will keep dying at 60 minutes. It validates the released state,
not the release candidate, so it does not block this line — but it is the same
fix, and it is not done.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment on main-green was left describing sharding as a future
option, which is no longer true of the release path above — it IS
sharded now — and it still carried `timeout-minutes: 180`, the number
run 35496465106 disproved.

main-green is knowingly NOT fixed. It is off by default
(VALIDATE_MAIN_BRANCH) and this fork ships from release/v* and never
merges into main, so main is a stale upstream snapshot nobody here
releases. Splitting it would duplicate ~80 lines of matrix for a job
that does not run.

What changes is the failure mode. 55 minutes rather than 180 means
whoever enables it gets a stated timeout with the log, instead of the
opaque exit 143 that four consecutive full sweeps died of — and the
comment now names the fix (the slow-suite matrix above) instead of
describing it as unreachable.

The budget test covers all three jobs now, main-green included.

  47/47 tests/unit/validate-release-green.test.ts
  YAML parses, main-green timeout: 55; prettier clean

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LMPrado-DZ23

Copy link
Copy Markdown
Owner Author

Pushed one more commit: main-green.

I said in the PR description that it was still broken and untouched. It is still broken, but leaving a comment there that described sharding as an unreachable future option — when the job right above it is now sharded — would have been the same kind of stale explanation #86 had to correct.

So it now says plainly that it cannot finish, why it is knowingly left that way (off by default; this fork ships from release/v* and never merges into main, so main is a stale upstream snapshot), and what the fix is when main starts to matter — the slow-suite matrix above.

The one real change: timeout-minutes 180 → 55. It will still fail if someone enables it, but it will fail saying so, with its log, instead of at an opaque exit 143. The budget test now covers all three jobs.

47/47 tests, YAML parses, prettier clean.

The sweep's new shape is only usable if the flags that split it are
written down next to the ones they extend — and the three refusals that
keep a split verdict honest (--expect-slow, unknown suite, zero gates)
belong with them, not only in the script's header.

  [doc-links] PASS — 172 docs, 1044 internal links
  fabricated-claim gate: clean

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LMPrado-DZ23
LMPrado-DZ23 merged commit a67c36f into release/v3.8.55 Sep 20, 2026
15 checks passed
@LMPrado-DZ23
LMPrado-DZ23 deleted the ci/shard-release-green branch September 20, 2026 09:14
LMPrado-DZ23 added a commit that referenced this pull request Sep 20, 2026
…l reds (#100)

A-H1 said the release-green sweep could not produce a verdict, and I
wrote that the two ways out were "neither reachable by editing a
workflow". One of them was. #87 split the sweep — resolve → seven
slow-suite jobs → an aggregator that merges their reports — and this
line now has the full-CI verdict it never had.

What that verdict found is the point of having had it:

  · api-routes-critical.test.ts was making a LIVE HTTPS request to
    aihorde.net on every run
  · api-keys.test.ts was a false positive of the network guard I wrote
    in #56 — cloud.example is RFC-reserved and resolves nowhere

Both fixed in #96; the second sweep passed all seven shards.

Also recorded, because it is the honest remainder: the aggregator passes
every static and drift gate and then dies at check:pack-artifact, six
minutes of silence and exit 143, in both sweeps. That gate falls back to
a full `next build` and the hosted runner cannot fit this tree —
build.yml has been manual-only since diegosouzapw#11946 for exactly that reason.
#99 stops it discarding thirteen green gates and seven green suites on
the way out, by recording the gate as unmeasured instead.

A fully green verdict needs USE_VPS_RUNNER with that runner online.
That is an external dependency and the owner's call, not pending work,
and the document now says so rather than leaving a HIGH that reads like
something I still owe.

  [doc-links] PASS — 172 docs, 1044 internal links

Co-authored-by: zodyp <zodyprado@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants