Skip to content

fix(quality): report the real failure line and stop double-counting ci.yml gates - #11321

Merged
diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.50from
pacocartones:fix/release-green-verdict-accuracy
Aug 24, 2026
Merged

diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.50from
pacocartones:fix/release-green-verdict-accuracy

Conversation

@pacocartones

Copy link
Copy Markdown
Contributor

What

Two accuracy bugs in scripts/quality/validate-release-green.mjs, both found while reading the release-green verdict posted on #9985.

1. The verdict blamed a line that had passed

firstFailureLine() picked the first line matching /✖|✗|not ok|AssertionError|error TS|FAIL|Error:|REGRESS/i. That regex is unanchored and case-insensitive, so FAIL matches the fail inside a test file name. The 2026-08-23 verdict reported this as the cause of the unit red:

✓ tests/unit/runtime/fail-fast-concurrency-gate.test.ts (4 tests) 203ms

That is a passing line. The gate had failed elsewhere, so whoever picked up the red was pointed at a green test. Same class of miss on the summary line Test Files 1 failed, which matches before the real FAIL <file> line ever gets a look.

Markers that also occur inside file names or prose (FAIL, not ok, ✖, REGRESS) are now anchored to the start of the line, lines starting with ✓/✔/√ are excluded outright, and markers that tools legitimately emit mid-line (error TS1234, AssertionError, Error:, REGRESS) stay unanchored so nothing that used to be caught gets lost.

2. The same gate reported as both a failure and as drift

--full-ci records every gate it extracts from ci.yml as kind: "hard", and the dedupe just above compares raw ids. The ci.yml id is the npm script name (check:file-size) while the curated id is not (file-size), so the dedupe never fires and the gate is recorded twice under two contradictory classifications. The verdict then prints, in one table:

HARD failures (block — real defects): …
  ❌ ci.yml:quality-gate → npm run check:file-size: …
  ❌ ci.yml:quality-gate → npm run check:compression-budget: …
Ratchet drift (non-blocking — rebaseline at release): …
  🟡 File-size ratchet: …
  🟡 Compression budget (ratchet): …

Same gate, same run, two verdicts. It affects six: file-size, compression-budget, dead-code, type-coverage, openapi-coverage and check:workflows/workflow-lint.

fullCiKindFor() now resolves the ci.yml script id to its curated equivalent (strip check:, plus a small alias map for the three that don't follow that shape) and reuses the curated kind when the curated pass already ran the gate. Anything the curated list does not cover still defaults to hard — that is the point of --full-ci and it is unchanged. check:bundle-size, for instance, has no curated counterpart, so it stays hard, and there is a test pinning that.

What this is not

This does not change which gates run, what any gate does, or what counts as a gate failure. No thresholds, no baselines, no gate list. It changes what the report says about a result it already had.

One consequence worth naming rather than burying: a gate the script already classifies as drift no longer flips the exit code through the --full-ci path. That restores the policy the file documents at the top — "Drift NEVER changes the exit code, so wiring this as a check can never block anyone on drift" — which --full-ci was silently overriding. If you would rather those ratchets block, the fix is a one-line change to the default in fullCiKindFor() and I am happy to flip it.

Tests

tests/unit/validate-release-green.test.ts already existed with 23 tests. It was green on release/v3.8.50 before this change and is green after. Seven added:

  • the real ✓ …fail-fast-concurrency-gate.test.ts line from the 🔴 Release branch not green: release/v3.8.50 #9985 verdict is never reported as a cause, and the real failing line is
  • every legitimate marker still resolves (not ok, FAIL <file>, error TS2322, ✗, Error:, REGRESSÃO, REGRESSED), including the pt-BR ratchet output
  • the last-line fallback is unchanged
  • curatedEquivalentId / fullCiKindFor mapping, aliases, and the hard default for uncurated gates
  • a verdict-level test that no gate can appear in both buckets of one report
ℹ tests 30
ℹ pass 30
ℹ fail 0

…i.yml gates

Two verdict-accuracy bugs found reviewing the release-green report of diegosouzapw#9985.

1. firstFailureLine() matched /FAIL/i unanchored, so it hit the substring
   "fail" inside a test FILE NAME and reported a PASSING line as the cause of
   the red (2026-08-23 verdict blamed
   "OK tests/.../fail-fast-concurrency-gate.test.ts (4 tests) 203ms").
   Markers that also occur inside file names or summary prose are now anchored
   to the start of the line and success lines are excluded; markers that tools
   legitimately emit mid-line (error TSxxxx, AssertionError, Error:, REGRESS)
   stay unanchored.

2. The --full-ci pass recorded every gate extracted from ci.yml as kind:"hard"
   while the dedupe compared raw ids only, and the ci.yml id ("check:file-size")
   never equals the curated id ("file-size"). file-size and compression-budget
   were therefore printed as a hard failure AND as drift in the same verdict.
   A gate that the curated pass already ran now keeps its curated kind.

Reporting only: which gates run and whether they pass is unchanged.
@diegosouzapw
diegosouzapw merged commit 6984676 into diegosouzapw:release/v3.8.50 Aug 24, 2026
13 of 16 checks passed
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…i.yml gates (diegosouzapw#11321)

Validated on a 17-PR combined board: validate-release-green within the board's 287/287, typecheck:core clean. Two accuracy bugs in the release-green verdict tool: an unanchored regex blamed a passing test line (matching a filename containing 'fail'), and 6 gates were double-recorded as both hard-failure and drift due to an id-format mismatch (ci.yml script name vs curated id). Found while reading the diegosouzapw#9985 verdict — good catch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants