fix(quality): report the real failure line and stop double-counting ci.yml gates - #11321
Merged
diegosouzapw merged 1 commit intoAug 24, 2026
Conversation
…i.yml gates Two verdict-accuracy bugs found reviewing the release-green report of diegosouzapw#9985. 1. firstFailureLine() matched /FAIL/i unanchored, so it hit the substring "fail" inside a test FILE NAME and reported a PASSING line as the cause of the red (2026-08-23 verdict blamed "OK tests/.../fail-fast-concurrency-gate.test.ts (4 tests) 203ms"). Markers that also occur inside file names or summary prose are now anchored to the start of the line and success lines are excluded; markers that tools legitimately emit mid-line (error TSxxxx, AssertionError, Error:, REGRESS) stay unanchored. 2. The --full-ci pass recorded every gate extracted from ci.yml as kind:"hard" while the dedupe compared raw ids only, and the ci.yml id ("check:file-size") never equals the curated id ("file-size"). file-size and compression-budget were therefore printed as a hard failure AND as drift in the same verdict. A gate that the curated pass already ran now keeps its curated kind. Reporting only: which gates run and whether they pass is unchanged.
diegosouzapw
merged commit Aug 24, 2026
6984676
into
diegosouzapw:release/v3.8.50
13 of 16 checks passed
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…i.yml gates (diegosouzapw#11321) Validated on a 17-PR combined board: validate-release-green within the board's 287/287, typecheck:core clean. Two accuracy bugs in the release-green verdict tool: an unanchored regex blamed a passing test line (matching a filename containing 'fail'), and 6 gates were double-recorded as both hard-failure and drift due to an id-format mismatch (ci.yml script name vs curated id). Found while reading the diegosouzapw#9985 verdict — good catch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two accuracy bugs in
scripts/quality/validate-release-green.mjs, both found while reading the release-green verdict posted on #9985.1. The verdict blamed a line that had passed
firstFailureLine()picked the first line matching/✖|✗|not ok|AssertionError|error TS|FAIL|Error:|REGRESS/i. That regex is unanchored and case-insensitive, soFAILmatches thefailinside a test file name. The 2026-08-23 verdict reported this as the cause of the unit red:That is a passing line. The gate had failed elsewhere, so whoever picked up the red was pointed at a green test. Same class of miss on the summary line
Test Files 1 failed, which matches before the realFAIL <file>line ever gets a look.Markers that also occur inside file names or prose (
FAIL,not ok,✖,REGRESS) are now anchored to the start of the line, lines starting with✓/✔/√are excluded outright, and markers that tools legitimately emit mid-line (error TS1234,AssertionError,Error:,REGRESS) stay unanchored so nothing that used to be caught gets lost.2. The same gate reported as both a failure and as drift
--full-cirecords every gate it extracts fromci.ymlaskind: "hard", and the dedupe just above compares raw ids. The ci.yml id is the npm script name (check:file-size) while the curated id is not (file-size), so the dedupe never fires and the gate is recorded twice under two contradictory classifications. The verdict then prints, in one table:Same gate, same run, two verdicts. It affects six:
file-size,compression-budget,dead-code,type-coverage,openapi-coverageandcheck:workflows/workflow-lint.fullCiKindFor()now resolves the ci.yml script id to its curated equivalent (stripcheck:, plus a small alias map for the three that don't follow that shape) and reuses the curatedkindwhen the curated pass already ran the gate. Anything the curated list does not cover still defaults tohard— that is the point of--full-ciand it is unchanged.check:bundle-size, for instance, has no curated counterpart, so it stays hard, and there is a test pinning that.What this is not
This does not change which gates run, what any gate does, or what counts as a gate failure. No thresholds, no baselines, no gate list. It changes what the report says about a result it already had.
One consequence worth naming rather than burying: a gate the script already classifies as drift no longer flips the exit code through the
--full-cipath. That restores the policy the file documents at the top — "Drift NEVER changes the exit code, so wiring this as a check can never block anyone on drift" — which--full-ciwas silently overriding. If you would rather those ratchets block, the fix is a one-line change to the default infullCiKindFor()and I am happy to flip it.Tests
tests/unit/validate-release-green.test.tsalready existed with 23 tests. It was green onrelease/v3.8.50before this change and is green after. Seven added:✓ …fail-fast-concurrency-gate.test.tsline from the 🔴 Release branch not green: release/v3.8.50 #9985 verdict is never reported as a cause, and the real failing line isnot ok,FAIL <file>,error TS2322,✗,Error:,REGRESSÃO,REGRESSED), including the pt-BR ratchet outputcuratedEquivalentId/fullCiKindFormapping, aliases, and the hard default for uncurated gates