Skip to content

fix(ci): bundle-size ratchet no longer passes when it cannot measure (#15159 G-04) - #15282

Merged
diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.52from
jonlwheat2-gif:fix/g04-bundle-size-unmeasurable
Oct 2, 2026
Merged

diegosouzapw merged 1 commit into
diegosouzapw:release/v3.8.52from
jonlwheat2-gif:fix/g04-bundle-size-unmeasurable

Conversation

@jonlwheat2-gif

Copy link
Copy Markdown
Contributor

Problem

check-bundle-size.mjs --ratchet printed bundleSize=SKIP reason=… and exited
0 on five separate "could not measure" paths. The ci.yml step is labelled
"Bundle size (ratchet, blocking)".

A ratchet that cannot take a measurement and then reports OK converts unknown
into verified — worse than no gate at all. The five paths, each with a comment
stating the behaviour was intentional:

Path Old behaviour
size-limit not installed SL_NO_BIN → fallback-stat → exit 0
size-limit without plugins SL_NO_PLUGINS → fallback-stat → exit 0
unexpected size-limit error console.error an Aviso, print SKIP, exit 0
no metrics.bundleSize baseline SKIP, sai 0
fallback-stat (raw bytes vs gzip baseline) SKIP (medição não-comparável…) → exit 0

The catalog named :250-256. The other four had the same defect and are fixed
here too — a partial fix would leave the gate able to pass without measuring.

The fix surfaced a second, larger bug

Once an unmeasurable run stopped exiting 0, the real gate started failing, and
the cause was not the ratchet:

node_modules\.bin\size-limit:2
basedir=$(dirname "$(echo "$0" | sed -e 's,\,/,g')")
SyntaxError: missing ) after argument list

runSizeLimit invoked node node_modules/.bin/size-limit --json. On Windows that
path is a shell shim, not JavaScript, so node <shim> died on every
invocation. The preferred measurement mode was unreachable on any Windows
machine — every run fell through to the raw-byte fallback, which is precisely the
path that could never produce a ratchet verdict. It went unnoticed only because
the failure exited 0.

Fixed by resolving the package's own bin entry through the repo's existing,
already-tested scripts/build/buildToolRunner.mjs — the helper written for the
esbuild Windows postbuild ENOENT incident — rather than inventing a second
resolver. It prefers the JS entry, detects native binaries, and falls back to the
.cmd shim with a shell on win32.

The preferred measurement now works on Windows: 9394 bytes vs the 10384
baseline, exit 0.

Also fixed while threading the root through runSizeLimit(cwd): resolution used a
module-level ROOT and silently ignored its cwd argument.

The rule, now

Under --ratchet, "not measured" is exit 1 with the reason and the remedy.
Advisory mode still exits 0 — a developer running the gate locally without a
build should not be blocked.

A CI ::error:: annotation is emitted so the reason is visible in the run
summary. The bundleSize=SKIP reason=<token> stdout token is unchanged, so log
scrapers keep working.

Verification (release/v3.8.52 @ dbe703a)

Check Result
check-bundle-size-unmeasurable.test.ts (new) 12/12 pass
check-bundle-size.test.ts (existing) 21/21 pass
build-tool-runner-win-shim.test.ts pass (reused helper unaffected)
npm run check:bundle-size -- --ratchet exit 0, real measurement 9394
npm run check:bundle-size (advisory) exit 0
tests/unit/build/*.test.ts 534 tests, 12 fail — the same 12 that fail at base

The new suite drives the real CLI as a subprocess against fixture repos with a
stubbed size-limit package. It keeps four guard cases so the fix cannot
over-correct into a gate that blocks on legitimate conditions: advisory still
exits 0, a measured regression still blocks, and a clean measurement still
passes.

Two fixture bugs were caught and fixed during the cycle rather than papered over:
writeFileSync does not create missing parent dirs, and stubbing the .bin shim
stops working once the gate correctly bypasses it — the stub has to be the
package entry.

⚠️ base-red inherited: #15246 — release/v3.8.52 is not green (6 hard failures in
unit, vitest, integration, package-artifact and tarball jobs). None originate here.

Scope

scripts/check/check-bundle-size.mjs ·
tests/unit/build/check-bundle-size-unmeasurable.test.ts (new) ·
tests/unit/build/check-bundle-size.test.ts (one test updated for the new
runSizeLimit signature) · .github/workflows/ci.yml (comment only — the step was
already correct, its description was the lie).

Refs #15159 (G-04)

…iegosouzapw#15159 G-04)

`check-bundle-size.mjs --ratchet` printed `bundleSize=SKIP reason=…` and exited
**0** on five separate "could not measure" paths, each with a comment saying so
on purpose — e.g. :252 "SKIP sai 0 mesmo com --ratchet (erro de medição nunca
bloqueia)". The ci.yml step is labelled "Bundle size (ratchet, blocking)".

A ratchet that cannot take a measurement and then reports OK converts "unknown"
into "verified". The five paths: size-limit missing, size-limit without plugins,
unexpected size-limit error, no baseline in quality-baseline.json, and a
fallback-stat measurement (raw bytes) not comparable to the gzip baseline.

**The rule is now one sentence:** under `--ratchet`, "not measured" is exit 1
with the reason and the remedy; advisory mode still exits 0 (a developer running
the gate locally without a build should not be blocked). A CI `::error::`
annotation is emitted so the reason is visible in the run summary.

## The fix surfaced a second, larger bug

Once an unmeasurable run stopped exiting 0, the real gate started failing — and
the cause was not the ratchet at all:

    node_modules\.bin\size-limit:2
    basedir=$(dirname "$(echo "$0" | sed -e 's,\,/,g')")
    SyntaxError: missing ) after argument list

`runSizeLimit` invoked `node node_modules/.bin/size-limit --json`. On Windows
that path is a **shell shim, not JavaScript**, so `node <shim>` died on every
invocation. The preferred measurement mode was unreachable on any Windows
machine — always falling through to the raw-byte fallback, which is exactly the
path that could never produce a ratchet verdict. It was invisible only because
the failure exited 0.

Fixed by resolving the package's own bin entry via the repo's existing, already
tested `scripts/build/buildToolRunner.mjs` (the helper written for the esbuild
Windows postbuild ENOENT incident) rather than inventing a second resolver. It
prefers the JS entry, detects native binaries, and falls back to the `.cmd` shim
with a shell on win32.

The preferred measurement now works on Windows: **9394 bytes** measured against
the 10384 baseline, exit 0.

Also fixed while threading the root through `runSizeLimit(cwd)`: resolution
previously used a module-level `ROOT` and silently ignored its `cwd` argument.

## Verification (release/v3.8.52 @ dbe703a)

| Check | Result |
| --- | --- |
| `check-bundle-size-unmeasurable.test.ts` (new) | 12/12 pass |
| `check-bundle-size.test.ts` (existing) | 21/21 pass |
| `build-tool-runner-win-shim.test.ts` | pass (reused helper unaffected) |
| `npm run check:bundle-size -- --ratchet` | exit 0, real measurement 9394 |
| `npm run check:bundle-size` (advisory) | exit 0 |
| `tests/unit/build/*.test.ts` | 534 tests, 12 fail — the same 12 that fail at base |

The new suite drives the real CLI as a subprocess against fixture repos with a
stubbed size-limit package, and keeps four guard cases so the fix cannot
over-correct: advisory still exits 0, a measured regression still blocks, and a
clean measurement still passes.

⚠️ base-red inherited: diegosouzapw#15246.

Refs diegosouzapw#15159 (G-04)
@jonlwheat2-gif

Copy link
Copy Markdown
Contributor Author

Ordering note (not a defect in this PR): the lint job will show red on this PR until #15283 (G-05) lands. The base branch carries six stale entries in config/quality/eslint-suppressions.json, which makes npm run lint exit 2 on every PR regardless of its contents. Merging #15283 first clears it. No code in this PR contributes to that failure.

@diegosouzapw
diegosouzapw merged commit dd66b15 into diegosouzapw:release/v3.8.52 Oct 2, 2026
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants