Skip to content

fix(quality): ratchet-bank verifier rejects object-valued notes — nightly banking has been aborting since 2026-08-11 - #11702

Closed
MumuTW wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
MumuTW:chore/file-size-baseline-reclaim
Closed

MumuTW wants to merge 3 commits into
cheaper-inference:release/v3.8.51from
MumuTW:chore/file-size-baseline-reclaim

Conversation

@MumuTW

@MumuTW MumuTW commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

scripts/quality/verify-ratchet-bank.mjs — the safety gate that lets the automated
banking lane write to the quality baselines — exits 1 on a pristine
release/v3.8.51 working tree
, with no local changes at all:

$ git status --porcelain          # empty
$ node scripts/quality/verify-ratchet-bank.mjs; echo $?
Nothing to bank — every baseline already matches the code.
[verify-ratchet-bank] UNSAFE — 1 problem(s); refusing to bank:
  ✗ frozen: note "_rebaseline_2026_08_11_v3850_merge_storm_provider_registry: …" was rewritten
1

Root cause

verifyFrozenMap compares note values referentially:

else if (after[key] !== prev)
  problems.push(`${label}: note "${key}" was rewritten — …`);

That is correct for the string notes the function and its tests were written
against. It is wrong for the object-valued note added on 2026-08-11, where the
merge-storm rebaseline folded four file → LOC pairs into the value of a single
note key. Two structurally identical objects parsed from two JSON.parse calls are
never ===, so the note is reported "rewritten" unconditionally.

Why it matters

bank-ratchet-shrinks in .github/workflows/nightly-release-green.yml:398 runs this
verifier and aborts on any non-zero exit. So the automation added in #8612 —
specifically to stop caps outliving the shrinks that earned them (#8584) — has
banked nothing since 2026-08-11.

The measured consequence, against release/v3.8.51:

baseline entries unbanked slack
frozen 90 26,907 lines
testFrozen 37 14,657 lines

That slack is not academic. A "shrink-only" ceiling set above the file's real size is
a growth allowance:

file frozen at actually headroom
dashboard/combos/page.tsx 6,114 4,803 +1,311
open-sse/executors/chatgpt-web.ts 4,213 3,143 +1,070
open-sse/services/adobeFireflyClient.ts 3,899 2,953 +946
dashboard/api-manager/ApiManagerPageClient.tsx 4,052 3,128 +924
open-sse/handlers/chatCore.ts 6,579 5,856 +723

The change

1 — scripts/quality/verify-ratchet-bank.mjs (one line + comment): compare notes
with the module's own jsonEqual() instead of !==. Notes are still required
verbatim; a genuinely edited note, string or object, is still rejected.

2 — tests/unit/verify-ratchet-bank.test.ts: two regression tests. The
object-valued-note case fails without the fix (not ok 8) and passes with it; the
companion test asserts a changed object note is still rejected, so the fix cannot
silently weaken the gate.

3 — config/quality/file-size-baseline.json: the output of
node scripts/check/check-file-size.mjs --update, now that the verifier permits it.
Data only.

Verification

node --test tests/unit/verify-ratchet-bank.test.ts     24 pass / 0 fail   (22 before, +2)
node --test tests/unit/check-file-size.test.ts \
            tests/unit/quality-ratchet.test.ts         34 pass / 0 fail
node scripts/check/check-file-size.mjs                 OK — 164 frozen, cap 1000
node scripts/quality/verify-ratchet-bank.mjs           OK — 110 lowered, 4 removed, nothing raised   (exit 0)
npx prettier --check <the three files>                 clean

Adversarial checks run before opening this PR:

  • Reverted the fix, re-ran the tests — the new object-note test fails, confirming it
    actually covers the change rather than passing either way.
  • Diffed the baseline structurally — 0 entries raised, 0 notes altered
    (JSON.stringify byte-equal on the object-valued note), 0 entries added, 4 removed
    and each verified to now sit under the 1000 cap.
  • Re-ran --update after banking — idempotent, no further movement.
  • Rebased and re-banked against release/v3.8.51 @ 2e8326d531; carrying the original
    numbers across the rebase produced 9 violations, so the baseline is regenerated against
    the current tip (see the comment below).
  • Confirmed the failure pre-dates this branch by running the verifier on an
    untouched checkout of release/v3.8.51: exit 1 there too.

The malformed 2026-08-11 entry itself is left exactly as the owner annotated it. This
PR makes the verifier tolerate it rather than rewriting an audit-trail note.

@MumuTW
MumuTW requested a review from diegosouzapw as a code owner August 26, 2026 20:39
@MumuTW

MumuTW commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

CI red here is base-red, not this branch

Same failing checks appear on unrelated open PRs against release/v3.8.51:

check #11702 (this) #11718 #11716
Docs Gates (fast-path) fail fail fail
No new ESLint warnings fail fail fail
Unit Tests fast-path fail fail fail
Fast Quality Gates pass fail fail
Merge integrity (changelog + generated skills) pass fail fail

The unit failures on this branch are chatCore Codex passthrough, live-ws-eaddrinuse-6324, runtime/live-ws-default and the APIKEY_PROVIDERS merge count. This PR touches four files: config/quality/file-size-baseline.json, scripts/quality/verify-ratchet-bank.mjs, tests/unit/verify-ratchet-bank.test.ts, and a new changelog fragment. The baseline JSON is read only by scripts/check/*, so none of it is reachable from those suites — and the two live-ws files are the port-binding flake pattern from #8577.

Directly affected suites, run locally against this branch:

node --test tests/unit/verify-ratchet-bank.test.ts    24 pass / 0 fail
node --test tests/unit/check-file-size.test.ts \
            tests/unit/quality-ratchet.test.ts        34 pass / 0 fail

Happy to rebase once the base goes green.

MumuTW added 3 commits August 27, 2026 09:49
…ting every nightly bank

`verifyFrozenMap` compared `_rebaseline_*` note values with `after[key] !== prev`.
That is a referential comparison: correct for the string notes it was written
against, wrong for the object-valued one that landed on 2026-08-11, where the
merge-storm rebaseline folded four file->LOC pairs under a single note key. Two
structurally identical objects off two `JSON.parse` calls are never `===`, so the
verifier reported that note "rewritten" on a tree nobody had touched.

Effect: `node scripts/quality/verify-ratchet-bank.mjs` exited 1 on pristine
release/v3.8.51, and the `bank-ratchet-shrinks` job in nightly-release-green.yml
(cheaper-inference#8612) aborts on any non-zero exit. The automation added precisely to stop caps
outliving their shrinks (cheaper-inference#8584) has therefore banked nothing since 2026-08-11.

Fix uses the module's own `jsonEqual()` helper, so notes are still compared
verbatim -- a genuinely edited note, string or object, is still rejected. Both
new tests fail without the one-line change and pass with it.

Claude-Session: https://claude.ai/code/session_01WXi3P7fMBXdVyZdnEeDYwq
Output of `node scripts/check/check-file-size.mjs --update` with the verifier fix
in place -- data only, no source or gate behavior changes.

  frozen      90 -> 86 entries   26,907 lines reclaimed   4 entries dropped (now under cap)
  testFrozen  37 -> 37 entries   14,657 lines reclaimed

`verify-ratchet-bank.mjs --ref <this commit's parent>` confirms the direction:
110 lowered, 4 removed, nothing raised, no note touched.

These are shrinks the codebase already earned; the ceilings simply outlived them
because the nightly banking lane has been aborting since 2026-08-11 (see the
preceding commit). Until they are banked the "shrink-only" gate is not binding --
`adobeFireflyClient.ts` was frozen at 3,899 against an actual 2,953, i.e. 946
lines of headroom to grow inside a ratchet that is supposed to only descend.
Four other files carried 900+ each.

Recomputed against release/v3.8.51 @ 2e8326d. A banked baseline is tied to the
tip it was measured on: on the previous tip nine files had since grown past the
tightened ceilings, so this must be regenerated if the base moves again before
merge. That is precisely the job the nightly lane is supposed to do unattended.

Claude-Session: https://claude.ai/code/session_01WXi3P7fMBXdVyZdnEeDYwq
@MumuTW
MumuTW force-pushed the chore/file-size-baseline-reclaim branch from 6dcaf10 to e170edd Compare August 27, 2026 01:51
@MumuTW

MumuTW commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto release/v3.8.51 @ 2e8326d531 and re-banked against that tip.

The reclaimed total moved slightly, 41,672 → 41,564 lines (frozen 26,907 + testFrozen 14,657), because the tip advanced 24 commits.

That shift is worth flagging, because it is a property of the change rather than noise: a banked baseline is tied to the tip it was measured on. Rebasing the original commit unchanged produced 9 violations — files that had grown past the newly tightened ceilings while the old, loose ones still permitted it:

✗ src/app/(dashboard)/dashboard/combos/page.tsx: 4803 > congelado 4797
✗ src/app/api/v1/models/catalog.ts:               1961 > congelado 1952
✗ open-sse/executors/base.ts:                     1748 > congelado 1732
   … 6 more

So this baseline has to be regenerated if the base moves again before merge — which is exactly the unattended job the nightly lane is meant to do, and cannot while the verifier exits 1.

Re-verified on the new tip:

check:file-size                                OK — 164 frozen, cap 1000
verify-ratchet-bank.mjs --ref <parent>         OK — 110 lowered, 4 removed, nothing raised (exit 0)
node --test verify-ratchet-bank / check-file-size / quality-ratchet    58 pass / 0 fail
prettier --check (all four files)              clean

Note the verifier must be pointed at the commit's parent (--ref), not the default HEAD, when the branch already carries a banked baseline.

diegosouzapw added a commit that referenced this pull request Aug 28, 2026
Recreated from #11702 (MumuTW) onto the current release/v3.8.51 — fixes the object-note
comparator bug that has silently blocked the automated ratchet-bank lane since 2026-08-11;
24/24 focused tests + file-size gate green on this tip. Thanks for tracking this down!
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for finding this! The release tip moved (further file-size-baseline.json rebaselines
landed) since you opened this, and this fork rejects maintainer pushes, so a direct
rebase-and-merge wasn't possible. Recreated onto the current tip (author preserved via merge,
baseline conflict re-resolved by re-measuring every affected file directly on the tree) and
merged as #11892 — thank you for unblocking the nightly ratchet-bank lane.

@MumuTW
MumuTW deleted the chore/file-size-baseline-reclaim branch September 5, 2026 10:18
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…per-inference#11892)

Recreated from cheaper-inference#11702 (MumuTW) onto the current release/v3.8.51 — fixes the object-note
comparator bug that has silently blocked the automated ratchet-bank lane since 2026-08-11;
24/24 focused tests + file-size gate green on this tip. Thanks for tracking this down!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants