diff --git a/.verify-floor.json b/.verify-floor.json index 0dcedbe..f1174c0 100644 --- a/.verify-floor.json +++ b/.verify-floor.json @@ -4,5 +4,5 @@ "skipped_max": 26, "selftest_skipped_max": 7, "gate_skipped_max": 2, - "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use — the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct." + "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that)." } diff --git a/CLAUDE.md b/CLAUDE.md index efec381..fe920cd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -207,6 +207,20 @@ Do not create a second event log, model registry, or capability inventory. real pytest, reads the COUNTS rather than the exit status, enforces a collection floor so tests silently ceasing to run cannot look like tests passing, treats a silent zero-exit selftest as a failure, and runs the five capability gates. CI runs the same command on a clean machine. +- **The collection floor is an EQUALITY, so adding tests means bumping it in the same PR.** + `collected` in `.verify-floor.json` must EQUAL what pytest collects: too few fails (tests + stopped running), and since 2026-08-23 too many fails as well. A floor BEHIND reality is + permissive by exactly the gap, and that direction was silent for as long as it existed — four + drifts (21 low at the worst, then 8, then 1, then 2) each caught only because somebody happened + to look, because nothing required a test-adding PR to touch the file at all. CI prints the two + integers to write. **If a branch merged under you, REBASE before re-measuring** — the number is + a property of the merge result, not of your branch. You will rarely have to remember that: once + every test-adding branch edits these same two lines, two concurrent branches conflict in git, + and the second cannot merge without rebasing onto the first. `passed` stays a MINIMUM on + `passed + skipped`, because only collection is machine-invariant — a skipped test is still a + collected one, so a bare runner and the owner's machine collect the same number while their + pass/skip split differs. `--update-floor` is not blocked by drift (that would be a gate + forbidding its own drain) and now APPENDS to the note rather than replacing it. - **A check whose PREREQUISITE is absent skips with the missing thing NAMED, and skipping is bounded.** Some checks need what only a running instance has: the populated capability ledger, an installed agent CLI, `~/.codex/skills`, the version-capable Codex binary. Those gates live in diff --git a/verify.py b/verify.py index ed176a4..99f379f 100644 --- a/verify.py +++ b/verify.py @@ -25,7 +25,10 @@ * a **floor** (`.verify-floor.json`) on tests collected and passed. A silent collection drop — an import error making a file uncollectable, a renamed file, a deleted test — fails instead of reading as green. This is the same trap in a different costume: fewer tests running looks - identical to all tests passing. + identical to all tests passing. `collected` is an EQUALITY, not a minimum (2026-08-23): a + floor that has fallen BEHIND reality is permissive by exactly the gap, and that direction + was silent for as long as it existed. See `_floor_problems` for why only `collected` can be + strict, and for the merge-conflict property the equality buys. * **zero collected is always a failure**, whatever the exit status. * the summary states what actually executed, never "the suite passed". * a **SKIP CEILING** (added 2026-08-21, with the first CI run). Skipping is the other way to @@ -45,6 +48,7 @@ from __future__ import annotations import argparse +import datetime as _dt import json import pathlib import re @@ -272,6 +276,45 @@ def load_floor() -> dict: return {} +# The drift message's opening words, named ONCE. The message is built from it and the +# `--update-floor` unblock predicate matches on it; a matching pair of literals would drift, and a +# drifted pair here would silently re-latch the gate closed. (House rule: one constant, defined +# once, consumed by both the measuring and the draining side.) +DRIFT_PREFIX = "floor is BEHIND reality" + + +def _blocks_floor_update(problems: list[str]) -> list[str]: + """Which problems must stop `--update-floor` from writing? NOT the drift one. + + LATCHED-GATE FIX. `--update-floor` used to require a fully green run. The moment `collected` + became an equality that stopped being safe: a floor behind reality is now a PROBLEM, so the + one command that fixes it would have been refused for the existence of the very condition it + exists to clear — the clear path blocked by the thing the gate measures. Real failures still + block the write, because recording a floor from a broken run would bake the breakage in. + """ + return [p for p in problems if not p.startswith(DRIFT_PREFIX)] + + +def _appended_note(prior: str | None, collected: int, passed: int, today: str) -> str: + """Append the recorded counts to the EXISTING note. Never replace it. Pure, so the selftest + can hold the preservation property rather than trusting it. + + Until 2026-08-23 `--update-floor` overwrote the note with a generic sentence. The note is the + only record of WHY each ceiling is the number it is — which missing prerequisite justifies + each agreed skip — so overwriting it destroyed the rationale on every use. The file had to + carry a warning about its own tool ("must be restored by hand after every use"), and a tool + whose correct use requires undoing part of what it just did is a footgun, not a tool. It also + made the honest path — hand-editing two integers — the only safe one, which is how the floor + came to be updated rarely enough to fall behind in the first place. + """ + stamp = ( + f"FLOOR RECORDED by verify.py --update-floor on {today}: collected={collected}, " + f"passed={passed}. Ceilings preserved, never re-measured — they are edited by hand." + ) + prior = (prior or "").strip() + return f"{prior} {stamp}" if prior else stamp + + # Ceiling keys, and what each bounds. Named once so the check below and `--update-floor` cannot # disagree about which number they mean. CEILINGS = ( @@ -282,15 +325,41 @@ def load_floor() -> dict: def _floor_problems(floor: dict, py: dict) -> list[str]: - """Did the amount of CHECKING drop? Pure, so the selftest exercises the real rule. + """Is the amount of CHECKING wrong in either direction? Pure, so the selftest exercises the + real rule. - Two independent drops, both of which look like passing: + Three problems, all of which look like passing: * fewer tests COLLECTED — an import error, a rename, a deletion; + * MORE tests collected than the floor records — the floor has fallen behind reality; * fewer tests passed-or-consciously-skipped — a test that stopped running without becoming a named skip. `passed` alone cannot be the floor once skipping is legitimate, or the machine missing a prerequisite fails for being honest; `passed + skipped` can be, and the ceiling is what stops the skipped side swallowing everything. """ + # WHY `collected` IS AN EQUALITY AND `passed` IS NOT. + # + # Until 2026-08-23 this fired only downward, so a branch could add tests and never touch the + # floor: silently green, with the floor left permissive by exactly the number added. That is + # not hypothetical — #34 and #37 each added a test and left the file alone, and every one of + # the recorded drifts (21 low at the worst, then 8, then 1, then 2) was caught only because + # somebody happened to look. A floor below reality is the hole this file exists to close, so + # falling BEHIND it has to be exactly as loud as dropping below it. + # + # The equality also buys what no amount of discipline could. Once every test-adding branch + # must edit these same two lines, two concurrent branches CONFLICT IN GIT. The second cannot + # merge without rebasing onto the first, and the rebased run reports the true merge-result + # count. Git's own conflict detection is what enforces "measure on the merge result, not on + # the branch" — the rule the note in .verify-floor.json had to repeat three times precisely + # because nothing enforced it. (This very change was rebased that way: #42 landed underneath + # it and moved the floor 368 -> 387.) + # + # Only `collected` can be strict, and the asymmetry is load-bearing. Collection is + # machine-invariant: a skipped test is still a collected test, so a runner with none of this + # instance's prerequisites collects exactly what the owner's machine collects — measured on + # CI and locally on 2026-08-23, both 368, with pass/skip splits of 344/24 against 368/0. + # `passed` is NOT invariant — it trades against `skipped` machine by machine — so it stays a + # MINIMUM on `passed + skipped`. Making that one strict too would fail every machine for + # being honest about a named skip. problems = [] fc, fp = int(floor.get("collected", 0)), int(floor.get("passed", 0)) if fc and py["collected"] < fc: @@ -298,6 +367,17 @@ def _floor_problems(floor: dict, py: dict) -> list[str]: f"collection DROPPED: {py['collected']} < floor {fc} — tests stopped " f"running, which looks identical to tests passing" ) + elif fc and py["collected"] > fc: + # Both numbers AND the remedy, per the house rule that a gate must say what would clear + # it. "387 > 386" alone invites a shrug; naming the exact integer to write does not. + problems.append( + f"{DRIFT_PREFIX}: {py['collected']} collected > floor {fc} — the floor is " + f"{py['collected'] - fc} test(s) permissive, so that many could silently stop being " + f"collected and still read as green. Set \"collected\": {py['collected']} and " + f"\"passed\": {py['passed'] + py.get('skipped', 0)} in .verify-floor.json, keeping " + f"the existing note, or run `python3 verify.py --update-floor`. If a branch merged " + f"under you, REBASE FIRST: the number must be measured on the merge result." + ) if fp and py["passed"] + py.get("skipped", 0) < fp: problems.append( f"executed-or-skipped count dropped: {py['passed']} passed + " @@ -367,11 +447,27 @@ def _cap(key: str) -> str: limit = floor.get(key) return f"{actual[key]}" + (f"/{limit} max" if limit is not None else " (no ceiling set)") + # The floor reports its RELATIONSHIP to reality, not just its value. "floor 386" reads as + # fine at a glance; "floor 386 — 1 BEHIND" cannot be misread, which is the same house rule + # that makes each ceiling print its count against its limit. + floor_state = ( + "unset" + if not fc + else ( + f"{fc}" + if py["collected"] == fc + else ( + f"{fc} — {py['collected'] - fc} BEHIND" + if py["collected"] > fc + else f"{fc} — NOT MET" + ) + ) + ) lines = ["# verify.py", ""] lines.append( f" pytest: {py['passed']} passed, {py['failed']} failed, " f"{_cap('skipped_max')} skipped " - f"({py['collected']} collected; floor {fc or 'unset'})" + f"({py['collected']} collected; floor {floor_state})" ) lines.append( f" selftests: {len(st['ok'])} of {len(mods)} modules ran, " @@ -429,7 +525,8 @@ def _cap(key: str) -> str: ) ) - if update_floor and not problems: + # Drift does NOT block the write — see `_blocks_floor_update`. A real failure still does. + if update_floor and not _blocks_floor_update(problems): # `collected` and `passed` are re-measured; the CEILINGS are NOT. A ceiling re-recorded # from whatever the last run happened to skip is not a ceiling, it is a ratchet that # follows the leak — and on the machine that has every prerequisite it would record 0 and @@ -443,11 +540,11 @@ def _cap(key: str) -> str: for key, _label in CEILINGS: if floor.get(key) is not None: blob[key] = int(floor[key]) - blob["note"] = ( - "floor recorded by verify.py --update-floor; a later run collecting fewer " - "tests FAILS, because silently running fewer tests looks exactly like " - "passing. `passed` is compared against passed+skipped. The *_max ceilings " - "bound skipping and are NOT re-measured here — edit them by hand." + blob["note"] = _appended_note( + str(floor.get("note", "")), + py["collected"], + blob["passed"], + _dt.datetime.now(_dt.timezone.utc).date().isoformat(), ) FLOOR.write_text(json.dumps(blob, indent=1) + "\n", encoding="utf-8") lines.append( @@ -573,6 +670,68 @@ def _selftest() -> None: ) assert any("collection DROPPED" in p for p in shrank), shrank + # ---- and the floor may not fall BEHIND reality either (2026-08-23) ------------------------ + # The permissive direction, silent until this was added: a branch adds tests, leaves the file + # alone, and the floor is now slack by exactly the number added. DELIBERATE-BREAK DEMO: revert + # the `elif` in `_floor_problems` and this assert fails while every other check here still + # passes — which is precisely the shape of the bug, a real hole that reads as green. + behind = _floor_problems( + {"collected": 366, "passed": 366}, {"collected": 368, "passed": 368, "skipped": 0} + ) + assert len(behind) == 1 and "BEHIND reality" in behind[0], behind + # It must name the exact integers to write; "too low" that does not say the number is how a + # gate becomes something people shrug at rather than clear. + assert '"collected": 368' in behind[0] and '"passed": 368' in behind[0], behind + assert "REBASE FIRST" in behind[0], behind + # Exact agreement is the only clean state, and a machine that SKIPS is still exact: skipped + # tests are collected, so the runner and the owner's machine hit the same equality. + assert ( + _floor_problems( + {"collected": 368, "passed": 368}, {"collected": 368, "passed": 344, "skipped": 24} + ) + == [] + ) + # An UNSET floor still means "nothing agreed yet" — it must not suddenly demand equality. + assert _floor_problems({}, {"collected": 368, "passed": 368, "skipped": 0}) == [] + + # ---- --update-floor can run while the floor gate is CLOSED (latched-gate fix) ------------- + # DELIBERATE-BREAK DEMO: change the guard back to `not problems` and the remedy the drift + # message names becomes unreachable — the gate forbidding its own drain, which is the defect + # this repo hits most. The predicate is what keeps the two windows the same size. + assert _blocks_floor_update(behind) == [], behind + assert _blocks_floor_update(behind + ["3 pytest failure(s)/error(s)"]) == [ + "3 pytest failure(s)/error(s)" + ] + # A collection DROP is not drift and must keep blocking: recording a floor from a run that + # lost tests would bake the loss in as the new normal. + assert _blocks_floor_update(shrank) != [] + # ...and the predicate must be WIRED, not merely present. The three asserts above all pass + # while the call site still reads `not problems` -- the helper exists, nothing calls it, + # built-but-not-wired, which is this repo's founding defect wearing yet another hat. It got + # through the first draft of this very change and was caught only by the deliberate-break + # demo, so the guard line itself is now the assertion. + _src = pathlib.Path(__file__).read_text(encoding="utf-8") + # The needle is BUILT FROM TWO PIECES on purpose. Written as one literal it would appear in + # this very line, so `_src` would contain it no matter what the call site said and the assert + # could never fail -- a test that cannot fail, guarding against a defect that already + # happened once. Split, the joined form exists only at the real guard. + _guard = "if update_floor and not " + "_blocks_floor_update(problems):" + assert _guard in _src, ( + "the --update-floor guard no longer calls _blocks_floor_update — a floor behind reality " + "would once again block the one command that fixes it" + ) + + # ---- --update-floor APPENDS to the note, so the ceiling rationale survives the tool ------- + # DELIBERATE-BREAK DEMO: restore the old `blob["note"] = (...)` literal and the first assert + # fails — the prior text, which is the only record of why each ceiling is what it is, is gone. + kept = _appended_note( + "26/7/2 is what a bare runner skips, measured 2026-08-21.", 387, 387, "2026-08-23" + ) + assert kept.startswith("26/7/2 is what a bare runner skips, measured 2026-08-21."), kept + assert "collected=387" in kept and "2026-08-23" in kept, kept + assert _appended_note("", 387, 387, "2026-08-23").startswith("FLOOR RECORDED") + assert _appended_note(None, 1, 1, "2026-08-23") # a missing note is not a crash + # ---- the absent-module summary line ------------------------------------------------------- # A row registered by another checkout makes three checks fail with messages that read # "registered with no implementation — retire it", and retiring it discards finished work. The @@ -621,7 +780,9 @@ def _selftest() -> None: print( "verify.py selftest: OK (count parsing, selftest discovery, silent-zero-exit is a " "FAILURE, a loud skip is not a pass, skip ceiling fails when exceeded and holds when " - "not, floor counts passed+skipped, absent-module line is silent when clean and is " + "not, floor counts passed+skipped, floor fails BEHIND reality as loudly as below " + "it and names the integers to write, --update-floor appends to the note instead of " + "clobbering the ceiling rationale, absent-module line is silent when clean and is " "never counted as a skip)" )