Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .verify-floor.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"collected": 366,
"passed": 366,
"collected": 370,
"passed": 370,
"skipped_max": 24,
"selftest_skipped_max": 7,
"gate_skipped_max": 2,
"note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time."
"note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use — the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 370 on 2026-08-22 (entrypoint-diagnosis work). Measured, not assumed: `pytest --collect-only -q` reports 370 on this tree, and `^def test_` counts confirm this branch adds exactly TWO test functions (both in test_capability_set_coverage.py: test_an_absent_entrypoint_diagnoses_itself_differently_from_a_real_defect and test_the_capability_gates_all_consult_the_entrypoint_diagnosis). So 366 was already 2 BELOW what this branch's base collects -- the same permissive gap this note has recorded twice before, from a floor measured on a branch cut before another PR merged. No ceiling moved: both new tests build their own synthetic ledger and read source files, so they need no populated ledger, no agent CLI and no ~/.codex/skills, and they run identically on a bare runner."
}
13 changes: 13 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,19 @@ Do not create a second event log, model registry, or capability inventory.
and the Brain. Pointing only the first at an empty directory and concluding "the suite is
state-independent" is exactly the mistake that made the first CI run red — the ledger never
moved. Set both when testing a fresh-machine claim.
- **The LEDGER is shared across branches; the CODE is not — so a sibling branch reds your gates.**
`$ORCH_LOCAL_RUNTIME/capabilities.json` is one file per machine, while every branch has its own
tree. Any branch that registers a capability therefore makes every OTHER branch's `verify.py`
red on three gates (admission / recurrence fixture / heartbeat call site), for a capability that
is perfectly well implemented somewhere you cannot see. **That red is not evidence of a
badly-declared capability, and the two remedies it invites — retiring the ledger row, or masking
it with a WAIVER — are both wrong and have both been proposed for a live capability.** The gates
now say which case they are in (`capability_activation_audit.entrypoint_diagnosis`): an entrypoint
absent from the tree is reported as such, with the branch check to run. Two rules that follow:
an empty `git log --all --oneline -- <file>` proves nothing until the refs are fetched — that
false negative is what produced the wrong verdict; and the honest verdict on a branch carrying
someone else's ledger row comes from a fresh-state run (`ORCH_STATE_DIR` **and**
`ORCH_LOCAL_RUNTIME` pointed at empty directories, per the bullet above), which is what CI does.
- **The split is TOOL vs EVIDENCE.** Generic capabilities, gates and tests are committed. This
instance's evidence is not: `IMPROVEMENT_BACKLOG.md`, `CAPABILITY_USEFULNESS.md`,
`LOCAL_POLICY.md`, `*.local.md`, `experiments/`, `ux_reviews/`, `data/`, `Audits/`. When adding a
Expand Down
Loading