diff --git a/.verify-floor.json b/.verify-floor.json index f1174c0..40f6c1c 100644 --- a/.verify-floor.json +++ b/.verify-floor.json @@ -1,8 +1,8 @@ { - "collected": 402, - "passed": 402, - "skipped_max": 26, - "selftest_skipped_max": 7, - "gate_skipped_max": 2, - "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that)." + "collected": 407, + "passed": 407, + "skipped_max": 26, + "selftest_skipped_max": 7, + "gate_skipped_max": 2, + "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that). FLOOR 402 -> 407 on 2026-08-23 (findability admission requirement). (findability admission requirement). (findability admission requirement). (findability admission requirement). Exactly five new pytest tests, all in test_capability_admission.py: test_findability_distinguishes_its_three_sub_causes, test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt, test_unreadable_reach_is_not_evaluated_and_never_a_failure, test_findability_exemption_is_declared_in_code_not_in_a_live_ledger, test_consult_sites_are_falsifiable_claims_about_real_callers. NO CEILING MOVED and nothing new skips: all five build synthetic ledgers in a tempdir or read committed tables, so none needs a populated capability ledger, an agent CLI or ~/.claude/skills. The one machine-dependent thing they touch -- an external consult site declared in capability_advisor.CONSULT_SITES whose skill prompt is not on this machine -- is reported as UNVERIFIED rather than skipped, because absence of the caller is not refutation of the claim; the in-tree site (tick) is asserted verified on every machine so the check can never degrade into 'everything unverified, nothing tested'. Measured on the merge result per the rule above: this file CONFLICTED three times while the branch was open, as main went 387 -> 391 -> 402 (#61, #64, #65, #60). Each time it was resolved as the UNION rather than by taking a side, and the count was RE-MEASURED on the new merge result rather than either number being carried forward: 402 (main at bd6da2e) + 5 (this branch's new tests) = 407. That is the rule this file already states -- measure the floor on the merge result, not on the branch -- and it mattered here, because #60 both deleted test_ci_gate_config.py and added more than it removed, so guessing in either direction would have been wrong." } diff --git a/ADDING_CAPABILITIES.md b/ADDING_CAPABILITIES.md index dc5313e..d242e96 100644 --- a/ADDING_CAPABILITIES.md +++ b/ADDING_CAPABILITIES.md @@ -41,11 +41,12 @@ Enforced by `capability_admission.py` + `test_capability_admission.py`. Run **be python3 capability_admission.py --preflight '{"capability_id":"capability:my-thing", ...}' ``` -`preflight` answers the five declarable requirements immediately and returns the other three as -explicit **obligations** rather than silently skipping them — because silently skipping is how they -got skipped. +`preflight` answers the six declarable requirements immediately — including findability, which is the +one most worth learning before the code exists — and returns the other three (caller, heartbeat, +fixture) as explicit **obligations** rather than silently skipping them, because silently skipping is +how they got skipped. -### The eight parts +### The nine parts | # | Requirement | The failure it prevents | |---|---|---| @@ -57,23 +58,68 @@ got skipped. | 6 | **A kill switch** | An undeclared switch cannot be found in an emergency. | | 7 | **A rollback path** | — | | 8 | **An expiry or a cadence** | Nothing may sit unexamined forever; that is how dormancy survives two audits. | +| 9 | **A surface that can OFFER it** (`findable`) | FM10 — 22 of 43 capabilities were bound to no surface at all, so nothing could offer them and no amount of running could produce evidence for them. All 43 had passed admission. The rule existed — as prose, in this file. | -Enforcement binds on capabilities registered from 2026-08-21. The 36 pre-gate capabilities are -reported as **legacy debt on every run** and do not fail the suite — a gate that is red on arrival -gets switched off, and then it protects nothing. Legacy rows still print exactly what they are -missing; the exemption lives on the row, never inside the predicates, so debt can never read as -compliance. +Enforcement binds on capabilities registered from 2026-08-21, and **each requirement carries its own +date**: findability binds from 2026-08-23 (`capability_admission.REQUIREMENT_ENFORCED_FROM`). The +pre-gate capabilities are reported as **legacy debt on every run** and do not fail the suite — a gate +that is red on arrival gets switched off, and then it protects nothing. The same reasoning applies to +a requirement added later, which is why the date is per-requirement rather than one global cutoff. +Legacy and pre-cutoff rows still print exactly what they are missing; the exemption lives on the row, +never inside the predicates, so debt can never read as compliance. -### Say which surfaces bind it (or why none does) +### Requirement 9 in detail: which surface can OFFER it (or why none can) -The eight parts make a capability *invocable and observable*. They do not make it *findable*. A -capability nothing binds is offered from a 40-plus catalogue queried generically, which is the +The first eight parts make a capability *invocable and observable*. None of them makes it *findable*. +A capability nothing binds is offered from a 40-plus catalogue queried generically, which is the measured 13.62% selection condition — built, admitted, and still not chosen. +**This was prose in this file until 2026-08-23, and the measurement is what a prose rule is worth +here: 37 of 43 capabilities had no usefulness evidence, and 22 of those were bound to no surface at +all.** It is now the ninth predicate in `capability_admission.REQUIREMENTS`, and +`test_capability_admission.py` fails on it. + So when adding or reviving one, name its surfaces in `capability_advisor.SURFACE_BINDINGS` — the -skills or automations for which it should be in the small declared set — with a one-line reason each, -or state that no surface binds it yet and what would change that. Keep a bound set to 3–7 entries; -past ~10 it reintroduces the problem the binding removes, and a selftest enforces the ceiling. +skills or automations for which it should be in the small declared set — with a one-line reason each. +Keep a bound set to 3–7 entries; past ~10 it reintroduces the problem the binding removes, and a +selftest enforces the ceiling. Run `--preflight` first: findability is **declarable**, so the answer +arrives before the code is written rather than after. + +**Three sub-causes, because the fixes differ.** The predicate names which one applies: + +| cause | what it means | the fix | +|---|---|---| +| `bound_nowhere` | no surface declares it | add one entry to a surface's 3–7, with its reason | +| `bound_to_unconsulted_surface` | every binding names a surface no caller ever consults | bind a surface listed in `capability_advisor.CONSULT_SITES`, or make that surface consult | +| *(invoked without attribution)* | a surface runs the entrypoint directly and the invocation is credited to nobody | **not checked** — see below | + +`CONSULT_SITES` is the other half of a binding, and until 2026-08-23 nothing declared it: `ci` bound +two capabilities and no caller anywhere consults a `ci` surface, while `opener-lane` and `closer-lane` +bind ten between them and both lane prompts consult with **no `--surface` at all**, so the declared +set never reaches the caller it was written for. A declared consult site is a falsifiable claim about +a file — the selftest opens it. Absent on this machine means *unverified*, never refuted; present and +no longer naming its surface is DRIFT and fails. + +**A surface has exactly three honest states, and the selftest enforces it**: a caller consults it +(`CONSULT_SITES`), it deliberately binds nothing (`NO_BINDING` with the reason), or it holds bindings +nothing can reach and that is *recorded* (`KNOWN_UNCONSULTED`, with the reason AND the fix). A fourth +state — bindings nothing can reach and nobody wrote down — is what `ci` was, and it is invisible until +a capability is stranded on it, so it now fails `capability_advisor._selftest_findability` naming the +SURFACE rather than only the capability. `KNOWN_UNCONSULTED` is a record, not a waiver: capabilities +bound only there still fail requirement 9. And a fixed entry may not linger — the same selftest fails +on a stale one, because a cached reason that outlives its evidence is this workspace's named defect. + +**What requirement 9 deliberately does NOT check**, stated here because a gate that cannot say what +it skipped is the same defect as one that cannot say what would clear it: a surface that invokes the +entrypoint *directly* without surface attribution. The `orchestrate` skill already runs `capacity.py` +while `windowed-capacity-policy`'s heartbeat sits behind `ORCH_CAPABILITY_HEARTBEATS`, which only a +live tick sets — so it is used and entirely uncredited. +`capability_activation_audit.heartbeat_reachable` was checked first and answers a different question: +it reports that row `reachable` via `orchestrate.sh (CLI)`, because it asks whether *some* driver +reaches the heartbeat, not whether *this surface's* invocation is attributed to the surface. Deciding +that needs the surface's own prompt, which lives outside this repository. Likewise +`repo-audit:fix` is *named* by its skill and never *entered* by a run; only trial records can show +that, never a table of files. Binding is prioritisation, not concealment: an unbound capability is still returned, ranked after the bound ones. That is deliberate — a capability that could never be selected could never earn the @@ -122,3 +168,4 @@ write it down with the criterion that would change it. | FM7 circular measurement | the activation/recurrence split | | FM8 tests asserting bugs | fixture-must-not-error assertion; synthetic-fixture rule | | FM9 wrong tree | mirror re-run after every sync | +| FM10 admitted but unofferable | `capability_admission.req_findable` (requirement 9), plus `capability_advisor.consulting_surfaces()` for the surfaces that strand a binding | diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index b411f9f..c10f60d 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -141,8 +141,9 @@ the two axes as one. They are orthogonal: capability routinely spans both (`adversarial-review` is role judgment, invoked by a rail gate, recorded over a deterministic acceptance edge). -The eight admission parts (`ADDING_CAPABILITIES.md`) are **not** the definition of a capability. They -are what must be present for one to work with this system — invocable, observable, improvable. +The nine admission parts (`ADDING_CAPABILITIES.md`) are **not** the definition of a capability. They +are what must be present for one to work with this system — invocable, observable, findable, +improvable. **Two kinds, and their measurement stories differ.** *Workflow* capabilities run implementation code and have a definable success condition, so effectiveness is a pass/fail rate. *Sub-agent* capabilities @@ -167,7 +168,7 @@ are ordered by when each starts working: | Layer | Mechanism | Works from | |---|---|---| -| 1 | `capability_advisor.SURFACE_BINDINGS` — declared, per surface, 3–7 entries, each with its reason; plus `CAPABILITY_PRECONDITIONS`, which explains an offer without changing it | day one; no classifier, no history | +| 1 | `capability_advisor.SURFACE_BINDINGS` — declared, per surface, 3–7 entries, each with its reason; `CONSULT_SITES`, which declares who actually ASKS at each surface; plus `CAPABILITY_PRECONDITIONS`, which explains an offer without changing it | day one; no classifier, no history | | 2 | `capability_propensity.rank` — orders *within* the bound set by measured usefulness | first resolved trials | | 3 | `capability_advisor.learned_associations` — corrects the table from what a surface actually reaches for | once observations accumulate | @@ -178,6 +179,58 @@ committed table is the seed (tool); instance promotions live in the ledger (evid bound set and flagged `bound: false`. A concealed capability could never be selected, so it could never earn the evidence that would bind it — the gate would starve its own drain. +**And a binding is only half of layer 1: `CONSULT_SITES` is the other half, and nothing declared it +until 2026-08-23.** `SURFACE_BINDINGS` says which capabilities a surface should be offered; nothing +said which surfaces are ever ASKED, and the two are independent — from a capability's point of view, +a binding to a surface no caller consults is indistinguishable from no binding at all. Measured over +the 43-row ledger: `ci` bound two capabilities and no caller anywhere consults a `ci` surface; +`opener-lane` and `closer-lane` bind ten between them and both lane prompts consult the advisor with +**no `--surface`**, so `binding_for("")` returns `{}` and the declared set never reaches the caller it +was written for. `repo-audit` is the control case — never consulted under its bare name, and +correctly so, because every consult happens at a phase key whose resolution merges the parent's +entries. So "not consulted" is a defect only for a key that is not a PREFIX of a consulted key. +A consult site is a **falsifiable claim about a file**: the selftest opens it. Present-and-no-longer- +naming-its-surface is DRIFT and fails; absent on this machine is *unverified*, never refuted — the +same "no ledger, no verdict" rule `capability_admission.commitments()` uses, because treating absence +as refutation would strand every skill-bound capability on a fresh clone. + +**A surface therefore has exactly three declared states, and a fourth is a selftest failure.** It is +consulted (`CONSULT_SITES`), it deliberately binds nothing (`NO_BINDING` with the reason), or it holds +bindings nothing can reach and that is recorded with the reason and the fix (`KNOWN_UNCONSULTED` — +currently `opener-lane` and `closer-lane`, whose fix is a `--surface` flag in a lane TOML outside this +repository). Bindings nothing can reach that nobody wrote down is the fourth state, it is what `ci` +was, and it is invisible until a capability is stranded on it — so it now fails naming the SURFACE, +not just the capability. `KNOWN_UNCONSULTED` is a record and not a waiver: a capability bound only +there still fails requirement 9. A fixed entry may not linger either; a stale one fails, because a +cached reason outliving its evidence is the prose-cache defect under a different hat. + +**Findability is the ninth admission requirement (2026-08-23), because the eight before it make a +capability invocable and observable and none of them makes it findable.** 37 of 43 capabilities had +no usefulness evidence and 22 of those were bound to no surface, so nothing could offer them and no +amount of running could produce evidence for them — every one had passed admission, and the rule +against it existed as prose in the document that argues prose does not survive the next session. +`capability_admission.req_findable` consumes `capability_advisor.surfaces_binding` (the inverse of +`binding_for`, so ONE resolver) and `consulting_surfaces()`, and it distinguishes `bound_nowhere` from +`bound_to_unconsulted_surface` because the fixes differ: declare a surface, versus bind a consulted +one or make the surface consult. A capability a rail invokes UNCONDITIONALLY rather than offers is +exempt by declaration — `findability_category: no_surface` plus a rationale in +`capabilities.KNOWN_DECLARATIONS`, both halves required, which is where `capability-admission-gate` +and `docs-drift-fix-agent` now sit instead of on a `ci` surface nothing consults. + +Two things it deliberately does **not** decide, named rather than omitted. A surface that invokes the +entrypoint DIRECTLY without surface attribution: the `orchestrate` skill already runs `capacity.py` +while `windowed-capacity-policy`'s heartbeat sits behind `ORCH_CAPABILITY_HEARTBEATS`, which only a +live tick sets, so the capability is used and entirely uncredited. +`capability_activation_audit.heartbeat_reachable` answers a different question — it calls that row +`reachable` via `orchestrate.sh (CLI)`, because it asks whether *some* driver reaches the heartbeat, +not whether *this surface's* invocation is attributed to the surface — and deciding it needs the +surface's own prompt, which lives outside this repository. And a surface that is NAMED but never +ENTERED (`repo-audit:fix`, which the skill's table lists and no audit run reaches, since an audit ends +at phase 5 and hands implementation to the lanes): only trial records can show that, never a table of +files. Enforcement is per-requirement dated (`REQUIREMENT_ENFORCED_FROM`) so the 43 pre-existing rows +are reported as drainable debt instead of failing the suite; the report prints the debt, its causes, +the surfaces that strand a binding, and the drainable count beside it. + **And a fourth input, orthogonal to all three: the per-repo contraindication.** The three layers above rank a capability by how well it fits the SURFACE. None of them can say *this tool does not work against this particular repository* — a fact that lives in the repo's own record, not in the ledger. diff --git a/CLAUDE.md b/CLAUDE.md index fe920cd..075df05 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,9 +43,10 @@ gates / feedback surfaces — and "capability" is a different axis entirely. is FOR. One capability routinely spans both (`adversarial-review` is role judgment invoked by a rail gate and recorded over a rail acceptance edge). - **The admission parts are not the definition of a capability.** A caller, heartbeat, outcome path, - fixture, kill switch, rollback, expiry and dedup finding are the components that must be present - for a capability to WORK WITH THIS SYSTEM — to be invocable, observable and improvable. Do not - describe a capability by its admission parts; describe it by what it does, then check the parts. + fixture, kill switch, rollback, expiry, dedup finding and a surface that can offer it are the + components that must be present for a capability to WORK WITH THIS SYSTEM — to be invocable, + observable, findable and improvable. Do not describe a capability by its admission parts; describe + it by what it does, then check the parts. **Two kinds of capability, and their measurement stories differ — do not average across them:** @@ -68,7 +69,12 @@ Three layers, ordered by when each starts working — see `ARCHITECTURE.md` for 1. **`capability_advisor.SURFACE_BINDINGS`** — declared per surface, 3–7 entries, each carrying its reason. Works on day one with no classifier and no history. When adding or reviving a capability, - say which surfaces bind it, or say why none does. + say which surfaces bind it, or say why none does — **and this is now the ninth admission + requirement, not advice**: `capability_admission.req_findable` fails a new capability that no + surface can offer, distinguishing `bound_nowhere` from `bound_to_unconsulted_surface`, because the + fixes differ. A binding is only half of it; `capability_advisor.CONSULT_SITES` declares which + surfaces a caller actually NAMES, and a binding to a surface nobody consults is indistinguishable + from no binding at all. 2. **`capability_propensity.rank`** — orders *within* the bound set by measured usefulness. 3. **`capability_advisor.learned_associations`** — corrects the table from observed use. @@ -169,10 +175,16 @@ after it was written. **`ADDING_CAPABILITIES.md` is the procedure, and it is ENFORCED.** Run `python3 capability_admission.py --preflight ''` before writing code: a capability must arrive with a dedup finding, a caller, a heartbeat, a recurrence fixture, an outcome path, a kill -switch, a rollback and an expiry-or-cadence. `test_capability_admission.py` fails the suite -otherwise, and also fails on a citation to a dated record that does not exist or a deadline that -passed with no record. That doc lists the nine failure modes behind those eight requirements; read -it before adding or reviving a capability. +switch, a rollback, an expiry-or-cadence, **and a surface that can offer it**. +`test_capability_admission.py` fails the suite otherwise, and also fails on a citation to a dated +record that does not exist or a deadline that passed with no record. That doc lists the failure modes +behind those nine requirements; read it before adding or reviving a capability. + +The ninth is the newest and the one `--preflight` exists for: findability is **declarable**, so a +capability no surface will ever offer is a design answer you get before writing code rather than a +second project afterwards. Each requirement carries its own enforcement date +(`capability_admission.REQUIREMENT_ENFORCED_FROM`), because a rule added later is red on arrival for +everything that predates it, and a gate red on arrival gets switched off. For model/profile or compiler work, also inspect `execution_profiles.py`, `completion_event_adapter.py`, `pattern_miner.py`, `capability_compiler.py`, `evidence_schema.py`, diff --git a/README.md b/README.md index a32b482..27afeec 100644 --- a/README.md +++ b/README.md @@ -363,18 +363,28 @@ Remote CI: `docs/CI_LINT_BASELINE.md` (what the Gate checks, what is deferred, a `capability_admission.py` is the admission gate, and unlike the other capability tooling it is PROSPECTIVE: everything else checks the capabilities that already exist. A capability must arrive -with eight parts — a recorded dedup finding, a caller, a heartbeat on the executed path, a +with nine parts — a recorded dedup finding, a caller, a heartbeat on the executed path, a recurrence fixture, an outcome path (declared consumer **and** learning sink), a kill switch, a -rollback, and an expiry-or-cadence. `--preflight ''` answers this before the code is -written, returning caller/heartbeat/fixture as explicit obligations rather than skipping them. +rollback, an expiry-or-cadence, and **a surface that can offer it**. `--preflight ''` +answers six of them before the code is written, returning caller/heartbeat/fixture as explicit +obligations rather than skipping them. + +The ninth (`findable`, 2026-08-23) exists because the first eight make a capability invocable and +observable and none of them makes it findable: 22 of 43 capabilities were bound to no surface at all, +so nothing could offer them and no amount of running could produce evidence for them. It distinguishes +`bound_nowhere` from `bound_to_unconsulted_surface` — `capability_advisor.CONSULT_SITES` declares +which surfaces a caller actually names — because the fixes differ, and it states what it does not +check (a surface invoking the entrypoint with no surface attribution needs that surface's own prompt, +which is outside this repo). It also tracks **commitments**: a citation to a dated record that does not exist, or a deadline that passed with no record naming its subject, fails `test_capability_admission.py`. That check exists because a scheduled trial review fired on time, wrote nothing, and let a flag revert by timeout for 36 days while live code cited the record nobody wrote. -Enforcement binds on capabilities registered from 2026-08-21; the pre-gate set is reported as legacy -debt on every run and does not fail the suite. Rationale and the nine failure modes: +Enforcement binds on capabilities registered from 2026-08-21, and each requirement carries its own +date (findability from 2026-08-23); the pre-cutoff set is reported as drainable debt on every run, +with its causes and its drainable count, and does not fail the suite. Rationale and the failure modes: `ADDING_CAPABILITIES.md`. ## Capability activation inventory diff --git a/capabilities.py b/capabilities.py index 00491cd..299ddad 100644 --- a/capabilities.py +++ b/capabilities.py @@ -1110,6 +1110,11 @@ def _expire_in_place(capabilities: dict[str, dict[str, Any]], now: int) -> list[ "kill_switch_category", "control_point", "kill_switch_rationale", + # The findability exemption, carried here for the same reason as the kill-switch categories: a + # declaration typed into the running instance's JSON is machine-local and invisible to review. + # `capability_admission.req_findable` reads these two, and both are required together. + "findability_category", + "findability_rationale", ) @@ -1154,6 +1159,29 @@ def _expire_in_place(capabilities: dict[str, dict[str, Any]], now: int) -> list[ "invokes it, ORCH_DISABLE_STEPS=redirect-sweep." ), }, + # THE FINDABILITY EXEMPTION, and the two capabilities that legitimately hold it. Both are + # INVOKED by a rail rather than OFFERED to a reasoning context, so selection pressure cannot + # reach them and a surface binding would be theatre — which is exactly what the `ci` binding in + # `capability_advisor.SURFACE_BINDINGS` was until 2026-08-23. Declared here rather than in a + # live ledger for the same reason the kill-switch categories moved: a machine-local declaration + # is green where it was typed, absent on a fresh checkout, and invisible to review. + "capability-admission-gate": { + "findability_category": "no_surface", + "findability_rationale": ( + "verify.py runs this gate on every PR unconditionally, as one of its five gates. No " + "agent ever chooses it, so binding it to a surface could not change how often it runs; " + "the thing that would change its reach is a CI-side consult, which does not exist." + ), + }, + "docs-drift-fix-agent": { + "findability_category": "no_surface", + "findability_rationale": ( + "its entrypoint is a Workflows GitHub Actions workflow that fires per PR in another " + "repository; invocations arrive here through " + "capability_outcome_bridge.ingest_external_ci_invocations. There is no local reasoning " + "context to offer it to, so no surface can make it more findable." + ), + }, "feedback-store": { "kill_switch_category": "compute_only", "control_point": "ORCH_DISABLE_STEPS", diff --git a/capability_admission.py b/capability_admission.py index 4fef4b4..e60040a 100644 --- a/capability_admission.py +++ b/capability_admission.py @@ -24,8 +24,12 @@ So this module enforces two things the others cannot: - A. ADMISSION — a capability must arrive with all eight parts, or the suite fails. Checked from a + A. ADMISSION — a capability must arrive with all NINE parts, or the suite fails. Checked from a proposed spec too (`preflight`), so the answer arrives before the code is written, not after. + The ninth, findability, was added 2026-08-23: the first eight make a capability invocable and + observable, and none of them makes it FINDABLE. 22 of 43 rows were bound to no surface at all, + so nothing could offer them and no amount of running could produce evidence for them — the + rule against that existed, in prose, in the very document that argues prose does not survive. B. COMMITMENTS — a dated promise must resolve to an artifact that exists. A citation to a decision record that was never written, or a trial deadline that passed with nothing recorded, fails here instead of rotting quietly. @@ -252,6 +256,125 @@ def req_expiry_or_cadence(cap: dict, ctx: dict) -> tuple[bool, str]: return False, "neither expiry nor cadence: nothing will ever re-examine this" +# -------------------------------------------------------------------------------------------- +# The ninth part — FINDABILITY. The eight above make a capability invocable and observable. None of +# them makes it FINDABLE, and a capability nothing can offer can never earn the evidence that would +# improve it: the gate would starve its own drain. +# -------------------------------------------------------------------------------------------- + +# `ADDING_CAPABILITIES.md` has carried "say which surfaces bind it (or why none does)" as PROSE since +# 2026-08-21. That document's own opening argues a rule living only in prose does not survive the +# next session, and THIS module exists because one did not. Measured 2026-08-23 over the 43-row +# ledger: 37 capabilities have no usefulness evidence at all, and 22 of those are bound to NO +# surface — nothing can offer them, so no amount of running will ever produce evidence for them. +# Every one of the 43 passed admission. That is the same failure the file warns about, one layer up. +FINDABILITY_ENFORCED_FROM = 1787443200 # 2026-08-23T00:00:00Z, the day findability became a gate + +# PER-REQUIREMENT ENFORCEMENT DATES. The row-level `legacy` flag answers "was this capability +# registered before the GATE existed". A requirement added LATER needs its own date, or it is red on +# arrival for every capability that predates it — and this module already made that trade once, in +# writing: a gate red on arrival gets switched off, and then it protects nothing. ONE constant per +# requirement, defined here and read only by `admit()`, so the window a requirement MEASURES and the +# window it can be DRAINED over are the same window by construction. +REQUIREMENT_ENFORCED_FROM: dict[str, int] = {"findable": FINDABILITY_ENFORCED_FROM} + +# The one declared exemption, shaped exactly like `kill_switch_category`: a category AND a written +# rationale, neither sufficient alone. It is for a capability that is INVOKED rather than OFFERED — +# a rail runs it unconditionally, so selection pressure cannot reach it and a binding would be +# theatre. Deliberately narrow, and a drift guard in `test_capability_admission.py` stops a live +# ledger from out-declaring the committed table, exactly as it does for the two kill-switch +# categories. +FINDABILITY_CATEGORY = "no_surface" + +# The verdicts that are not failures. `reach_not_evaluated` is here for the same reason +# `commitments()` returns "cannot judge" with no audit ledger: an unreadable advisor must not +# reclassify the whole catalogue as unfindable. +FINDABLE_OK = frozenset({"declared_no_surface", "bound_and_consulted", "reach_not_evaluated"}) + +# Each detectable cause and the DATA EDIT that clears it. Declared, so `report()` can print a +# drainable count that is falsifiable rather than tautological: add a cause with no drain here and +# `drainable` drops below `failing`, which is the alarm. Both fixes are things an agent performs in +# the same PR, and neither requires the thing the gate forbids — so this gate fails toward motion. +FINDABILITY_DRAIN: dict[str, str] = { + "bound_nowhere": "add one entry to a surface's 3-7 in capability_advisor.SURFACE_BINDINGS with " + f"its reason, or declare findability_category={FINDABILITY_CATEGORY!r} + findability_rationale", + "bound_to_unconsulted_surface": "bind a surface listed in capability_advisor.CONSULT_SITES, or " + "make the bound surface consult (pass --surface / the `surface` field)", +} + +# What this requirement DOES NOT CHECK, stated where the check is rather than nowhere. A gate that +# cannot say what would clear it is already defective; so is one that cannot say what it never +# looked at. +FINDABILITY_NOT_CHECKED = ( + "a surface that invokes the entrypoint DIRECTLY without surface attribution — the `orchestrate` " + "skill already runs capacity.py (SKILL.md:31, listed as a tool at :116) while " + "`windowed-capacity-policy`'s heartbeat sits in `capacity.build` behind " + "ORCH_CAPABILITY_HEARTBEATS, which only a live tick sets, so the capability is used and " + "entirely uncredited. `capability_activation_audit.heartbeat_reachable` was checked first and " + "answers a DIFFERENT question — it reports that row `reachable` via `orchestrate.sh (CLI)`, " + "because it asks whether SOME driver reaches the heartbeat, not whether THIS surface's " + "invocation is attributed to the surface. Answering that needs the surface's own prompt, which " + "lives outside this repository, so no predicate over the tree can see it" +) + + +def findability_cause(cap: dict, ctx: dict) -> tuple[str, str]: + """Which sub-cause applies, and the sentence naming the fix. ONE classifier, two readers. + + `req_findable` turns this into a verdict; `report()` counts the causes. A second classifier + would be the parallel inventory this tree keeps paying for. + + THE THREE SUB-CAUSES HAVE DIFFERENT FIXES, so they are never collapsed into one "unfindable": + + 1. `bound_nowhere` — no surface declares it (22 of 43 on 2026-08-23). DETECTED EXACTLY: the + binding table is committed data and `capability_advisor.surfaces_binding` inverts it. + 2. `bound_to_unconsulted_surface` — every binding names a surface no caller ever consults. + `capability-admission-gate` and `docs-drift-fix-agent` are bound to `ci`, which nothing + consults; ten more are bound to `opener-lane`/`closer-lane`, whose prompts consult with no + surface at all. DETECTED from `capability_advisor.consulting_surfaces()`. One shape of this + is NOT detectable and the table says so on the entry: `repo-audit:fix` is NAMED by the + skill and never ENTERED by a run, which only trial records can show. + 3. Invoked without attribution — see `FINDABILITY_NOT_CHECKED`. Deliberately out of reach of + any predicate over this tree, and named rather than omitted. + """ + cap_id = cap.get("capability_id") + # Checked FIRST, and therefore also for a capability bound only to an unconsulted surface: a + # capability a rail invokes unconditionally is exempt whatever the binding table says about it. + if cap.get("findability_category") == FINDABILITY_CATEGORY and _has( + cap, "findability_rationale" + ): + return "declared_no_surface", ( + "declared unofferable, with a rationale: " + str(cap.get("findability_rationale"))[:110] + ) + surfaces = list((ctx.get("bound_surfaces") or {}).get(cap_id) or []) + if not surfaces: + return "bound_nowhere", ( + "bound_nowhere: no surface in capability_advisor.SURFACE_BINDINGS offers it, so it is " + "only ever drawn from the full catalogue queried generically — the measured 13.62% " + "selection condition. " + FINDABILITY_DRAIN.get("bound_nowhere", "") + ) + reached = ctx.get("reached_surfaces") + if not reached: + return "reach_not_evaluated", ( + f"bound to {len(surfaces)} surface(s); consult reach NOT EVALUATED " + "(capability_advisor.consulting_surfaces unavailable) — never read as a pass or a fail" + ) + consulted = [s for s in surfaces if s in reached] + if consulted: + return "bound_and_consulted", "offered at " + ", ".join(sorted(consulted)[:3]) + return "bound_to_unconsulted_surface", ( + "bound_to_unconsulted_surface: every binding names a surface no caller consults " + f"({', '.join(sorted(surfaces)[:4])}). A binding nothing asks for is indistinguishable from " + "no binding. " + FINDABILITY_DRAIN.get("bound_to_unconsulted_surface", "") + ) + + +def req_findable(cap: dict, ctx: dict) -> tuple[bool, str]: + """Can any surface OFFER it? Thin wrapper; `findability_cause` owns the logic.""" + cause, detail = findability_cause(cap, ctx) + return cause in FINDABLE_OK, detail + + REQUIREMENTS = ( ("dedup_recorded", req_dedup_recorded), ("caller_exists", req_caller_exists), @@ -261,6 +384,7 @@ def req_expiry_or_cadence(cap: dict, ctx: dict) -> tuple[bool, str]: ("kill_switch", req_kill_switch), ("rollback", req_rollback), ("expiry_or_cadence", req_expiry_or_cadence), + ("findable", req_findable), ) @@ -289,15 +413,42 @@ def known_controls() -> set[str]: return controls +def _findability_context(capability_ids, *, path: pathlib.Path | None = None) -> dict: + """The two inputs `req_findable` reads, both CONSUMED from `capability_advisor`. + + Nothing here re-derives a binding or a reach: `surfaces_binding` inverts `binding_for`, and + `consulting_surfaces` owns the consult table. If the advisor cannot be imported at all, the + reach set is left empty and the predicate reports `reach_not_evaluated` rather than failing the + whole catalogue on an ImportError. + """ + try: + import capability_advisor as advisor + + reach = advisor.consulting_surfaces() + return { + "bound_surfaces": advisor.surfaces_binding(capability_ids, path=path), + "reached_surfaces": set(reach["reached"]), + "consult_reach": reach, + } + except Exception as exc: # noqa: BLE001 + return { + "bound_surfaces": {}, + "reached_surfaces": set(), + "consult_reach": {"unreadable": f"{type(exc).__name__}: {str(exc)[:80]}"}, + } + + def _context(path: pathlib.Path | None = None) -> dict: import capability_activation_audit as audit import test_capability_set_coverage as coverage rows = {r["capability_id"]: r for r in audit.audit(use_cache=True)["rows"]} + ledger = capabilities.load(path or capabilities.REG) return { "audit_rows": rows, "fixtures": coverage._fixture_capabilities(), "known_controls": known_controls(), + **_findability_context(sorted(ledger), path=path), } @@ -321,11 +472,24 @@ def admit(capability_id: str, *, path: pathlib.Path | None = None, ctx: dict | N waived = bool(waiver and int(waiver.get("expires", 0)) > _now()) created = _created_ts(cap) legacy = bool(created and created < GRANDFATHERED_BEFORE) + # PER-REQUIREMENT SCOPING, alongside the row-level kind. A requirement added after this + # capability was registered is DEFERRED: still reported in `missing` so the debt is visible, + # but excluded from `blocking` so a newly added rule is not red on arrival for the whole + # catalogue. Same trade as `legacy`, one level finer, and it lives on the ROW rather than + # inside a predicate for the same reason — debt must never read as compliance. + deferred = [ + name + for name in missing + if created and created < REQUIREMENT_ENFORCED_FROM.get(name, GRANDFATHERED_BEFORE) + ] + blocking = [name for name in missing if name not in deferred] return { "capability_id": capability_id, "status": cap.get("status"), "admitted": not missing, "missing": missing, + "deferred": deferred, + "blocking": blocking, "checks": checks, "waived": waived, "waiver": waiver, @@ -349,7 +513,16 @@ def preflight(spec: dict) -> dict: **capabilities._blank_capability(spec.get("capability_id") or "capability:proposed"), **spec, } - ctx = {"audit_rows": {}, "fixtures": set()} + # FINDABILITY IS DECLARABLE, so preflight ANSWERS it rather than deferring it — which is the + # whole point of running this before writing code. The binding table and the consult table are + # both committed, so the question needs no ledger and no built module: "which surface will offer + # this?" is a design question the author can settle now, and the alternative is discovering + # after the build that nothing can reach it. + ctx = { + "audit_rows": {}, + "fixtures": set(), + **_findability_context([stub["capability_id"]]), + } checks, missing = {}, [] # Caller/heartbeat/fixture cannot be verified for code that does not exist; they are reported as # OBLIGATIONS rather than silently skipped, because silently skipping is how they got skipped. @@ -539,20 +712,63 @@ def _capability_heartbeat(event: str) -> None: pass -def report(*, path: pathlib.Path | None = None) -> dict: +def findability_report(ledger: dict, ctx: dict, rows: list[dict] | None = None) -> dict: + """Findability's own numbers: what it BLOCKS, what it defers, and what would drain each. + + BOTH QUANTITIES IN ONE PLACE, which is this workspace's standing rule for any gate: `25/43` + reads as "be patient" indefinitely, while `25/43, drainable 25` says the fix is available now + and `drainable 0` would be an instant deadlock report. `drainable` is computed from + `FINDABILITY_DRAIN`, so a future cause with no declared fix makes it fall BELOW `failing` + instead of silently inheriting a comfortable number. + """ + by_cause: dict[str, list[str]] = {} + for cap_id in sorted(ledger): + cause, _detail = findability_cause(ledger[cap_id], ctx) + if cause in FINDABLE_OK: + continue + by_cause.setdefault(cause, []).append(cap_id) + failing = sorted(cap for ids in by_cause.values() for cap in ids) + blocking = sorted( + r["capability_id"] for r in (rows or []) if r["enforced"] and "findable" in r["blocking"] + ) + drainable = sorted( + cap for cause, ids in by_cause.items() if cause in FINDABILITY_DRAIN for cap in ids + ) + reach = ctx.get("consult_reach") or {} + return { + "total": len(ledger), + "failing": failing, + "by_cause": {cause: sorted(ids) for cause, ids in sorted(by_cause.items())}, + "blocking": blocking, + "deferred": sorted(set(failing) - set(blocking)), + "drainable": drainable, + "drain": dict(FINDABILITY_DRAIN), + "not_checked": FINDABILITY_NOT_CHECKED, + "bound_unconsulted_surfaces": list(reach.get("bound_unconsulted") or []), + "consult_sites_unverified": list(reach.get("unverified") or []), + "consult_sites_drifted": list(reach.get("drifted") or []), + "enforced_from": FINDABILITY_ENFORCED_FROM, + } + + +def report(*, path: pathlib.Path | None = None, ctx: dict | None = None) -> dict: _capability_heartbeat("invocation") ledger = capabilities.load(path or capabilities.REG) - ctx = _context(path) + ctx = ctx or _context(path) rows = [admit(cid, path=path, ctx=ctx) for cid in sorted(ledger)] enforced = [r for r in rows if r["enforced"]] return { "total": len(rows), "admitted": sum(1 for r in rows if r["admitted"]), "enforced_total": len(enforced), - "enforced_failing": [r["capability_id"] for r in enforced if not r["admitted"]], + # BLOCKING, not merely "not admitted": a requirement that postdates the capability is + # reported below and does not fail the suite. Without the distinction, adding the ninth + # part would have turned four rows red the day it landed. + "enforced_failing": [r["capability_id"] for r in enforced if r["blocking"]], "legacy_debt": sorted( r["capability_id"] for r in rows if r["legacy"] and not r["admitted"] ), + "findability": findability_report(ledger, ctx, rows), "rows": rows, "commitments": commitments(), } @@ -581,6 +797,30 @@ def format_report(rep: dict) -> str: out.append( " no enforced failures: every capability added since the gate carries its parts" ) + find = rep.get("findability") or {} + if find: + causes = ", ".join(f"{len(ids)} {cause}" for cause, ids in sorted(find["by_cause"].items())) + out += [ + "", + f" FINDABILITY: {len(find['failing'])} of {find['total']} cannot be OFFERED" + + (f" — {causes}" if causes else ""), + f" blocking {len(find['blocking'])}, " + f"pre-cutoff debt {len(find['deferred'])}, " + f"DRAINABLE {len(find['drainable'])} (a data edit each, no code)", + ] + if find["bound_unconsulted_surfaces"]: + out.append( + " surfaces bound but never consulted: " + + ", ".join(find["bound_unconsulted_surfaces"]) + ) + for d in find["consult_sites_drifted"]: + out.append(f" DRIFT {d['surface']}: {d['why']} ({d['caller']})") + if find["consult_sites_unverified"]: + out.append( + " consult sites not verifiable here: " + + ", ".join(u["surface"] for u in find["consult_sites_unverified"]) + ) + out.append(f" NOT CHECKED: {find['not_checked'][:96]}...") if rep["legacy_debt"]: out += ["", " legacy debt (pay down opportunistically; never a human queue):"] counts: dict[str, int] = {} @@ -664,6 +904,11 @@ def _selftest() -> None: ctx_full = { "audit_rows": {"capability:nothing-declared": {"reachable": True, "defects": []}}, "fixtures": {"capability:nothing-declared"}, + # Findability is satisfied by DATA, so the satisfiable case is a synthetic binding to a + # synthetic consulted surface. Driving it from ctx rather than from the real table keeps + # this assertion independent of whatever the live binding table happens to say. + "bound_surfaces": {"capability:nothing-declared": ["t-consulted"]}, + "reached_surfaces": {"t-consulted"}, } for name, fn in REQUIREMENTS: if name == "heartbeat": @@ -671,6 +916,51 @@ def _selftest() -> None: ok, detail = fn(full, ctx_full) assert ok, f"requirement {name!r} cannot be satisfied even when declared: {detail}" + # EVERY CAUSE THE CLASSIFIER CAN RETURN IS EITHER A PASS OR CARRIES ITS OWN FIX. A gate that + # cannot say what would clear it is already defective, and `FINDABILITY_DRAIN` is where that + # answer lives — so the two must not be able to drift apart. Driven over a synthetic matrix that + # walks every branch of `findability_cause`, so it holds on any machine and does not consult the + # live ledger. + probe_cap = {"capability_id": "t-probe"} + exempt = { + **probe_cap, + "findability_category": FINDABILITY_CATEGORY, + "findability_rationale": "a rail invokes it unconditionally", + } + bound = {"bound_surfaces": {"t-probe": ["t-x"]}} + matrix = [ + (probe_cap, {}), # bound_nowhere + (probe_cap, {**bound, "reached_surfaces": {"t-x"}}), # bound_and_consulted + (probe_cap, {**bound, "reached_surfaces": {"t-other"}}), # bound_to_unconsulted_surface + (probe_cap, {**bound, "reached_surfaces": set()}), # reach_not_evaluated + (exempt, {}), # declared_no_surface + ] + seen = set() + for cap_probe, ctx_probe in matrix: + cause, detail = findability_cause(cap_probe, ctx_probe) + seen.add(cause) + assert detail.strip(), f"cause {cause!r} returns no detail at all" + assert cause in FINDABLE_OK or cause in FINDABILITY_DRAIN, ( + f"cause {cause!r} is a failure with no entry in FINDABILITY_DRAIN — the report would " + "then name a backlog it cannot say how to clear" + ) + if cause in FINDABILITY_DRAIN: + assert ( + FINDABILITY_DRAIN[cause] in detail + ), f"cause {cause!r} does not carry its own fix into the message a caller reads" + assert len(seen) == len(matrix), f"the matrix does not reach every branch: {sorted(seen)}" + + # THE ENFORCEMENT DATE MUST NOT BE IN THE FUTURE. `GRANDFATHERED_BEFORE` was first written four + # days ahead, which grandfathered brand-new capabilities and made the gate check nothing; the + # same mistake in a per-requirement date would be invisible, since it silences only one + # requirement rather than all of them. + for name, when in REQUIREMENT_ENFORCED_FROM.items(): + assert when <= _now(), ( + f"{name!r} is enforced from {when}, which is in the FUTURE — every capability, " + "including brand-new ones, would pass it as pre-cutoff" + ) + assert dict(REQUIREMENTS).get(name), f"{name!r} has an enforcement date but no predicate" + # GRANDFATHERING MUST BE VISIBLE, NOT SILENT, and must not weaken the predicates themselves. # Legacy scoping lives on the ROW (`legacy`/`enforced`), so a pre-gate capability still reports # exactly what it is missing — it simply does not block the suite. Were the exemption pushed @@ -723,27 +1013,44 @@ def _selftest() -> None: # preflight must report obligations rather than pretending it verified them. + spec = { + "capability_id": "capability:proposed-thing", + "notes": "dedup: checked X; absent", + "downstream_consumer": "a.py:b", + "learning_sink": "feedback.outcomes", + "kill_switch": "F=0", + "rollback": "revert", + "trigger_cadence": "daily", + } + # FINDABILITY IS THE ONE THE AUTHOR MUST LEARN HERE, before writing code — a proposed + # capability no surface will offer is the 13.62% case, and discovering that after the build is + # a second project. So a spec with the first eight parts and no surface is NOT ready to build. + pf_unbound = preflight(spec) + assert not pf_unbound["ready_to_build"], pf_unbound + assert pf_unbound["declarable_missing"] == ["findable"], pf_unbound + assert "bound_nowhere" in pf_unbound["checks"]["findable"]["detail"], pf_unbound + # ...and declaring the exemption (category AND rationale) clears it. pf = preflight( { - "capability_id": "capability:proposed-thing", - "notes": "dedup: checked X; absent", - "downstream_consumer": "a.py:b", - "learning_sink": "feedback.outcomes", - "kill_switch": "F=0", - "rollback": "revert", - "trigger_cadence": "daily", + **spec, + "findability_category": FINDABILITY_CATEGORY, + "findability_rationale": "a rail invokes it unconditionally; it is never offered", } ) assert pf["ready_to_build"], pf assert set(pf["obligations"]) == {"caller_exists", "heartbeat", "fixture"}, pf assert all(pf["checks"][o]["ok"] is None for o in pf["obligations"]), pf + # The category ALONE must not clear it, exactly as for the kill-switch categories. + pf_bare_category = preflight({**spec, "findability_category": FINDABILITY_CATEGORY}) + assert not pf_bare_category["ready_to_build"], pf_bare_category pf2 = preflight({"capability_id": "capability:bare"}) assert not pf2["ready_to_build"] and "kill_switch" in pf2["declarable_missing"], pf2 env_prereq.report_gaps("capability_admission.py", gaps) print( "capability_admission.py selftest: OK (every requirement can fail and can pass, " - "grandfathering visible, waivers expire, dangling + overdue commitments detected)" + "grandfathering visible, per-requirement cutoffs are in the past, findability is " + "declarable pre-build, waivers expire, dangling + overdue commitments detected)" + (f" — {len(set(gaps))} section(s) skipped, see above" if gaps else "") ) diff --git a/capability_advisor.py b/capability_advisor.py index e0f0876..99b3ebf 100644 --- a/capability_advisor.py +++ b/capability_advisor.py @@ -33,6 +33,7 @@ import argparse import json +import pathlib import re import sys @@ -880,10 +881,27 @@ def should_reask(previous: dict | None, current_context: dict) -> dict: "capability-activation-audit": "and where can-it-fire is observed", "capability-propensity": "the tick is the highest-volume unattended surface (~91 writes/day), " "so it is where propensity evidence should accrue fastest", + "evidence-acquisition": "a `tick_phase` matcher whose declared consumer IS the " + "`orchestrate.sh:evidence-acquisition` cadence step — the same shape as `switch-review` " + "above, bound so the tick can consult it rather than only schedule it. Added 2026-08-23 " + "because the findability requirement blocked it: registered after that cutoff and bound " + "nowhere, it is the first row the new gate actually drained", }, + # `ci` WAS A BINDING TO A SURFACE NOTHING CONSULTS, which is the first finding the findability + # requirement produced about the tree it was added to (2026-08-23). Both former entries were + # right about the mechanism and wrong about the axis: neither capability is ever OFFERED to a + # reasoning context, so no amount of binding could raise its selection odds. `verify.py` runs + # the admission gate on every PR unconditionally, and `docs-drift-fix-agent` is a Workflows + # workflow whose invocations arrive through `capability_outcome_bridge`. Both now declare + # `findability_category: no_surface` in `capabilities.KNOWN_DECLARATIONS`, which is the + # honest statement, and this entry records why the surface is deliberately empty rather than + # leaving a deleted key that reads as an oversight. "ci": { - "capability-admission-gate": "runs in verify.py on every PR; the gate IS the CI surface", - "docs-drift-fix-agent": "a Workflows CI workflow that fires per PR without local invocation", + NO_BINDING: "no caller anywhere consults a `ci` surface — not verify.py, not a workflow, " + "not a skill — so a binding here could never reach a reasoning context. Both " + "capabilities that used to sit here are invoked UNCONDITIONALLY by a rail and " + "declare `findability_category: no_surface` instead. If a CI-side consult is " + "ever added, this is the entry to restore.", }, "repo-audit": { "offload": "whole-repo reads are the canonical offload case, and the audit is the biggest " @@ -980,7 +998,7 @@ def should_reask(previous: dict | None, current_context: dict) -> dict: } -def binding_for(surface: str, *, path=None) -> dict[str, str]: +def binding_for(surface: str, *, path=None, promoted: dict | None = None) -> dict[str, str]: """The declared bound set for a surface, plus any promotions this instance has learned. PHASE-SCOPED. A surface may be a bare name (`closer-lane`) or a phase within a long process @@ -994,6 +1012,11 @@ def binding_for(surface: str, *, path=None) -> dict[str, str]: Seed comes from the committed table; promotions are read from the ledger, so an instance can grow its own bindings without a code change and without touching any prompt. + + `promoted` is an OPTIONAL pre-read promotion index (see `_promoted_index`). Passing it makes a + sweep over every surface read the ledger once instead of once per prefix per call — a 43x30 + inverse lookup took minutes against the live ledger without it. Omitted, the behaviour is + exactly as before. """ if not surface: return {} @@ -1012,7 +1035,7 @@ def binding_for(surface: str, *, path=None) -> dict[str, str]: if cap_id == NO_BINDING: continue out[cap_id] = reason - for cap_id, reason in _promoted_bindings(key, path=path).items(): + for cap_id, reason in _promoted_bindings(key, path=path, index=promoted).items(): out.setdefault(cap_id, reason) return out @@ -1032,18 +1055,225 @@ def binding_suppressed(surface: str) -> str: return str(entry.get(NO_BINDING) or "") -def _promoted_bindings(surface: str, *, path=None) -> dict[str, str]: - """Bindings this instance promoted from observed use. Machine-local evidence, never committed.""" +def _promoted_index(path=None) -> dict[str, dict[str, str]]: + """Every promotion this instance has learned, surface -> {capability: reason}. ONE ledger read. + + Split out of `_promoted_bindings` so a sweep over every surface pays the ledger cost once. The + per-surface function still exists and still answers identically; this is the shared read, not a + second source. + """ caps = capabilities.load_declared(path or capabilities.REG) - out: dict[str, str] = {} + index: dict[str, dict[str, str]] = {} for cap_id, cap in caps.items(): for event in cap.get("event_history") or []: meta = event.get("metadata") or {} - if meta.get("source") == "binding_promotion" and meta.get("surface") == surface: - out[cap_id] = str(meta.get("reason") or "promoted from observed use") + if meta.get("source") == "binding_promotion" and meta.get("surface"): + index.setdefault(str(meta["surface"]), {})[cap_id] = str( + meta.get("reason") or "promoted from observed use" + ) + return index + + +def _promoted_bindings(surface: str, *, path=None, index: dict | None = None) -> dict[str, str]: + """Bindings this instance promoted from observed use. Machine-local evidence, never committed.""" + idx = _promoted_index(path) if index is None else index + return dict(idx.get(surface) or {}) + + +# --------------------------------------------------------------------------- +# WHO ACTUALLY ASKS — the other half of a binding, and the half nothing declared. +# +# The table above says which capabilities a surface should be OFFERED. Nothing said which surfaces +# are ever ASKED, and the two are independent: from a capability's point of view, a binding to a +# surface no caller consults is indistinguishable from no binding at all. Measured 2026-08-23 over +# the 43-row ledger, and all three shapes are live: +# +# * `ci` binds `capability-admission-gate` and `docs-drift-fix-agent`, and NOTHING consults a `ci` +# surface anywhere — not `verify.py`, not a workflow, not a skill. +# * `opener-lane` and `closer-lane` bind ten capabilities between them, and both lane prompts DO +# consult the advisor — with no surface at all: +# `capability_advisor.py --json --lane opener --repository ''`. `binding_for("")` is +# `{}`, so the declared set never reaches the caller the declaration was written for. +# * `repo-audit` binds `offload` surface-wide and is never consulted under its bare name — and +# that one is CORRECT, because every consult happens at a phase key whose resolution merges the +# parent's entries. So "not consulted" is a defect only for a key that is not a PREFIX of a +# consulted key, and `consulting_surfaces()` encodes exactly that difference. +# +# DECLARED, AND FALSIFIABLE. A consult site is a claim about a file, so each entry names the caller +# and the surface literal it passes, and `_selftest_findability` opens the file and checks. In-tree +# callers are verified on every machine, CI included. External callers — skill prompts under +# `~/.claude/skills`, lane prompts under `~/.codex/automations` — are verified where they exist and +# reported UNVERIFIED where they do not. Absence is never read as refutation: doing so would strand +# every skill-bound capability on a fresh clone, which is a gate red on arrival. Same three-valued +# discipline as the precondition axis, same rule as `capability_admission.commitments()` — no +# ledger, no verdict. A caller that is PRESENT and no longer names its surface is different: that is +# DRIFT, it drops out of `reached`, and the selftest fails on it. +# +# WHY NOT DERIVE IT ENTIRELY. The consulting callers are prompts OUTSIDE this repository, so a fresh +# clone can derive nothing at all. A committed table answers the same way on every machine and shows +# a change in a diff; a derivation that silently returned the empty set on CI would make the +# findability gate inert exactly where it has to bite. +# --------------------------------------------------------------------------- +CONSULT_SITES: dict[str, dict] = { + "tick": { + "caller": "capability_propensity.py", + "literal": "TICK_SURFACE", + "how": "`tick-evidence` consults on every tick with surface=TICK_SURFACE. IN-TREE, so this " + "site is verified on every machine, CI included", + }, + "orchestrate": { + "caller": "~/.claude/skills/orchestrate/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + "ux-review": { + "caller": "~/.claude/skills/ux-review/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + "file-agent-issue": { + "caller": "~/.claude/skills/file-agent-issue/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + "implementation-verification": { + "caller": "~/.claude/skills/implementation-verification/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + "cross-env-test-doctor": { + "caller": "~/.claude/skills/cross-env-test-doctor/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + "latched-gate-check": { + "caller": "~/.claude/skills/latched-gate-check/SKILL.md", + "how": "task-initiation consult naming its own surface", + }, + # THE PHASE FAMILIES. The skill's own surface table says "pass the phase as the surface" and + # "pass this in each dimension AGENT's prompt, N=1..8", so the literal in the file is the family + # name while the surfaces actually passed are its instances. The instances are enumerated rather + # than matched by prefix, so a diff reviews them — a prefix rule would silently admit a phase + # nobody consults, which is the very thing being measured. + "repo-audit:phase-N": { + "caller": "~/.claude/skills/repo-audit/SKILL.md", + "instances": [f"repo-audit:phase-{n}" for n in range(1, 6)], + "how": "the skill consults once per phase, passing `repo-audit:phase-N`", + }, + "repo-audit:dimension-N": { + "caller": "~/.claude/skills/repo-audit/SKILL.md", + "instances": [f"repo-audit:dimension-{n}" for n in range(1, 9)], + "how": "phase 2 fans out to eight dimension agents, each passed its own surface", + }, + "repo-audit:fix": { + "caller": "~/.claude/skills/repo-audit/SKILL.md", + "how": "named in the skill's surface table for the follow-up arc. NOTE THE LIMIT: the " + "surface is NAMED, but an audit run ends at phase 5 and hands implementation to the " + "lanes, so no run actually ENTERS it. A table of files cannot see that difference; " + "only `capability_propensity`'s trials can, from what a surface really consulted", + }, +} + + +# BOUND, AND KNOWN NOT TO BE CONSULTED — the third declared state, and an incident record rather +# than a permission. A surface has exactly three honest states: a caller consults it +# (`CONSULT_SITES`), it deliberately binds nothing (`NO_BINDING`), or it holds bindings that nothing +# can ever reach. The third one is what `ci` was, silently, and `_selftest_findability` now fails on +# any surface in that state which is not listed here — so restoring a binding to an unconsulted +# surface is a loud failure at the point of the change instead of a wrong verdict months later. +# +# NOT A WAIVER. Capabilities bound only to a surface listed here still fail `req_findable` as +# `bound_to_unconsulted_surface`; this table records that WE KNOW, with the reason and the fix, so +# the difference between an acknowledged defect and an oversight is legible. When a consult is added, +# MOVE the entry into `CONSULT_SITES` — the selftest will tell you if you forget. +KNOWN_UNCONSULTED: dict[str, str] = { + "opener-lane": "the lane prompt DOES consult " + "(`capability_advisor.py --json --lane opener --repository ''`) but passes no " + "--surface, so `binding_for('')` returns {} and these five bindings have never reached it. " + "FIX: add `--surface opener-lane` to the lane TOML, which is outside this repository — " + "CLAUDE.md forbids a loop that edits lane prompts, so this is recorded, not patched here.", + "closer-lane": "same defect, same lane family: the closer TOML consults with --lane and " + "--context and no --surface. FIX: add `--surface closer-lane` to the lane TOML.", +} + + +def consult_keys() -> set[str]: + """Every concrete surface some caller passes, families expanded to their instances.""" + out: set[str] = set() + for key, site in CONSULT_SITES.items(): + out |= set(site.get("instances") or [key]) return out +def consulting_surfaces() -> dict: + """Which surfaces a caller actually NAMES, and which bound surfaces none of them does. + + Returns `reached` (the surfaces an offer can travel through), `verified` / `unverified` / + `drifted` (the three states of a declared claim), and `bound_unconsulted` — the bound surfaces + that strand whatever is declared on them. + """ + reached: set[str] = set() + verified: list[str] = [] + unverified: list[dict] = [] + drifted: list[dict] = [] + here = pathlib.Path(__file__).resolve().parent + for key, site in sorted(CONSULT_SITES.items()): + instances = set(site.get("instances") or [key]) + literal = str(site.get("literal") or key) + caller = str(site.get("caller") or "") + target = pathlib.Path(caller).expanduser() + if not target.is_absolute(): + target = here / target + try: + text = target.read_text(encoding="utf-8", errors="ignore") + except OSError: + unverified.append( + {"surface": key, "caller": caller, "why": "caller not present on this machine"} + ) + reached |= instances + continue + if literal in text: + verified.append(key) + reached |= instances + else: + drifted.append( + {"surface": key, "caller": caller, "why": f"caller no longer names {literal!r}"} + ) + promoted = _promoted_index() + stranded: list[str] = [] + for surface in sorted(SURFACE_BINDINGS): + if not binding_for(surface, promoted=promoted): + continue # suppressed, or empty — there is nothing here to strand + if surface in reached: + continue + # A PARENT WHOSE PHASES ARE CONSULTED IS NOT STRANDED. `repo-audit` is the case: its + # surface-wide entries are resolved at every `repo-audit:phase-N` consult. + if any(r.startswith(surface + ":") for r in reached): + continue + stranded.append(surface) + return { + "reached": sorted(reached), + "verified": sorted(verified), + "unverified": unverified, + "drifted": drifted, + "bound_unconsulted": stranded, + "site_count": len(CONSULT_SITES), + } + + +def surfaces_binding(capability_ids, *, path=None) -> dict[str, list[str]]: + """The inverse of `binding_for`: which surfaces bind each of these capabilities. + + CONSUMES `binding_for`, so prefix inheritance, `NO_BINDING` suppression and this instance's + ledger promotions are resolved by exactly ONE model — a second resolver here would be the + parallel inventory this tree keeps paying for. The promotion index is read once and handed to + every resolution. + """ + wanted = set(capability_ids) + promoted = _promoted_index(path) + out: dict[str, list[str]] = {cap_id: [] for cap_id in wanted} + for surface in sorted(set(SURFACE_BINDINGS) | consult_keys()): + for cap_id in binding_for(surface, promoted=promoted): + if cap_id in wanted: + out[cap_id].append(surface) + return {cap_id: sorted(surfaces) for cap_id, surfaces in out.items()} + + # --------------------------------------------------------------------------- # THE `applies_to` AXIS, AND PRECONDITIONS GENERALLY — evaluate the condition; do not weaken the # binding. @@ -2419,6 +2649,148 @@ def _selftest_preconditions() -> None: ) +def _selftest_findability() -> None: + """The consult table and the inverse lookup — the two inputs the admission gate reads. + + SYNTHETIC SURFACES, not the live table, per the standing rule to test the mechanism rather than + the bug: `ci` is stranded today and `repo-audit` is not, but asserting on those two would make + this selftest fail the moment either is fixed, which is the opposite of a regression guard. + Every assertion below was written by breaking it first — see the docstring of each block. + """ + import tempfile + from pathlib import Path + + # ---- 1. THE INVERSE LOOKUP RESOLVES THROUGH `binding_for`, INHERITANCE INCLUDED. + # Broken first by having `surfaces_binding` read `SURFACE_BINDINGS[surface]` directly: the + # parent-inherited capability then reported ZERO surfaces, which would have failed every + # surface-wide binding as `bound_nowhere` — a second resolver disagreeing with the first. + saved = {k: dict(v) for k, v in SURFACE_BINDINGS.items() if k.startswith("t-find")} + saved_sites = {k: dict(v) for k, v in CONSULT_SITES.items() if k.startswith("t-find")} + SURFACE_BINDINGS["t-find"] = {"wide-cap": "declared surface-wide"} + SURFACE_BINDINGS["t-find:asked"] = {"asked-cap": "phase that a caller consults"} + SURFACE_BINDINGS["t-find:silent"] = {"silent-cap": "phase nobody consults"} + SURFACE_BINDINGS["t-find:empty"] = {NO_BINDING: "deliberately empty, with a reason"} + try: + with tempfile.TemporaryDirectory(prefix="adv-find-") as td: + ledger = Path(td) / "capabilities.json" + capabilities.save({}, ledger) + site = Path(td) / "fake-skill.md" + site.write_text('consult with surface: "t-find:asked"\n') + CONSULT_SITES["t-find:asked"] = {"caller": str(site), "how": "synthetic"} + + inv = surfaces_binding( + ["wide-cap", "asked-cap", "silent-cap", "absent-cap"], path=ledger + ) + assert "t-find:asked" in inv["wide-cap"], inv["wide-cap"] + assert "t-find:silent" in inv["wide-cap"], inv["wide-cap"] + assert inv["asked-cap"] == ["t-find:asked"], inv["asked-cap"] + assert inv["silent-cap"] == ["t-find:silent"], inv["silent-cap"] + # A SUPPRESSED PHASE MUST NOT INHERIT. Broken first by dropping the NO_BINDING check + # from `binding_for`, which made every deliberately-empty surface look bound. + assert "t-find:empty" not in inv["wide-cap"], inv["wide-cap"] + # A capability nothing declares gets an EMPTY LIST, not a KeyError — the caller reads + # this as `bound_nowhere` and must not have to guard. + assert inv["absent-cap"] == [], inv + + # ---- 2. A DECLARED CONSULT SITE IS A FALSIFIABLE CLAIM ABOUT A FILE. + reach = consulting_surfaces() + assert "t-find:asked" in reach["reached"], reach["reached"][:8] + assert "t-find:asked" in reach["verified"], reach["verified"] + # PRESENT BUT NO LONGER NAMING ITS SURFACE IS DRIFT, and drift must LEAVE `reached`. + # Broken first by treating any readable file as verification, which let a renamed + # surface keep counting as consulted forever. + site.write_text("this file no longer mentions the surface at all\n") + drifted = consulting_surfaces() + assert "t-find:asked" not in drifted["reached"], drifted["reached"][:8] + assert any(d["surface"] == "t-find:asked" for d in drifted["drifted"]), drifted + # ABSENT IS NOT REFUTED. A caller this machine does not have stays reached and is + # reported unverified — the same "no ledger, no verdict" rule the commitment check + # uses. Broken first by treating absence as drift, which stranded every skill-bound + # capability on a fresh clone: a gate red on arrival. + CONSULT_SITES["t-find:asked"] = { + "caller": str(Path(td) / "does-not-exist.md"), + "how": "synthetic", + } + absent = consulting_surfaces() + assert "t-find:asked" in absent["reached"], absent["reached"][:8] + assert any(u["surface"] == "t-find:asked" for u in absent["unverified"]), absent + + # ---- 3. A PARENT WHOSE PHASES ARE CONSULTED IS NOT STRANDED. + # This is the `repo-audit` case and the assertion that keeps the check from crying wolf + # about every surface-wide declaration. Broken first by testing plain membership in + # `reached`, which reported `t-find` stranded while its own phase was being consulted. + stranded = set(absent["bound_unconsulted"]) + assert "t-find" not in stranded, sorted(stranded) + assert "t-find:silent" in stranded, sorted(stranded) + # ...and a deliberately-empty surface strands nothing, so it is not reported either. + assert "t-find:empty" not in stranded, sorted(stranded) + + # ---- 4. FAMILIES EXPAND TO THE SURFACES ACTUALLY PASSED, not to a prefix rule. + fam_site = Path(td) / "fam.md" + fam_site.write_text("pass t-find:step-N as the surface\n") + CONSULT_SITES["t-find:step-N"] = { + "caller": str(fam_site), + "instances": ["t-find:step-1", "t-find:step-2"], + "how": "synthetic family", + } + fam = consulting_surfaces() + assert {"t-find:step-1", "t-find:step-2"} <= set(fam["reached"]), fam["reached"][:12] + assert ( + "t-find:step-3" not in fam["reached"] + ), "a family must not admit an un-listed step" + # The family KEY is a label for the declaration, never a surface anyone passes — so it + # must not leak into the reachable set and quietly satisfy a binding to it. + assert "t-find:step-N" not in consult_keys(), sorted(consult_keys())[:8] + assert "t-find:step-1" in consult_keys(), sorted(consult_keys())[:8] + finally: + for key in [k for k in SURFACE_BINDINGS if k.startswith("t-find")]: + SURFACE_BINDINGS.pop(key) + SURFACE_BINDINGS.update(saved) + for key in [k for k in CONSULT_SITES if k.startswith("t-find")]: + CONSULT_SITES.pop(key) + CONSULT_SITES.update(saved_sites) + + # ---- 5. THE REAL TABLE MUST NOT HAVE DRIFTED, and the in-tree site must verify EVERYWHERE. + live = consulting_surfaces() + assert not live[ + "drifted" + ], f"a consult site's caller no longer names its surface: {live['drifted']}" + # EVERY BOUND SURFACE IS IN ONE OF THREE DECLARED STATES. A fourth — bindings that nothing can + # reach and nobody wrote down — is precisely what `ci` was, and it is invisible until a + # capability is stranded on it. Broken first by adding a binding to a surface with no consult + # site and no entry below: without this the gate reports `bound_to_unconsulted_surface` on the + # capability and never on the SURFACE that caused it. + unaccounted = set(live["bound_unconsulted"]) - set(KNOWN_UNCONSULTED) + assert not unaccounted, ( + f"surface(s) {sorted(unaccounted)} hold bindings that no caller can reach. Either declare " + "the consult in CONSULT_SITES, make the binding deliberately empty with NO_BINDING, or " + "record the defect in KNOWN_UNCONSULTED with its reason and fix" + ) + # ...and the reverse: an acknowledged defect that has been FIXED must not linger, or the record + # outlives the evidence — the prose-cache failure this workspace names explicitly. + stale = set(KNOWN_UNCONSULTED) - set(live["bound_unconsulted"]) + assert not stale, ( + f"KNOWN_UNCONSULTED still lists {sorted(stale)}, which is no longer stranded — move the " + "entry to CONSULT_SITES (or drop it) rather than leaving a cached reason behind" + ) + for surface, why in KNOWN_UNCONSULTED.items(): + assert "FIX:" in why, f"{surface} records the defect without naming the fix" + assert "tick" in live["verified"], ( + "the in-tree consult site must verify on every machine, CI included; " + f"verified={live['verified']}" + ) + for key, site in CONSULT_SITES.items(): + assert str(site.get("caller") or "").strip(), key + assert str(site.get("how") or "").strip(), f"{key} declares no reason" + assert set(site.get("instances") or [key]) <= set(live["reached"]) | { + u["surface"] for u in live["unverified"] + } | {d["surface"] for d in live["drifted"]}, key + print( + "capability_advisor findability selftest: OK (inverse lookup reuses binding_for, drift " + "leaves reach, absence is not refutation, consulted parents are not stranded)" + ) + + def _selftest() -> None: import tempfile from pathlib import Path @@ -2644,6 +3016,7 @@ def main(argv: list[str]) -> int: _selftest_bindings() _selftest_contraindications() _selftest_preconditions() + _selftest_findability() _selftest_reach() return 0 if not args.task: diff --git a/langsmith-fleet-worker-attempt.json b/langsmith-fleet-worker-attempt.json index 18b24e5..2595427 100644 --- a/langsmith-fleet-worker-attempt.json +++ b/langsmith-fleet-worker-attempt.json @@ -1,13 +1,13 @@ { "agent": "codex", "cli_version": "0.144.1", - "emitted_at": "2026-08-23T14:07:18.355766Z", + "emitted_at": "2026-08-23T15:04:55.656311Z", "execution_profile": "codex-default", "fallback_models": [ "gpt-5.5" ], "operation_role": "worker", - "pr_number": "61", + "pr_number": "68", "requested_model": "gpt-5.6-terra", "runner": "reusable-codex-run", "schema": "langsmith-fleet/v1", diff --git a/test_capability_admission.py b/test_capability_admission.py index a6eee9f..ee756c8 100644 --- a/test_capability_admission.py +++ b/test_capability_admission.py @@ -14,6 +14,9 @@ from __future__ import annotations +import pathlib +import tempfile + import capabilities import capability_activation_audit as audit import capability_admission as admission @@ -210,6 +213,286 @@ def test_compute_only_category_must_name_a_control_that_exists(): assert not stray, f"ledger declares compute_only for undeclared capabilities: {stray}" +def _findability_ledger(tmp_dir): + """A SYNTHETIC ledger covering every findability verdict, with controlled registration dates. + + Never the live ledger: it holds 43 rows on this machine and far fewer on a clean runner, so an + assertion about counts or membership there passes locally and fails in CI. Dates are explicit + because the whole point of the requirement is that it binds from a cutoff, and `_created_ts` + reads the earliest `event_history` timestamp. + """ + import capabilities as caps + + after = admission.FINDABILITY_ENFORCED_FROM + 3600 + before = admission.FINDABILITY_ENFORCED_FROM - 86400 + rows = { + # Everything except findability is declared, so the only verdict under test is the ninth. + "t-consulted": dict(bound="asked", when=after), + "t-stranded": dict(bound="silent", when=after), + "t-nowhere": dict(bound=None, when=after), + "t-declared": dict(bound=None, when=after, category=True, rationale=True), + "t-category-only": dict(bound=None, when=after, category=True), + "t-old-nowhere": dict(bound=None, when=before), + } + ledger = {} + for cap_id, spec in rows.items(): + cap = caps._blank_capability(cap_id) + cap.update( + { + "notes": "dedup: checked A/B/C; not present; building new", + "downstream_consumer": "x.py:consume", + "learning_sink": "feedback.outcomes", + "kill_switch": "ORCH_X=0", + "rollback": "revert PR", + "trigger_cadence": "daily", + "event_history": [{"timestamp": spec["when"], "type": "migrated"}], + } + ) + if spec.get("category"): + cap["findability_category"] = admission.FINDABILITY_CATEGORY + if spec.get("rationale"): + cap["findability_rationale"] = "a rail invokes it unconditionally; never offered" + ledger[cap_id] = cap + path = pathlib.Path(tmp_dir) / "capabilities.json" + caps.save(ledger, path) + ctx = { + "audit_rows": {cap_id: {"reachable": True, "defects": []} for cap_id in rows}, + "fixtures": set(rows), + "known_controls": set(), + "bound_surfaces": { + cap_id: ([spec["bound"]] if spec.get("bound") else []) for cap_id, spec in rows.items() + }, + "reached_surfaces": {"asked"}, + "consult_reach": {"reached": ["asked"], "bound_unconsulted": ["silent"]}, + } + return path, ctx + + +def test_findability_distinguishes_its_three_sub_causes(): + """The verdicts a CALLER receives, per sub-cause — because the fixes are different. + + `bound_nowhere` is fixed by declaring a surface; `bound_to_unconsulted_surface` is fixed by + binding a surface someone asks at, or by making that surface ask. Collapsing them into one + "unfindable" licenses the wrong repair, which is why the causes are asserted to DIFFER rather + than merely to be failures. Asserted through `admit()` and `report()`, the two entry points + anything outside this module calls — never through `findability_cause` directly. + """ + with tempfile.TemporaryDirectory(prefix="find-cause-") as td: + path, ctx = _findability_ledger(td) + verdicts = { + cap_id: admission.admit(cap_id, path=path, ctx=ctx) + for cap_id in ( + "t-consulted", + "t-stranded", + "t-nowhere", + "t-declared", + "t-category-only", + ) + } + ok = {cap_id: v["checks"]["findable"]["ok"] for cap_id, v in verdicts.items()} + assert ok == { + "t-consulted": True, + "t-stranded": False, + "t-nowhere": False, + "t-declared": True, + "t-category-only": False, + }, ok + # The two failures must be DISTINGUISHABLE, not just both red. + causes = { + cap_id: v["checks"]["findable"]["detail"].split(":")[0] + for cap_id, v in verdicts.items() + if not v["checks"]["findable"]["ok"] + } + assert causes["t-nowhere"] == "bound_nowhere", causes + assert causes["t-stranded"] == "bound_to_unconsulted_surface", causes + assert causes["t-nowhere"] != causes["t-stranded"] + # The stranded verdict must NAME the surface, or the reader cannot act on it. + assert "silent" in verdicts["t-stranded"]["checks"]["findable"]["detail"] + # A category with no rationale must not clear the requirement — both halves are required, + # the same rule as the two kill-switch categories. + assert "bound_nowhere" in verdicts["t-category-only"]["checks"]["findable"]["detail"] + # ...and the aggregate a report reader sees carries the causes AND the drain, so a debt + # count can never read as "be patient indefinitely". + # + # SCOPED TO THE `t-` ROWS ON PURPOSE. `capabilities.load()` seeds every `KNOWN_GATES` row + # into any ledger it opens, synthetic ones included (verified: a one-row file comes back + # with 15). Asserting on the whole population would therefore be asserting on the committed + # gate table, which is a different test and would move whenever that table does. + rep = admission.report(path=path, ctx=ctx) + find = rep["findability"] + mine = {cap_id for cap_id in find["failing"] if cap_id.startswith("t-")} + assert mine == {"t-category-only", "t-nowhere", "t-old-nowhere", "t-stranded"}, mine + by_cause = { + cause: [c for c in ids if c.startswith("t-")] for cause, ids in find["by_cause"].items() + } + assert by_cause["bound_nowhere"] == [ + "t-category-only", + "t-nowhere", + "t-old-nowhere", + ], by_cause + assert by_cause["bound_to_unconsulted_surface"] == ["t-stranded"], by_cause + # EVERY FAILURE MUST BE DRAINABLE. A gate reporting a backlog with no stated way to clear it + # is a latched gate; `drainable` falls below `failing` the moment a cause has no declared + # fix, which is the alarm this equality arms. + assert len(find["drainable"]) == len(find["failing"]), find + assert set(find["by_cause"]) <= set(find["drain"]), ( + "a failing cause with no declared fix in FINDABILITY_DRAIN — the report would then " + f"name a backlog it cannot say how to clear: {set(find['by_cause']) - set(find['drain'])}" + ) + assert find["bound_unconsulted_surfaces"] == ["silent"], find + assert find["not_checked"], "the requirement must say what it does NOT check" + assert "heartbeat_reachable" in find["not_checked"], find["not_checked"] + + +def test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt(): + """Scoped like the gate it joins: block the next mistake, keep the old debt visible. + + A requirement added later is red on arrival for everything that predates it, and this module + already recorded what happens then — the gate gets switched off and protects nothing. So a row + registered before the requirement's own cutoff still REPORTS `findable` as missing and does not + BLOCK, and a row registered after it does both. + """ + with tempfile.TemporaryDirectory(prefix="find-scope-") as td: + path, ctx = _findability_ledger(td) + new = admission.admit("t-nowhere", path=path, ctx=ctx) + assert "findable" in new["missing"] and "findable" in new["blocking"], new + assert new["deferred"] == [], new + + old = admission.admit("t-old-nowhere", path=path, ctx=ctx) + assert "findable" in old["missing"], "pre-cutoff debt must still be REPORTED" + assert "findable" in old["deferred"], old + assert "findable" not in old["blocking"], "a pre-cutoff row must not block the suite" + + # AND THE SAME SPLIT IN THE AGGREGATE. Scoped to the `t-` rows because + # `capabilities.load()` seeds the committed `KNOWN_GATES` rows into any ledger; and read + # from `findability`, not from `enforced_failing`, because these synthetic rows have no + # module so `req_heartbeat` fails them too — a row can be blocking for another reason + # entirely, and this test is about the ninth requirement's scoping, not about all nine. + rep = admission.report(path=path, ctx=ctx) + find = rep["findability"] + assert {c for c in find["blocking"] if c.startswith("t-")} == { + "t-nowhere", + "t-stranded", + "t-category-only", + }, find["blocking"] + assert "t-old-nowhere" in find["deferred"], find["deferred"] + assert "t-old-nowhere" not in find["blocking"], find["blocking"] + # ...and a row blocking on findability must be in `enforced_failing`, or the gate reports a + # violation it does not act on. + assert "t-nowhere" in rep["enforced_failing"], rep["enforced_failing"] + # THE CUTOFF MUST BE IN THE PAST. `GRANDFATHERED_BEFORE` was first written four days ahead, + # which grandfathered brand-new capabilities and made the gate check nothing. + assert admission.FINDABILITY_ENFORCED_FROM <= admission._now() + assert set(admission.REQUIREMENT_ENFORCED_FROM) <= {n for n, _ in admission.REQUIREMENTS} + + +def test_unreadable_reach_is_not_evaluated_and_never_a_failure(): + """Three-valued, like every other verdict here — and every failure must name its own fix. + + If `capability_advisor` cannot be read at all, the consult-reach question is UNANSWERABLE. + Collapsing that into False would fail every bound capability in the catalogue on an ImportError, + which is a gate red on arrival; collapsing it into True would be a silent pass. So a bound + capability passes with the reason stated, while `bound_nowhere` still fails, because that + question needs no reach at all. + + The same test carries the DRAIN-COVERAGE invariant, because the two break together: dropping + `reach_not_evaluated` from `FINDABLE_OK` both fails a bound capability for an unreadable module + AND makes the report name a cause `FINDABILITY_DRAIN` has no fix for — a backlog it cannot say + how to clear, which is this workspace's definition of an already-defective gate. + """ + with tempfile.TemporaryDirectory(prefix="find-blind-") as td: + path, ctx = _findability_ledger(td) + blind = { + **ctx, + "reached_surfaces": set(), + "consult_reach": {"unreadable": "ImportError: no capability_advisor"}, + } + for cap_id in ("t-consulted", "t-stranded"): + v = admission.admit(cap_id, path=path, ctx=blind)["checks"]["findable"] + assert v["ok"], f"{cap_id}: an unreadable advisor must not fail a bound capability: {v}" + assert "NOT EVALUATED" in v["detail"], v + nowhere = admission.admit("t-nowhere", path=path, ctx=blind)["checks"]["findable"] + assert not nowhere["ok"], "bound_nowhere needs no reach and must still fail" + assert "bound_nowhere" in nowhere["detail"], nowhere + + rep = admission.report(path=path, ctx=blind) + find = rep["findability"] + assert set(find["by_cause"]) <= set(find["drain"]), ( + "the report names a failing cause that FINDABILITY_DRAIN has no fix for, so it reports " + "a backlog it cannot say how to clear: " + f"{sorted(set(find['by_cause']) - set(find['drain']))}" + ) + assert len(find["drainable"]) == len(find["failing"]), find + + +def test_findability_exemption_is_declared_in_code_not_in_a_live_ledger(): + """The `no_surface` category, pinned like the two kill-switch categories. + + A declaration applied straight to the running instance's `capabilities.json` is green where it + was typed, absent on a fresh checkout, and invisible in a diff — the exact defect that moved + `kill_switch_category` into `KNOWN_DECLARATIONS`. So the exempt set is asserted from the CODE + table, and the ledger may not out-declare it. + """ + declared = { + cid + for cid, d in capabilities.KNOWN_DECLARATIONS.items() + if d.get("findability_category") == admission.FINDABILITY_CATEGORY + } + assert declared == {"capability-admission-gate", "docs-drift-fix-agent"}, declared + for cap_id in declared: + assert str( + capabilities.KNOWN_DECLARATIONS[cap_id].get("findability_rationale") or "" + ).strip(), cap_id + # Reconciliation can only seed these if the fields are declaration-owned. + for field in ("findability_category", "findability_rationale"): + assert field in capabilities.DECLARATION_FIELDS, field + # DRIFT GUARD: nothing may carry the exemption that the code table does not declare. + ledger = capabilities.load_declared(capabilities.REG) + stray = { + cid + for cid, cap in ledger.items() + if cap.get("findability_category") == admission.FINDABILITY_CATEGORY + } - declared + assert ( + not stray + ), f"ledger declares the findability exemption for undeclared capabilities: {stray}" + + +def test_consult_sites_are_falsifiable_claims_about_real_callers(): + """A declared consult site names a file; this opens it. + + Without this, `CONSULT_SITES` would be prose in a dict — and the requirement it feeds would + inherit the failure mode this whole module exists to stop. Three states, and only one is a + failure: verified, unverified (the caller is not on this machine, so no verdict), and DRIFTED + (the caller is here and no longer names its surface), which is the real regression. + """ + import capability_advisor as advisor + + reach = advisor.consulting_surfaces() + assert not reach["drifted"], ( + "a consult site's caller no longer names its surface — update CONSULT_SITES or restore the " + f"consult: {reach['drifted']}" + ) + # The in-tree site must verify on EVERY machine, CI included, so this check can never degrade + # into "everything unverified, nothing checked". + assert "tick" in reach["verified"], reach["verified"] + assert advisor.CONSULT_SITES["tick"]["caller"].endswith( + ".py" + ), "the in-tree site must be in-tree" + # Every declared instance must be accounted for in exactly one of the three states. + accounted = ( + set(reach["reached"]) + | {u["surface"] for u in reach["unverified"]} + | {d["surface"] for d in reach["drifted"]} + ) + for key, site in advisor.CONSULT_SITES.items(): + assert str(site.get("how") or "").strip(), f"{key} declares no reason" + assert set(site.get("instances") or [key]) <= accounted or key in accounted, key + # A family key is a label for the declaration, not a surface anyone passes. + keys = advisor.consult_keys() + assert "repo-audit:phase-1" in keys and "repo-audit:phase-N" not in keys, sorted(keys) + + def test_the_gate_admits_itself(): """Dogfooding, and not for style: a gate exempt from its own rule is the rule being optional.""" # `admit()` raises ValueError on a capability the ledger has never heard of, so the gate can