feat(tick): the tick consults the capability advisor and records whether one helped - #37
Merged
Merged
Conversation
…her one helped 21 of the 43 capabilities live on the hourly tick and four are bound to the `tick` surface in `capability_advisor.SURFACE_BINDINGS`, yet `orchestrate.sh` had never called `advise()` and nothing had ever recorded an `invocation`/`outcome` edge against an advisory `match`. That is exactly why the capability-propensity cadence step printed PRIOR-ONLY on every run: the loop had a measurement and no producer. This adds the producer on the highest-volume unattended surface in the system, so evidence accrues hourly with no further human attention. DEDUP (CLAUDE.md §0, also recorded in the `capability-propensity` ledger notes): `advise()` and `record_trigger`/`record_usefulness` all EXIST and had no production caller — a tree-wide grep found none outside the two modules' own selftests. `capability_firing_monitor` stores firing HISTORY (did it RUN) and `capability_effectiveness` measures DELIVERY via `influence_edges`, which no observer can populate; neither answers "did the run say anything new". Nothing was rebuilt: this is the missing caller. The one new part is the observer verdict, because no module compared an observer's output across runs. WHAT "IT HELPED" MEANS FOR AN OBSERVER. `capabilities.is_observer()` is the single source of the distinction, consumed rather than re-declared. An observer HELPED when its report's finding set changed since its own previous run (a defect newly reported, a regression flagged, a switch verdict that moved, a finding resolved) and did NOT help when it re-emitted an identical set — silence is not usefulness, and an empty set that stays empty is explicitly not useful. Never a delivery verdict: that is the category error that parked eight capabilities in a measurement gap they could not leave. A bound capability the observer test does not confirm records that it ran and gets no verdict, so two different questions never average into one rate. IT CANNOT MANUFACTURE EVIDENCE — treated as a correctness requirement, since 24 ticks/day over 4 capabilities is 96 potential data points and a per-run verdict would make the ranking measure the cadence. Two independent bounds: (1) STRUCTURAL — the experiment id is scoped to the UTC day, so the ledger idempotency keys admit at most one match/trigger/verdict per capability per day however many ticks run, and ticks 2..24 write nothing; (2) SUBSTANTIVE — a verdict additionally requires that capability's own cadence artifact to have been regenerated since the last evaluation, which ties one verdict to one production. The declared cadences are daily and 6-daily, so the graded ceiling is 1.29 verdicts/day against 96 unconditional — a 74x reduction, and the step prints that arithmetic in its own report. A first observation establishes a baseline and records nothing. The finding projection is the whole safety mechanism: `overdue` rows carry `silent_days`, which rises daily on its own, so hashing a row whole would score the firing monitor "useful" on every run it will ever make. The projection keeps identity and verdict fields and drops everything else. Artifact filenames come from `cadence_registry.STEP_BY_KEY`, not a second table, so they cannot drift from the step that writes them. IT CANNOT STALL THE TICK. No gh, no network, no subprocess, no dispatch; per-capability guards, a whole-run wrapper, and a SIGALRM budget over the one blocking wait in the path (the ledger flock — writes are tmp+os.replace, so an interrupt cannot tear it). Handled failures exit 0. Placed below `ORCH-ANCHOR: heartbeat-export` (a producer above it records nothing) and below all four steps it grades; a selftest asserts the caller exists and sits in both of those positions, which is the guard against this becoming another built-and-forgotten subsystem. Also fixes a pre-existing defect found while wiring it: `advise()`'s binding-only branch returned real capabilities with `useful: true` and recorded NO match events, so the fix that made a declared binding survive a classification miss covered the answer and not the evidence. A surface whose words never hit the keyword vocabulary — the tick's cadence pass exactly — could never accumulate a candidate set, leaving trials with no control arm and no attributable skill, which also meant the demotion drain could never run for such a surface. Kill switches (both tested): `ORCH_TICK_EVIDENCE_DISABLED=1` module-side, and `ORCH_DISABLE_STEPS=tick-capability-evidence` shell-side. Either alone makes the tick behave exactly as before — no consult, no ledger event, no state file. No new store: evidence goes to the existing ledger events. The state file holds only the last-seen artifact mtime and finding fingerprint, which is what "did the output change" needs to compare against — the same shape and reason as `capability_firing_monitor`'s history file. No new pytest test, so `.verify-floor.json` is untouched; the guards live in `capability_propensity --selftest`, which `verify.py` already runs and counts. `ARCHITECTURE.md` gains prose only — no stage, component, rail/role classification or feedback surface changed, so `orchestrator-loop.svg` is correctly unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 25 minutes Limit details: You’ve used the included review currently available. Your 70 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (6)
Comment |
This was referenced Aug 23, 2026
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
`.verify-floor.json` recorded 366 while main collects 368. The drift came from #34 (evidence-acquisition landed) and #37 (tick capability evidence), each measured against a branch cut before the other merged — the same mistake the file's own note already documents from the 345->353 episode. A floor BELOW reality is the permissive direction this file exists to close: two tests could have stopped being collected and still cleared the check, which is precisely the hole it guards. Measured on the merge result, not on a branch: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. No ceiling moved and nothing new is skipped. Co-authored-by: Tim Stranske <tim@stranskemo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself: #42 merged underneath it, moved the floor 368 -> 387 and the ceiling 24 -> 26, and the resulting conflict forced the rebase that produced the 387 recorded here. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant -- a skipped test is still collected, measured the same day at 368 on both CI and this machine with pass/skip splits of 344/24 against 368/0. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and the break demo is what caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified: 387 passed, 0 failed, 0/26 skipped, 387 collected (floor 387), 83/83 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself TWICE inside an hour: #42 landed underneath it (floor 368 -> 387, ceiling 24 -> 26) and then #59 did (387 -> 391), and each conflict forced a rebase and a fresh measurement. Under the old one-directional rule both would have merged green with a floor below reality. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant -- a skipped test is still collected, measured the same day at 368 on both CI and this machine with pass/skip splits of 344/24 against 368/0. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and the break demo is what caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified on the merge result: 391 passed, 0 failed, 0/26 skipped, 391 collected (floor 391), 84/84 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2; floor stays at main's 391, since this adds selftest assertions rather than pytest tests. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself repeatedly: #42 landed underneath it (floor 368 -> 387, ceiling 24 -> 26, verify.py reformatted), then #59 (387 -> 391), then #61, all in one afternoon. Each conflict forced a rebase and a fresh measurement; under the old one-directional rule each would have merged green with a floor below reality. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant, measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and only the break demo caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified on the merge result: 391 passed, 0 failed, 0/26 skipped, 391 collected (floor 391), 84/84 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2; the floor stays at main's 391, since this adds selftest assertions rather than pytest tests. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself, at length: six merges landed on main during the two hours it took to write, moving the floor 368 -> 387 -> 391 -> 402 and the ceiling 24 -> 26. Under the old one-directional rule every one of them would have left the floor below reality and merged green. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant, measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and only the break demo caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified on the merge result: 402 passed, 0 failed, 0/26 skipped, 402 collected (floor 402), 84/84 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2; the floor stays at main's 402, since this adds selftest assertions rather than pytest tests. black clean at line-length 100. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
22 of 43 capabilities were bound to NO surface, so nothing could ever offer them and no amount of running produced evidence: the gate starving its own drain, one layer below the concealment rule the table already guards. DEDUP FINDING (CLAUDE.md §0). Searched "binding", "surface", "SURFACE_BINDINGS", "consult", "NO_BINDING", "phase" across the tree and the improvement log. The MECHANISM all exists — `binding_for()`'s prefix resolution already sub-surfaces `repo-audit` by phase and dimension; `capability_propensity.tick_evidence` (PR #37) is already a tick-surface consult; the skills already carry a uniform consult stanza; `verify.py` already has the diagnostic-line pattern (`absent_entrypoint_line`). So this is DATA + CALLERS: no second binding table, no second event log, no second consult mechanism. 1. THE TICK IS SUB-SURFACED BY PHASE, and every phase has a caller. 18 of the 43 capabilities live on the tick; one context holding 18 is the measured 13.62% selection condition. Five phases — `tick:capacity`, `tick:dispatch`, `tick:experiments`, `tick:redirect`, `tick:learning` — named from orchestrate.sh's own "capacity -> discover -> plan -> dispatch" line and its `--- Learning cadence ---` / `[cadence] redirect ...` / `[cadence] experiment follow-up` blocks, with most bound capabilities carrying a `{kind: tick_phase, name: ...}` matcher naming the phase they land in. Each resolves to 6-8 instead of 18. THE BARE `tick` SET DOES NOT MOVE. `capability_propensity.TICK_SURFACE` is "tick", `tick_evidence()` grades exactly `binding_for("tick")`, and its selftest requires every capability with a finding projection to be in it — moving those four into a phase would silently zero the only producer of layer-2 evidence. A selftest now pins that, and pins that each phase declares something the bare set does not. The caller is `capability_advisor.py --consult-tick-phases` at `ORCH-ANCHOR: tick-phase-consult`, immediately below the #37 step and below `ORCH-ANCHOR: heartbeat-export`. It writes NO verdicts — `tick_evidence` remains the only writer and reads an unchanged binding, so its ~1.3/day ceiling is untouched. Its own volume is bounded by construction: the consult text is stable per (surface, UTC day) and the match heartbeat is idempotent on its digest, so 34 events land on the first tick of a day and 0 on the other 23. Fails open per phase, SIGALRM-bounded, always exits 0. Kill switch: ORCH_DISABLE_STEPS=tick-phase-consult (registered in cadence_registry.py, so the switch is real rather than a no-op that WARNs). 2. `verify.py` IS THE `ci` SURFACE AND NOW CONSULTS AS ONE. Three capabilities were declared for a surface no caller reached. verify.py runs on every PR and already executes the admission gate. The line reports both quantities — what the table DECLARES (identical on every machine) beside what this machine's ledger could OFFER — plus the findability pair (rows bound to some surface / rows bound to none). record=False: a verifier must not write to the ledger its own gates read. Never a skip (no PREREQ_ABSENT_MARK), never in `problems`, so exit semantics and every reported count are unchanged. 3. THE SURFACE WAS DROPPED ON THE BINDING-ONLY BRANCH. `advise()` passed only `skill` to `_record_matches` there, so a `--surface` consult that missed the classifier wrote surface:null AND skill:null — the candidate set recorded and belonging to nobody, so no drain could locate it. That is the branch an unclassifiable cadence consult ALWAYS takes. 4. AN UNSUBSTITUTED SURFACE TEMPLATE NOW SAYS SO. `binding_for` resolves by prefix, so the literal `repo-audit:phase-N` returned `repo-audit`'s surface-wide set — one capability where the phase declares four — with nothing to distinguish it from a correct answer. Three audits under identical instructions consulted 13, 9 and 2 distinct surfaces. `surface_template` is reported on all four answer paths and printed loudly; it changes neither the set nor the order. Also bound: role-adjudicator → implementation-verification, role-prompt → file-agent-issue, role-{redirect,prompt} + agy-runtime-isolation → orchestrate, redirect-{policy,plan} → closer-lane (not the opener: the opener's own prompt raises `drain_needed` and relays a stalled PR to the closer), capability:reference-sync-hygiene-test-gate → ci. Unbound is now 1: `local-model-profile-trial`, declared with `NO_BINDING` and its quarantine reason rather than left silently absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
…77) * feat(advisor): give every declared binding a caller — 22 unbound → 1 22 of 43 capabilities were bound to NO surface, so nothing could ever offer them and no amount of running produced evidence: the gate starving its own drain, one layer below the concealment rule the table already guards. DEDUP FINDING (CLAUDE.md §0). Searched "binding", "surface", "SURFACE_BINDINGS", "consult", "NO_BINDING", "phase" across the tree and the improvement log. The MECHANISM all exists — `binding_for()`'s prefix resolution already sub-surfaces `repo-audit` by phase and dimension; `capability_propensity.tick_evidence` (PR #37) is already a tick-surface consult; the skills already carry a uniform consult stanza; `verify.py` already has the diagnostic-line pattern (`absent_entrypoint_line`). So this is DATA + CALLERS: no second binding table, no second event log, no second consult mechanism. 1. THE TICK IS SUB-SURFACED BY PHASE, and every phase has a caller. 18 of the 43 capabilities live on the tick; one context holding 18 is the measured 13.62% selection condition. Five phases — `tick:capacity`, `tick:dispatch`, `tick:experiments`, `tick:redirect`, `tick:learning` — named from orchestrate.sh's own "capacity -> discover -> plan -> dispatch" line and its `--- Learning cadence ---` / `[cadence] redirect ...` / `[cadence] experiment follow-up` blocks, with most bound capabilities carrying a `{kind: tick_phase, name: ...}` matcher naming the phase they land in. Each resolves to 6-8 instead of 18. THE BARE `tick` SET DOES NOT MOVE. `capability_propensity.TICK_SURFACE` is "tick", `tick_evidence()` grades exactly `binding_for("tick")`, and its selftest requires every capability with a finding projection to be in it — moving those four into a phase would silently zero the only producer of layer-2 evidence. A selftest now pins that, and pins that each phase declares something the bare set does not. The caller is `capability_advisor.py --consult-tick-phases` at `ORCH-ANCHOR: tick-phase-consult`, immediately below the #37 step and below `ORCH-ANCHOR: heartbeat-export`. It writes NO verdicts — `tick_evidence` remains the only writer and reads an unchanged binding, so its ~1.3/day ceiling is untouched. Its own volume is bounded by construction: the consult text is stable per (surface, UTC day) and the match heartbeat is idempotent on its digest, so 34 events land on the first tick of a day and 0 on the other 23. Fails open per phase, SIGALRM-bounded, always exits 0. Kill switch: ORCH_DISABLE_STEPS=tick-phase-consult (registered in cadence_registry.py, so the switch is real rather than a no-op that WARNs). 2. `verify.py` IS THE `ci` SURFACE AND NOW CONSULTS AS ONE. Three capabilities were declared for a surface no caller reached. verify.py runs on every PR and already executes the admission gate. The line reports both quantities — what the table DECLARES (identical on every machine) beside what this machine's ledger could OFFER — plus the findability pair (rows bound to some surface / rows bound to none). record=False: a verifier must not write to the ledger its own gates read. Never a skip (no PREREQ_ABSENT_MARK), never in `problems`, so exit semantics and every reported count are unchanged. 3. THE SURFACE WAS DROPPED ON THE BINDING-ONLY BRANCH. `advise()` passed only `skill` to `_record_matches` there, so a `--surface` consult that missed the classifier wrote surface:null AND skill:null — the candidate set recorded and belonging to nobody, so no drain could locate it. That is the branch an unclassifiable cadence consult ALWAYS takes. 4. AN UNSUBSTITUTED SURFACE TEMPLATE NOW SAYS SO. `binding_for` resolves by prefix, so the literal `repo-audit:phase-N` returned `repo-audit`'s surface-wide set — one capability where the phase declares four — with nothing to distinguish it from a correct answer. Three audits under identical instructions consulted 13, 9 and 2 distinct surfaces. `surface_template` is reported on all four answer paths and printed loudly; it changes neither the set nor the order. Also bound: role-adjudicator → implementation-verification, role-prompt → file-agent-issue, role-{redirect,prompt} + agy-runtime-isolation → orchestrate, redirect-{policy,plan} → closer-lane (not the opener: the opener's own prompt raises `drain_needed` and relays a stalled PR to the closer), capability:reference-sync-hygiene-test-gate → ci. Unbound is now 1: `local-model-profile-trial`, declared with `NO_BINDING` and its quarantine reason rather than left silently absent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(verify): bound the ci consult's ledger wait, and format at the pinned width TWO REAL DEFECTS in the ci consult as first written. 1. IT COULD HANG THE VERDICT. The consult reads the ledger and `capabilities.load()` takes a BLOCKING flock. Unbounded, a stuck lock would hang `verify.py` — the project's verdict — indefinitely. Not hypothetical: it actually timed out during this session, with a dozen concurrent verify runs and the hourly tick contending for that lock. Now SIGALRM-bounded at 20s, the same shape and the same reason as `capability_propensity.tick_evidence_guarded`; a wait that expires becomes the `NOT CHECKED` line the guard already produced for an import failure, so a hang is a reported line and never a hung run. The line still carries no `PREREQ_ABSENT_MARK`, so a timeout is not a skip and cannot spend the zero-headroom ceiling. Selftested by substituting the inner call, with the healthy path re-asserted so the guard cannot swallow it. 2. `black -l 100` WOULD HAVE REFORMATTED IT. `ruff.toml` records that the Gate's format check and Autofix both run `black --line-length 100`; main is clean under it and these two files were not, so the Gate would have gone red and Autofix would have rewritten them. Reformatted at the pinned width — a bare `black` (default 88) disagrees with the Gate and is the wrong instrument here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(verify): the ci-consult selftest asserted against the live ledger lock `assert "NOT CHECKED" not in ci_consult_line()` was meant to prove the guard does not swallow a healthy answer. It asserted it through the REAL ledger, so it went red the same afternoon it was written: a dozen concurrent verify runs and the hourly tick were contending for the flock, the 20s budget expired, and the guard did exactly its job — failing a test written to prove the guard works. A machine-dependent assertion inside a check about robustness is this repo's green-locally/red-on-CI defect one level up, and I was warned about it twice. Both directions now go through a SUBSTITUTED inner call — a raising one for the timeout branch, a known-good one for the healthy branch — so neither depends on what the ledger lock is doing. The real call is still exercised, but only on the SHAPE both branches satisfy (starts with the label, carries no PREREQ_ABSENT_MARK, so a timeout can never be miscounted as a skip). Break→revert, both discriminating: making the guard return NOT CHECKED for a healthy inner call fails the new healthy assertion; deleting the except clause lets the TimeoutError escape and the selftest dies on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(advisor): drop the ci consult — #68 showed the surface has nothing to select Two sessions reached opposite conclusions about `ci` and only one can ship. I instructed the binding work to add a `surface: "ci"` consult to verify.py on the reasoning that verify.py runs on every PR and therefore IS the CI surface. The findability requirement (#68), landing concurrently, showed that reasoning is wrong: `capability-admission-gate` and `docs-drift-fix-agent` are invoked UNCONDITIONALLY by a rail. They are never OFFERED to a reasoning context, so no binding could raise their selection odds and no consult could change what runs — verify.py does not CHOOSE to run the admission gate. Both now declare `findability_category: no_surface`, which is the honest statement, and `ci` is NO_BINDING with that reason. Shipping both would have left the merged tree self-contradictory: the `ci` reason says "no caller anywhere consults a `ci` surface — not verify.py" on the same commit that adds a verify.py consult. So the consult is removed (185 lines) and #68's verdict kept. Two further reasons to prefer that direction: a consult against a suppressed surface returns `confidence: "suppressed"` and an empty set on every run, so it was a no-op in the one script whose exit semantics and printed counts CI parses; and it had already needed two follow-up commits to bound a ledger wait and to stop a selftest asserting against the live ledger lock. That is real risk for no signal. Kept from the binding work, unchanged: the five tick sub-surfaces (tick:capacity/dispatch/experiments/learning/redirect) with their consult sites, the lane `--surface` fix, and the NO_BINDING verdict for `local-model-profile-trial`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(advisor): declare the tick phase consults, and retire the lane defect record Integration between two concurrent sessions, and #68's ninth requirement caught it within minutes of landing — which is the first thing it did. The binding work added five tick sub-surfaces and a real in-tree consult (`ORCH-ANCHOR: tick-phase-consult` iterating `tick_phase_surfaces()`, below the heartbeat export). #68 added `CONSULT_SITES`, which requires every bound surface to name a caller that can be VERIFIED — the caller file must actually contain the declared literal. Neither branch knew about the other, so all five landed bound and undeclared, and the findability selftest failed with them named. Declared as a FAMILY, not five entries. `tick_phase_surfaces()` derives the phases from `TICK_PHASE_PREFIX`, so no caller names them literally and a per-surface entry could never be verified — that is exactly what `instances` exists for. One claim, five instances, caller `capability_advisor.py` with literal `TICK_PHASE_PREFIX`. Also retires both `KNOWN_UNCONSULTED` entries. #68 independently found the same defect I did — the opener and closer TOMLs consulted with `--lane` and no `--surface`, so `binding_for("")` returned {} and eleven bindings never reached the two highest-volume surfaces — and recorded it rather than patching it, correctly, since CLAUDE.md forbids a loop that edits lane prompts. The TOMLs now carry `--surface`, re-rendered and verified (bound_count 0 -> 5 and 0 -> 6), so the record is stale. The table is kept EMPTY rather than deleted, so the next stranded surface has an obvious home and the record shows the defect existed. Two mistakes of mine while resolving this, both caught before commit: a regex scoped to the file rather than to `CONSULT_SITES` deleted the tick bindings out of `SURFACE_BINDINGS` (caught by `tick_phase_surfaces()` returning `['tick:*']`), and the first attempt declared five per-surface entries the verifier could never confirm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style: black at the pinned width (100) The list was hand-wrapped; black wants one element per line. The Gate's lint-format job runs black, not ruff format, and the two disagree on assert-style wrapping — black is the authority here per ruff.toml's own note. --------- Co-authored-by: Tim Stranske <tim@stranskemo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this wires
21 of the 43 capabilities live on the hourly tick, four of them are bound to the
ticksurface incapability_advisor.SURFACE_BINDINGS— andorchestrate.shhad never calledadvise()and had never recorded aninvocation/outcomeedge against an advisorymatch. That is precisely why thecapability-propensitycadence step printedPRIOR-ONLYon every run: the loop had a measurement and no producer.New step, every tick, below
ORCH-ANCHOR: heartbeat-exportand below all four steps it grades:Live dry-run against this instance's real artifacts:
Dedup finding (CLAUDE.md §0)
Recorded in the module docstring and in the
capability-propensityledgernotes, because a plan is not durable.capability_advisor.advise()tickbinding EXISTS; tree-wide grep found no caller outside the two modules' own selftestsrecord_trigger/record_usefulnesscapability_firing_monitorcapability_effectivenessinfluence_edgesaccepted-vs-counterfactual, which no observer can ever populateNothing was rebuilt. This is the missing caller. The one genuinely new part is the observer verdict, because no module compared an observer's own output across runs.
What "it helped" means for an observer
capabilities.is_observer()is the single source of the distinction — consumed, never re-declared, andOBSERVER_MATCHER_KINDSis deliberately not widened (capabilities.pysays at the set itself why widening hides real linkage gaps).capability-propensity,cli_subcommand) records that it ran and gets no verdict, so an output-change question and a delivery question never average into one rate.It cannot manufacture evidence
Treated as a correctness requirement, not a nicety. Two independent bounds:
advise()'sadvice:<cap>:<digest>and thetrigger:/useful:idempotency keys are day-unique. Ceiling 4 match + 4 trigger + 4 outcome events/day, whatever happens; ticks 2..24 of a day write nothing at all (one small state-file read and return).Arithmetic, printed by the step itself in
report["ceiling"]:A 74x reduction, and the realised rate is lower still. A first observation establishes a baseline and records nothing.
The finding projection is the whole safety mechanism.
capability-firing-monitor'soverduerows carrysilent_days, which rises every day on its own; hashing a row whole would score the monitor `useful` on every run it will ever make — a 100% rate that measures the calendar. The projection keeps identity and verdict fields and drops everything else, and a selftest re-feeds identical findings with moved counters and timestamps and asserts NOT useful. Artifact filenames come from `cadence_registry.STEP_BY_KEY`, not a second table.Latched-gate answers for the freshness gate
Runtime rule: every run reports `verdicts_recorded` beside `gradable` / `awaiting_regeneration`, so `0 verdicts` reads as cadence when 3 are gradable and as a deadlock when 0 are.
It cannot stall the tick
No gh, no network, no subprocess, no dispatch. Per-capability guards, a whole-run wrapper, and a SIGALRM budget over the one blocking wait in the path (the ledger flock; ledger writes are tmp+
os.replace, so an interrupt cannot tear it). Handled failures exit 0 and the shell'sifcovers the rest.Pre-existing defect fixed on the way
advise()'s binding-only branch returned real capabilities withuseful: trueand recorded no match events — so the fix that made a declared binding survive a classification miss covered the answer and not the evidence. A surface whose words never hit the keyword vocabulary (the tick's cadence pass, exactly) could never accumulate a candidate set:experiments()saw triggers and outcomes on trials with zero candidates, no control arm, and no attributable skill — which also meant the demotion drain could never run for such a surface.Human involvement
Zero. Report-only, nothing queues, nothing accumulates, no owner touchpoint added.
Kill switch
ORCH_TICK_EVIDENCE_DISABLED=1— module-side, works from any caller.ORCH_DISABLE_STEPS=tick-capability-evidence— shell-side, the repo's one mechanism (registered incadence_registry, so a typo still WARNs).Either alone makes the tick behave exactly as it did before: no consult, no ledger event, no state file. Both are exercised by selftests, including a positive control that the same fresh artifact IS evaluated once the switch is off, so the assertion discriminates rather than describing an inert path.
No second store
Evidence goes to the existing capability ledger events via
record_trigger/record_usefulnessand the advisor'smatchheartbeat. The state file holds only the last-seen artifact mtime and finding fingerprint — what "did the output change" needs to compare against, the same shape and reason ascapability_firing_monitor's history file.Verification
python3 verify.pyis byte-identical to the pre-change baseline on this machine:365 passed, 3 failed, 368 collected,82 of 82 selftest modules ran,0/24skipped. The 3 failures are the documented pre-existing ones, all traced to theevidence-acquisitionledger row whose module is not committed (another session's in-flight work) — confirmed by re-running against the pre-change tree.No new pytest test, so
.verify-floor.jsonis untouched; the guards live incapability_propensity --selftest, whichverify.pyruns and counts.ARCHITECTURE.mdgains prose only. No stage, component, rail/role classification or feedback surface changed, soorchestrator-loop.svgis correctly unchanged.break -> revert, all 11 verified to discriminate
useful: True>->>=is_observercheck removedunprojectablegraded as unchanged--budget-seconds 0promoted to the defaultbudget_exhaustednever reportedReverted clean afterwards; both modules' selftests green.
End-to-end proof against this instance's real artifacts (isolated ledger + state dir, nothing written to the live ledger)
🤖 Generated with Claude Code