From 86a8d7696688970590f5f45cbbc53f45c09144c0 Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 00:08:21 -0500 Subject: [PATCH 1/2] feat(propensity): make a reasoned DECLINE learnable, as a third state with a kind Two independent audit rounds on 2026-08-23 reached the same conclusion: propensity carried information exactly once, because most candidates sat at the uninformative prior. The missing input was not more consults but DECLINES -- a capability declined on repo-specific grounds looked IDENTICAL in the ledger to one nobody ever considered (`trig 0, use 0, no 0`). Four of nine decisions in one round and 21 of the offers in another were reasoned rejections and none was learnable. A DECLINE IS A THIRD STATE, NOT A NEGATIVE OUTCOME, and the separation is structural rather than conventional. A decline means the capability did NOT run, so recording it as an `outcome` would bucket it into `not_useful` -- asserting we tried it and it did not help, about something that never executed, corrupting the one signal declines exist to sharpen. So a decline rides on a `match` event (it genuinely WAS offered, which is the only claim the event type makes) tagged `source=capability_decline`, and `usefulness()` reads `outcome` events only. There is no code path from a decline to the posterior. No new event type, no new store: `record_promotion` already carries a non-match fact on `match` distinguished by `metadata.source`. `triggered` / `declined` / `not_triggered_silently` now partition the candidate set, so "rejected on stated grounds", "offered and ignored" and "never considered" are three different findings. `propensity()` reports the decline count BESIDE the posterior -- "prior, no evidence" and "prior, no evidence, four reasoned rejections" were previously identical readings. AND A DECLINE HAS A KIND, because one undifferentiated count licenses the wrong correction. A third audit round declined 25 offers across six classes with opposite fixes: `testgen-lane` matched CORRECTLY three times and was structurally impossible (a read-only audit has no commit target) -- fix: nothing; `offload` was declined at nine surfaces because it is declared surface-wide and a one-subsystem audit has nothing big enough to hand off -- fix: a precondition or a narrower declaration; and `frontend-verifier`, declined on two frontend-less repos, then produced the second-strongest finding of an audit on a repo that DOES have a display surface -- fix: evaluate the condition, do not weaken the binding. `demotable` is therefore a property of the KIND (`DECLINE_KINDS`), declared once: only `wrong_match` and `scope_too_small` can propose a demotion. An unknown kind is refused rather than coerced. Demotion consumes declines as a second, much lower-floor rule, and the two rules read DISJOINT populations: `never_triggered` counts only offers where nothing was said. The first draft counted every offer, so eight honest `no_landing_zone` declines tripped the silent-non-use rule and demoted a correct match through the back door -- the exact wrong correction the taxonomy exists to prevent, arriving via the other rule. Attribution is now on the event: the advisor records `surface` on each `match`. It recorded only `skill`, and the CLI has no `--skill` flag, so every `--surface` consult wrote `skill: null` and the entire control arm of those audit rounds was unattributable to the surface that produced it. Reachable from bash (`decline --capability --experiment --reason --kind --surface`) and from MCP (`capability_decline`), because the surfaces that make these decisions are skills and lane automations, not Python. The MCP guard is AST-based: a substring check over the handler body was written first and did NOT discriminate -- deleting `surface=` from the call left the word in the response dict and the break stayed green. Break->revert demonstrations, all confirmed to fail without the fix: decline-as-outcome moves the posterior; a blank reason is accepted; demotion ignores declines; a triggered capability is still demoted; surface attribution dropped; the partition collapses; every kind treated as demotable; the silent rule counting declines; an unknown kind coerced; `no_landing_zone` and `precondition_unmet` declared demotable; `surface` advertised but not forwarded; `reason` dropped from the schema; `capability_id` hardcoded. Pre-existing on this machine and unrelated: 3 pytest failures for the `evidence-acquisition` ledger row (machine-local, 0 references in committed capabilities.py), confirmed identical with the change stashed. Co-Authored-By: Claude Opus 5 --- ARCHITECTURE.md | 50 ++- README.md | 9 +- capability_advisor.py | 12 +- capability_propensity.py | 743 +++++++++++++++++++++++++++++++++++++-- mcp_server.py | 188 +++++++++- 5 files changed, 963 insertions(+), 39 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 051a093..ad5c1a6 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -256,8 +256,54 @@ daily on its own and hashing a row whole would score the monitor useful on every make. **Demotion is the drain.** Bindings that could only grow end with every surface holding all 43 — -the exact condition binding prevents. A capability bound to a surface that never triggers it across -`DEMOTION_MIN_TRIALS` resolved experiments is proposed for removal. +the exact condition binding prevents. Two rules propose removal, and they read **disjoint +populations**: `never_triggered` counts offers where nothing was said (`not_triggered_silently`) +across `DEMOTION_MIN_TRIALS`; `declined_with_reason` counts *demotable* declines across +`DEMOTION_MIN_DECLINES`, a much lower floor because a stated reason is much better evidence. A single +trigger at that surface disqualifies both — something actually used there is not a demotion candidate +however often it is passed over. The disjointness is not tidiness: counting declines as silent offers +lets an honest decline of a *correct* match trip the rule meant for capabilities nobody spoke about. + +**A DECLINE IS A THIRD STATE, NOT A NEGATIVE OUTCOME** (`capability_propensity.record_decline`, CLI +`decline`, MCP `capability_decline`). Two independent audit rounds on 2026-08-23 reached the same +finding: propensity carried information exactly once, because most candidates sat at the +uninformative prior, and the missing input was not more consults but reasoned rejections — a +capability declined on repo-specific grounds looked identical in the ledger to one nobody ever +considered. So `triggered` / `declined` / `not_triggered_silently` now partition the candidate set, +and *never considered* is the fourth case of not being a candidate at all. + +The discipline that makes this safe is a separation, not a convention. A decline means the capability +did NOT run, so recording it as an `outcome` would bucket it into `not_useful` — asserting we tried +it and it did not help, about something that never executed, corrupting the one signal declines +exist to sharpen. A decline is therefore carried on a `match` event (it genuinely *was* offered) +tagged `source=capability_decline`, and `usefulness()` reads `outcome` events only. There is no code +path from a decline to the posterior. `propensity()` reports the decline count **beside** the +posterior for exactly this reason: "prior, no evidence" and "prior, no evidence, four reasoned +rejections" are opposite readings that were previously identical. A decline requires a reason and is +refused without one, the same way `record_usefulness` refuses an unevidenced verdict. + +Attribution is on the event: the advisor records the `surface` on each `match`, because it recorded +only `skill` before and the CLI has no `--skill` flag — so every `--surface` consult wrote +`skill: null` and its whole control arm was unattributable to the surface that produced it. + +**And a decline has a KIND, because the kinds imply opposite corrections.** One undifferentiated +"declined" column licenses the wrong fix. A third audit round on 2026-08-23 separated them and the +separation is the finding: `testgen-lane` matched **correctly** three times in a read-only audit and +was structurally impossible every time (no commit target), while `offload` was declined at 9 of 12 +surfaces because it is declared surface-wide and a one-subsystem audit has nothing big enough to hand +off. The first calls for no change at all; the second calls for a precondition or a narrower +declaration. And `frontend-verifier` — declined on two frontend-less repos, then the +second-strongest finding of an audit on a repo that *does* have a display surface — is the same +lesson from the other side: two negatives are not a verdict on a binding. + +So `demotable` is a property of the **kind**, declared once in `DECLINE_KINDS` and read nowhere +else. `wrong_match` and `scope_too_small` may demote; `precondition_unmet`, `no_landing_zone`, +`gated_off`, `deferred` and the `unspecified` default may not — they are counted and reported, and +cannot clear the floor. `precondition_unmet` is the load-bearing one: the correct response to a +capability whose condition does not hold here is to **evaluate the condition, not to weaken the +binding**, which is why it is recorded and inert. An unknown kind is refused rather than coerced, +because a typo silently becoming `unspecified` would discard the classification the caller believed +it had made. First live run found a real gap: `deliberate-break-verifier` showed 69 hand-done instances in 1,765 closer rounds while bound only to the opener, and the loop promoted it. **It must not ratchet:** raising selection pressure whenever a capability was not chosen, diff --git a/README.md b/README.md index 217702e..58ed9d0 100644 --- a/README.md +++ b/README.md @@ -270,8 +270,13 @@ safety switch, not dead code. "Not logged in" with a live one). Prints the per-seat refresh hint; never prints secret values. `--json` for machines; exit 1 only when a seat is *definitively* broken, never on UNKNOWN. - **mcp_server.py** — exposes the fleet to any MCP client (registered user-scope as `orchestrator`): - capacity, fleet summary, route weights, owner-question list/answer, resume hints. Read-only plus - the two bounded owner-question actions; no dispatch through this door. + capacity, fleet summary, route weights, capability advice, owner-question list/answer, resume + hints. Read-only plus three bounded actions — the two owner-question ones and + `capability_decline`, which records that an OFFERED capability was rejected, why, and of which + `kind`. No dispatch through this door. A decline is append-only evidence and never a verdict: it + cannot reach the usefulness posterior, because the capability did not run. Only the kinds that + indict a binding (`wrong_match`, `scope_too_small`) can propose a demotion — a correct match with + nowhere to land is recorded and never counted against the capability. - **Cadence resilience** — failing daily/weekly steps back off (`.fail-` stamps, `ORCH_CADENCE_RETRY_HOURS`) and ALERT after N consecutive failures instead of retrying hourly. - **Per-step kill switch** — `ORCH_DISABLE_STEPS="feature-scan,redirect-sweep"` (comma or space diff --git a/capability_advisor.py b/capability_advisor.py index 7f34836..4dab5bc 100644 --- a/capability_advisor.py +++ b/capability_advisor.py @@ -481,7 +481,8 @@ def advise(text: str, *, repository: str = "", lane: str = "opener", skill: str } if record and matched: # Asking the question is itself the observation that improves the answer. - result["recorded_matches"] = _record_matches(result, skill=skill, path=path) + result["recorded_matches"] = _record_matches(result, skill=skill, + surface=surface or skill, path=path) return result @@ -848,7 +849,7 @@ def experiment_id(task: str) -> str: return "advice:" + hashlib.sha1(str(task or "").encode()).hexdigest()[:12] -def _record_matches(advice: dict, *, skill: str = "", path=None) -> int: +def _record_matches(advice: dict, *, skill: str = "", surface: str = "", path=None) -> int: """Record that these capabilities matched REAL work, with the skill that surfaced it. Uses the existing `match` heartbeat rather than a new store: a capability whose declared trigger @@ -859,6 +860,12 @@ def _record_matches(advice: dict, *, skill: str = "", path=None) -> int: The skill is carried in the event metadata, so `learned_associations()` can later aggregate skill -> capability from accumulated observations. Idempotent per (capability, exact task), so repeating the same query does not inflate frequency while distinct tasks still accumulate. + + THE SURFACE IS RECORDED TOO (2026-08-23). It was not, and the CLI has no `--skill` flag at all — + so every `--surface` consult wrote `skill: null`, and the two audit rounds of 2026-08-23 produced + 33 candidate-offers that `propose_demotions` and `missed_selection` could not attribute to any + surface. The control arm existed in the ledger and was unreachable, which is the same + "recorded but unusable" defect one level down from the declines this change is about. """ import hashlib digest = hashlib.sha1(str(advice.get("task") or "").encode()).hexdigest()[:12] @@ -871,6 +878,7 @@ def _record_matches(advice: dict, *, skill: str = "", path=None) -> int: path=path or capabilities.REG, idempotency_key=f"advice:{entry['capability_id']}:{digest}", metadata={"source": "capability_advisor", "skill": skill or None, + "surface": surface or None, "task_type": entry.get("matched_task_type")}, ) written += 1 if ok else 0 diff --git a/capability_propensity.py b/capability_propensity.py index ad464f8..7a96836 100644 --- a/capability_propensity.py +++ b/capability_propensity.py @@ -55,8 +55,57 @@ not write to the Brain. It ranks advice. Triggering stays with the caller that already had the authority to trigger, so a bad propensity can misorder a recommendation list and nothing else. +DECLINES ARE A THIRD STATE, NOT A NEGATIVE OUTCOME (2026-08-23). Two independent audit rounds the +same day reached the same conclusion: propensity carried information exactly once because 11 of 13 +candidates sat at the uninformative 0.5 floor, and the missing input was not more consults but +DECLINES. Four of nine decisions in one round and 20 of 22 candidate-offers in the other were +reasoned rejections, and none was learnable, because a capability declined on repo-specific grounds +looked IDENTICAL in the ledger to one nobody ever considered (`trig 0, use 0, no 0`). + +THE TRAP, and it is the whole difficulty. A decline means the capability was NOT TRIGGERED. Recording +it as an `outcome` would land it in `not_useful` — a false statement that we tried it and it did not +help, about something that never ran, corrupting the exact signal this module exists to sharpen. So a +decline is carried on a `match` event (it WAS offered; that part is true) tagged +`source=capability_decline`, and `experiments()` buckets useful/not_useful from `outcome` events +ONLY. The separation is structural, not conventional: there is no code path by which a decline can +reach the posterior. `propensity()` reports the decline count beside the posterior precisely so the +two can be read together without being mixed. + +WHAT A DECLINE FEEDS, exhaustively: + 1. THE DISTINCTION between `declined` (offered, rejected, reason stated) and + `not_triggered_silently` (offered, ignored) — the reported defect above. The three states + partition `candidates`, and the selftest asserts that partition. + 2. `propose_demotions` — a binding declined at a surface across `DEMOTION_MIN_DECLINES` runs is a + demotion candidate. This is far better evidence than silent non-use, so its floor is much lower + than `DEMOTION_MIN_TRIALS`, and the reasons travel with the proposal. + 3. NOTHING ELSE. It must not, and structurally cannot, move `propensity`. + +AND A DECLINE HAS A KIND, because one undifferentiated count licenses the wrong correction. A third +audit round the same day declined 25 offers across six reason classes, and the classes imply OPPOSITE +fixes: + + * `testgen-lane` matched CORRECTLY three times and was structurally impossible each time -- a + read-only audit has no commit target. Fix: NOTHING. + * `offload` was declined at nine surfaces, always structurally, because it is declared + surface-wide and a one-subsystem audit has nothing big enough to hand off. Fix: a precondition, + or a narrower declaration. + * `frontend-verifier` was declined on two frontend-less repos and then, on a repo that DOES have a + display surface, produced the second-strongest finding of that audit -- one the code-reading path + had missed. Fix: EVALUATE THE CONDITION, do not weaken the binding. Down-weighting it on the two + negatives alone would have cost that finding. + +So `demotable` is a property of the KIND (`DECLINE_KINDS`), declared once and read nowhere else. +Exactly two kinds indict a binding; the rest are counted, reported, and cannot clear the demotion +floor. An unknown kind is refused rather than coerced, because a typo silently becoming +`unspecified` would discard the classification the caller believed it had made. + +NO NEW STORE, AGAIN. `capabilities.EVENT_FIELDS` already has `match`; `record_promotion` already +carries a non-match fact on it distinguished by `metadata.source`. A decline follows that precedent +rather than adding an eighth event type or a second table. + python3 capability_propensity.py report python3 capability_propensity.py experiments + python3 capability_propensity.py decline --capability X --experiment advice:abc --reason "..." python3 capability_propensity.py --json report python3 capability_propensity.py --selftest """ @@ -89,6 +138,64 @@ # An outcome heartbeat carries the verdict in metadata under this key. USEFUL_KEY = "useful" ADVICE_REF_PREFIX = "advice:" +# A DECLINE rides on a `match` event, tagged by source. `match` is the honest carrier: the capability +# genuinely WAS offered, which is the only claim the event type itself makes. Everything that +# distinguishes a decline lives in metadata, and no reader of `outcome` events can see it -- which is +# why a decline cannot reach the usefulness posterior even by accident. +DECLINE_SOURCE = "capability_decline" +DECLINE_REASON_KEY = "reason" +DECLINE_KIND_KEY = "decline_kind" + +# THE KIND OF DECLINE, because the kinds imply OPPOSITE corrections and one undifferentiated +# "declined" column would license the wrong one. Measured, not theorised: a third audit round on +# 2026-08-23 declined 22 offers across six reason classes, and only some are the binding's fault. +# `demotable` is therefore a property OF THE KIND, declared here once, and read by nothing else. +DECLINE_KINDS: dict[str, dict] = { + # It does not fit this work. The binding or the matcher is wrong. + "wrong_match": {"demotable": True, "fix": "the matcher or the binding"}, + # A CORRECT match declared too broadly. `offload` was offered at 9 of 12 surfaces in one run and + # declined at all 9, always structurally, because a one-subsystem audit has no read big enough + # to pay for a dispatch. Narrowing the declaration IS a demotion, so this counts -- and the + # proposal carries the fix text, because "add a precondition" is the other valid answer. + "scope_too_small": {"demotable": True, + "fix": "a precondition or a narrower declaration, not a lower rank"}, + # A CORRECT match whose declared PRECONDITION does not hold here: the instrument is aimed at + # another system (`switch-review` audits THIS repo's switches; the gate under audit was in + # another), or at a surface this repo does not have (`frontend-verifier` on a repo with no UI). + # NOT DEMOTABLE, and this is the most important row in the table. `frontend-verifier` was + # declined on two frontend-less repos and then, on a repo that DOES have a display surface, + # produced the second-strongest finding of that audit -- one the code-reading path had missed. + # Down-weighting it on the two negatives alone would have cost that finding. THE FIX IS TO + # EVALUATE THE CONDITION, NOT TO WEAKEN THE BINDING. + "precondition_unmet": {"demotable": False, + "fix": "declare and EVALUATE the capability's precondition (applies_to, " + "an observable surface); the binding is right where it holds"}, + # A CORRECT match the deliverable shape made impossible: `testgen-lane` matched correctly three + # times in a read-only audit with no commit target. THE FIX IS NOTHING, so this must never + # demote -- down-weighting here would punish a capability for being right. + "no_landing_zone": {"demotable": False, + "fix": "nothing — the match was correct and the deliverable had nowhere to " + "put the result"}, + # Correct match held behind a deliberate default-OFF switch or a shadow status. The gate is the + # subject, and it moves on its own evidence, not on this. + "gated_off": {"demotable": False, "fix": "the capability's own gate, on its own evidence"}, + # Wanted and not affordable this run ("the one I most regret declining"). + "deferred": {"demotable": False, "fix": "nothing — wanted, not affordable this run"}, + # The caller did not classify it. Recorded, so offered-vs-never-considered still works, and NOT + # demotable: an unclassified decline that could demote is precisely the wrong correction arriving + # by default, which is the failure this vocabulary exists to prevent. + "unspecified": {"demotable": False, "fix": "unknown — the caller did not classify it"}, +} +DECLINE_KIND_DEFAULT = "unspecified" + + +def decline_kind_demotable(kind: str) -> bool: + """Whether a decline of this kind may drive a demotion. One lookup, so it cannot drift.""" + return bool((DECLINE_KINDS.get(str(kind)) or DECLINE_KINDS[DECLINE_KIND_DEFAULT])["demotable"]) +# The surface a decline (or a match) was recorded for. Attribution has to be on the EVENT: the +# advisor's own consults recorded `skill=None` for every `--surface` call, so the control arm of the +# two 2026-08-23 audit rounds was unattributable to `repo-audit:*` at all. +SURFACE_KEY = "surface" def _events(cap: dict) -> list[dict]: @@ -123,13 +230,32 @@ def experiments(*, path=None, window_days: int = WINDOW_DAYS, now: int | None = if not exp or not _within_window(event, now=now, window_days=window_days): continue trial = trials.setdefault(exp, {"experiment_id": exp, "candidates": [], "triggered": [], - "useful": [], "not_useful": [], "skills": set()}) + "useful": [], "not_useful": [], "declined": [], + "decline_reasons": {}, "decline_kinds": {}, + "skills": set()}) meta = event.get("metadata") or {} - if meta.get("skill"): - trial["skills"].add(str(meta["skill"])) + # SURFACE and SKILL are the same attribution axis read from two keys. `--surface` calls + # recorded no `skill` at all, so a surface-attributed run was invisible to + # `propose_demotions` and `missed_selection` -- the control arm existed and could not be + # located. Reading both keys fixes that without a second attribution field. + for key in ("skill", SURFACE_KEY): + if meta.get(key): + trial["skills"].add(str(meta[key])) etype = event.get("type") or event.get("event_type") - if etype == "match" and cap_id not in trial["candidates"]: - trial["candidates"].append(cap_id) + if etype == "match": + if cap_id not in trial["candidates"]: + trial["candidates"].append(cap_id) + # A DECLINE. It is a candidate (it was offered) and it is NOT an outcome. This branch + # is the only place a decline is read, and it sits inside `match` on purpose: the + # `outcome` branch below cannot see it, so `useful`/`not_useful` cannot absorb it. + if meta.get("source") == DECLINE_SOURCE: + if cap_id not in trial["declined"]: + trial["declined"].append(cap_id) + reason = str(meta.get(DECLINE_REASON_KEY) or "").strip() + if reason: + trial["decline_reasons"].setdefault(cap_id, reason) + trial["decline_kinds"].setdefault( + cap_id, str(meta.get(DECLINE_KIND_KEY) or DECLINE_KIND_DEFAULT)) elif etype == "invocation" and cap_id not in trial["triggered"]: trial["triggered"].append(cap_id) elif etype == "outcome": @@ -143,6 +269,24 @@ def experiments(*, path=None, window_days: int = WINDOW_DAYS, now: int | None = # named for this exact task and NOT triggered. Reporting it is not optional -- an experiment # with an unreported control arm is a testimonial. trial["not_triggered"] = sorted(set(trial["candidates"]) - set(trial["triggered"])) + # A capability that was declined and LATER triggered in the same trial ran; the trigger + # wins. Otherwise a change of mind would be counted as a rejection forever. + trial["declined"] = sorted(set(trial["declined"]) - set(trial["triggered"])) + trial["decline_reasons"] = {c: r for c, r in sorted(trial["decline_reasons"].items()) + if c in trial["declined"]} + trial["decline_kinds"] = {c: k for c, k in sorted(trial["decline_kinds"].items()) + if c in trial["declined"]} + # THE DEMOTABLE SUBSET, separated here so no downstream reader has to remember which kinds + # are the binding's fault. `no_landing_zone` was a CORRECT match; it belongs in `declined` + # and must never appear here. + trial["declined_demotable"] = sorted( + c for c in trial["declined"] if decline_kind_demotable(trial["decline_kinds"].get(c))) + # THE THIRD STATE, named. `triggered` + `declined` + `not_triggered_silently` partition + # `candidates` exactly, which is the property that makes "rejected on stated grounds" + # distinguishable from "offered and ignored" from "never considered" (not a candidate). + trial["not_triggered_silently"] = sorted(set(trial["not_triggered"]) - set(trial["declined"])) + # RESOLVED means an OUTCOME landed. A decline resolves nothing -- the capability never ran, + # so there is nothing to have been useful or useless about. trial["resolved"] = bool(trial["useful"] or trial["not_useful"]) out.append(trial) return sorted(out, key=lambda t: t["experiment_id"]) @@ -157,7 +301,8 @@ def usefulness(*, path=None, window_days: int = WINDOW_DAYS, now: int | None = N caps = capabilities.load_declared(path or capabilities.REG) rows: dict[str, dict] = { cap_id: {"capability_id": cap_id, "candidates": 0, "triggered": 0, - "useful": 0, "not_useful": 0, "status": cap.get("status")} + "useful": 0, "not_useful": 0, "declined": 0, "declined_demotable": 0, + "declines_by_kind": {}, "status": cap.get("status")} for cap_id, cap in sorted(caps.items()) } for trial in experiments(path=path, window_days=window_days, now=now): @@ -167,11 +312,23 @@ def usefulness(*, path=None, window_days: int = WINDOW_DAYS, now: int | None = N for cap_id in trial["triggered"]: if cap_id in rows: rows[cap_id]["triggered"] += 1 + # DECLINES ARE COUNTED AND KEPT OUT OF EVERY RATE BELOW. `resolved` is deliberately + # `useful + not_useful` and nothing else, so this column can never leak into the posterior. + for cap_id in trial["declined"]: + if cap_id in rows: + rows[cap_id]["declined"] += 1 + kind = trial["decline_kinds"].get(cap_id, DECLINE_KIND_DEFAULT) + rows[cap_id]["declines_by_kind"][kind] = \ + rows[cap_id]["declines_by_kind"].get(kind, 0) + 1 + for cap_id in trial["declined_demotable"]: + if cap_id in rows: + rows[cap_id]["declined_demotable"] += 1 for key in ("useful", "not_useful"): for cap_id in trial[key]: if cap_id in rows: rows[cap_id][key] += 1 for row in rows.values(): + row["declines_by_kind"] = dict(sorted(row["declines_by_kind"].items())) resolved = row["useful"] + row["not_useful"] row["resolved"] = resolved row["trigger_rate"] = (row["triggered"] / row["candidates"]) if row["candidates"] else None @@ -202,6 +359,18 @@ def propensity(capability_id: str, *, path=None, window_days: int = WINDOW_DAYS, "floored": value > posterior, # BLOCKING quantity and DRAINABLE quantity, together, always. "evidence_count": resolved, + # REPORTED, NEVER MIXED. "0.5 with 0 evidence" and "0.5 with 0 evidence and 4 reasoned + # rejections" are opposite findings that were indistinguishable until declines existed. + # Printing them side by side is the point; the posterior above is computed from `resolved`, + # which is `useful + not_useful` and cannot include this number. + "declines": row["declined"], + # THE KIND SPLIT, not just the count. "9 declines" invites narrowing a binding; "9 declines, + # 0 of them the binding's fault" forbids it. A third audit round found two of its three + # decline classes were CORRECT matches, so a bare number licenses the wrong fix two times + # in three. + "declines_by_kind": dict(row["declines_by_kind"]), + "declines_demotable": row["declined_demotable"], + "declines_excluded_from_posterior": True, # DERIVED, never asserted. This field was a hardcoded True until a break-test removed the # floor and it still claimed the gate was drainable -- a predicate that cannot fail is # decoration, and decoration is exactly what this repo's prose rules turned out to be. @@ -210,7 +379,10 @@ def propensity(capability_id: str, *, path=None, window_days: int = WINDOW_DAYS, "basis": ("no resolved outcomes yet — optimistic prior plus an unconditional floor, so this " "can still be sampled and can therefore still earn evidence" if not resolved else - f"{row['useful']} of {resolved} resolved trials were useful"), + f"{row['useful']} of {resolved} resolved trials were useful") + + (f"; declined with a stated reason {row['declined']} time(s) " + f"({row['declined_demotable']} of them attributable to the binding), which is " + f"recorded but never scored — it never ran" if row["declined"] else ""), "window_days": window_days, } @@ -276,6 +448,18 @@ def report(*, path=None, window_days: int = WINDOW_DAYS, now: int | None = None) # dashboard that looks informative while reporting nothing. "capabilities_with_evidence": len(resolved_caps), "capabilities_without_evidence": stats["capability_count"] - len(resolved_caps), + # THE POPULATION THE 0.5 FLOOR USED TO HIDE. A capability with no outcome evidence but + # several reasoned rejections is not "unmeasured"; it is measured on a different axis. This + # count says how much of the un-evidenced population is actually of that kind. + "capabilities_declined_with_reason": sum(1 for r in ranked if r["declined"]), + "decline_count": sum(len(t["declined"]) for t in trials), + # Both quantities again: how many declines exist, and how many of them are actually a + # statement about the binding rather than about the work's shape. + "decline_demotable_count": sum(len(t["declined_demotable"]) for t in trials), + "declines_by_kind": {k: sum(v for r in ranked + for kk, v in r["declines_by_kind"].items() if kk == k) + for k in sorted(DECLINE_KINDS) + if any(k in r["declines_by_kind"] for r in ranked)}, "ranked": ranked, "experiments": trials, } @@ -1238,6 +1422,353 @@ def daily(overdue, regressed, tick): "shape change reported not scored, one verdict per capability per day, kill switch inert)") +def record_decline(capability_id: str, experiment_id: str, *, reason: str, surface: str = "", + kind: str = DECLINE_KIND_DEFAULT, path=None, + metadata: dict | None = None) -> bool: + """This candidate was OFFERED and deliberately NOT used, for a stated reason. + + THE THIRD STATE. `record_trigger` says it ran; `record_usefulness` says whether running helped. + Neither can express "it was the wrong tool here, and here is why" — and until this existed a + reasoned rejection was byte-identical in the ledger to a capability nobody ever considered. + + WHAT THIS MUST NOT DO, and structurally cannot. It writes a `match`, never an `outcome`, so it + can never enter the `useful`/`not_useful` buckets `propensity()` is computed from. Recording a + decline as a negative outcome would assert that we tried it and it did not help — a false + statement about something that never ran, and it would corrupt the one signal declines exist to + sharpen. + + `reason` is REQUIRED and refused when blank, exactly as `record_usefulness` refuses an + unevidenced verdict: an unexplained decline is indistinguishable from inattention, which is the + state this replaces. `surface` is optional but load-bearing — without it the decline is recorded + and readable but cannot be attributed to a surface, so it cannot feed `propose_demotions`. + + `kind` says WHAT KIND of decline, from `DECLINE_KINDS`, because the kinds imply opposite fixes: + `wrong_match` indicts the binding, while `no_landing_zone` says the match was correct and the + deliverable had nowhere to put the result. An unknown kind is refused rather than coerced — a + typo silently becoming `unspecified` would hide the classification the caller thought it made. + Omitting it yields `unspecified`, which is recorded and can never demote. + + Idempotent per (capability, experiment), so replaying a backfill cannot inflate the count. + """ + if not str(reason).strip(): + raise ValueError("a decline requires a reason naming why this capability was not the right " + "tool here; an unexplained decline is indistinguishable from inattention") + if str(kind) not in DECLINE_KINDS: + raise ValueError(f"unknown decline kind {kind!r}; expected one of {sorted(DECLINE_KINDS)}") + if not experiment_id.startswith(ADVICE_REF_PREFIX): + raise ValueError(f"experiment_id must start with {ADVICE_REF_PREFIX!r}: {experiment_id!r}") + return capabilities.heartbeat( + capability_id, "match", ref=experiment_id, path=path or capabilities.REG, + idempotency_key=f"decline:{capability_id}:{experiment_id}", + metadata={"source": DECLINE_SOURCE, DECLINE_REASON_KEY: str(reason)[:400], + DECLINE_KIND_KEY: str(kind), SURFACE_KEY: surface or None, + **(metadata or {})}) + + +def _selftest_declines() -> None: + """A DECLINE IS A THIRD STATE. It must be visible, attributable, and inert on the posterior. + + The last property is the whole difficulty and the reason this function exists. Bucketing a + decline as a negative outcome would assert "we tried it and it did not help" about a capability + that never ran — a false statement, and it would corrupt the exact signal declines were added to + sharpen. Every assertion below was written by breaking it first: + + * routing `record_decline` through an `outcome` heartbeat moves the posterior -> caught here; + * dropping the blank-reason guard -> caught here; + * dropping the decline rule from `propose_demotions` -> caught here; + * dropping the `triggered` guard so a used capability is still demoted -> caught here; + * dropping the surface key from `experiments()` -> the demotion loses its attribution and is + caught here. + """ + import tempfile + from pathlib import Path + import capability_advisor + + with tempfile.TemporaryDirectory(prefix="decline-selftest-") as td: + ledger = Path(td) / "capabilities.json" + rows = {} + for cid in ("helper", "wrong-tool", "used-here"): + cap = capabilities._blank_capability(cid) + cap["status"] = "generated" + cap["matcher"] = {"field": "task_type", "operator": "in", "value": ["testgen"]} + rows[cid] = cap + capabilities.save(rows, ledger) + + exp = "advice:decline000001" + for cid in ("helper", "wrong-tool"): + capabilities.heartbeat(cid, "match", ref=exp, path=ledger, + idempotency_key=f"m:{cid}", metadata={"skill": "t-dec"}) + record_trigger("helper", exp, path=ledger) + record_usefulness("helper", exp, useful=True, evidence="found a real defect", path=ledger) + + # ---- 1. THE POSTERIOR MUST NOT MOVE. Measured on a capability that HAS evidence, so a + # decline leaking in as `not_useful` would visibly drag a real number down rather than + # merely appearing beside a prior. + before = propensity("helper", path=ledger) + assert before["evidence_count"] == 1 and before["propensity"] == 0.6667, before + for i in range(3): + e = f"advice:helperdecl{i:03d}" + assert record_decline("helper", e, reason=f"wrong phase for this work ({i})", + surface="t-dec", kind="wrong_match", path=ledger) + after = propensity("helper", path=ledger) + assert after["propensity"] == before["propensity"], (before, after) + assert after["posterior_mean"] == before["posterior_mean"], (before, after) + assert after["evidence_count"] == before["evidence_count"], (before, after) + # ...and it is nonetheless VISIBLE. Inert must not mean invisible: "0.5, no evidence" and + # "0.5, no evidence, three reasoned rejections" are the two readings the audits could not + # tell apart. + assert after["declines"] == 3, after + assert after["declines_excluded_from_posterior"] is True, after + assert "never scored" in after["basis"], after["basis"] + u = usefulness(path=ledger)["rows"]["helper"] + assert u["declined"] == 3 and u["useful"] == 1 and u["not_useful"] == 0, u + assert u["usefulness_rate"] == 1.0, u # not 0.25 — declines are not failures + + # ---- 2. THE THREE STATES PARTITION THE CANDIDATE SET. This is what makes "declined with a + # reason", "offered and ignored" and "never considered" three different findings. + d_exp = "advice:decline000002" + for cid in ("helper", "wrong-tool", "used-here"): + capabilities.heartbeat(cid, "match", ref=d_exp, path=ledger, + idempotency_key=f"m2:{cid}", metadata={"surface": "t-dec"}) + record_trigger("used-here", d_exp, path=ledger) + record_decline("wrong-tool", d_exp, reason="this repo has no front end", surface="t-dec", + kind="wrong_match", path=ledger) + trial = next(t for t in experiments(path=ledger) if t["experiment_id"] == d_exp) + assert trial["declined"] == ["wrong-tool"], trial + assert trial["decline_reasons"]["wrong-tool"] == "this repo has no front end", trial + assert trial["decline_kinds"]["wrong-tool"] == "wrong_match", trial + assert trial["declined_demotable"] == ["wrong-tool"], trial + assert trial["triggered"] == ["used-here"], trial + assert trial["not_triggered_silently"] == ["helper"], trial + assert (set(trial["triggered"]) | set(trial["declined"]) + | set(trial["not_triggered_silently"])) == set(trial["candidates"]), trial + assert not (set(trial["triggered"]) & set(trial["declined"])), trial + assert not (set(trial["declined"]) & set(trial["not_triggered_silently"])), trial + # A DECLINE RESOLVES NOTHING. `resolved` gates the usefulness population, so a decline that + # resolved a trial would make the denominator lie in the other direction. + assert trial["resolved"] is False, trial + assert trial["useful"] == [] and trial["not_useful"] == [], trial + # It IS a candidate: a decline is evidence the capability was offered. + assert "wrong-tool" in trial["candidates"], trial + # ...and it is NOT an invocation. + assert "wrong-tool" not in trial["triggered"], trial + + # ---- 3. THE TRIGGER WINS. Declining and then using it is a change of mind, not a rejection. + both = "advice:decline000003" + capabilities.heartbeat("helper", "match", ref=both, path=ledger, + idempotency_key="m3:helper", metadata={"surface": "t-dec"}) + record_decline("helper", both, reason="looked wrong at first", surface="t-dec", + kind="wrong_match", path=ledger) + record_trigger("helper", both, path=ledger) + t3 = next(t for t in experiments(path=ledger) if t["experiment_id"] == both) + assert t3["declined"] == [] and t3["triggered"] == ["helper"], t3 + assert t3["decline_reasons"] == {} and t3["decline_kinds"] == {}, t3 + assert t3["declined_demotable"] == [], t3 + + # ---- 4. A REASON IS MANDATORY, exactly as an evidenced verdict is. + for bad in ("", " ", "\n"): + try: + record_decline("helper", "advice:decline000004", reason=bad, path=ledger) + except ValueError: + pass + else: + raise AssertionError("an unexplained decline must be refused") + # And the experiment must be a real advisory digest, or declines accrue against no trial. + try: + record_decline("helper", "not-an-advice-ref", reason="x", path=ledger) + except ValueError: + pass + else: + raise AssertionError("a non-advisory experiment id must be refused") + # AN UNKNOWN KIND IS REFUSED, never coerced. A typo silently becoming `unspecified` would + # hide the classification the caller believed it had made, and `unspecified` cannot demote — + # so the coercion would quietly discard the one signal the taxonomy exists to carry. + try: + record_decline("helper", "advice:decline0000ff", reason="x", kind="wrong-match", + path=ledger) + except ValueError: + pass + else: + raise AssertionError("an unknown decline kind must be refused, not coerced") + # Omitting the kind is allowed and yields the non-demotable default: no silence, no wrong + # correction. Failing toward motion, not toward a demotion nobody classified. + assert record_decline("helper", "advice:decline0000aa", reason="did not classify it", + surface="t-dec", path=ledger) + assert decline_kind_demotable(DECLINE_KIND_DEFAULT) is False + # IDEMPOTENT per (capability, experiment): replaying a backfill cannot inflate the count. + assert record_decline("wrong-tool", d_exp, reason="repeat", surface="t-dec", + path=ledger) is False + assert usefulness(path=ledger)["rows"]["wrong-tool"]["declined"] == 1 + + # ---- 5. DEMOTION CONSUMES DECLINES. A binding rejected repeatedly at one surface is the + # drain the binding table needs, and it must carry the caller's own words. + real = capability_advisor.SURFACE_BINDINGS.get("t-dec") + capability_advisor.SURFACE_BINDINGS["t-dec"] = {"wrong-tool": "bound for now", + "used-here": "bound and used", + "helper": "bound and used"} + try: + # LITERAL boundary, deliberately not `DEMOTION_MIN_DECLINES - 1`: an assertion written + # in terms of the constant it guards moves with the constant and can never fail. + assert DEMOTION_MIN_DECLINES == 2, "boundary cases below assume the floor is 2" + # One decline so far for wrong-tool -> below the floor, no proposal, and the accumulating + # count must still be REPORTED. "no proposal" beside "1/2 accumulating" reads completely + # differently from "no proposal" beside nothing. + assert propose_demotions("t-dec", path=ledger) == [], "1 decline must not demote" + counts = surface_decline_counts("t-dec", path=ledger) + assert counts["declined"]["wrong-tool"] == 1, counts + record_decline("wrong-tool", "advice:decline000005", + reason="code-mutating tool offered inside a read-only audit", + surface="t-dec", kind="wrong_match", path=ledger) + dem = propose_demotions("t-dec", path=ledger) + assert [d["capability_id"] for d in dem] == ["wrong-tool"], dem + assert dem[0]["basis"] == "declined_with_reason", dem[0] + assert dem[0]["declined"] == 2 and dem[0]["triggered"] == 0, dem[0] + assert dem[0]["declined_demotable"] == 2, dem[0] + assert dem[0]["declines_by_kind"] == {"wrong_match": 2}, dem[0] + assert dem[0]["implied_fixes"] == ["the matcher or the binding"], dem[0] + assert len(dem[0]["decline_reasons"]) == 2, dem[0] + assert "read-only audit" in dem[0]["reason"], dem[0] + # BOTH QUANTITIES on the proposal, so the floor it cleared is legible. + assert dem[0]["declines_floor"] == DEMOTION_MIN_DECLINES, dem[0] + + # A CAPABILITY THAT IS ACTUALLY USED HERE IS NEVER DEMOTED, however often it is passed + # over. Enough declines to clear the floor on its own, or removing the trigger guard + # would leave this below the floor and the assertion could not discriminate. + for i in range(DEMOTION_MIN_DECLINES + 1): + record_decline("used-here", f"advice:usedheredec{i:02d}", + reason="not this time", surface="t-dec", kind="wrong_match", + path=ledger) + assert surface_decline_counts("t-dec", path=ledger)["declined"]["used-here"] > \ + DEMOTION_MIN_DECLINES + assert "used-here" not in [d["capability_id"] for d in + propose_demotions("t-dec", path=ledger)], \ + "a capability triggered at this surface is not a demotion candidate" + + # ATTRIBUTION IS ON THE EVENT. A decline with no surface is still recorded and still + # readable, and it must not feed a demotion for a surface it never named. + record_decline("helper", "advice:decline000006", reason="no surface given", + kind="wrong_match", path=ledger) + assert usefulness(path=ledger)["rows"]["helper"]["declined"] >= 4 + assert "helper" not in [d["capability_id"] for d in + propose_demotions("t-dec", path=ledger)], \ + "an unattributed decline must not demote a surface it never named" + + # ---- THE TAXONOMY'S WHOLE POINT: A CORRECT MATCH MUST NOT BE PUNISHED FOR BEING + # RIGHT. `testgen-lane` matched correctly three times in one read-only audit and was + # structurally impossible every time (no commit target). Its fix is NOTHING, so however + # many times it is declined that way it can never clear the demotion floor. + # + # DELIBERATELY MANY TIMES OVER THE FLOOR, and asserted against `wrong_match` in the same + # ledger: if `demotable` were ignored, this capability would demote and the assertion + # would fire. A count merely equal to the floor could not tell "the kind was honoured" + # apart from "the floor was not reached". + right_but_impossible = capabilities._blank_capability("right-but-impossible") + right_but_impossible["status"] = "generated" + right_but_impossible["matcher"] = {"field": "task_type", "operator": "in", + "value": ["testgen"]} + all_rows = capabilities.load_declared(ledger) + all_rows["right-but-impossible"] = right_but_impossible + capabilities.save(all_rows, ledger) + capability_advisor.SURFACE_BINDINGS["t-dec"]["right-but-impossible"] = "bound, correct" + for i in range(DEMOTION_MIN_DECLINES * 4): + record_decline("right-but-impossible", f"advice:nolanding{i:04d}", + reason="correct match, read-only run has no commit target", + surface="t-dec", kind="no_landing_zone", path=ledger) + counts = surface_decline_counts("t-dec", path=ledger) + # The decline IS recorded and IS visible -- inert must not mean invisible. + assert counts["declined"]["right-but-impossible"] == DEMOTION_MIN_DECLINES * 4, counts + assert counts["declined_demotable"].get("right-but-impossible", 0) == 0, counts + # THE TWO RULES READ DISJOINT POPULATIONS. This probe deliberately exceeds the SILENT + # floor as well, so it proves the never-triggered rule cannot be reached through + # declines. Without that, eight honest declines demote a correct match via the other + # rule -- which is what the first draft of this function actually did. + assert DEMOTION_MIN_DECLINES * 4 >= DEMOTION_MIN_TRIALS, ( + "this probe must exceed the silent-offer floor too, or it cannot discriminate") + assert counts["silent"].get("right-but-impossible", 0) == 0, counts + assert counts["declines_by_kind"]["right-but-impossible"] == \ + {"no_landing_zone": DEMOTION_MIN_DECLINES * 4}, counts + assert "right-but-impossible" not in [ + d["capability_id"] for d in propose_demotions("t-dec", path=ledger)], \ + ("a CORRECT match blocked by the deliverable's shape must never be demoted — the " + "fix for no_landing_zone is nothing") + # ...and it must not reach the posterior either, on any kind. + prop = propensity("right-but-impossible", path=ledger) + assert prop["evidence_count"] == 0 and prop["declines"] == DEMOTION_MIN_DECLINES * 4 + assert prop["declines_demotable"] == 0, prop + assert prop["propensity"] >= EXPLORATION_FLOOR and prop["explorable"] is True, prop + + # THE frontend-verifier STORY, asserted. Declined at two surfaces because its + # precondition did not hold, then USEFUL at a third on a repo that has the surface. The + # two negatives must not demote it anywhere -- "evaluate the condition, do not weaken + # the binding". Exactly at the floor, so a demotable `precondition_unmet` would fire. + precond = capabilities._blank_capability("surface-gated") + precond["status"] = "generated" + precond["matcher"] = {"field": "task_type", "operator": "in", "value": ["testgen"]} + rows2 = capabilities.load_declared(ledger) + rows2["surface-gated"] = precond + capabilities.save(rows2, ledger) + capability_advisor.SURFACE_BINDINGS["t-dec"]["surface-gated"] = "bound, conditional" + for i in range(DEMOTION_MIN_DECLINES): + record_decline("surface-gated", f"advice:precond{i:05d}", + reason="this repository has no observable surface at all", + surface="t-dec", kind="precondition_unmet", path=ledger) + pc = surface_decline_counts("t-dec", path=ledger) + assert pc["declined"]["surface-gated"] == DEMOTION_MIN_DECLINES, pc + assert pc["declined_demotable"].get("surface-gated", 0) == 0, pc + dem_ids = [d["capability_id"] for d in propose_demotions("t-dec", path=ledger)] + assert "surface-gated" not in dem_ids, ( + "an unmet PRECONDITION must never demote the binding — the fix is to evaluate the " + "condition, and two negatives are not a verdict on a binding that fires elsewhere") + assert DECLINE_KINDS["precondition_unmet"]["demotable"] is False + + # EVERY non-demotable kind behaves the same way, so the guarantee is a property of the + # table rather than of one branch. Iterating the table also means a NEW kind cannot be + # added as demotable-by-accident without this failing. + for kind, spec in sorted(DECLINE_KINDS.items()): + if spec["demotable"]: + continue + cid = f"nd-{kind}" + rows_now = capabilities.load_declared(ledger) + blank = capabilities._blank_capability(cid) + blank["status"] = "generated" + blank["matcher"] = {"field": "task_type", "operator": "in", "value": ["testgen"]} + rows_now[cid] = blank + capabilities.save(rows_now, ledger) + capability_advisor.SURFACE_BINDINGS["t-dec"][cid] = f"bound to probe {kind}" + for i in range(DEMOTION_MIN_DECLINES * 3): + record_decline(cid, f"advice:{kind[:6]}nd{i:04d}", + reason=f"declined as {kind}", surface="t-dec", kind=kind, + path=ledger) + assert cid not in [d["capability_id"] + for d in propose_demotions("t-dec", path=ledger)], \ + f"a non-demotable kind ({kind}) demoted a binding" + + # `detect()` prints the drainable quantity for the surface even when nothing fires. + rep = detect(path=ledger) + assert "t-dec" in rep["surfaces"], sorted(rep["surfaces"]) + assert rep["surfaces"]["t-dec"]["declines"]["wrong-tool"] == 2, rep["surfaces"]["t-dec"] + assert rep["surfaces"]["t-dec"]["declines_floor"] == DEMOTION_MIN_DECLINES + assert "wrong-tool" in [d["capability_id"] for d in rep["demotions"]], rep["demotions"] + finally: + if real is None: + capability_advisor.SURFACE_BINDINGS.pop("t-dec", None) + else: + capability_advisor.SURFACE_BINDINGS["t-dec"] = real + + # ---- 6. ONE WINDOW. Declines age out with the trials they belong to, so the measuring and + # the draining window cannot drift apart into permanent debt. + old = capabilities._now() + (WINDOW_DAYS + 2) * 86400 + assert usefulness(path=ledger, now=old)["rows"]["wrong-tool"]["declined"] == 0 + rep_old = report(path=ledger, now=old) + assert rep_old["decline_count"] == 0, rep_old + rep_now = report(path=ledger) + assert rep_now["decline_count"] >= 6, rep_now + assert rep_now["capabilities_declined_with_reason"] >= 3, rep_now + + print("capability_propensity decline selftest: OK (a decline is a candidate, never an outcome, " + "never moves the posterior, partitions the third state, and drains a binding)") + + def _selftest_detection() -> None: """The recursive loop: detect a pass-over, propose, promote — and never ratchet. @@ -1441,23 +1972,29 @@ def _fmt(rep: dict) -> str: f" experiments: {rep['experiment_count']} " f"({rep['resolved_experiment_count']} resolved)", f" capabilities with usefulness evidence: {rep['capabilities_with_evidence']} " - f"of {rep['capability_count']}"] + f"of {rep['capability_count']}", + f" reasoned declines recorded: {rep['decline_count']} across " + f"{rep['capabilities_declined_with_reason']} capability(ies) — counted, never scored; " + f"{rep['decline_demotable_count']} attributable to a binding", + f" decline kinds: {rep['declines_by_kind'] or '(none)'}"] if not rep["capabilities_with_evidence"]: lines.append(" NOTE: no resolved outcomes yet — every propensity below is the PRIOR, " "not a measurement") lines.append("") - lines.append(f" {'capability':34s} {'prop':>6s} {'cand':>5s} {'trig':>5s} {'use':>4s} {'no':>3s}") + lines.append(f" {'capability':34s} {'prop':>6s} {'cand':>5s} {'trig':>5s} {'use':>4s} {'no':>3s}" + f" {'decl':>5s}") for row in rep["ranked"][:60]: lines.append(f" {row['capability_id']:34s} {row['propensity']:6.3f} " f"{row['candidates']:5d} {row['triggered']:5d} {row['useful']:4d} " - f"{row['not_useful']:3d}" + (" (floored)" if row["floored"] else "")) + f"{row['not_useful']:3d} {row['declined']:5d}" + + (" (floored)" if row["floored"] else "")) return "\n".join(lines) + "\n" def main(argv: list[str]) -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("command", nargs="?", default="report", - choices=["report", "experiments", "trigger", "useful", "detect", + choices=["report", "experiments", "trigger", "useful", "decline", "detect", "tick-evidence"]) # A loop that can only be closed from Python cannot be closed by a lane, which runs bash. These # two subcommands are the whole reason the recording edges are reachable from an automation. @@ -1466,6 +2003,16 @@ def main(argv: list[str]) -> int: ap.add_argument("--evidence", default="", help="what the capability CHANGED (required by useful)") ap.add_argument("--not-useful", action="store_true", help="record that triggering it did NOT help") + # THE CALLERS ARE BASH. Both lane automations and every skill reach this module from a shell, so + # a verb that exists only in Python is a verb the surfaces that make these decisions cannot use. + ap.add_argument("--reason", default="", + help="decline: why this capability was NOT the right tool here (required)") + ap.add_argument("--surface", default="", + help="decline: the surface that declined (e.g. repo-audit:phase-2). Optional, " + "but a decline without it cannot feed propose_demotions") + ap.add_argument("--kind", default=DECLINE_KIND_DEFAULT, choices=sorted(DECLINE_KINDS), + help="decline: WHICH KIND of decline. The kinds imply opposite fixes, and only " + "wrong_match/scope_too_small can ever demote a binding") # ISOLATION FOR PROOFS. Wiring this up, I recorded a trial into the LIVE ledger whose evidence # described the wiring rather than the capability's review value -- a mislabeled trial, and the # system's first data point. A proof belongs on a throwaway ledger; without this flag the only @@ -1484,6 +2031,7 @@ def main(argv: list[str]) -> int: args = ap.parse_args(argv) if args.selftest: _selftest() + _selftest_declines() _selftest_detection() _selftest_tick_evidence() return 0 @@ -1515,10 +2063,28 @@ def main(argv: list[str]) -> int: if rep["applied"]: print(f" APPLIED: {rep['applied']}") return 0 - if args.command in {"trigger", "useful"}: + if args.command in {"trigger", "useful", "decline"}: if not args.capability or not args.experiment: ap.error("--capability and --experiment are required") ledger = pathlib.Path(args.ledger) if args.ledger else None + if args.command == "decline": + if not args.reason.strip(): + ap.error("--reason is required: an unexplained decline is indistinguishable from " + "inattention, which is the state this verb exists to replace") + ok = record_decline(args.capability, args.experiment, reason=args.reason, + surface=args.surface, kind=args.kind, path=ledger) + print(json.dumps({"recorded": bool(ok), "command": "decline", + "ledger": str(ledger) if ledger else "live", + "capability": args.capability, "experiment": args.experiment, + "surface": args.surface or None, + "kind": args.kind, + "kind_implies_fix": DECLINE_KINDS[args.kind]["fix"], + "can_demote_the_binding": decline_kind_demotable(args.kind), + # SAY WHAT THIS DID NOT DO. A decline is not a verdict, and a caller + # that thinks it scored the capability has been misled. + "affects_propensity": False, + "attributable_to_surface": bool(args.surface)})) + return 0 if args.command == "trigger": ok = record_trigger(args.capability, args.experiment, path=ledger) else: @@ -1595,6 +2161,21 @@ def main(argv: list[str]) -> int: } PROMOTION_MIN_HAND_WORK = 3 # below this, one anecdote could widen a bound set DEMOTION_MIN_TRIALS = 8 # resolved experiments a binding gets before non-use counts +# A REASONED DECLINE IS MUCH STRONGER EVIDENCE THAN SILENT NON-USE, so its floor is much lower. A +# phase surface is consulted at most ONCE per run, so two declines are two independent runs by +# construction, where `DEMOTION_MIN_TRIALS` trials can all come from one high-volume lane. +# +# LATCHED-GATE ANSWERS (it is a threshold, so it needs all three in writing): +# 1. WHAT DECREMENTS IT? A single TRIGGER at that surface removes the proposal outright, and the +# binding keeps offering the capability while the count sits below the floor. So the gate fails +# toward motion: the capability stays selectable either way. +# 2. CAN THE DRAIN RUN WHILE CLOSED? Yes. Demotion is itself the drain on the binding table, and +# recording a decline requires nothing the proposal forbids -- the capability is still offered +# on every consult, so it can always be either declined again or used. +# 3. SAME WINDOW BOTH WAYS? Yes: `WINDOW_DAYS`, the one constant, drives the decline count and the +# trial count alike. And `detect()` reports each bound capability's decline count even when it +# is BELOW the floor, so "no proposal" can never read as "nothing is accumulating". +DEMOTION_MIN_DECLINES = 2 # Where a surface's own records live. INSTANCE paths, so they are resolved at runtime and @@ -1641,9 +2222,20 @@ def detect(*, path=None, apply_promotions: bool = False) -> dict: recs = surface_records(surface) proms = propose_bindings(surface, recs, path=path) if recs else [] dems = propose_demotions(surface, path=path) - if recs or proms or dems: - out["surfaces"][surface] = {"records": len(recs), - "bound": sorted(capability_advisor.binding_for(surface, path=path))} + counts = surface_decline_counts(surface, path=path) + if recs or proms or dems or counts["declined"]: + out["surfaces"][surface] = { + "records": len(recs), + "bound": sorted(capability_advisor.binding_for(surface, path=path)), + # THE DRAINABLE QUANTITY, printed whether or not the floor was reached. "0 proposals" + # beside "3 declines accumulating, floor 2" reads completely differently from "0 + # proposals" beside nothing at all, and only one of those is a healthy silence. + "declines": dict(sorted(counts["declined"].items())), + "declines_demotable": dict(sorted(counts["declined_demotable"].items())), + "declines_by_kind": {c: dict(sorted(k.items())) + for c, k in sorted(counts["declines_by_kind"].items())}, + "declines_floor": DEMOTION_MIN_DECLINES, + } out["promotions"].extend(proms) out["demotions"].extend(dems) if apply_promotions: @@ -1737,27 +2329,116 @@ def propose_bindings(surface: str, records: list, *, path=None) -> list[dict]: return sorted(out, key=lambda r: -r["hand_work"]) -def propose_demotions(surface: str, *, path=None, window_days: int = WINDOW_DAYS) -> list[dict]: - """Bound capabilities this surface never triggers. The drain, without which bindings only grow.""" - import capability_advisor - bound = capability_advisor.binding_for(surface, path=path) +def surface_decline_counts(surface: str, *, path=None, window_days: int = WINDOW_DAYS) -> dict: + """Per capability at this surface: offered / triggered / declined, plus the stated reasons. + + Split out of `propose_demotions` so the DRAINABLE quantity is reportable on its own. A threshold + that only speaks when it fires cannot be told apart from one that will never fire. + """ seen: dict[str, int] = {} used: dict[str, int] = {} + silent: dict[str, int] = {} + declined: dict[str, int] = {} + demotable: dict[str, int] = {} + kinds: dict[str, dict[str, int]] = {} + reasons: dict[str, list[str]] = {} for trial in experiments(path=path, window_days=window_days): if surface not in (trial.get("skills") or []): continue - for cap_id in bound: - if cap_id in (trial.get("candidates") or []): - seen[cap_id] = seen.get(cap_id, 0) + 1 - if cap_id in (trial.get("triggered") or []): - used[cap_id] = used.get(cap_id, 0) + 1 - return sorted( - ({"capability_id": c, "surface": surface, "offered": seen[c], "triggered": used.get(c, 0), - "action": "demote", - "reason": (f"bound and offered in {seen[c]} resolved experiments for this surface, " - f"triggered {used.get(c, 0)} times")} - for c in bound if seen.get(c, 0) >= DEMOTION_MIN_TRIALS and not used.get(c)), - key=lambda r: -r["offered"]) + for cap_id in trial.get("candidates") or []: + seen[cap_id] = seen.get(cap_id, 0) + 1 + for cap_id in trial.get("triggered") or []: + used[cap_id] = used.get(cap_id, 0) + 1 + # SILENT is the population the never-triggered rule is ABOUT: offered, and nothing said. It + # must exclude declines, or a capability declined honestly enough times trips a rule meant + # for capabilities nobody spoke about -- which is how a `no_landing_zone` decline would have + # demoted a correct match through the back door. (It did, in the first draft of this.) + for cap_id in trial.get("not_triggered_silently") or []: + silent[cap_id] = silent.get(cap_id, 0) + 1 + for cap_id in trial.get("declined") or []: + declined[cap_id] = declined.get(cap_id, 0) + 1 + kind = (trial.get("decline_kinds") or {}).get(cap_id, DECLINE_KIND_DEFAULT) + bucket = kinds.setdefault(cap_id, {}) + bucket[kind] = bucket.get(kind, 0) + 1 + why = (trial.get("decline_reasons") or {}).get(cap_id) + if why and why not in reasons.setdefault(cap_id, []): + reasons[cap_id].append(why) + for cap_id in trial.get("declined_demotable") or []: + demotable[cap_id] = demotable.get(cap_id, 0) + 1 + return {"surface": surface, "offered": seen, "triggered": used, "silent": silent, + "declined": declined, + # BOTH QUANTITIES. `declined` is how often it was turned down; `demotable` is how much of + # that is a statement about the BINDING. Reporting only the first is what would license + # unbinding `testgen-lane` for matching correctly three times. + "declined_demotable": demotable, "declines_by_kind": kinds, + "decline_reasons": reasons} + + +def propose_demotions(surface: str, *, path=None, window_days: int = WINDOW_DAYS) -> list[dict]: + """Bound capabilities this surface rejects or never triggers. The drain on the binding table. + + TWO RULES, and the decline rule is the sharper one. Silent non-use across `DEMOTION_MIN_TRIALS` + offers says only that nobody reached for it, which has two causes with opposite fixes. A + REASONED DECLINE says which one it is, in the caller's own words, so it clears at + `DEMOTION_MIN_DECLINES` and carries its evidence into the proposal. + + THE TWO RULES READ DISJOINT POPULATIONS, and that is load-bearing. `never_triggered` counts + only offers where NOTHING WAS SAID (`not_triggered_silently`), never declines. The first draft + counted every offer, so eight honest `no_landing_zone` declines tripped the silent-non-use rule + and demoted a correct match through the back door — the exact wrong correction the taxonomy + exists to prevent, arriving via the other rule. + + ONLY DEMOTABLE KINDS COUNT, and that qualifier is the whole point of the taxonomy. + `testgen-lane` matched CORRECTLY three times in one audit and was structurally impossible + (`no_landing_zone`, read-only run, no commit target); demoting it would punish a capability for + being right. So a non-demotable decline is counted, reported on the row, and cannot clear the + floor. `frontend-verifier`, declined on two frontend-less repos and then producing the + second-strongest finding of a third audit on a repo that has a display surface, is the same + lesson from the other side: two negatives are not a verdict on a binding. + + A single trigger at this surface disqualifies the capability from both rules: something that + actually gets used here is not a demotion candidate however often it is passed over. + """ + import capability_advisor + bound = capability_advisor.binding_for(surface, path=path) + counts = surface_decline_counts(surface, path=path, window_days=window_days) + seen, used, silent = counts["offered"], counts["triggered"], counts["silent"] + declined, reasons = counts["declined"], counts["decline_reasons"] + demotable, by_kind = counts["declined_demotable"], counts["declines_by_kind"] + out = [] + for c in bound: + if used.get(c): + continue + n_dec, n_seen = declined.get(c, 0), seen.get(c, 0) + n_dem, n_silent = demotable.get(c, 0), silent.get(c, 0) + kinds = dict(sorted((by_kind.get(c) or {}).items())) + fixes = sorted({DECLINE_KINDS[k]["fix"] for k in kinds if k in DECLINE_KINDS}) + if n_dem >= DEMOTION_MIN_DECLINES: + basis = "declined_with_reason" + why = (f"declined with a stated reason in {n_dec} of {n_seen} offers at this surface, " + f"{n_dem} of them attributable to the binding (floor " + f"{DEMOTION_MIN_DECLINES}), never triggered: " + + " | ".join(reasons.get(c, [])[:3])) + elif n_silent >= DEMOTION_MIN_TRIALS: + basis = "never_triggered" + why = (f"bound and offered in {n_seen} experiments for this surface, " + f"{n_silent} of them passed over with nothing said, triggered " + f"{used.get(c, 0)} times") + else: + continue + out.append({"capability_id": c, "surface": surface, "offered": n_seen, + "triggered": used.get(c, 0), "silent": n_silent, "declined": n_dec, + # THE QUALIFIED COUNT, beside the raw one, plus the fix each kind implies -- so + # a reader can choose "add a precondition" over "unbind" where that is the + # correct answer, instead of inferring one action from one number. + "declined_demotable": n_dem, "declines_by_kind": kinds, + "implied_fixes": fixes, + "decline_reasons": reasons.get(c, []), "basis": basis, + # BLOCKING quantity and the floor it is measured against, together. + "declines_floor": DEMOTION_MIN_DECLINES, + "silent_offers_floor": DEMOTION_MIN_TRIALS, + "action": "demote", "reason": why}) + return sorted(out, key=lambda r: (-r["declined_demotable"], -r["offered"], r["capability_id"])) def record_promotion(capability_id: str, surface: str, reason: str, *, path=None) -> bool: diff --git a/mcp_server.py b/mcp_server.py index 8ce99b7..37a1fea 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -6,8 +6,9 @@ fleet steerable from ANY session on this machine: check capacity, read the fleet summary and route weights, list/answer owner questions, look up resume hints. -Deliberately SAFE surface: read-only tools plus exactly two bounded actions — -answer_owner_question (feeds the 16h decision loop) and record_owner_question. No dispatch, no +Deliberately SAFE surface: read-only tools plus exactly three bounded actions — +answer_owner_question (feeds the 16h decision loop), record_owner_question, and capability_decline +(append-only evidence that an offered capability was rejected, and why). No dispatch, no claims, no config mutation through this door; steering that mutates the fleet stays with the CLIs. `--selftest` drives the server as a subprocess through a real initialize/tools/list/tools/call round-trip.""" @@ -28,6 +29,20 @@ STATE_DIR = Path(os.environ.get("ORCH_STATE_DIR", Path.home() / ".codex" / "orchestrator")) PROTOCOL_VERSION = "2025-06-18" +def _decline_kinds() -> list[str]: + """The decline vocabulary, read from its owner so the MCP enum cannot drift from it. + + Imported here rather than at module scope for the same reason `capability_advice` imports the + advisor lazily: a capability-registry problem must never take down the capacity/fleet reads. A + hardcoded list would be a second inventory of the thing this function exists to mirror. + """ + try: + import capability_propensity + return sorted(capability_propensity.DECLINE_KINDS) + except Exception: # noqa: BLE001 + return [] + + TOOLS = [ { "name": "capacity_status", @@ -79,6 +94,48 @@ "required": ["task"], }, }, + { + "name": "capability_decline", + "description": ( + "Record that a capability capability_advice OFFERED was deliberately NOT used, and why. " + "Pass the experiment_id from the advice you are responding to. A decline is NOT a " + "negative verdict: it never touches the usefulness posterior, because the capability " + "did not run. It makes 'offered and rejected on stated grounds' distinguishable from " + "'never considered' — which were byte-identical in the ledger before this existed — and " + "repeated declines at one surface propose demoting that binding. A reason is REQUIRED." + ), + "inputSchema": { + "type": "object", + "properties": { + "capability_id": {"type": "string", + "description": "the offered capability you are turning down"}, + "experiment_id": {"type": "string", + "description": "the `experiment_id` from the capability_advice " + "response (an 'advice:' string)"}, + "reason": {"type": "string", + "description": "why this capability was the wrong tool for THIS work. " + "Refused when blank — an unexplained decline is " + "indistinguishable from inattention"}, + "surface": {"type": "string", + "description": "the surface declining, matching the one you passed to " + "capability_advice (e.g. 'repo-audit:phase-2'). Optional, " + "but a decline without it cannot feed demotion"}, + "kind": {"type": "string", + "enum": sorted(_decline_kinds()), + "description": "WHICH KIND of decline, because the kinds imply opposite " + "fixes. wrong_match = it does not fit this work (the " + "binding is wrong). scope_too_small = a correct match " + "declared too broadly. no_landing_zone = a CORRECT match " + "the deliverable shape made impossible (e.g. a test " + "generator in a read-only audit) — this never counts " + "against the capability. gated_off = held behind a " + "deliberate switch or shadow status. deferred = wanted, " + "not affordable. Omitted means 'unspecified', which is " + "recorded and can never demote a binding."}, + }, + "required": ["capability_id", "experiment_id", "reason"], + }, + }, { "name": "capability_associations", "description": ( @@ -188,6 +245,28 @@ def _call_tool(name: str, args: dict): "capabilities_ready": result.get("dispatch_ready_count") or 0, }) return result + if name == "capability_decline": + import capability_propensity + recorded = capability_propensity.record_decline( + str(args["capability_id"]), str(args["experiment_id"]), + reason=str(args.get("reason") or ""), + surface=str(args.get("surface") or ""), + kind=str(args.get("kind") or capability_propensity.DECLINE_KIND_DEFAULT), + ) + kind = str(args.get("kind") or capability_propensity.DECLINE_KIND_DEFAULT) + return {"recorded": bool(recorded), "capability_id": str(args["capability_id"]), + "experiment_id": str(args["experiment_id"]), + "surface": str(args.get("surface") or "") or None, + # SAY WHAT IT DID NOT DO, in the response. A caller that believes it just scored the + # capability has been misinformed by a successful call. + "affects_propensity": False, + "attributable_to_surface": bool(args.get("surface")), + "kind": kind, + "can_demote_the_binding": capability_propensity.decline_kind_demotable(kind), + "kind_implies_fix": capability_propensity.DECLINE_KINDS[kind]["fix"], + "note": ("recorded as a reasoned rejection; the usefulness posterior is untouched " + "because the capability never ran"), + "already_recorded": not recorded} if name == "capability_associations": import capability_advisor return capability_advisor.learned_associations() @@ -299,6 +378,110 @@ def _selftest_advice_schema_matches_advise() -> None: "and forwarded)") +def _forwarded_args(handler, callee: str) -> set[str]: + """Which `args[...]` keys actually reach `callee(...)` inside `handler`. From the AST. + + Positional arguments are read through their `args["key"]` subscript and keyword arguments + through their names, so this answers "what did the caller's field become" rather than "does the + word appear somewhere nearby" — the difference between a guard and decoration. + """ + import ast + import inspect as _inspect + tree = ast.parse(_inspect.getsource(handler)) + keys: set[str] = set() + + def _subscript_key(node) -> str | None: + for sub in ast.walk(node): + if isinstance(sub, ast.Subscript) and isinstance(sub.value, ast.Name) \ + and sub.value.id == "args" and isinstance(sub.slice, ast.Constant): + return str(sub.slice.value) + return None + + for node in ast.walk(tree): + if not isinstance(node, ast.Call): + continue + func = node.func + name = func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", "") + if name != callee: + continue + for positional in node.args: + key = _subscript_key(positional) + if key: + keys.add(key) + for kw in node.keywords: + if kw.arg: + keys.add(kw.arg) + return keys + + +def _selftest_decline_schema_matches_record_decline() -> None: + """The same guard as above, for the decline verb — because the same mistake is available again. + + `record_decline` has a REQUIRED keyword (`reason`) and an attribution keyword (`surface`) that + silently degrades the evidence when dropped: the decline is still written, still readable, and + can no longer feed `propose_demotions`. That is the worst possible failure shape — a successful + call that quietly produces weaker evidence — so both the advertisement and the FORWARDING are + asserted. + + Every probe below is a REFUSAL path, so this selftest writes nothing to any ledger. Asserting + forwarding through a successful write would mean writing a fake decline into the live ledger of + whatever machine runs the suite, which is the mislabeled-trial mistake this project already made + once with `record_usefulness`. + """ + import inspect + import capability_propensity + + tool = next(t for t in TOOLS if t["name"] == "capability_decline") + advertised = set(tool["inputSchema"]["properties"]) + sig = inspect.signature(capability_propensity.record_decline) + internal = {"path", "metadata"} + callable_kw = {n for n, prm in sig.parameters.items() + if prm.kind is prm.KEYWORD_ONLY and n not in internal} + missing = sorted(callable_kw - advertised) + assert not missing, ( + f"capability_decline does not advertise {missing}, so a caller setting them is silently " + f"ignored. Add them to the inputSchema AND pass them through in _call_tool.") + # The positional arguments must be reachable too, under the names the tool advertises. + assert {"capability_id", "experiment_id"} <= advertised, sorted(advertised) + assert set(tool["inputSchema"]["required"]) == {"capability_id", "experiment_id", "reason"}, tool + + # ADVERTISED IS NOT FORWARDED. `surface` is the dangerous one: dropping it still returns a + # SUCCESSFUL call that writes a weaker decline, so no refusal probe can see the loss. Assert it + # against the CALL ITSELF — code vs code, no ledger, so it runs on any machine and catches + # exactly the original defect (callers passing a field the callee silently discarded). + # + # Read off the AST, not the source text. A substring search over the handler body was written + # first and DID NOT DISCRIMINATE: deleting `surface=` from the call left the word in the + # response dict, so the break stayed green. A guard that cannot fail is decoration. + forwarded = _forwarded_args(_call_tool, "record_decline") + assert forwarded == advertised, ( + f"capability_decline advertises {sorted(advertised)} and forwards {sorted(forwarded)} to " + f"record_decline; the difference is silently dropped and the call still succeeds") + + def _fails(args) -> str: + try: + _call_tool("capability_decline", args) + except Exception as exc: # noqa: BLE001 + return str(exc) + raise AssertionError(f"capability_decline accepted {args!r}") + + # `reason` is forwarded: blank reaches record_decline's own refusal, which names the reason. + assert "requires a reason" in _fails( + {"capability_id": "x", "experiment_id": "advice:abc", "reason": " "}) + # `experiment_id` is forwarded: a non-advisory id reaches the prefix check. + assert "advice:" in _fails( + {"capability_id": "x", "experiment_id": "not-an-advice-ref", "reason": "why"}) + # `capability_id` is forwarded: an id no ledger can hold reaches the ledger's own guard. This + # deliberately does NOT assert which capabilities exist -- the ledger is machine-local (43 rows + # on the owner's machine, 14 on a clean runner) and asserting its contents inside a guard about + # schema agreement is the exact cross-machine mistake the sibling selftest documents. + assert "unknown capability" in _fails( + {"capability_id": "__mcp-selftest-absent-capability__", + "experiment_id": "advice:0000000000ff", "reason": "probe, writes nothing"}) + print("mcp_server decline-schema selftest: OK (every caller-settable record_decline field is " + "advertised AND read by the handler; every probe is a refusal, so nothing was written)") + + def _selftest() -> None: import subprocess import tempfile @@ -342,5 +525,6 @@ def _selftest() -> None: if "--selftest" in sys.argv: _selftest() _selftest_advice_schema_matches_advise() + _selftest_decline_schema_matches_record_decline() raise SystemExit(0) raise SystemExit(serve()) From 63799502936473bc8ede22c58ed69a80b7fd69ae Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 00:10:38 -0500 Subject: [PATCH 2/2] fix(propensity): detect() must see the surfaces that only INHERIT a binding MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `detect` enumerated `SURFACE_RECORD_GLOBS | SURFACE_BINDINGS` — the DECLARED keys. But `repo-audit:dimension-1` has no table entry of its own; it inherits `offload` surface-wide. So when three independent 2026-08-23 audit rounds each declined `offload` at that surface with a stated reason, all three were written to the ledger and none was ever read. A drain that cannot see a surface cannot drain it, and the surfaces most likely to be OVER-BOUND are exactly the ones that only inherit. `observed_surfaces()` derives the set from the trials themselves rather than adding a second list, which would drift from the surfaces that exist and would still omit the inheriting ones. The first version of the selftest asserted `observed_surfaces()` and stayed GREEN when `detect()` was reverted to the declared-keys-only enumeration — a test of the helper instead of the answer, which is this project's most-repeated testing mistake. It now asserts what the CALLER receives, and the break fails as required. Effect on the live ledger, report-only as always: demotion proposals go 2 -> 3, the new one being `repo-audit:dimension-1 -> offload`, declined in 3 of 3 offers across three audits, all three attributable to the binding. Co-Authored-By: Claude Opus 5 --- ARCHITECTURE.md | 6 ++++++ capability_propensity.py | 38 ++++++++++++++++++++++++++++++++++++-- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index ad5c1a6..64627c2 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -282,6 +282,12 @@ posterior for exactly this reason: "prior, no evidence" and "prior, no evidence, rejections" are opposite readings that were previously identical. A decline requires a reason and is refused without one, the same way `record_usefulness` refuses an unevidenced verdict. +`detect` enumerates every surface that has either a declaration or evidence, not only the declared +keys: `repo-audit:dimension-1` has no table entry of its own — it inherits `offload` surface-wide — +so three independent audits declining `offload` there were recorded and never read. A drain that +cannot see a surface cannot drain it, and the surfaces most likely to be over-bound are exactly the +ones that only inherit. + Attribution is on the event: the advisor records the `surface` on each `match`, because it recorded only `skill` before and the CLI has no `--skill` flag — so every `--surface` consult wrote `skill: null` and its whole control arm was unattributable to the surface that produced it. diff --git a/capability_propensity.py b/capability_propensity.py index 7a96836..c84062f 100644 --- a/capability_propensity.py +++ b/capability_propensity.py @@ -1743,9 +1743,27 @@ def _selftest_declines() -> None: for d in propose_demotions("t-dec", path=ledger)], \ f"a non-demotable kind ({kind}) demoted a binding" - # `detect()` prints the drainable quantity for the surface even when nothing fires. + # `detect()` prints the drainable quantity for the surface even when nothing fires -- + # and it must find a surface that has EVIDENCE BUT NO TABLE ENTRY, because the surfaces + # most likely to be over-bound are the ones that only inherit a binding. `t-dec` has a + # stubbed entry, so assert the derived path on a surface that has none. + assert "t-dec" in observed_surfaces(path=ledger), sorted(observed_surfaces(path=ledger)) + record_decline("helper", "advice:inherited0001", reason="inherited-surface probe", + surface="t-inherited-only", kind="wrong_match", path=ledger) + assert "t-inherited-only" in observed_surfaces(path=ledger), \ + "a surface with evidence and no table entry must still be enumerated" + assert "t-inherited-only" not in capability_advisor.SURFACE_BINDINGS rep = detect(path=ledger) assert "t-dec" in rep["surfaces"], sorted(rep["surfaces"]) + # ASSERT ON WHAT THE CALLER RECEIVES, not on the helper. The first version of this + # checked `observed_surfaces()` alone and stayed GREEN when `detect()` was reverted to + # enumerating only declared keys -- a test of the table instead of the answer, which is + # this project's most-repeated testing mistake. + assert "t-inherited-only" in rep["surfaces"], ( + "detect() must REPORT a surface that has decline evidence and no table entry; " + f"it reported {sorted(rep['surfaces'])}") + assert rep["surfaces"]["t-inherited-only"]["declines"] == {"helper": 1}, \ + rep["surfaces"]["t-inherited-only"] assert rep["surfaces"]["t-dec"]["declines"]["wrong-tool"] == 2, rep["surfaces"]["t-dec"] assert rep["surfaces"]["t-dec"]["declines_floor"] == DEMOTION_MIN_DECLINES assert "wrong-tool" in [d["capability_id"] for d in rep["demotions"]], rep["demotions"] @@ -2209,6 +2227,16 @@ def surface_records(surface: str) -> list[str]: return [r for r in re.split(RECORD_SPLIT, text, flags=re.M) if r.strip()] +def observed_surfaces(*, path=None, window_days: int = WINDOW_DAYS) -> set[str]: + """Surfaces that actually consulted, read from the trials themselves. + + Derived, never a second list: a declared table of surfaces would drift from the surfaces that + exist, and the ones that only INHERIT a binding would never appear in it at all. + """ + return {surface for trial in experiments(path=path, window_days=window_days) + for surface in (trial.get("skills") or [])} + + def detect(*, path=None, apply_promotions: bool = False) -> dict: """Run detection across every surface whose records are resolvable here. @@ -2218,7 +2246,13 @@ def detect(*, path=None, apply_promotions: bool = False) -> dict: """ import capability_advisor out = {"surfaces": {}, "promotions": [], "demotions": [], "applied": []} - for surface in sorted(set(SURFACE_RECORD_GLOBS) | set(capability_advisor.SURFACE_BINDINGS)): + # EVERY SURFACE THAT HAS EITHER A DECLARATION OR EVIDENCE. Enumerating only the declared keys + # missed the inherited ones entirely: `repo-audit:dimension-1` has no table entry of its own -- + # it inherits `offload` surface-wide -- so three independent audits declining `offload` there + # were recorded and never read. A drain that cannot see a surface cannot drain it, and the + # surfaces most likely to be over-bound are exactly the ones that only inherit. + for surface in sorted(set(SURFACE_RECORD_GLOBS) | set(capability_advisor.SURFACE_BINDINGS) + | observed_surfaces(path=path)): recs = surface_records(surface) proms = propose_bindings(surface, recs, path=path) if recs else [] dems = propose_demotions(surface, path=path)