From 1dbfa21730b6c2c09582c01358b987889cb91a39 Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 18:16:09 -0500 Subject: [PATCH] fix(salvage): #42's catalog-resolution commit and #34's README item, both stranded post-merge MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both branches were held back from the branch cleanup because their tips carried commits pushed AFTER their PR merged, so "PR merged" did not mean "work landed". Verified per-symbol rather than by diff size — both branches are thousands of lines behind main, so a raw diff conflates stale with unlanded. #42 / commit 4e0d6ae — `adapters.py` catalog resolution. Main has `advertised_models` and NONE of the generalisation around it: `advertised_catalog`, `_advertised_catalog`, `_cached_catalog`, `agy_log_for`, `AGY_LOG_SUFFIX`, `CATALOG_ROUTING_TAGS`, `_catalog_model_id` were all absent. This is learning-loop provenance code (CLAUDE.md 2: "never treat a generic trace model as provider resolution"), and its whole point is that THE CATALOG IS THE AUTHORITY — a label resolves against the ids the CLI actually advertises, with routing TAGS (`auto`, `default`, `cli-default`) refused as non-identities. The commit's own note records that `VENDOR_MODEL_RE` rejects 42 of 204 real cursor ids, so shape-matching an id the CLI itself advertised is both redundant and wrong. Cherry-picked; `adapters.py`, `dispatcher.py` and `ledger_reconcile.py` applied clean. Two conflicts: * `.verify-floor.json` — took main's. A floor is a property of the MERGE RESULT, never carried in from a branch, so it is re-measured below. * `test_feedback_model_provenance.py` — TWO DIFFERENT tests in one region: main's `test_late_sweep_completes_terminal_attempts_never_one_in_flight` (from #63) and the branch's `test_gemini_provenance_reads_the_per_run_log_before_the_conversation_store`. Kept BOTH; they are independent. #34 / commit c1dc9a7 — README item 11 for `evidence_acquisition.py`, which main documented nowhere (zero occurrences). Every factual claim was re-verified against main's code rather than trusted: `capabilities.unblock()` exists; `ORCH_EVIDENCE_ACQUISITION_MAX_FEEDS`/`_MAX_ITEMS` default to 1 and 3; `LIVE_FLAG = "ORCH_EVIDENCE_ACQUISITION"` with SHADOW as the documented default; and the quoted summary line matches the format string verbatim (`feedable {n} / capped {n} / candidates {n} / fed {n}`). It is the drainable-vs-blocking line the latched-gate rule asks for, and it was the only place that reported it. DELIBERATE-BREAK -> REVERT: emptying `CATALOG_ROUTING_TAGS` fires `assert model_id_for_label("cursor", "Auto (default)") is None` in adapters' OWN selftest; reverted clean. Worth recording that `pytest test_feedback_model_provenance.py` did NOT catch that break — the guard is covered by a `--selftest`, not by a test_*.py, which is precisely why `verify.py` is the gate and a pytest subset is not. A redundant pytest test written before checking was dropped. FLOOR 427 -> 428, one new test, note appended not replaced. Verified FRESH-STATE (both ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME at empty dirs, reproducing CI): VERIFIED — 420 passed, 0 failed, 79 selftests, 3/5 gates green, 8 tests + 5 selftests + 2 gates skipped for named prerequisites; 420 + 8 = 428 = floor. ruff + black -l 100 clean. NOT FIXED HERE, and not caused here: on this machine `test_capabilities.py`'s `test_gate_blocks_execution_is_opt_in_and_narrow` and `test_evidence_gate_kind_is_not_blanket_observer` fail on PRISTINE main too — the hourly fleet tick mutated the machine-local ledger and range-lane-rollout now classifies `matched_not_invoked` instead of `deliberately_gated`. Ledger STATE, not code; they skip with a named reason under a fresh ledger, which is what CI uses. Co-Authored-By: Claude Opus 5 --- .verify-floor.json | 12 +- README.md | 10 + adapters.py | 418 +++++++++++++++++++++++++++--- dispatcher.py | 8 +- ledger_reconcile.py | 13 +- test_feedback_model_provenance.py | 122 +++++++++ 6 files changed, 530 insertions(+), 53 deletions(-) diff --git a/.verify-floor.json b/.verify-floor.json index debbbda..1aaed9c 100644 --- a/.verify-floor.json +++ b/.verify-floor.json @@ -1,8 +1,8 @@ { - "collected": 427, - "passed": 427, - "skipped_max": 26, - "selftest_skipped_max": 7, - "gate_skipped_max": 2, - "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that). FLOOR 402 -> 407 on 2026-08-23 (findability admission requirement). (findability admission requirement). (findability admission requirement). (findability admission requirement). Exactly five new pytest tests, all in test_capability_admission.py: test_findability_distinguishes_its_three_sub_causes, test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt, test_unreadable_reach_is_not_evaluated_and_never_a_failure, test_findability_exemption_is_declared_in_code_not_in_a_live_ledger, test_consult_sites_are_falsifiable_claims_about_real_callers. NO CEILING MOVED and nothing new skips: all five build synthetic ledgers in a tempdir or read committed tables, so none needs a populated capability ledger, an agent CLI or ~/.claude/skills. The one machine-dependent thing they touch -- an external consult site declared in capability_advisor.CONSULT_SITES whose skill prompt is not on this machine -- is reported as UNVERIFIED rather than skipped, because absence of the caller is not refutation of the claim; the in-tree site (tick) is asserted verified on every machine so the check can never degrade into 'everything unverified, nothing tested'. Measured on the merge result per the rule above: this file CONFLICTED three times while the branch was open, as main went 387 -> 391 -> 402 (#61, #64, #65, #60). Each time it was resolved as the UNION rather than by taking a side, and the count was RE-MEASURED on the new merge result rather than either number being carried forward: 402 (main at bd6da2e) + 5 (this branch's new tests) = 407. That is the rule this file already states -- measure the floor on the merge result, not on the branch -- and it mattered here, because #60 both deleted test_ci_gate_config.py and added more than it removed, so guessing in either direction would have been wrong. -> re-measured on 2026-08-23 (PR #62, the four deferred #42 review findings): three new tests, all machine-independent (each builds its own tmp_path Brain and manifests), so NO CEILING MOVED and nothing new is skipped. Fourth conflict for this branch, and the first one under the EQUALITY -- which is the point: the equality's own rationale says git conflict detection is what enforces 'measure on the merge result', and that is exactly what happened here. Under the old minimum the three earlier conflicts could each have been resolved by keeping the larger number; under the equality the count MUST be measured, and it was. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's three-test entry beside it -- and the count RE-MEASURED on the merge result rather than keeping either side's number. main fc1fd42 collects 407; this branch adds 3; 410 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record whose two sides differed only in `emitted_at` and `pr_number` (62 here, 68 on main). Main's NEWER record was kept rather than this branch's older one -- discarding a newer provenance observation to win a merge would corrupt exactly the causal-provenance evidence CLAUDE.md 2 protects, and this branch's own run re-emits its record anyway. FLOOR 410 -> 411 on 2026-08-23 (CodeRabbit follow-up on PR #42, thread 3837879039; re-measured again after #56 made `collected` an EQUALITY, which makes an assumed number a hard RED rather than a quiet pass -- main stayed at 402 across #56, and the merge result measures 403, so #56 added no collected tests and this is main's 402 plus this branch's one): exactly one new test, test_feedback_model_provenance.test_late_sweep_completes_terminal_attempts_never_one_in_flight, which pins that ledger_reconcile.resolve_unresolved_worker_attempts completes only TERMINAL unresolved worker attempts and never one still in flight. No ceiling moved and nothing new is skipped -- the test builds its own tmp ledger and codex rollout fixture and monkeypatches adapters.CODEX_SESSIONS, so it needs no agent CLI and no populated capability ledger and runs on a bare runner. RESOLVED AGAINST #59 (improvement-log accessor), which raised the floor 387 -> 391 on main while this branch raised it to 388: taken as the UNION -- #59's rationale is retained above and the count was RE-MEASURED on the new merge result rather than keeping either number, which is the rule this file states and the mistake that once put the floor 8 below reality. 391 (main, incl. #59's four tests) + 1 (this branch's one new test) = 392 measured, not assumed: 392 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. Three sibling follow-up branches are in flight against this same main (CI/ruff config, arm-attribution + durability, adapters label->ID); if this file conflicts with one of them, resolve as the UNION and RE-MEASURE on the new merge result rather than taking either number -- that is what #42 and #50 did, and taking a side is what put the floor 8 below reality earlier. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's one-test entry beside it -- and the count RE-MEASURED on the merge result. main fc1fd42 collects 407; this branch adds 1; 408 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped -- the one new test builds its own tmp ledger and codex rollout fixture, so it runs on a bare runner. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record differing only in `emitted_at` and `pr_number`; main's NEWER record was kept, since discarding a newer provenance observation to win a merge would corrupt the causal-provenance evidence CLAUDE.md 2 protects. FLOOR 411 -> 415 on 2026-08-23 (PR #70 diagnostics salvage): four new collected tests in test_capability_set_coverage.py from the PR #43 salvage plus CodeRabbit follow-ups on PR #51/#70 \u2014 union/missing-candidate fetch command, truncation after six modules, AST-scoped gate-call audit, and entrypoint-diagnosis coverage. NO CEILING MOVED and nothing new is skipped; all inject synthetic ledgers or read committed source. Measured on the merge result at 91d37fa: 389 passed + 26 skipped = 415 collected on CI, not assumed. FLOOR 415 -> 416 on 2026-08-23 (the dangling-citation follow-up, PR #74): exactly ONE new test, test_ci_gate_config.test_every_cited_repo_path_resolves, which reads the two committed config files this repo OWNS (the pin file and ruff.toml) and asserts every repo-relative path they cite exists. It exists because the pin file shipped citing docs/ci/LINT_BASELINE.md when the real path was docs/CI_LINT_BASELINE.md: the sibling checks read that file's CONTENTS thoroughly and its PROSE not at all, and the prose is the only pointer telling a reader where to re-measure before bumping a pin. Scoped to the two owned files deliberately -- scanning pr-00-gate.yml yields six findings that are all correct as written (guarded by hashFiles or a .agents check, or upstream paths), and a test that cries wolf gets waived. NO ceiling moved. RE-MEASURED SIX TIMES as the base moved under this ONE-LINE change: bd6da2e 402 -> ddb0928 402 -> fc1fd42 407 -> 0d661e3 407 -> 0593eeb 411 -> 6fed4ad 415, each +1 with this test, and the branch was rebuilt on each rather than the number carried forward. THIS BRANCH IS THE WORKED EXAMPLE of the equality's concurrency cost, so record it rather than rediscover it: main moved EIGHT times in the ~2.5 hours a one-line comment fix was open (#56, #68, #73, #69, #62, #70 and two direct commits), the floor line conflicted THREE separate times, and two merges overlapped the change directly -- #73 landed a byte-identical copy of the backplane-conformance.yml guard this branch also carried (dropped as redundant), and #69 edited this very test file in a neighbouring region. The equality is still the right call and should stay: every entry above this one records the floor being found BELOW reality, which is the permissive direction. But no amount of author care wins this race, because the correct value is only knowable on the merge result. The durable fix is CI running `verify.py --update-floor` on the merge commit, which keeps the equality and removes the race; until then a test-adding PR must be merged promptly after going green, because it re-conflicts on roughly every subsequent merge. FLOOR 416 -> 427 on 2026-08-23 (PR #72 hygiene untrack, rebased after #71 merged): exactly 11 new tests from test_repo_artifact_hygiene.py with root-anchored gitignore patterns. NO ceiling moved. Measured on merge result after #71 landed on main: 416 (main) + 11 = 427 collected via pytest --collect-only -q, not assumed. #71's simpler untrack landed first; this branch carries the full hygiene test suite and corrected root-anchored patterns." + "collected": 428, + "passed": 428, + "skipped_max": 26, + "selftest_skipped_max": 7, + "gate_skipped_max": 2, + "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that). FLOOR 402 -> 407 on 2026-08-23 (findability admission requirement). (findability admission requirement). (findability admission requirement). (findability admission requirement). Exactly five new pytest tests, all in test_capability_admission.py: test_findability_distinguishes_its_three_sub_causes, test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt, test_unreadable_reach_is_not_evaluated_and_never_a_failure, test_findability_exemption_is_declared_in_code_not_in_a_live_ledger, test_consult_sites_are_falsifiable_claims_about_real_callers. NO CEILING MOVED and nothing new skips: all five build synthetic ledgers in a tempdir or read committed tables, so none needs a populated capability ledger, an agent CLI or ~/.claude/skills. The one machine-dependent thing they touch -- an external consult site declared in capability_advisor.CONSULT_SITES whose skill prompt is not on this machine -- is reported as UNVERIFIED rather than skipped, because absence of the caller is not refutation of the claim; the in-tree site (tick) is asserted verified on every machine so the check can never degrade into 'everything unverified, nothing tested'. Measured on the merge result per the rule above: this file CONFLICTED three times while the branch was open, as main went 387 -> 391 -> 402 (#61, #64, #65, #60). Each time it was resolved as the UNION rather than by taking a side, and the count was RE-MEASURED on the new merge result rather than either number being carried forward: 402 (main at bd6da2e) + 5 (this branch's new tests) = 407. That is the rule this file already states -- measure the floor on the merge result, not on the branch -- and it mattered here, because #60 both deleted test_ci_gate_config.py and added more than it removed, so guessing in either direction would have been wrong. -> re-measured on 2026-08-23 (PR #62, the four deferred #42 review findings): three new tests, all machine-independent (each builds its own tmp_path Brain and manifests), so NO CEILING MOVED and nothing new is skipped. Fourth conflict for this branch, and the first one under the EQUALITY -- which is the point: the equality's own rationale says git conflict detection is what enforces 'measure on the merge result', and that is exactly what happened here. Under the old minimum the three earlier conflicts could each have been resolved by keeping the larger number; under the equality the count MUST be measured, and it was. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's three-test entry beside it -- and the count RE-MEASURED on the merge result rather than keeping either side's number. main fc1fd42 collects 407; this branch adds 3; 410 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record whose two sides differed only in `emitted_at` and `pr_number` (62 here, 68 on main). Main's NEWER record was kept rather than this branch's older one -- discarding a newer provenance observation to win a merge would corrupt exactly the causal-provenance evidence CLAUDE.md 2 protects, and this branch's own run re-emits its record anyway. FLOOR 410 -> 411 on 2026-08-23 (CodeRabbit follow-up on PR #42, thread 3837879039; re-measured again after #56 made `collected` an EQUALITY, which makes an assumed number a hard RED rather than a quiet pass -- main stayed at 402 across #56, and the merge result measures 403, so #56 added no collected tests and this is main's 402 plus this branch's one): exactly one new test, test_feedback_model_provenance.test_late_sweep_completes_terminal_attempts_never_one_in_flight, which pins that ledger_reconcile.resolve_unresolved_worker_attempts completes only TERMINAL unresolved worker attempts and never one still in flight. No ceiling moved and nothing new is skipped -- the test builds its own tmp ledger and codex rollout fixture and monkeypatches adapters.CODEX_SESSIONS, so it needs no agent CLI and no populated capability ledger and runs on a bare runner. RESOLVED AGAINST #59 (improvement-log accessor), which raised the floor 387 -> 391 on main while this branch raised it to 388: taken as the UNION -- #59's rationale is retained above and the count was RE-MEASURED on the new merge result rather than keeping either number, which is the rule this file states and the mistake that once put the floor 8 below reality. 391 (main, incl. #59's four tests) + 1 (this branch's one new test) = 392 measured, not assumed: 392 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. Three sibling follow-up branches are in flight against this same main (CI/ruff config, arm-attribution + durability, adapters label->ID); if this file conflicts with one of them, resolve as the UNION and RE-MEASURE on the new merge result rather than taking either number -- that is what #42 and #50 did, and taking a side is what put the floor 8 below reality earlier. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's one-test entry beside it -- and the count RE-MEASURED on the merge result. main fc1fd42 collects 407; this branch adds 1; 408 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped -- the one new test builds its own tmp ledger and codex rollout fixture, so it runs on a bare runner. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record differing only in `emitted_at` and `pr_number`; main's NEWER record was kept, since discarding a newer provenance observation to win a merge would corrupt the causal-provenance evidence CLAUDE.md 2 protects. FLOOR 411 -> 415 on 2026-08-23 (PR #70 diagnostics salvage): four new collected tests in test_capability_set_coverage.py from the PR #43 salvage plus CodeRabbit follow-ups on PR #51/#70 \u2014 union/missing-candidate fetch command, truncation after six modules, AST-scoped gate-call audit, and entrypoint-diagnosis coverage. NO CEILING MOVED and nothing new is skipped; all inject synthetic ledgers or read committed source. Measured on the merge result at 91d37fa: 389 passed + 26 skipped = 415 collected on CI, not assumed. FLOOR 415 -> 416 on 2026-08-23 (the dangling-citation follow-up, PR #74): exactly ONE new test, test_ci_gate_config.test_every_cited_repo_path_resolves, which reads the two committed config files this repo OWNS (the pin file and ruff.toml) and asserts every repo-relative path they cite exists. It exists because the pin file shipped citing docs/ci/LINT_BASELINE.md when the real path was docs/CI_LINT_BASELINE.md: the sibling checks read that file's CONTENTS thoroughly and its PROSE not at all, and the prose is the only pointer telling a reader where to re-measure before bumping a pin. Scoped to the two owned files deliberately -- scanning pr-00-gate.yml yields six findings that are all correct as written (guarded by hashFiles or a .agents check, or upstream paths), and a test that cries wolf gets waived. NO ceiling moved. RE-MEASURED SIX TIMES as the base moved under this ONE-LINE change: bd6da2e 402 -> ddb0928 402 -> fc1fd42 407 -> 0d661e3 407 -> 0593eeb 411 -> 6fed4ad 415, each +1 with this test, and the branch was rebuilt on each rather than the number carried forward. THIS BRANCH IS THE WORKED EXAMPLE of the equality's concurrency cost, so record it rather than rediscover it: main moved EIGHT times in the ~2.5 hours a one-line comment fix was open (#56, #68, #73, #69, #62, #70 and two direct commits), the floor line conflicted THREE separate times, and two merges overlapped the change directly -- #73 landed a byte-identical copy of the backplane-conformance.yml guard this branch also carried (dropped as redundant), and #69 edited this very test file in a neighbouring region. The equality is still the right call and should stay: every entry above this one records the floor being found BELOW reality, which is the permissive direction. But no amount of author care wins this race, because the correct value is only knowable on the merge result. The durable fix is CI running `verify.py --update-floor` on the merge commit, which keeps the equality and removes the race; until then a test-adding PR must be merged promptly after going green, because it re-conflicts on roughly every subsequent merge. FLOOR 416 -> 427 on 2026-08-23 (PR #72 hygiene untrack, rebased after #71 merged): exactly 11 new tests from test_repo_artifact_hygiene.py with root-anchored gitignore patterns. NO ceiling moved. Measured on merge result after #71 landed on main: 416 (main) + 11 = 427 collected via pytest --collect-only -q, not assumed. #71's simpler untrack landed first; this branch carries the full hygiene test suite and corrected root-anchored patterns. FLOOR 427 -> 428 on 2026-08-23 (PR salvaging #34/#42 remnants): exactly one new test, test_feedback_model_provenance.test_gemini_provenance_reads_the_per_run_log_before_the_conversation_store, recovered from #42's post-merge commit 4e0d6ae along with the adapters catalog work it exercises. No ceiling moved and nothing new is skipped -- it seeds adapters._ADVERTISED_MEMO instead of letting the catalog probe shell out, so it runs on any machine and adds no prerequisite. `passed` is 428 rather than the 426 verify.py suggested on this machine: two test_capabilities liveness tests (test_gate_blocks_execution_is_opt_in_and_narrow, test_evidence_gate_kind_is_not_blanket_observer) currently fail HERE on pristine main as well, because the hourly fleet tick mutated the machine-local ledger and range-lane-rollout now classifies matched_not_invoked rather than deliberately_gated. That is ledger STATE, not this branch and not the code -- CI bootstraps an empty ledger and counts 428/428. Recording 426 would have baked a local environment failure into the floor as though it were the expected result." } diff --git a/README.md b/README.md index c2a0308..25d1e5e 100644 --- a/README.md +++ b/README.md @@ -126,6 +126,16 @@ safety switch, not dead code. active, cooldown, per-subject, and global unevaluated duplicates; repeated runs on one subject sum to one effective learner observation. +11. **evidence_acquisition.py** — daily lane that acts on `capabilities.unblock()`, which named the + capabilities whose only blocker is missing evidence while nothing consumed the list. Bounded by + construction: one feed per cycle and three items per capability + (`ORCH_EVIDENCE_ACQUISITION_MAX_FEEDS` / `_MAX_ITEMS`). SHADOW by default — it computes the plan + and routes nothing; live routing is behind `ORCH_EVIDENCE_ACQUISITION`. A capability held by a + documented default-off switch is never fed: feeding one manufactures work it cannot execute, so + the durable reuse its own gate needs could never be produced and it would be fed again every + cycle forever. Reports the blocking AND drainable quantities on one line + (`feedable 0 / capped 1 / candidates 0 / fed 0`), so "nothing to feed" and "blocked from feeding" + are distinguishable instead of a shared silence. ### The Brain (feedback.py, SQLite at ~/.codex/orchestrator/feedback/orchestrator.db) - **Capability attribution at dispatch** — a run is tagged with the infrastructure capabilities it actually exercises (`dispatcher._exercised_capability_ids`: the gemini adapter path for diff --git a/adapters.py b/adapters.py index 88bb9c7..d60c1da 100644 --- a/adapters.py +++ b/adapters.py @@ -280,26 +280,81 @@ def parse_model_catalog_pairs(text: str) -> dict[str, str]: return pairs +# Catalog entries that are ROUTING TAGS, not model identities. cursor advertises `auto` in the same +# list as its 200-odd real ids, and recording `auto` as provider-resolved identity is precisely the +# "generic trace model is not provider resolution" CLAUDE.md section 2 forbids. +CATALOG_ROUTING_TAGS = frozenset({"auto", "default", "cli-default"}) + + +def _catalog_model_id(candidate: str | None) -> str | None: + """A catalog-sourced model id, or None when it is a routing tag rather than an identity. + + THE CATALOG IS ITS OWN VALIDATION. The id came out of the CLI's advertised list, so it exists by + construction; what it still needs is the identity/tag distinction and the placeholder check the + rest of the provenance path already applies (`_first_real_model` -> `feedback`). + + NOT `VENDOR_MODEL_RE`, and this is measured rather than argued: that regex is an allowlist of + vendor families for guarding strings we did NOT get from an authority (a slug we formed, a + transcript we grepped), and against the live `cursor-agent --list-models` on 2026-08-23 it + rejects 42 of 204 REAL ids -- every `claude-fable-*`, `cursor-grok-*`, `kimi-*` and `glm-*`, + plus the version-first `claude-4.6-opus-*` spelling. Using it as the catalog's validator would + trade fabricated ids for lost ones and re-break on the next vendor family -- the same + maintenance-treadmill shape as the bug above. `auto` is the ONE rejection it got right, and that + one is nameable. + """ + value = str(candidate or "").strip() + if not value or value.lower() in CATALOG_ROUTING_TAGS: + return None + return _first_real_model([value]) + + def model_id_for_label(agent: str, label: str) -> str | None: """Turn a CLI's human model label into its model id, or None if it cannot be trusted. - Prefers the CLI's OWN catalog, because that is the authority on its own ids. Falls back to the - obvious slug (`Gemini 3.7 Flash (High)` -> `gemini-3.7-flash-high`, `Composer 2.5` -> - `composer-2.5`), which happens to be exactly how both vendors form them -- but the result is - only returned if it looks like a real vendor model, so a chatty log line cannot become an id. + THE CATALOG IS THE AUTHORITY, and it has to be the CLI's RAW catalog to be one. This built its + lookup as `parse_model_catalog_pairs("\\n".join(f"{mid}\\t{mid}" for mid in + advertised_models(agent)))` -- a map of id->id, because `advertised_models` returns only ids. + `catalog.get(text.lower())` was then handed a human LABEL, which can never be an id key, so the + catalog branch was DEAD for every real label, the "prefers the CLI's own catalog" claim was + inoperative, and every call fell through to the slug heuristic. Measured against the live + `cursor-agent --list-models` on 2026-08-23, 4 of 5 real labels resolved wrongly: `Codex 5.3 + High` and `Claude Fable 5 1M Thinking (NO ZDR)` resolved to None (provenance simply lost), and + `Claude Opus 5 1M Thinking` / `GPT-5.6 Sol 1M High` resolved to `claude-opus-5-1m-thinking` / + `gpt-5.6-sol-1m-high` -- ids that DO NOT EXIST, written into + `execution_attempts.resolved_model` as provider-resolved identity. Only `Composer 2.5`, whose + label happens to slug into its own id, worked. + + Precedence, and the last two rungs are the point: + 1. the catalog's own `label -> id` pair (`advertised_catalog`); + 2. the label IS an advertised id -- some CLIs report the id, and the catalog confirms it; + 3. REFUSE, when the catalog was readable and lists neither. The slug would then be a guess + the authority contradicts, and `VENDOR_MODEL_RE` cannot catch it -- + `claude-opus-5-1m-thinking` is perfectly vendor-shaped and perfectly fictional. CLAUDE.md + section 2 forbids exactly this: a fabricated identity is worse than a skipped event. + 4. the vendor slug (`Gemini 3.7 Flash (High)` -> `gemini-3.7-flash-high`), ONLY while the + catalog is UNKNOWN (probe off, CLI missing, auth failed), because an unreadable catalog + must not cost us provenance we can still name. Still regex-guarded, so a chatty log line + cannot become an id. + + Rungs 1-2 are validated by `_catalog_model_id`, NOT by `VENDOR_MODEL_RE`. See the note there: + shape-matching an id the CLI itself advertised is both redundant and wrong. """ text = str(label or "").strip() if not text: return None try: - catalog = parse_model_catalog_pairs( - "\n".join(f"{mid}\t{mid}" for mid in advertised_models(agent)) - ) + catalog = advertised_catalog(agent) except Exception: # noqa: BLE001 - a probe failure must not block resolution - catalog = {} - direct = catalog.get(text.lower()) - if direct and VENDOR_MODEL_RE.fullmatch(direct): + catalog = {"models": [], "pairs": {}} + direct = _catalog_model_id(catalog["pairs"].get(text.lower())) + if direct: return direct + if catalog["models"]: + # The catalog ANSWERED. Accept the label only if it is itself an advertised id, then stop -- + # rung 3. `models` is the readability test, not `pairs`: both come from one probe, and + # non-empty ids mean the CLI was read. + exact = {mid.lower(): mid for mid in catalog["models"]}.get(text.lower()) + return _catalog_model_id(exact) slug = re.sub(r"[^a-z0-9.]+", "-", re.sub(r"\s*\([^)]*\)\s*$", "", text).lower()).strip("-") stripped = re.sub(r"\s*\(([^)]*)\)\s*$", r"-\1", text).lower() stripped = re.sub(r"[^a-z0-9.]+", "-", stripped).strip("-") @@ -312,6 +367,28 @@ def model_id_for_label(agent: str, label: str) -> str | None: AGY_MODEL_LABEL_RE = re.compile( r"Propagating selected model override to backend:\s*label=\"([^\"]+)\"" ) +# The per-run agy log's suffix, defined ONCE and consumed by both ends: `dispatcher` rewrites agy's +# `--log-file` to it, `cli_reported_model` reads it back. A reader that spells the writer's filename +# itself silently stops resolving the day the writer changes it, and the symptom is a model that +# never resolves -- which is exactly how the first version of that rewrite sat inert. +AGY_LOG_SUFFIX = ".agy.log" + + +def agy_log_for(log_file: str | Path | None) -> Path | None: + """The per-run agy log for a dispatch log, or None when there is no log to derive it from. + + Accepts the agy log itself, so a caller that already holds it is not made to derive it twice and + cannot accidentally produce `...agy.agy.log`. + """ + if not log_file: + return None + path = Path(str(log_file)).expanduser() + if path.name.endswith(AGY_LOG_SUFFIX): + return path + try: + return path.with_suffix(AGY_LOG_SUFFIX) + except ValueError: # a name `with_suffix` refuses (empty, or a trailing dot) + return path.with_name(path.name + AGY_LOG_SUFFIX) def model_label_from_agy_log(text: str) -> str | None: @@ -402,32 +479,66 @@ def _probe_env(agent: str) -> dict | None: return env -def advertised_models(agent: str, *, refresh: bool = False, timeout_s: int = 30) -> list[str]: - """Model ids the installed CLI for `agent` actually offers, or [] when it can't be read. +def _cached_catalog(agent: str, now: float, cache_path: Path, *, need_pairs: bool) -> dict | None: + """A fresh cached catalog, or None when the cache cannot answer THIS question. - Catalog probes are ~3s network calls, so results are cached on disk per agent (TTL) and shared - by the dispatcher and capacity's preflight. An EMPTY list means UNKNOWN, never "nothing is - advertised" — callers must not read it as evidence that a model is missing. Agents with no - probe registered always return [] and are therefore never judged unresolvable. + Hydrates the memo from disk BEFORE deciding, so an id-only caller still warms the memo for the + next one, and so a legacy blob's absent `pairs` key survives into the memo rather than being + flattened to an empty dict that would then look like a real answer. + """ + memo = _ADVERTISED_MEMO.get(agent) or {} + if memo.get("models") and now - float(memo.get("ts") or 0) <= GEMINI_MODEL_CACHE_TTL_S: + if not need_pairs or "pairs" in memo: + return {"models": list(memo["models"]), "pairs": dict(memo.get("pairs") or {})} + return None # memo predates the pairs migration; re-probe rather than answer blind + try: + cached = json.loads(cache_path.read_text()) + except (OSError, ValueError, TypeError): + return None + try: + if now - float(cached.get("ts") or 0) > GEMINI_MODEL_CACHE_TTL_S: + return None + except (TypeError, ValueError): + return None + models = [str(m) for m in (cached.get("models") or [])] + if not models: + return None + entry: dict = {"ts": cached.get("ts"), "models": models} + if "pairs" in cached: + entry["pairs"] = {str(k): str(v) for k, v in (cached.get("pairs") or {}).items()} + _ADVERTISED_MEMO[agent] = entry + if need_pairs and "pairs" not in entry: + return None + return {"models": models, "pairs": dict(entry.get("pairs") or {})} + + +def _advertised_catalog(agent: str, *, refresh: bool, timeout_s: int, need_pairs: bool) -> dict: + """The installed CLI's own model catalog: `{"models": [ids], "pairs": {label: id}}`. + + ONE probe and ONE cache behind both projections, because "which ids exist" and "which label + means which id" are the same question asked twice and must never disagree — they did, and the + label half was answered with `id -> id` (see `model_id_for_label`). + + Both empty means UNKNOWN, never "nothing is advertised". + + CACHE MIGRATION, STATED RATHER THAN SILENT (this file is on the provenance path). The on-disk + blob gains a `pairs` key ALONGSIDE the unchanged `models` list, so every existing reader of + `models` — including `capacity`'s preflight and the `test_capacity_profiles` prerequisite + check — is unaffected. A blob written before this carries no `pairs` KEY (presence, not + truthiness, so a label-less catalog still caches and cannot cause a probe per call): it serves + id requests from cache, while a PAIRS request treats it as a miss and re-probes, rewriting it in + the new shape. The migration therefore self-heals within one TTL per agent, and until it does + `model_id_for_label` degrades to the slug heuristic it already used — never to a wrong id. """ probe = MODEL_CATALOG_PROBES.get(agent) if not probe or not _model_probe_enabled(): - return [] + return {"models": [], "pairs": {}} now = time.time() cache_path = _catalog_cache_path(agent) if not refresh: - memo = _ADVERTISED_MEMO.get(agent) or {} - if memo.get("models") and now - float(memo.get("ts") or 0) <= GEMINI_MODEL_CACHE_TTL_S: - return list(memo["models"]) - try: - cached = json.loads(cache_path.read_text()) - if now - float(cached.get("ts") or 0) <= GEMINI_MODEL_CACHE_TTL_S: - models = [str(m) for m in (cached.get("models") or [])] - if models: - _ADVERTISED_MEMO[agent] = {"ts": cached.get("ts"), "models": models} - return models - except (OSError, ValueError, TypeError): - pass + hit = _cached_catalog(agent, now, cache_path, need_pairs=need_pairs) + if hit is not None: + return hit try: proc = subprocess.run( probe, @@ -438,19 +549,48 @@ def advertised_models(agent: str, *, refresh: bool = False, timeout_s: int = 30) stdin=subprocess.DEVNULL, ) except (OSError, subprocess.SubprocessError): - return [] # CLI missing/hung => unknown, not "unavailable" + return {"models": [], "pairs": {}} # CLI missing/hung => unknown, not "unavailable" if proc.returncode != 0: - return [] # includes 'Authentication required' => unknown + return {"models": [], "pairs": {}} # includes 'Authentication required' => unknown models = parse_agy_models(proc.stdout) if not models: - return [] - _ADVERTISED_MEMO[agent] = {"ts": int(now), "models": models} + return {"models": [], "pairs": {}} + # THE RAW TEXT, parsed twice. Rebuilding pairs from `models` is what broke the resolver: the + # labels only exist in the CLI's own output, so they have to be kept while it is in hand. + pairs = parse_model_catalog_pairs(proc.stdout) + _ADVERTISED_MEMO[agent] = {"ts": int(now), "models": models, "pairs": pairs} try: cache_path.parent.mkdir(parents=True, exist_ok=True) - cache_path.write_text(json.dumps({"ts": int(now), "models": models})) + cache_path.write_text(json.dumps({"ts": int(now), "models": models, "pairs": pairs})) except OSError: pass # cache is an optimization, never a hard dependency - return models + return {"models": list(models), "pairs": dict(pairs)} + + +def advertised_catalog(agent: str, *, refresh: bool = False, timeout_s: int = 30) -> dict: + """`{"models": [ids], "pairs": {label: id}}` the installed CLI advertises. + + Both empty means UNKNOWN. The `pairs` half is what `model_id_for_label` always needed and never + had: the CLIs report a LABEL at runtime (`"model":"Composer 2.5"`, + `label="Gemini 3.7 Flash (High)"`) and the id only in their catalog. + """ + return _advertised_catalog(agent, refresh=refresh, timeout_s=timeout_s, need_pairs=True) + + +def advertised_models(agent: str, *, refresh: bool = False, timeout_s: int = 30) -> list[str]: + """Model ids the installed CLI for `agent` actually offers, or [] when it can't be read. + + Catalog probes are ~3s network calls, so results are cached on disk per agent (TTL) and shared + by the dispatcher and capacity's preflight. An EMPTY list means UNKNOWN, never "nothing is + advertised" — callers must not read it as evidence that a model is missing. Agents with no + probe registered always return [] and are therefore never judged unresolvable. + + Ids ONLY. For label resolution use `advertised_catalog` — building pairs out of this list + yields `id -> id`, which is the bug `model_id_for_label` documents. + """ + return _advertised_catalog(agent, refresh=refresh, timeout_s=timeout_s, need_pairs=False)[ + "models" + ] def agy_advertised_models(*, refresh: bool = False, timeout_s: int = 30) -> list[str]: @@ -1057,6 +1197,7 @@ def cli_reported_model( *, started_ts: int | None = None, window_s: int = 7200, + log_file: str | Path | None = None, ) -> dict: """Identity the agent's own CLI recorded for the run in `workspace`. @@ -1064,6 +1205,9 @@ def cli_reported_model( ``{"model": str | None, "cli_version": str | None, "source": str | None, "reason": str | None}``. An unresolved answer with a named reason is the whole point -- silence here was indistinguishable from "no such run". + + `log_file` is THIS run's dispatch log. It is what makes the gemini answer per-run rather than a + workspace-and-window guess (see that branch); callers without one still get the store fallback. """ blank = {"model": None, "cli_version": None, "source": None, "reason": None} if agent in NO_SESSION_LOG_AGENTS: @@ -1084,12 +1228,50 @@ def cli_reported_model( except (OSError, json.JSONDecodeError): version = None return {"model": model, "cli_version": version, "source": str(path), "reason": None} - if agent in ("cursor", "vibe", "gemini"): - reader = { - "cursor": _cursor_model_for, - "vibe": _vibe_model_for, - "gemini": _agy_model_for, - }[agent] + if agent == "gemini": + # PER-RUN LOG FIRST, STORE SECOND -- the precedence the comment above + # `NO_SESSION_LOG_AGENTS` has always claimed while this function did the opposite: `gemini` + # mapped only to `_agy_model_for` and `model_label_from_agy_log` was never called from here, + # so a gemini run resolved from the conversation store ONLY. Commit fe59bc7 ("the run + # reports its own model") settles which way to fix it: agy's log line belongs to THIS run, + # because the + # dispatcher gives each run its own `--log-file`, where the store has to be matched by + # workspace AND time window and can pick up a neighbour's session. + agy_log = agy_log_for(log_file) + if agy_log is not None: + label = model_label_from_agy_log(_read_head(agy_log)) + # `model_id_for_label` resolves against agy's OWN catalog. That resolver had to be fixed + # FIRST: while it built an id->id map it always fell through to a slug guess, so routing + # gemini provenance through it would have persisted a heuristic as provider-resolved + # identity -- CLAUDE.md section 2's exact prohibition. + model = model_id_for_label("gemini", label) if label else None + if model: + return { + "model": model, + "cli_version": None, + "source": str(agy_log), + "reason": None, + } + model = _agy_model_for(str(workspace), started_ts, window_s) + if model: + return { + "model": model, + "cli_version": None, + "source": "gemini-session-store", + "reason": None, + } + # The reason names what was actually SEARCHED, so a run dispatched without a log is + # distinguishable from one whose log named nothing. + return { + **blank, + "reason": ( + "no_gemini_model_in_run_log_or_session_store" + if agy_log is not None + else "no_gemini_session_matched_workspace" + ), + } + if agent in ("cursor", "vibe"): + reader = {"cursor": _cursor_model_for, "vibe": _vibe_model_for}[agent] model = reader(str(workspace), started_ts, window_s) if not model: return {**blank, "reason": f"no_{agent}_session_matched_workspace"} @@ -1597,6 +1779,160 @@ def _selftest_inner(*, gaps: list[str] | None = None): "gemini-3.6-flash-low\tGemini 3.6 Flash (Low)\n" ) assert parsed == ["gemini-3.1-pro-high", "gemini-3.6-flash-low"], parsed + # LABEL -> ID, AGAINST THE CLI'S RAW CATALOG. The resolver used to build its lookup from + # `advertised_models()`, which returns only ids, so the map was `id -> id` and a human LABEL + # could never key it: the catalog branch was dead and every call fell through to the slug. + # Fixtures are verbatim `cursor-agent --list-models` / `agy models` lines, because the whole + # point is that the real catalog's labels do NOT slug into their ids. + cursor_catalog = ( + "Fetching available models...\n" + "auto - Auto (default)\n" + "composer-2.5 - Composer 2.5 (current)\n" + "gpt-5.3-codex-high - Codex 5.3 High\n" + "claude-opus-5-thinking-high - Claude Opus 5 1M Thinking\n" + "claude-fable-5-thinking-high - Claude Fable 5 1M Thinking (NO ZDR)\n" + ) + import shutil as _probe_shutil + import tempfile as _probe_tmp + + old_memo_pairs = dict(_ADVERTISED_MEMO) + old_probe_flag = os.environ.pop("ORCH_MODEL_PROBE", None) + try: + # ISOLATION, NOT A SKIP: a memo entry means `_advertised_catalog` answers from memory, so + # nothing here shells out to `cursor-agent` (the probe-leak this file has already been + # bitten by twice). The memo needs the probe flag ON: the kill-switch short-circuits first. + _ADVERTISED_MEMO["cursor"] = { + "ts": time.time(), + "models": parse_model_catalog(cursor_catalog), + "pairs": parse_model_catalog_pairs(cursor_catalog), + } + # Every one of these was WRONG before the fix, verified against the live catalog on + # 2026-08-23: two resolved to None and two to vendor-shaped ids that do not exist. + for label, expected in ( + ("Composer 2.5", "composer-2.5"), + ("Codex 5.3 High", "gpt-5.3-codex-high"), + ("Claude Opus 5 1M Thinking", "claude-opus-5-thinking-high"), + ("Claude Fable 5 1M Thinking (NO ZDR)", "claude-fable-5-thinking-high"), + ("composer-2.5", "composer-2.5"), # the CLI may report the id; the catalog confirms it + ): + assert model_id_for_label("cursor", label) == expected, (label, expected) + # A ROUTING TAG IS NOT AN IDENTITY. cursor advertises `auto` beside its real ids. + assert model_id_for_label("cursor", "Auto (default)") is None + assert model_id_for_label("cursor", "auto") is None + # THE ACCEPTANCE CASE. A vendor-renamed LABEL against an unchanged id: the slug + # (`composer-pro-2.5`) is not advertised, so only the catalog can answer. + _ADVERTISED_MEMO["cursor"] = { + "ts": time.time(), + "models": ["composer-2.5"], + "pairs": parse_model_catalog_pairs("composer-2.5 - Composer Pro 2.5 (current)"), + } + assert model_id_for_label("cursor", "Composer Pro 2.5") == "composer-2.5" + # DELIBERATE BREAK: the old id->id construction, which is what made the branch dead. + _broken = {"models": ["composer-2.5"], "pairs": {"composer-2.5": "composer-2.5"}} + _real_catalog = advertised_catalog + try: + globals()["advertised_catalog"] = lambda *_a, **_k: dict(_broken) + assert ( + model_id_for_label("cursor", "Composer Pro 2.5") is None + ), "the break must restore the pre-fix behaviour: a label cannot key an id->id map" + finally: + globals()["advertised_catalog"] = _real_catalog # REVERTED + assert model_id_for_label("cursor", "Composer Pro 2.5") == "composer-2.5" + # A READABLE CATALOG THAT LISTS NEITHER REFUSES, rather than persisting a guess it + # contradicts -- `VENDOR_MODEL_RE` cannot catch a fictional-but-vendor-shaped id. + assert model_id_for_label("cursor", "Composer Ultra 9.9") is None + assert ( + VENDOR_MODEL_RE.fullmatch("claude-opus-5-1m-thinking") is not None + ), "the shape guard genuinely cannot do the catalog's job" + finally: + _ADVERTISED_MEMO.clear() + _ADVERTISED_MEMO.update(old_memo_pairs) + if old_probe_flag is not None: + os.environ["ORCH_MODEL_PROBE"] = old_probe_flag + # THE PROBE MUST KEEP THE LABELS. The cases above seed the memo, so they exercise the + # RESOLVER but not the wiring the finding was actually about -- and with only those, rebuilding + # `pairs` from `models` (the pre-fix construction) leaves the selftest green. Drive the real + # `subprocess.run` seam with a verbatim catalog and assert on the pairs it stored. + _probed_cache = Path(_probe_tmp.mkdtemp(prefix="adapters-catalog-probe-")) + _old_runtime, _old_run = AGENT_RUNTIME, subprocess.run + try: + globals()["AGENT_RUNTIME"] = _probed_cache + + class _CatalogCompleted: + returncode = 0 + stdout = cursor_catalog + stderr = "" + + globals()["subprocess"].run = lambda *_a, **_k: _CatalogCompleted() + _ADVERTISED_MEMO.pop("cursor", None) + probed = advertised_catalog("cursor", refresh=True) + assert probed["models"][:2] == ["auto", "composer-2.5"], probed["models"] + assert probed["pairs"]["composer 2.5"] == "composer-2.5", probed["pairs"] + assert probed["pairs"]["codex 5.3 high"] == "gpt-5.3-codex-high", probed["pairs"] + # THE REGRESSION GUARD: an id->id map has ids for keys and no label keys at all. + assert "composer-2.5" not in probed["pairs"], ( + "pairs must be keyed by LABEL; an id key means the id->id construction is back", + probed["pairs"], + ) + assert model_id_for_label("cursor", "Codex 5.3 High") == "gpt-5.3-codex-high" + # And the pairs reach DISK, so the next process resolves labels without re-probing. + _on_disk = json.loads(_catalog_cache_path("cursor").read_text()) + assert _on_disk["models"] == probed["models"], _on_disk + assert _on_disk["pairs"]["claude opus 5 1m thinking"] == "claude-opus-5-thinking-high" + # `models` is byte-compatible with the pre-migration blob every other reader still uses. + assert set(_on_disk) == {"ts", "models", "pairs"}, _on_disk + finally: + globals()["AGENT_RUNTIME"] = _old_runtime + globals()["subprocess"].run = _old_run + _ADVERTISED_MEMO.pop("cursor", None) + _ADVERTISED_MEMO.update(old_memo_pairs) + _probe_shutil.rmtree(_probed_cache, ignore_errors=True) + # AN UNKNOWN CATALOG KEEPS THE SLUG, so an unreadable CLI costs no provenance we can still + # name. Removing the probe entry is how "UNKNOWN" is expressed without a subprocess. + old_probes = dict(MODEL_CATALOG_PROBES) + try: + MODEL_CATALOG_PROBES.pop("cursor", None) + assert model_id_for_label("cursor", "Composer 2.5") == "composer-2.5" + assert model_id_for_label("cursor", "Gemini 3.7 Flash (High)") == "gemini-3.7-flash-high" + assert model_id_for_label("cursor", "some log prose") is None + finally: + MODEL_CATALOG_PROBES.clear() + MODEL_CATALOG_PROBES.update(old_probes) + # THE CACHE MIGRATION IS EXPLICIT. A blob written before `pairs` existed still answers id + # questions from cache, and is a MISS for label questions rather than answering blind. + _cache_dir = Path(_probe_tmp.mkdtemp(prefix="adapters-catalog-cache-")) + try: + legacy = _cache_dir / "legacy.json" + legacy.write_text(json.dumps({"ts": int(time.time()), "models": ["composer-2.5"]})) + _ADVERTISED_MEMO.pop("cursor", None) + assert _cached_catalog("cursor", time.time(), legacy, need_pairs=False) == { + "models": ["composer-2.5"], + "pairs": {}, + } + _ADVERTISED_MEMO.pop("cursor", None) + assert _cached_catalog("cursor", time.time(), legacy, need_pairs=True) is None + # And the memo hydrated from that legacy blob must not then LOOK like a pairs answer. + assert "pairs" not in (_ADVERTISED_MEMO.get("cursor") or {}), _ADVERTISED_MEMO.get("cursor") + assert _cached_catalog("cursor", time.time(), legacy, need_pairs=True) is None + migrated = _cache_dir / "migrated.json" + migrated.write_text( + json.dumps( + { + "ts": int(time.time()), + "models": ["composer-2.5"], + "pairs": {"composer 2.5": "composer-2.5"}, + } + ) + ) + _ADVERTISED_MEMO.pop("cursor", None) + assert _cached_catalog("cursor", time.time(), migrated, need_pairs=True) == { + "models": ["composer-2.5"], + "pairs": {"composer 2.5": "composer-2.5"}, + } + finally: + _ADVERTISED_MEMO.pop("cursor", None) + _ADVERTISED_MEMO.update(old_memo_pairs) + _probe_shutil.rmtree(_cache_dir, ignore_errors=True) # Rename survival: pinned model gone => auto-pick the newest Pro/high seat, never die. old_memo = dict(_ADVERTISED_MEMO) try: diff --git a/dispatcher.py b/dispatcher.py index 50172cb..1bd7680 100644 --- a/dispatcher.py +++ b/dispatcher.py @@ -1430,7 +1430,9 @@ def offload( logf = DISPATCH_LOG_DIR / f"offload.{agent}.{time.time_ns()}.log" agy_log: Path | None = None if agent == "gemini" and "--log-file" in argv: - agy_log = logf.with_suffix(".agy.log") + # `adapters.agy_log_for` owns the name, because `adapters.cli_reported_model` reads it back. + # Two literals would drift, and the symptom of drift here is a model that never resolves. + agy_log = adapters.agy_log_for(logf) argv[argv.index("--log-file") + 1] = str(agy_log) auth_prelude = _auth_prelude(agent) agent_prelude = _agent_runtime_prelude(agent) @@ -1580,7 +1582,9 @@ def _record_complete(exit_code: int | None = None, error: str | None = None) -> probe = ( {"model": observed_model, "reason": None} if observed_model - else adapters.cli_reported_model(agent, run_cwd, started_ts=started_ts) + else adapters.cli_reported_model( + agent, run_cwd, started_ts=started_ts, log_file=str(logf) + ) ) if probe.get("model"): feedback.complete_profile_attempt( diff --git a/ledger_reconcile.py b/ledger_reconcile.py index 6886c70..3fff5aa 100644 --- a/ledger_reconcile.py +++ b/ledger_reconcile.py @@ -435,17 +435,22 @@ def _workspace_from_rows(run_rows) -> str | None: return None -def _cli_identity_for_run(run_rows, started_ts: int | None = None) -> dict: +def _cli_identity_for_run(run_rows, started_ts: int | None = None, log_file=None) -> dict: """CLI-reported identity for a run, or a NAMED reason there is none. Never guesses. A seat whose CLI leaves no per-session log returns the reason and the attempt stays unresolved -- `fallback_reason` then says which seat and why, instead of the single undifferentiated `resolved_model_not_reported_by_completion` that every seat used to get. + + `log_file` is the run's own dispatch log, which is what lets the gemini seat read the model from + THIS run's agy log instead of matching a conversation store by workspace and window. """ agent = next((str(row.get("agent")) for row in run_rows if row.get("agent")), "") if not agent: return {"model": None, "cli_version": None, "source": None, "reason": "no_agent_on_run"} - return adapters.cli_reported_model(agent, _workspace_from_rows(run_rows), started_ts=started_ts) + return adapters.cli_reported_model( + agent, _workspace_from_rows(run_rows), started_ts=started_ts, log_file=log_file + ) def record_completion( @@ -473,7 +478,7 @@ def record_completion( # caller left EMPTY -- an explicitly supplied resolved_model always wins, because the # caller may have provenance this reader cannot see. probed = adapters.cli_reported_model( - agent, target[len("offload:") :], started_ts=started_ts + agent, target[len("offload:") :], started_ts=started_ts, log_file=log_file ) probe_reason = probed.get("reason") if probed.get("model"): @@ -680,7 +685,7 @@ def reconcile( ), default=None, ) - identity = _cli_identity_for_run(run_rows, started_ts=started) + identity = _cli_identity_for_run(run_rows, started_ts=started, log_file=log_file) selected = next(iter(profile_ids)) if identity.get("model"): feedback.complete_profile_attempt( diff --git a/test_feedback_model_provenance.py b/test_feedback_model_provenance.py index 55da99c..f85b94b 100644 --- a/test_feedback_model_provenance.py +++ b/test_feedback_model_provenance.py @@ -1,3 +1,4 @@ +import json import sqlite3 import time @@ -993,3 +994,124 @@ def __exit__(self, *exc): # REVERTED by monkeypatch teardown; the filtered assertions above are the guard. finally: feedback.DB_PATH = old_db + + +def test_gemini_provenance_reads_the_per_run_log_before_the_conversation_store( + tmp_path, monkeypatch +): + """The precedence the code claimed in a comment but did not implement. + + `cli_reported_model` mapped `gemini` to `_agy_model_for` alone -- a conversation-store scrape + joined by workspace and time window -- while the comment above `NO_SESSION_LOG_AGENTS` said the + per-run agy log was primary and the store only a fallback. `model_label_from_agy_log` was never + called from here at all. Commit fe59bc7 ("the run reports its own model") settles the direction: + the per-run log wins, because the dispatcher gives each run its own `--log-file`, so that line + belongs to exactly one run where a store match can pick up a neighbour's session. + + Ordering matters and is asserted here: resolution runs through `model_id_for_label`, which until + the catalog fix returned a slug guess for every label. This test pins that the label resolves to + the id agy ADVERTISES, not to the slug of the label. + """ + import adapters + + workspace = tmp_path / "offloads" / "20260823T000000Z-issue-9-1" + workspace.mkdir(parents=True) + + # agy's OWN catalog, verbatim `agy models` shape. Seeding the memo is isolation, not a skip: + # `advertised_catalog` shells out to `agy models` on a cold cache, and that probe has already + # leaked into two monkeypatched-`subprocess.run` selftests in this branch. + catalog = ( + "Fetching available models...\n" + "gemini-3.6-flash-high\tGemini 3.6 Flash (High)\n" + "gemini-3.1-pro-high\tGemini 3.1 Pro (High)\n" + ) + monkeypatch.setitem( + adapters._ADVERTISED_MEMO, + "gemini", + { + "ts": time.time(), + "models": adapters.parse_model_catalog(catalog), + "pairs": adapters.parse_model_catalog_pairs(catalog), + }, + ) + # An EMPTY store, so a passing assertion can only have come from the log. + monkeypatch.setattr(adapters, "AGY_HOME", tmp_path / "agy-empty") + + dispatch_log = tmp_path / "offload.gemini.1787500000000000000.log" + agy_log = adapters.agy_log_for(dispatch_log) + # ONE name, both ends: the dispatcher derives the same path when it rewrites agy's --log-file. + assert agy_log == tmp_path / "offload.gemini.1787500000000000000.agy.log", agy_log + assert adapters.agy_log_for(agy_log) == agy_log, "already-an-agy-log must not double-suffix" + assert adapters.agy_log_for(None) is None + agy_log.write_text( + "I0823 20:08:19.863873 1 model_config_manager.go:311] Propagating selected " + 'model override to backend: label="Gemini 3.1 Pro (High)"\n' + "I0823 20:08:41.101010 1 model_config_manager.go:311] Propagating selected " + 'model override to backend: label="Gemini 3.6 Flash (High)"\n' + ) + + got = adapters.cli_reported_model("gemini", workspace, log_file=str(dispatch_log)) + # LAST OCCURRENCE WINS -- the line is re-emitted as the session settles. + assert got["model"] == "gemini-3.6-flash-high", got + assert got["source"] == str(agy_log), got + assert got["reason"] is None, got + + # THE LOG BEATS THE STORE. With a store that names a DIFFERENT model, the per-run log still + # wins -- which is the whole claim, and the reason a store-only mapping was wrong. + store_home = tmp_path / "agy" + brain = store_home / "brain" / "conv-1" / ".system_generated" / "logs" + brain.mkdir(parents=True) + (brain / "transcript_full.jsonl").write_text(json.dumps({"model": "claude-sonnet-4-6"}) + "\n") + index = store_home / "conversation_summaries.db" + conn = sqlite3.connect(index) + conn.execute( + "CREATE TABLE conversation_summaries " + "(conversation_id TEXT, workspace_uris TEXT, last_modified_time INTEGER)" + ) + conn.execute( + "INSERT INTO conversation_summaries VALUES (?, ?, ?)", + ("conv-1", f"file://{workspace.resolve()}", 1_787_500_000), + ) + conn.commit() + conn.close() + monkeypatch.setattr(adapters, "AGY_HOME", store_home) + both = adapters.cli_reported_model("gemini", workspace, log_file=str(dispatch_log)) + assert both["model"] == "gemini-3.6-flash-high", ("the per-run log is primary", both) + + # THE STORE IS STILL THE FALLBACK, so removing the log does not lose the seat's provenance. + no_log = adapters.cli_reported_model("gemini", workspace) + assert no_log["model"] == "claude-sonnet-4-6", no_log + assert no_log["source"] == "gemini-session-store", no_log + + # NEITHER PATH ANSWERS => a reason naming what was searched, distinguishable from a run that + # never had a log at all. + monkeypatch.setattr(adapters, "AGY_HOME", tmp_path / "agy-empty") + agy_log.write_text("nothing a model could be read from\n") + silent = adapters.cli_reported_model("gemini", workspace, log_file=str(dispatch_log)) + assert silent["model"] is None + assert silent["reason"] == "no_gemini_model_in_run_log_or_session_store", silent + assert ( + adapters.cli_reported_model("gemini", workspace)["reason"] + == "no_gemini_session_matched_workspace" + ) + + # DELIBERATE BREAK: gemini mapped to the store scrape only, as before the fix. The per-run log + # is then unreachable and the run resolves to the store's model -- or to nothing. + agy_log.write_text( + "model_config_manager.go:311] Propagating selected model override to backend: " + 'label="Gemini 3.6 Flash (High)"\n' + ) + monkeypatch.setattr(adapters, "AGY_HOME", store_home) + real_label_reader = adapters.model_label_from_agy_log + monkeypatch.setattr(adapters, "model_label_from_agy_log", lambda _text: None) + broken = adapters.cli_reported_model("gemini", workspace, log_file=str(dispatch_log)) + assert broken["model"] == "claude-sonnet-4-6", ( + "the break must restore the old behaviour: the store answers and the log is never read", + broken, + ) + # REVERTED, and the per-run log is primary again. + monkeypatch.setattr(adapters, "model_label_from_agy_log", real_label_reader) + assert ( + adapters.cli_reported_model("gemini", workspace, log_file=str(dispatch_log))["model"] + == "gemini-3.6-flash-high" + )