feat(profiles+provenance): a profile per agent, and the provenance chain that makes per-agent identity trustworthy - #42
Merged
Conversation
… a one-seat miner is visible PROFILES. Only codex had a registered execution profile, so only codex could ever record a worker attempt and only codex work could ever be minable. All six seats now have a profile and a capacity pool. Pool fields are COPIED from capacity.AGENTS -- execution_profiles cannot import capacity (the dependency runs the other way), so `test_capacity_pools_match_capacity_agents` asserts the two agree and fails if either drifts. A pool that misdescribed a real account would mis-report capacity to the dispatcher, so this is a guarded copy, never a guess. `test_registry_models_match_adapters` likewise fails if the registry requests a model the adapter never sends. Three seats (gemini, cursor, vibe) report a routing tag rather than a vendor model, so their attempts complete UNRESOLVED and their work stays unminable. Registered anyway, deliberately: an unresolved worker attempt is a visible, attributable gap, whereas no profile at all is silence -- and silence is what hid a dead miner for 43 days. COVERAGE. An up/down signal cannot distinguish a healthy miner from a healthy miner covering one seat of six. `mining_coverage()` reports, per agent that did work: whether a profile exists, whether its model can ever RESOLVE, and how many worker attempts actually resolved -- with a verdict naming the blocker (`no_profile`, `model_not_reportable`, `no_worker_attempt`, `minable`, `no_runs`). The headline is a FRACTION on purpose: "0 of 5 working agents minable" is a coverage problem you can see; "it ran" is not. Surfaced in periodic_report, the existing operator report, rather than a second auditor. Also fixes a bug in that report: `pattern compiler: status=` read a `status` key the miner's artifact has never had, so it printed None on every report. It now renders `mining_health`, which is the field that says what happened. WHAT THE COVERAGE HONESTLY SAYS TODAY: nothing is minable yet. `reported_model` is empty for every agent, and 0 worker attempts have ever resolved -- including codex. Every resolved_model in the store came from LangSmith EVALUATOR ingestion, and `langsmith_direct.py:267` deliberately refuses to promote a trace's generic `model` field to worker resolution without an explicit `resolved_model`. So the remaining work is worker TRACING, not per-agent adapter work: one route unblocks five seats. vibe is the real exception -- `vibe:default` names no model at all. Four existing assertions changed, each a literal that only held while codex was the only seat: `len(pools) == 1`, `COUNT(*) == 3`, `1/3` propensity, `== ["codex-subscription"]`. Each was re-scoped to the property it protects -- one agent never spans two accounts; a shared subscription is debited once per profile, not per model; every registered profile persists. The share-one-pool invariant is now asserted for EVERY agent, so it is stronger than before. Two pre-existing selftest stubs needed `**kwargs` because every seat now passes `profile=`; a narrower stub raised TypeError inside offload() and read like a dispatch bug rather than a stale double. My own earlier assertion "unprofiled agent must not get a fabricated worker attempt" was wrong once gemini had a profile. Replaced with the honest and stronger invariant: a routing-tag seat DOES record an attempt, that attempt stays unresolved, and `resolved_worker_identity_for_run` must still return None for it. Verified: 368 passed, 0 failed, 81 of 81 selftests, 5 of 5 gates green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…coverage
Only codex had a registered execution profile, so only codex could record a
worker attempt and only codex work could ever be minable. All six seats now have
a profile and a capacity pool, and every seat's model is NAMED from that seat's
own authority rather than a routing tag:
codex/claude tier research gpt-5.6-sol / claude-sonnet-5
gemini agy's OWN advertised-models probe gemini-3.1-pro-high
cursor its known default composer-2.5
vibe its own config.toml mistral-medium-3.5
aider floating alias mistral/codestral-latest
vibe was the one real discovery: `model_identity` returned `vibe:default`, which
said nothing, while `~/.vibe/config.toml` records
`active_model = "mistral-medium-3.5"` -- matching the 2026-08-08 tier research
already in this file. The config maps that alias to provider id
`mistral-vibe-cli-latest`, which FLOATS, so this names the REQUEST honestly; an
immutable resolved identity still has to come from the provider.
TWO THINGS I GOT WRONG, both caught by the suite:
1. I stripped the `agy:`/`cursor:` prefixes, believing they hid an identity.
They do not. `--model` is sent BARE and the prefix records WHICH ROUTER served
the work -- agy is a multi-provider router whose probe lists gemini, claude AND
gpt-oss ids, and both seats are metered separately. This module's own selftest
asserts the prefix and four other modules depend on it; removing it broke 5
selftests. Restored, with the attempt recorded in the code so it is not retried.
The correct split: `model_identity` is a DRIFT TAG that keeps the router, and a
profile's `requested_model` is the bare vendor id -- the field a
provider-resolved model is compared against. `test_registry_models_match_adapters`
now reconciles the two by stripping the router prefix rather than flattening either.
2. I changed `gemini-3.1-pro-high`'s definition after it had been persisted, which
`ensure_schema` correctly refuses (`immutable execution profile changed`) and
which broke 3 more selftests. The conflicting rows were my own unmerged test
writes from 15 minutes earlier -- timestamps proved it: the 3 codex profiles date
to 2026-07-09, the rest to today 13:20/13:35. Reverted those six local rows
(backed up first; published codex rows untouched) rather than bumping
PROFILE_SCHEMA_VERSION, which would have created a new registry generation for
every profile and conflated "the schema changed" with "I fixed my own typo".
COVERAGE. An up/down signal cannot distinguish a healthy miner from a healthy
miner covering one seat of six. `mining_coverage()` reports per agent: profile
present, model resolvable, worker attempts resolved -- with a verdict naming the
blocker, and a headline FRACTION ("0 of 5 working agents minable") because "it
ran" is what hid a dead miner for 43 days. Surfaced in periodic_report, the
existing operator surface, not a second auditor. Also fixes that report printing
`status=None` for the miner forever: it read a `status` key the artifact never had.
Drift guards, because execution_profiles cannot import capacity (the dependency
runs the other way): pools are asserted against capacity.AGENTS, registry models
against adapters, gemini's model against agy's advertised list, and vibe's against
its live config -- each skipping with the missing thing NAMED when absent.
Four existing assertions re-scoped from literals that only held while codex was
the only seat (`len(pools)==1`, `COUNT(*)==3`, `1/3`, `["codex-subscription"]`) to
the properties they protect. The share-one-pool invariant now covers EVERY agent,
so it is stronger than before.
Honest state: nothing is minable yet. `reported_model` is empty for every agent
and 0 worker attempts have ever resolved. Every resolved_model in the store came
from LangSmith EVALUATOR ingestion, and langsmith_direct.py:267 deliberately
refuses to promote a trace's generic `model` to worker resolution. The remaining
work is worker TRACING -- one route for all six seats, not six adapter problems.
Verified: 371 passed, 0 failed, 81 of 81 selftests, 5 of 5 gates green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…its are permanent, not pending Two backlog items: C (evidence-acquisition lane) and G (auth probe strength). C — PREREQUISITE FIXED, LANE DELIBERATELY NOT BUILT. `unblock()` already owns the feed decision and only one branch returns feed=True; nothing consumed it to route work, so the lane was genuinely missing. But before building it I checked WHAT it would feed: exactly two capabilities, `range-lane-rollout` and `synthesis-promotion`, both `deliberately_gated` with the kill switch "restore documented default-off gate" -- i.e. held by a default-OFF ORCH_* flag. Feeding a switched-off capability manufactures work it cannot execute, so no durable reuse is produced, so the debt never falls, so it is fed again every cycle. The drain is blocked by the very switch the feed ignored. Building a bounded lane on that decision would have automated this workspace's signature defect. So `unblock()` now refuses to feed a documented default-off switch, while STILL REPORTING the evidence debt -- the capability stays visible as starved instead of vanishing from the queue -- and hands the owner the only decision that can move it. `_has_default_off_switch` reads the capability's own declared kill_switch/rollback prose rather than a second list of flag names, so a rename cannot drift out of the check, and an unmatched switch reads as NOT default-off so a genuinely feedable capability stays feedable. **After the fix the feedable set is 0 of 42.** That is why the lane is not built: its input can only become non-empty when the owner flips a default-off switch. A lane with no possible input is the "built and forgotten" failure mode CLAUDE.md §0 names as this project's #1 defect, so shipping it would add to the dormancy inventory rather than the working system. C's design and approval stand; its precondition is now honest and tested. Break/revert: allowing a default-off switch to be fed fails, and hiding the evidence debt behind a shorter blocker string fails. G — RESOLVED TO ITS SECOND BRANCH, WITH EVIDENCE. The gate was "codex/claude move presence -> validates via a non-billing round-trip, OR the limit is documented as permanent." The first is unreachable: * codex exposes no non-billing round-trip. Command set is exec/review/login/ logout/mcp/plugin/app-server/doctor/sandbox/debug/apply/resume; `login` has only `status`; `doctor` is local and 12.2s; `exec` BILLS. * claude's `auth status` returns real account identity (email, orgId, orgName, subscriptionType) -- strictly more than presence -- but a claude.ai OAuth token can carry those in its own claims, so the output cannot distinguish a local decode from a server call. Proving it requires presenting an INVALID credential to the owner's live seat, which is not a safe experiment. The original note said "do not upgrade this on assumption"; that still holds. Both limits are now recorded ON the probe spec, not in a comment, and a selftest requires every presence-only probe to carry a limit stating whether it is permanent or pending -- while forbidding a limit on a validating probe, which would be a contradiction in the record. Without this, codex/claude read as a perpetual unfinished upgrade, which is precisely why they were re-investigated between 2026-08-09 and today. cursor/gemini are pinned at `validates` so neither can be silently downgraded. Break/revert: adding a presence-only probe with no documented limit fails. Verified: 368 passed, 81 of 81 selftests. Three pytest failures and the two capability gates are NOT from this change -- they reproduce identically on pristine HEAD and come from `capability-propensity`, registered in the local ledger at 14:08 by a concurrent session without a caller, heartbeat or fixture. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d for want of evidence `unblock()` names the capabilities whose only blocker is missing evidence, but nothing acted on that list — the report was read by no one and the debt did not move. This adds the lane that acts on it. Bounded by construction: one feed per cycle, three items per capability, shadow unless ORCH_EVIDENCE_ACQUISITION is set. It never feeds a capability whose switch is default-off or whose blocker is a failure rather than absent evidence — a default-off switch cannot be fed, and feeding a failing one would train on noise. The empty states are distinct, because "nothing to feed" and "blocked from feeding" look identical in a count: `nothing_to_feed — feedable 0 / capped 1 / candidates 0 / fed 0` reports the blocking AND drainable quantities in one line, per the latched-gate rule. Declared as a cadence lane (`tick_phase`), not task-routed. The first registration used a task_type matcher, which made capability_activation_audit classify it task_routed and judge reachability on issue labels that can never invoke a cadence step — it reported caller_exists FAIL and was right to. A matcher describes how a capability is INVOKED, not the work it nominates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d as a tie `rejecting — 203 of 12805 events rejected as malformed` told an operator that something was wrong and nothing about what to fix; the causes were in a JSON nobody opens. The detail line now carries the reason codes it is counting. Reporting a single "top blocker" would have been worse than the count. On the live corpus FOUR codes each hit 203 of 203 events — missing_base_sha, missing_joined_attempt_id, unresolved_model_provenance, worker_attempt_not_resolved — so naming the alphabetically first one invites fixing it and expecting the queue to drain, when it would not move by a single event. A tie at the top is the whole story, so the line states how many independent fixes stand between here and one episode. Pinned by a test with a deliberate break->revert: dropping the reason counts at the call site (what the pre-fix code did) keeps `actionable: true` while losing the cause, and actionable-with-nothing-named is the silence this repo exists to prevent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… attempt can resolve
`execution_attempts.resolved_model` had NO writer outside the quarantined trial bridge. 1,374
attempts, 25 of them `worker`, none resolved — so `unresolved_model_provenance` and
`worker_attempt_not_resolved` were unavoidable on every research-claiming event in the system
(203 of 203 on 2026-08-22, and per-agent coverage `0 of 5 minable`).
Syncing the mirror would NOT have fixed this. `b1e8543` makes `offload` create worker attempts,
but the only thing that completes them is `reconcile`, which completed every seat UNRESOLVED
unconditionally: the sole resolved path was a `--resolved-model` CLI flag that no caller anywhere
passes. That was a latched gate — `resolved_model_not_reported_by_completion` could never be
false, because nothing reported.
The source is the one §2 already names: "a local Codex session rollout may establish CLI-reported
identity". Codex writes `turn_context.payload.model` into its own rollout log, Claude writes
`"model"` into its transcript, and `session_meta.payload.cwd` joins to the run — via the
`target = f"offload:{run_cwd}"` the dispatcher has ALWAYS written. 1,614 offload runs already
carry that key; nothing read it.
Verified live: codex resolves `gpt-5.6-terra` (cli 0.149.0-alpha.4.1), claude `claude-opus-5`.
End to end an attempt now goes `started/None/None` -> `complete/openai/gpt-5.6-terra`, and
`latest_worker_identity_for_agent('codex')` returns real provenance for the first time.
WHAT IT REFUSES TO DO. It never falls back to the requested model, the catalog, or a lane tag —
`model_identity()` is request-side and says so. Grepping run stdout for a model-shaped string was
tried and rejected: the offload logs are full of `gpt-4o-mini` and `CostModel` because the AGENT
WAS EDITING CODE ABOUT MODELS. A fabricated identity is worse than a skipped event. Seats whose
CLI leaves no per-session log (cursor, gemini, vibe, aider) stay unresolved with the seat NAMED,
so a permanent limit no longer reads the same as a missing file.
Also fixes a real hole found on the way: `validate_resolved_worker_model` only rejected `agent:`
tags, so it accepted `<synthetic>`, `unknown`, `none` and `default` — and Claude's transcripts do
emit `"model": "<synthetic>"`. A placeholder admitted as provenance makes an unresolved attempt
look resolved, inverting the one guarantee the table offers.
Pinned by three tests including a break->revert: disabling the reader puts the chain back to
unresolved with no exact-model claim available.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`record_domain_research` passes `base_sha=None` deliberately — a study of model pricing or a tier comparison is not cut from a commit — and `research_subjects` handles a null base_sha throughout: its identity hash maps it to "unknown" and its cooldown and backlog queries carry explicit null branches. The completion-event adapter required it unconditionally, which would have made the ENTIRE domain-research namespace permanently unminable — the largest volume of research the system captures, and the whole point of the domain namespace. THIS IS NOT A RELAXATION OF THE IDENTITY CONTRACT, and the test pins both halves so it cannot become one. The exemption is scoped to one closed namespace, and within it base_sha would be constant-null across every subject, so it distinguishes nothing — dropping an inapplicable component removes no discriminating power. A repo-scoped subject with no base_sha is still rejected, because there you genuinely could not say which code was studied. It also clears none of today's 203 rejections: every one of them is repo-scoped, so this is not a route to moving the accepted count. Those need the historical backfill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…o evaluations_v2 stayed empty Backlog item D was recorded as outstanding and was half done. `tick.research_tick` had already been switched to `prepare_arms`, but `exploration_backfill.schedule_backfill` still called `exp_abcd.prepare` with a plain agent list. `prepare` writes only `meta["agents"]`, so `experiment_members()` takes its legacy fallback, every member comes back `legacy=True`, `record_evaluation_v2` never fires, and the arm/member/profile identity §2 requires is replaced by an `agent_parent_projection`. All 21 on-disk manifests have `schema_version: None` and no `members[]`; 2,556 evaluations are projections. `research_v2_arms` moves from `tick` to `exp_abcd`, beside the arm/member normaliser it feeds, because two launchers need it and a second copy is how one drifts back to the legacy shape. `tick.research_v2_arms` stays as a re-export so its callers and selftest are unchanged. Also fixes a vacuous test double found on the way — the fourth in this family. `fake_prepare` accepted a plain agent list and kept passing after the launcher started sending v2 arms: a fake that accepts anything cannot tell `prepare` from `prepare_arms`, which is exactly the confusion that left the v2 table empty. It now asserts the arm shape, and reverting the launcher to `exp_abcd.prepare` fails the selftest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…related offloads
Backlog item B, capture half. A corpus review fans one scope out to several agents over many
partitions — structurally the same shape as a UX-review panel, which is already captured — but
nothing bound those offloads together. Each partition was an unrelated run against an ephemeral
temp path, so 1,617 offload runs across six agents produced nothing the learner could compare.
`run_plan` now registers the round as one research subject before its first offload and stamps
the round id onto every partition attempt. Identity is `domain/<review_id>` — a corpus review is
not cut from a commit, which is exactly the case the preceding base_sha change recognises — with
the spec pinned to `plan_sha256`, so the same question asked of a different corpus is a different
subject while a resumed run of the same plan is the same one.
`--round-agents` declares the seats that actually work the round; the default is this invocation's
own agent alone. It never pads: a forged arm set would make one agent's opinion look like a
panel's agreement, and the test pins that a one-arm round yields a different subject identity
than a three-arm one.
Capture is SUBORDINATE to the review. A Brain failure is reported in the run summary as
`research_round: {registered: false, reason: ...}` and never loses the review — but it is never
silent either, because an absence nobody notices is the defect this repo is named after.
WHAT THIS DOES NOT DO: durability inheritance. The existing `influence_edges` propagation already
carries durability once an edge exists, so no new machinery is needed — but the finding->issue
linkage that would create the edge is produced OUTSIDE this repo (audit findings become issues via
the Workflows lane), and this tool cannot observe it. Capture without a fabricated label.
The test exists because every existing offload double takes `**kwargs` and would have swallowed
`research_round` silently — a binding that never happened would look identical to one that did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s misdeclared `subjectless_producer_carries_experiment` fired on ANY experiment_id, so a producer declared subjectless could never demonstrate otherwise: it was rejected as malformed until a human edited a Python dict, and the only signal that would prompt that edit was the rejection itself. A gate whose drain is blocked by the thing it measures. It is not hypothetical. `roles` was declared subjectless on the grounds that it has "no delivering arm", and that is empirically false for the `redirect` role — measured today: 8 target/role cells ran it across 2-3 agents each, acceptance is tracked on `influence_edges` (14 accepted / 211 counterfactual), and durability already propagates over those edges. The declaration's stated reason was wrong, and the gate would have kept rejecting the evidence that says so. The drain that works while the gate is shut is a SELF-CONSISTENT subject identity: an event whose supplied subject_id equals the id derived from its own declared target/spec/arms went through the registered-round path, so it is a declared identity rather than an undeclared one. A forged or borrowed experiment_id cannot satisfy it, and the test pins both halves — a coherent identity graduates, a borrowed one is still named. `roles` keeps its subjectless declaration, but for the true reason: the 1,397 events on record present no design set at all and would become rejections rather than episodes. Not incapable — and the entry now says so instead of asserting something measurably false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g a commit they never had Backlog item E, applied. DEDUP FIRST, because the same rule was broken earlier today: 24 of the 26 on-disk panels were ALREADY registered. One was genuinely new (`Trend_Model_Project_uxreview_2026-08-11`) and one must never be registered at all. Three correctness findings, each of which would have corrupted the subject population: 1. `_gate2_smoke` is a harness fixture — a 2.5KB rubric against a real panel's 8.7KB — that exists to prove the pipeline runs. Registering it would put a rehearsal into the population the miner learns from, and the learner cannot tell a rehearsal from a review. Excluded by name. 2. The new panel's arms read `['claude','codex','cursor','vibe','vibe-retry1', 'vibe.FAILED-http520']`. Those last three are ONE agent — an attempt, a retry, and a failure. Taken literally that is a forged arm set: it manufactures independence the evidence does not have and trebles that seat's weight in every comparison drawn from the subject. A `.FAILED-*` attempt produced no usable output and is not an arm at all. 3. `resolve_panel_base_sha` falls back to `git rev-parse HEAD`, which is right for a panel running NOW and catastrophic for a backfill — it would stamp August's commit onto a June review and fuse two states of the same app into one subject, the exact failure its own docstring warns about. Historical callers now declare `base_sha_unrecoverable` and get None. WHAT THIS DOES NOT ACHIEVE, stated plainly: 25 registered, 0 with a base commit. The panels are now identified and retrievable — which was half of E's purpose — but they are NOT minable, because their base commit was never captured and inventing one is worse than a skipped event. Verified that the 25 do not collide (25 panels -> 25 distinct subject ids): each rubric embeds that review's own captured evidence, so identity distinguishes them without the commit. That is NOT a licence to exempt ux_review from base_sha the way domain research is exempt — there a commit is inapplicable, here it merely went unrecorded, and relaxing it would be moving the accepted count by weakening the contract. Future panels resolve it live. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s findings produced Backlog item B, durability half. Everything downstream already existed and is untouched: an issue becomes a PR, a merged PR gets a durability label from `durability_sweep`, and `influence_edges` already back-propagates that label over accepted edges. The missing fact was "round R, arm A produced issue N". That fact CANNOT be inferred, which is why this is a recorder and not a parser. The only written trace of the link is a free-form prose line — `_Surfaced by the maint-69 outage investigated in #3007._` — and turning a sentence into a causal edge would credit or blame an agent on the strength of grammar. A wrong attribution trains the learner on a fiction, which is worse than no edge at all. So the filer records it at filing time, and `file-agent-issue` now does (one command, skipped entirely when the finding was not from a registered round; it is told never to invent a round id if the round is unregistered). WHY THE LABEL IS UN-GAMEABLE, which is the whole reason B was worth doing: the agent that found the defect decides neither of the two things that score it. Whether it gets filed is the filer's prove-before-file check; whether the fix HOLDS is decided later by real work landing on top of it. An audit finding that was wrong gets falsified by the codebase itself — a better terminal signal than production delivery, whose output contract is constant. Uses `influence_type='experiment'` — a round already has an exp_id, so no new type and no second durability path beside `influence_edges`. `apply_edges` is off by default: reading is safe, writing is a change. Resolved and unresolved counts are reported together, because "3 durable" alone reads as a verdict on the round when forty findings simply have not landed yet. Found by the test: the target join was case-sensitive — `canonical_target` lowercases the recorded issue while `runs.target` keeps the repo's real casing, so every finding silently looked unlanded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ile, and a profile is not a claim Syncing the mirror proved the resolution path was still dead, and for two more reasons. Both are now fixed and codex resolves 37 of 37. 1. THE LEDGER NEVER CARRIED A PROFILE. `reconcile`'s completion branch built `profile_ids` solely from capacity-ledger `start` rows, whose keys are (agent, cost_usd, count, event, log_file, mode, model, run_id, started_ts, target, task_type, ts) — no `selected_profile_id`, ever. So the branch that resolves an attempt could not execute: 250 marker backfills, 0 resolved, 0 unresolved, every run. `execution_attempts` already knows, because `offload` wrote the profile onto the attempt row, and reading it there works RETROACTIVELY where a new ledger field could not. 2. AN ATTEMPT IS VISITED ONCE. Every attempt recorded before the CLI reader existed was already completed `unresolved`, and nothing revisits it — so 53 attempts were stranded with `resolved_model_not_reported_by_completion` while their CLI logs sat on disk still naming what served them. `resolve-unresolved` sweeps them, dry-run by default. Applied: 37 codex resolved to `openai/gpt-5.6-terra`, 16 cursor blocked with `no_cli_session_log` NAMED. Per-agent coverage moved `0 of 5 minable` -> `1 of 5`, codex `minable`. 3. A REGRESSION MY OWN FIX CAUSED, found by measuring after the sync rather than assuming. `presents_identity` counted `subject_profiles`, which was safe only while profiles appeared exclusively on research runs. Once provenance resolved on ordinary offloads, 111 production events with no subject, no arms and no experiment flipped from correctly EXCLUDED to reported as malformed — for naming the model that served them. And `selected_profile_not_in_subject_set` fired on events with no subject at all, where a profile cannot be in a set that does not exist; because that code is not one of NO_SUBJECT_REASONS it dragged all 111 into the rejection count. Both are now judged only where a subject actually exists, the same rule `repository_target_mismatch` already follows. Rejections 279 -> 168, and the 168 that remain are real. The profile-set check stays a real check where a subject IS present, pinned. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… 1.0 — the actual root cause `offload` recorded `selected_profile_id=profile_id` — the caller's ARGUMENT, which is None whenever the profile was chosen internally, i.e. always, because no caller passes one. So every profiled offload wrote a ledger start row and routing metadata saying `selected_profile_id: null` next to `requested_model: gpt-5.6-luna`, `profile_policy_version`, and `propensity: 1.0` — a decision record with no subject. The worker attempt row, three lines below, always used `profile["profile_id"]`. The run disagreed with itself. THIS IS WHY THE RESOLUTION BRANCH WAS DEAD. `reconcile` builds its `profile_ids` set from that ledger field, so it was always empty: 250 marker backfills, 0 resolved, 0 unresolved, every run. The `_profile_id_from_attempt` fallback added earlier today was a workaround for this bug; it stays as a backstop for rows already written, but the primary path now works. Also explains the coverage split measured after the sync, which is NOT what I reported last turn: - claude's offloads were never failing to select a profile. Claude has 6 offload runs total and its last was 07-12 — it simply has not been offloaded to in six weeks. Corrected. - gemini (10 post-wiring runs), vibe (1) and aider got no profile because the RUNNING MIRROR knew only codex profiles until it was synced minutes ago — verified against the pre-sync mirror copy, whose PROFILE_REGISTRY contains codex ids and nothing else. Not a bug, a stale exec tree. - Those seats will now record worker attempts and stay honestly UNRESOLVED (`no_cli_session_log`), so the realistic ceiling is 2 of 5 seats minable, not 5. Pinned by extending the existing behavioural assertion to require that the ledger row, the routing metadata and the worker attempt all name the SAME profile. Reverting either line reproduces the production symptom exactly — `selected_profile_id: None` beside `requested_model: gpt-5.6-luna`. The ledger half reads through `ledger_reconcile._read_ledger`, not a `hasattr` fallback, because a guard there would have passed when the field was absent — the vacuous shape this file has already been burned by three times. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r held 22 of what it denied `mining_coverage` reported cursor as `no_worker_attempt` while the same row showed `worker_attempts: 22`. That sends a reader hunting a dispatch bug when the real answer is that the seat's CLI leaves no per-session log to read a model from — a permanent, named limit. A wrong verdict is worse than a missing one: it is confidently actionable in the wrong direction, and this is the surface built specifically to answer "how will I know if the miner only works on a subset". Adds `attempts_unresolved` for the real state — attempts exist, none carry a resolved model — and asserts across every row that a verdict agrees with its own counts. The first version of that assertion was VACUOUS and the break-revert caught it: it read whatever the ambient DB happened to hold, which under the selftest's isolated runtime is nothing, so it passed with the fix removed. It now builds the exact state in a temp DB, and reverting the branch fails with `verdict: no_worker_attempt` beside `worker_attempts: 1`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…er substantiate
A worker attempt exists to carry model provenance. Wiring profile selection for every seat made
cursor, gemini, vibe and aider write one on every offload — and none of them keeps a per-session
model log, so each row completed `unresolved` and could never become anything else. Cursor produced
23 in five hours; at ~230 offloads every two days it never stops.
Four defects, all mine, all from today:
1. UNBOUNDED, UNDRAINABLE ROWS. Nothing decrements a permanently-unresolvable attempt — the only
thing that would is the capability the seat lacks. `resolve-unresolved` swept them, so 23 of 23
candidates were rows it could never resolve, re-probing the filesystem for each on every run.
Now: the seat's capability is a single static property (`adapters.can_report_cli_identity`), no
attempt is written when it is False, and the sweep counts only drainable rows — it reports
`candidates: 0` with `excluded_unreportable: {cursor: 23}` beside it.
2. THE PROFILE STILL APPLIES. Withholding the attempt costs nothing auditable: the profile is what
puts the bare vendor id on the command line, and the ledger row and routing metadata still name
it. Only the claim we cannot back is withheld — and the reason is one static string per seat, not
the same prose re-cached on thousands of rows.
3. A METRIC THAT CONTRADICTED ITSELF. `complete_profile_attempt` never cleared `fallback_reason`, and
`resolved_model_coverage` counts that field — so codex-5.6-terra-high reported coverage 1.00 AND
fallback_rate 1.00 simultaneously. fallback_rate is how a profile's health is read. Cleared on
resolve, plus a one-time repair of the 37 rows already contradicting themselves; codex now reads
1.00 / 0.00.
4. A GREEN FIELD THAT HID THE LIMIT. `model_reportable` only asked whether the profile's
`requested_model` looked like an adapter tag, so giving every seat a bare vendor id flipped all
six to True while four still had nothing to read a model FROM. I had made a metric pass by
renaming its input — and a selftest ASSERTED that wrong state for all six seats, which is what
licensed this whole class of row. Reportability is now the reader, and the headline states the
achievable ceiling: `1 of 2 reportable seats minable (3 of 5 working seats can never report:
cursor, gemini, vibe)` instead of `1 of 5`, which invented a four-seat backlog out of a physical
limit.
The dispatch path is gated identically. It is latent today (0 worker attempts on non-offload runs)
but gemini, cursor and vibe all dispatch through it, so the first caller to assign them a profile
would reproduce this exactly; fixing one path and leaving its twin is how the defect returns.
The 23 rows already written STAY: 23 append-only completion events reference them, and orphaning
those to tidy a metric would trade a bounded artifact for a broken invariant. Growth is stopped,
they are excluded from the sweep, and every surface now names them as permanent rather than pending.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ILITY, not by the seat alone
Last commit gated the worker attempt on `can_report_cli_identity` alone. That stopped the junk but
silenced the case the row exists for: a gemini- or cursor-armed audit round could no longer record
the attempt its episode joins on, which broke the round-registration path built for it hours
earlier. Preventing the bad outcome is not the goal; the goal is that a round becomes minable
evidence.
Two independent reasons justify the row, and either is sufficient:
* the run is EVIDENCE — bound to a registered research round, where the attempt is the join that
lets the round become an episode. True for any seat, because a round's arms are whichever
agents actually did the work.
* the seat can RESOLVE — its CLI records what served the run, so the row carries real model
provenance and feeds drift detection. True for any run, bound or not.
Only unbound-and-unreportable writes nothing, and that is the only cell where the row could never
be used for anything — 472 production offloads a week, versus a handful per deliberate round.
Verified end to end through the REAL path, not a fixture: a round-bound codex offload now exports a
completion event the adapter ACCEPTS — `attempt_resolution: resolved`, `profile_id:
codex-5.6-terra-high`, `resolved_model: gpt-5.6-terra`, `subject_id` from the round, and `base_sha:
None` correctly exempt because a corpus review is not cut from a commit. Capability discovery is
live for codex- and claude-armed rounds.
All four cells are pinned, and both wrong predicates now fail the selftest: gating on `can_report`
alone loses the round-bound case, gating on nothing reproduces the 23 dead rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…per-session model record
I wrote `cursor-agent writes no per-session model log under ~/.cursor` after an eight-line
`find | head`. That was inferring a blocker instead of verifying it, and it was wrong for three of
the four seats I declared:
cursor ~/.cursor/chats/<h>/<session>/store.db -> providerOptions.cursor.modelName
joined by the sibling meta.json's `cwd` -- the PROVIDER's own record, not our --model
vibe ~/.vibe/logs/session/<id>/meta.json -> config.active_model
joined by environment.working_directory, with start_time/end_time
gemini conversation_summaries.db -> workspace_uris + conversation_id -> brain/<id>/.../transcript
Verified live: cursor resolves `composer-2.5`, vibe resolves `mistral-medium-3.5`.
GEMINI IS THE ONE REAL LIMIT, and it is verified absent rather than assumed: agy's index does give
a workspace join, but every file of two Orchestrator conversations records no model at all. It is
also the seat where this matters most, because agy is a multi-provider ROUTER -- one real
conversation was served by `claude-sonnet-4-6` while our profile requests `gemini-3.1-pro-high`.
The requested model genuinely is not what ran, which is why recording the request would fabricate
the attribution rather than approximate it. That retires the `profile_attributed` tier I proposed
last turn: it would have credited gemini for claude's work.
RESOLUTION NOW HAPPENS AT COMPLETION, not in a later sweep. `offload` closed every attempt
`unresolved` because the CLI's stdout carries no model -- true, but the CLI has just written its own
session record, and completion is the moment we know the agent, the workspace and the window.
Resolving post-hoc made the sweep the mechanism instead of a backstop and left rows stranded
whenever it did not run. It still never falls back to the requested model.
A vendor-family allowlist (`VENDOR_MODEL_RE`) replaces the earlier reject-list, because that was
the real reason log parsing looked unusable: offload logs are full of `gpt-4o-mini` and
`claude-fleet-list.sh` from agents editing code about models. `composer-2.5` matches;
`claude-fleet-list.sh` does not.
Two existing tests asserted the wrong state and are corrected -- they had pinned cursor and vibe as
permanently incapable, which is how a false declaration becomes enforced.
FOUND, NOT FIXED: cursor's 24 attempts still do not resolve, and the reason is dispatch
configuration rather than tooling. Every codex offload ran in a UNIQUE workspace
(`~/.codex/orchestrator/offloads/<stamp>-<name>-<pid>`) so its session joins exactly; all 24 cursor
offloads ran in the same reused `/private/tmp`, so no session can be attributed to a specific run.
A per-run workspace would make cursor join like codex does.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eport Third test this session to pin a wrong belief about a seat. It asserted cursor is `model_not_reportable`, which stopped being true the moment cursor's own chat store was read -- so a false declaration was being enforced by a test rather than caught by one. Retargeted to gemini, the verified case: its index gives a workspace join but its conversations record no served model anywhere. Caught by verify.py's selftest COUNT dropping 82 -> 81, not by the failure itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…m stdout, not a store scrape
RESEARCH FIRST. The industry answer to "which model served this?" is a standard, and the
Orchestrator's schema already mirrors it: OpenTelemetry's GenAI semantic conventions define
`gen_ai.request.model` (what was asked for) against `gen_ai.response.model` (what actually ran) —
exactly the `requested_model` / `resolved_model` split in `execution_attempts`, which simply had no
writer for the response side. The convention's premise is that the PROVIDER echoes the served model
in its response, and the CLI form of that is the `system/init` event of `--output-format stream-json`.
Every seat's CLI does it, and I had been scraping session stores instead:
cursor `{"type":"system","subtype":"init","cwd":...,"session_id":...,"model":"Composer 2.5"}`
agy `--log-file` + `model_config_manager.go: Propagating selected model override to backend:
label="Gemini 3.7 Flash (High)"`, with `agy models` giving the label -> id map
Cursor now resolves from its own stdout: exact, per-run, no workspace to match and no time window
to guess. That is what makes it resolvable AT ALL — every one of its 24 real offloads ran in the
same reused `/private/tmp`, so no session in cursor's chat store was attributable to any single run.
The store readers stay as a fallback for seats and runs that predate this.
Verified end to end through the real dispatcher: a live cursor offload records
`resolved_provider=cursor, resolved_model=composer-2.5, status=complete, fallback_reason=NULL`, and
the caller still receives plain `ok` — the stream is reduced to its final result text before anyone
sees it, so every existing consumer keeps the contract it already had. Scoped to
`transport="offload"`; the long-running dispatch path's output is parsed elsewhere and is untouched.
TWO BUGS FOUND BY RUNNING IT, not by reading it:
1. `nonlocal observed_model`. The parse happens inside `_run_offload_attempt`, so the assignment
made a local and the completion in the enclosing scope never saw it: the reader found
`composer-2.5` and the attempt still closed `unresolved`. The reader working while the row denies
it is the most deceptive failure available here.
2. A vacuous half in my own test. The double supplies stream-json stdout whatever was requested, so
reverting the transport to `text` left the test green while nothing would ever be reported in
production. Now the real argv is asserted, and both breaks fail.
ALSO FOUND: agy is serving `Gemini 3.7 Flash (High)` = `gemini-3.7-flash-high`, while our profile
requests `gemini-3.1-pro-high`. Live model drift on that seat, invisible until the response side was
read. Its resolution needs the `--log-file` path threaded through the offload; not in this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…emoved the choice TWO CORRECTIONS, both mine. 1. THERE IS NO DRIFT. I reported agy "serving Gemini 3.7 Flash (High) while our profile requests gemini-3.1-pro-high" as live drift, in a commit message and twice in the backlog. Wrong: that probe passed no `--model`, so agy used its own session default. Tested properly — `--model gemini-3.1-pro-high` propagates `label="Gemini 3.1 Pro (High)"`. agy honours the request exactly. 2. GEMINI HAS TWO LINES AND THEY ARE NOT ONE VERSION LADDER. Flash (3.7/3.6/3.5, each with high/medium/low effort) is the fast mid tier; Pro (3.1) is the higher-end reasoning tier. So `3.7 > 3.1` is newer FLASH, not better than PRO. Reading those numbers as a single sequence is the trap, and both lines are legitimately used depending on the task. THE REGRESSION THAT FOLLOWED. Registering one top-tier profile per seat made `_select_offload_profile` pin every gemini offload to Pro — silently overriding `DEFAULT_OFFLOAD_TIER = "mid"` and the comment beside it, which had ALREADY diagnosed this exact waste: "a codex offload burned Sol and a gemini offload burned Pro. The mid tier is the right home for that work." A profile pinning one rung of a three-rung ladder removes the choice the ladder exists to make, and gemini runs ~900 offloads. Fixed by registering the mid-tier profile (`gemini-3.6-flash-high`) and making offload selection take the rung the ladder names for `DEFAULT_OFFLOAD_TIER`: codex luna (cheap) -> terra (mid) claude sonnet-5 -> sonnet-5 (already mid) gemini pro (FULL) -> flash-high (mid) cursor/vibe/aider -> unchanged; single-lane seats have no ladder and keep their one profile **Pro stays registered.** The fix is choosing per task, not deleting the expensive rung — full-tier work can still ask for it, which is the whole point of having two lines. Both halves break-tested: ignoring the tier reselects codex luna, and removing the mid-tier profile reselects gemini Pro. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…unrepairable history are named Four items, in turn. 1. AGY RESOLVES (the last unreportable seat). agy's structured output carries no model, but its CLI log does — `Propagating selected model override to backend: label="..."` — and `--log-file` lets the dispatcher give each run its own log, so the join is direct. It already passed `--log-file`, but to ONE SHARED path for every gemini run: a line in a shared file belongs to no particular run, the same non-attributability that made cursor's reused `/private/tmp` useless. Verified live end to end: a gemini offload records `profile_id=gemini-3.6-flash-high, resolved_model=gemini-3.6-flash-high, status=complete, fallback_reason=NULL`. gemini moves out of NO_SESSION_LOG_AGENTS. A subtlety worth the comment it now carries: the argv rewrite has to happen BEFORE `wrapped` is composed, because that freezes argv into a shell string. The first version sat below it and was inert — the run still wrote to the shared log and the only symptom was a model that never resolved. 2. STALE RUNNERS, in the existing sweep. `orch-sync-mirror.sh` is treated as the deploy step, but a process that already imported the code keeps running the old copy for as long as it lives — observed: a cursor offload one minute after a sync still used the pre-sync dispatcher because four `mcp_server.py` processes held their own copies. `switch_review` now reports processes started before the current mirror was written (live: 3 of them, oldest 8.0h). FYI only; it never kills anything, because a live process may be serving a session. 3. AIDER, probed rather than shrugged at. Its `--analytics-log` records launched/repo/exit with NO model, and its stdout echoes `Model: mistral/codestral-latest` — the requested alias, which FLOATS, so writing it as resolved identity is exactly the `--model` copy §2 forbids. The declaration now carries that finding and names what would settle it (`--llm-history-file`, which needs a paid call on a seat with zero runs this week). 4. THE HISTORICAL REJECTIONS ARE HISTORY, and the health line now says so. Determined: 0 of 25 UX-panel subjects ever recorded a base commit, 0 execution attempts exist for those runs, all dated before provenance existed. `missing_joined_attempt_id` means there is no attempt row to join and the export builds the join FROM that row, so nothing downstream can ever supply it. `184 malformed` read as work somebody should do; it now reads `184 have no attempt row to join and can never be repaired, 0 drainable`. Both halves pinned — a rejection that DOES have an attempt row stays drainable, so this cannot become a way to make real defects vanish. FIFTH TIME PATTERN, now fixed structurally: four separate assertions hardcoded a seat as unable to report, and each went stale as soon as that seat's store or log was actually read. They now derive the seat from `NO_SESSION_LOG_AGENTS` and fail loudly if no seat lacks a reader, so the test can no longer enforce a belief the code has outgrown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 16 minutes Limit details: You’ve used the included review currently available. Your 70 included PR review attempts over the past 7 days set your current allowance at 1 review per hour. Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab. How can I continue?Wait for the limit to reset, then comment An organization admin can change what happens after included review limits in Billing. How do review limits work?CodeRabbit enforces per-developer PR review limits within each organization. For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThis PR adds CLI-reported model identity resolution across adapters and ledger reconciliation, extends capacity/execution profiles for multiple agents, adds research-round registration and durability tracking, adds mining-coverage reporting, adds ux_review historical panel backfill and switch_review stale-runner detection, and applies formatting-only refactors across most other automation modules and tests. ChangesModel Identity, Capability, and Research Tooling Update
Estimated code review effort: 4 (Complex) | ~75 minutes Merge Risk: 🟠 High · up to This PR changes how agent identity, provenance, and experiment evidence are recorded, but the current head can still associate results with the wrong model or agent, mutate active attempts, and omit v2 evaluations. That can corrupt comparative evidence and downstream attribution, so the PR is not merge-ready until these correctness issues are fixed or explicitly accepted. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
The dispatcher selftest passed on the owner's machine and failed on any bare runner, including
CI. Cause: `adapters.advertised_models` shells out to `agy models` when its disk cache
(`agent-runtime/gemini/advertised-models.json`) is cold. That cache is warm here and cold
everywhere else, so only elsewhere did the probe land INSIDE the monkeypatched `subprocess.run`
and overwrite `captured["cmd"]`. The per-run-log assertion then compared against
`['agy', 'models']` and failed:
AssertionError: agy argv lost its per-run log: type=list len=2 head="['agy', 'models']"
CLAUDE.md names this exact case — a monkeypatched call catching a model-catalog probe — and says
the fix is ISOLATION, not a skip, because isolation makes CI run MORE. The double now records
only the run under test (offload always shells through `bash -lc <wrapped>`); an incidental probe
returns an empty `ProbeCompleted` and changes nothing.
Also makes the assertion self-describing. The bare `captured["cmd"]` dump was truncated out of
the CI log, so the failure could not be attributed from the log alone; naming type/len/head
identified the cause on the first run. Its paired assertion reuses the same normalised join, so
neither depends on `cmd` being a list.
Verified in a runner sandbox built to the five documented differences, which reproduces CI
exactly (361 passed, 26 skipped, 387 collected): fails before this change with the probe command,
passes after. Selftests there go 75 -> 76 modules — dispatcher now speaks instead of failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…merge result Raising a ceiling means agreeing one more thing may go unchecked, so this names which and why. Both new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling — the preferred way to lower one: - agy advertised-models cache absent: comparing declared model ids against the catalogue agy actually advertises needs agy's catalogue. A fixture would exercise the comparison while detecting no real drift. - vibe config absent (~/.vibe/config.toml): active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was refused: dispatcher's per-run agy-log assertion was a stub leak, so the previous commit isolated the double instead. Never turn a real failure into a skip. Floor measured on the MERGE RESULT, per this file's own rule, in a sandbox that reproduces CI exactly — not on the branch, which is the mistake that once left the floor 8 tests below reality. Owner's machine: 387 passed, 0 skipped, 5/5 gates. Bare runner: 361 passed, 26 skipped, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
temporarily deployed
to
agent-standard
August 23, 2026 05:29 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
# Conflicts: # .verify-floor.json
stranske
had a problem deploying
to
agent-standard
August 23, 2026 06:03 — with
GitHub Actions
Failure
GitHub Actions
Failure
stranske
temporarily deployed
to
agent-standard
August 23, 2026 06:03 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-standard
August 23, 2026 06:54 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-standard
August 23, 2026 06:54 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Owner
Author
|
Added 🤖 Addressed by Claude Code |
Owner
Author
|
@coderabbitai — answering the repeated offer once, for all four deferred threads (ledger_reconcile, research_subjects x2, exp_abcd, and the coupled adapters pair): please do not open issues assigned to the owner. Not a rejection of the findings — I agree with all of them and recorded the reasoning on each thread. It is a constraint from this repo's own rules. CLAUDE.md §3 forbids adding any step that can accumulate a backlog for the owner, who is one person working in spare time. Issues auto-assigned to them are precisely that accumulating queue, and the volume here makes the point: six open threads from one review. There is also a documented pipeline for this. New work originates from the repo-review → approved-issue-queue → opener lane, which materialises queue items as issues and draft PRs under its own active-PR cap. Ad-hoc bot-filed issues bypass that cap and the prioritisation behind it. If issues are wanted later they should be unassigned, so the opener/closer lanes can pick them up under the normal cap rather than landing in a personal queue. For now all four are tracked outside GitHub with the dependency ordering preserved — notably that 3837879023 (label→ID resolution) must land before 3837879025 (gemini per-run provenance), or the second persists a slug guess as provider-resolved identity. Please treat the deferred threads as tracked and stop re-offering. 🤖 Addressed by Claude Code |
|
Tip For best results, initiate chat on the files or code changes.
If an issue is needed later, it must be unassigned and use the documented opener-lane workflow. The dependency order remains 3837879023 before 3837879025. 🐇 ✏️ Learnings added
You are interacting with an AI system. |
Owner
Author
Finding 3 —
|
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… then make gemini use it Two coupled CodeRabbit findings on PR #42, in the order that keeps provenance honest: the resolver first, the caller second. STEP 1 — `model_id_for_label` never consulted a catalog (comment 3837879023). It built its lookup as `parse_model_catalog_pairs(f"{mid}\t{mid}" ...)` over `advertised_models()`, which returns only ids — so the map was id->id and `catalog.get(text.lower())`, handed a human LABEL, could never match a key. The catalog branch was dead for every real label, the docstring's "prefers the CLI's OWN catalog" was inoperative, and every call fell through to the slug heuristic. Measured against the live `cursor-agent --list-models` (2026-08-23), 4 of 5 real labels resolved wrongly — and the harm is worse than the review predicted: Codex 5.3 High -> None (lost) Claude Fable 5 1M Thinking (NO ZDR) -> None (lost) Claude Opus 5 1M Thinking -> claude-opus-5-1m-thinking (FABRICATED) GPT-5.6 Sol 1M High -> gpt-5.6-sol-1m-high (FABRICATED) Those last two are vendor-shaped ids that do not exist, and `dispatcher` already writes this result into `execution_attempts.resolved_model` for cursor stream labels — a fabricated provider-resolved identity, which CLAUDE.md §2 forbids. (`Composer Pro 2.5` -> `composer-pro-2.5`, the case the review cited, is in fact refused by `VENDOR_MODEL_RE`; the real defect is the two fabrications above.) DEDUP (CLAUDE.md §0): `parse_model_catalog_pairs` already existed and is exactly the right parser — nothing new was built. Grepped `catalog_pairs|label_to_id| raw_catalog|catalog_text|model_labels`: no other label->id retention anywhere. The fix is to feed the existing parser the CLI's raw output instead of synthetic `id\tid` text. * one probe, one cache, two projections (`_advertised_catalog`), so "which ids exist" and "which label means which id" cannot disagree; * `advertised_models` keeps its exact contract ([] = UNKNOWN) and stays the monkeypatch seam `test_model_tier_resolution` already patches; * CACHE MIGRATION IS EXPLICIT, since this sits on the provenance path: the blob gains `pairs` beside the unchanged `models`, so every existing reader is unaffected; a pre-migration blob (no `pairs` KEY — presence, not truthiness) still serves id requests and is a MISS for label requests, re-probing and rewriting itself. Self-heals in one TTL per agent; until then the resolver degrades to the slug it already used, never to a wrong id; * precedence: catalog pair -> label-is-an-advertised-id -> REFUSE when the catalog was readable and lists neither -> slug only while UNKNOWN. Catalog-sourced ids are validated by `_catalog_model_id`, not `VENDOR_MODEL_RE`: that regex rejects 42 of cursor's 204 REAL ids (every `claude-fable-*`, `cursor-grok-*`, `kimi-*`, `glm-*`, plus version-first `claude-4.6-opus-*`), so using it as the catalog's validator would trade fabricated ids for lost ones and re-break on the next vendor family. `auto` was its one correct rejection, and it is nameable — hence `CATALOG_ROUTING_TAGS`. STEP 2 — the gemini comment described a path the code did not take (3837879025). `cli_reported_model` mapped `gemini` to `_agy_model_for` alone and never called `model_label_from_agy_log`, while the comment above `NO_SESSION_LOG_AGENTS` said the per-run log was primary. fe59bc7 settles the direction, so the code moved to match the comment: per-run `--log-file` -> `model_label_from_agy_log` -> `model_id_for_label` -> `_agy_model_for` only as fallback. `log_file` is threaded from `dispatcher.offload` and both `ledger_reconcile` call sites; the reason now names what was searched (`no_gemini_model_in_run_log_or_session_store` vs the unchanged `no_gemini_session_matched_workspace`). Step 2 depended on step 1: routing gemini through a resolver that returned a slug guess would have persisted a heuristic as provider-resolved identity. `adapters.AGY_LOG_SUFFIX` / `agy_log_for()` are now the ONE name for that log, consumed by the writer (dispatcher) and the reader — two literals would drift and the symptom of drift is a model that silently never resolves. TESTS — one per step, each with a deliberate-break->revert demonstration: * step 1 in `adapters --selftest`, driving the real `subprocess.run` seam with a verbatim catalog. The first version seeded only the memo and stayed GREEN when the pre-fix `id\tid` construction was restored — a real coverage hole in the exact line the finding is about; the probe-path case closes it and now fails on that break; * step 2 in `test_feedback_model_provenance` — log-beats-store, store-still-a- fallback, both-silent reasons; reverting the gemini branch to store-only makes it red. Both tests seed `_ADVERTISED_MEMO` rather than letting the catalog probe shell out, per the `agy models` stub leak this branch already fixed twice. FLOOR 387 -> 388, measured on the merge result (main added no test changes since the merge-base). No ceiling moved; nothing new is skipped. verify.py: 388 passed, 0 failed, 0 skipped, 83/83 selftests, 5/5 gates — green in the worktree and from a mirror-shaped copy on local disk. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
CLAUDE.md depended on IMPROVEMENT_BACKLOG.md in two places and both were
unfollowable by exactly the workers they bind. §0 step 3 ("check the improvement
log; many ideas are already DONE") is part of the MANDATORY dedup-before-develop
check, and §5 says append a status note there. The file was gitignored, and a
gitignored file does not exist in a git WORKTREE — it was verified absent from
every worktree on this machine. Agents work in worktrees. Three agents in one day
could not do step 3; two said so and fell back to ledger notes and docstrings.
481 KB of accumulated "already DONE" knowledge — the project's stated
WHICH STATE VARIABLE: ORCH_LOCAL_RUNTIME, not ORCH_STATE_DIR. CLAUDE.md §1
already partitions them — ORCH_STATE_DIR holds the audit cache, firing monitor
and redirect-sweep state, all derived and self-regenerating, while
ORCH_LOCAL_RUNTIME holds the capability LEDGER and the Brain: durable,
irreplaceable instance evidence. The improvement log is the second kind. It
cannot be regenerated from anything, and its §0 role is the prose twin of the
ledger's `notes` field (both answer "is this already DONE?"), so it belongs
beside the ledger rather than beside the caches. A useful consequence: ci.yml
already points ORCH_LOCAL_RUNTIME at an empty temp dir, so the named-absence
path is exercised on every runner by construction.
THE MOVE was copy -> verify -> remove, in that order: 482,865 bytes, sha256
f7eea973...d16d identical at source and destination before anything was deleted.
There is no git undo for unversioned history.
THE ACCESSOR is improvement_log.py (backlog.py is taken by the unrelated fleet
work-discovery lane and is not touched):
search <term> — §0 step 3 in one command. Prints each hit under the item
heading that owns it, plus the lines and sections read,
so "no matching items" describes a file actually read.
append <ref> <note> — §5 in one command. Places a dated note INSIDE the
matched item, atomically, keeping one rolling backup;
REFUSES on an ambiguous or unknown ref rather than
guessing, because a note filed against the wrong item
corrupts the record it exists to improve.
path — where it resolved to, and whether it is here.
A NAMED ABSENCE, NEVER A SILENT ONE. On a fresh clone, a runner, or a second
instance there is no log; every command then names what is missing, where it
would be, and both env vars that control it, and exits 2. Honest-empty is exit 1
and says so in words. A reason-less empty result is indistinguishable from "no
matches", which is this repo's founding defect.
THE POINTER is what actually fixes the invisibility: nothing in a worktree even
hinted the log existed. IMPROVEMENT_BACKLOG.md is now TRACKED and holds a short
pointer, so all thirteen citations of that filename still resolve and `ls` still
shows it. It is removed from .gitignore deliberately: left ignored, the 481 KB
file could reappear at that path silently — tracked, it announces itself in
`git status`. The real log lives outside the repository entirely, so `git add`
on it fails with "outside repository", which is stronger than an ignore rule.
test_improvement_log.py fails if the pointer grows into a log.
capability_admission.SKIP_NAMES drops its IMPROVEMENT_BACKLOG.md entry: the
allowlist reason ("the backlog narrates history and cites closed records") no
longer describes anything in the tree, so the gate now scans one more file.
DEDUP FINDING (CLAUDE.md §0), recorded before writing code and again in the
module docstring since plans are not durable: grepped by concept, not name.
`git grep IMPROVEMENT_BACKLOG` returns 13 references, every one prose or a
comment — no reader, no writer, no accessor. `improvement_backlog|
improvement-backlog|backlog_notes` over *.py returns nothing. The only machinery
touching the filename was capability_admission.SKIP_NAMES, an exclusion.
Genuinely absent, so new. NO LEDGER ROW, deliberately: this is documentation
access with no dispatch path, outcome or learning sink, so the admission gate
does not bind on it — the same reasoning env_prereq.py records for itself — and
registering one would turn every sibling worktree's verify.py red for a module
they cannot see (§1, ledger shared per machine / code per worktree).
Not an ARCHITECTURE.md change: this is repo tooling like verify.py and
env_prereq.py, not a loop stage, rail, role, feedback surface or registry entry.
VERIFIED on the MERGE RESULT, after rebasing onto af6654d (PR #42, 143 files):
391 passed, 0 failed, 0 skipped, 84/84 selftests, 43/43 can-fire, 5/5 gates.
391 = main's 387 + exactly these four tests. Under the CI condition (both
ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME at empty dirs): 383 passed + 8 skipped =
391, exit 0, and all four new tests RUN there — none reads a ledger, an agent
CLI or ~/.codex, so NO skip ceiling moved. Floor 387 -> 391 for exactly those
four tests, measured on the merge result rather than on the branch base.
Break->revert, nine cases, each confirmed to fail while broken and pass after
revert: absence-as-empty (under --selftest and under pytest separately);
ambiguous append guessing hits[0]; append writing at EOF instead of inside the
item; the two nothing-exit-codes collapsed into one; the pointer size gate; the
pointer no longer naming the accessor; CLAUDE.md §0 step 3 reverted to a bare
path; CLAUDE.md §5 reverted to "edit the file". The append-placement case did
NOT discriminate on the first attempt — the fixture appended to the LAST section,
where "section end" and "end of file" are the same position — so the fixture was
rewritten to target a middle item and assert the note lands above the next
heading. That hole is recorded in the selftest so it cannot come back.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
…findings
Every Python PR failed six checks — lint-ruff, lint-format, typecheck-mypy,
python 3.12, python 3.13, summary — and `ruff check .` on main reported 915
findings with 181 files needing reformatting. That reads as undrained lint debt
no single PR can clear. It was not. All five upstream jobs died at the SAME
shared install step, before any tool ran:
Error: .../.github/workflows/autofix-versions.env is required;
refusing to install unpinned tooling.
identical in jobs 97151941071 / 97151941074 / 97151941041 / 97151952706 /
97151952767. Ruff, Black and mypy had never executed on this repository, not
once, so every prior statement about its lint debt was inferred.
The cause is a contradiction between two lines of one upstream file.
`Workflows/.github/sync-manifest.yml:33` syncs `pr-00-gate.yml` to consumers;
line 967 of the same file lists `autofix-versions.env` under EXCLUDED —
"consumers copy or override per docs/ci/WORKFLOWS.md. Intentionally not synced".
So the sync delivered a Gate that hard-requires a file the sync deliberately
never delivers, and this repo never did the documented copy
(`Workflows/docs/ci/WORKFLOWS.md:83`). This commit does it.
Neither option in the brief would have worked alone. Setting lint/format_check/
typecheck/run-mypy to false leaves python 3.12, python 3.13 and summary red,
because `require_exact_pin` demands pytest and pytest-xdist pins
unconditionally. And a second latch sat behind the first: with coverage on, the
reusable appends `--cov-config=pyproject.toml` unconditionally, so pytest dies
at startup here (`ConfigError: Couldn't read 'pyproject.toml'`) — while adding a
pyproject.toml makes the same workflow append `-e '.[app,dev]'`, which 126 flat
root modules with no build backend cannot satisfy. That is why the Ruff and mypy
config land as ruff.toml + mypy.ini, and why coverage is off.
THE REAL LATCH: two windows that could never agree. With no config present the
two CI surfaces resolved "no config" differently — the Gate pinned the pre-0.16
family (`--select E4,E7,E9,F`, 79 findings) while Autofix took Ruff 0.16's own
much wider default plus `--select I` (915). So Autofix rewrote the tree to
satisfy rules the Gate never checked, on every Gate failure, and the Gate stayed
red anyway: four `chore(autofix)` commits on #42 and one on #51, ~143 files and
+20,562/-11,089 each. Reverting one got it re-pushed.
ruff.toml collapses them into one window — Ruff reads it from either surface, so
what Autofix fixes is exactly what the Gate checks — and this commit brings the
tree TO that canon, which is what makes the loop structurally dead rather than
dormant. Verified by replaying Autofix's three commands verbatim: `ruff check
--select I --fix`, `ruff check --fix`, `black -l 100 .` → zero changes.
Deferring format instead would have left the loop armed, and `.autofix-exclude`,
the only repo-owned lever, cannot express "do not format this repo".
Measured 2026-08-23 at the pinned versions, blocking / drainable. The first two
are against main under the ruff.toml this commit adds:
lint-ruff 141 -> 0 findings drainable 141 ON, green
lint-format 126 -> 0 files drainable 126 ON, green
python 3.12/3.13 0 failures ON, green
typecheck-mypy 587 errors in 88/186 drainable 0 OFF, annotated
coverage 1 startup error drainable 0 OFF, annotated
Both OFF toggles state blocking AND drainable AND "drains by" at the single
place the toggles are computed, and a test fails if any of the three fields goes
missing — the runtime rule enforced rather than written down. mypy.ini is
committed although the check is off, because without it `mypy .` aborts on a
duplicate-module setup error and the 587 would be unverifiable prose; it
silences no error code, by test. docs/CI_LINT_BASELINE.md records the baseline
and `scripts/ci_lint_baseline.py` regenerates it, refusing to print numbers
measured with unpinned tools. A `measured-with` line is asserted against the pin
file, so bumping a tool version without re-measuring goes red — the drain and
the measurement are wired to move together.
E501 is deliberately not selected: 1036 lines still exceed 100 columns after
`black -l 100`, all long strings, URLs and comment prose. Selecting it would be
selecting a rule with no drain.
Verified: `python3 verify.py` on the owner's machine — 378 passed, 0 failed, 1
skipped (cursor catalog offline), 83/83 selftests, 43/43 CAN FIRE, 5/5 gates.
The CI-equivalent parallel run (`pytest -n auto --dist=loadgroup`) with fresh
ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME: 371 passed, 8 skipped. Floor 368 -> 379
for the 11 new tests; no ceiling moved. The reformat was audited by AST: of 126
changed modules, 60 are identical in AST, 40 differ only in imports, 4 only in
docstring whitespace, and all 22 remaining deltas are the named lint fixes
(E741 renames, E731 lambda->def, F841, F601 duplicate dict key, F541). Black
rewrote 125 of the 126 files; the Ruff fixes had already normalised the other.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
CLAUDE.md depended on IMPROVEMENT_BACKLOG.md in two places and both were
unfollowable by exactly the workers they bind. §0 step 3 ("check the improvement
log; many ideas are already DONE") is part of the MANDATORY dedup-before-develop
check, and §5 says append a status note there. The file was gitignored, and a
gitignored file does not exist in a git WORKTREE — it was verified absent from
every worktree on this machine. Agents work in worktrees. Three agents in one day
could not do step 3; two said so and fell back to ledger notes and docstrings.
481 KB of accumulated "already DONE" knowledge — the project's stated
WHICH STATE VARIABLE: ORCH_LOCAL_RUNTIME, not ORCH_STATE_DIR. CLAUDE.md §1
already partitions them — ORCH_STATE_DIR holds the audit cache, firing monitor
and redirect-sweep state, all derived and self-regenerating, while
ORCH_LOCAL_RUNTIME holds the capability LEDGER and the Brain: durable,
irreplaceable instance evidence. The improvement log is the second kind. It
cannot be regenerated from anything, and its §0 role is the prose twin of the
ledger's `notes` field (both answer "is this already DONE?"), so it belongs
beside the ledger rather than beside the caches. A useful consequence: ci.yml
already points ORCH_LOCAL_RUNTIME at an empty temp dir, so the named-absence
path is exercised on every runner by construction.
THE MOVE was copy -> verify -> remove, in that order: 482,865 bytes, sha256
f7eea973...d16d identical at source and destination before anything was deleted.
There is no git undo for unversioned history.
THE ACCESSOR is improvement_log.py (backlog.py is taken by the unrelated fleet
work-discovery lane and is not touched):
search <term> — §0 step 3 in one command. Prints each hit under the item
heading that owns it, plus the lines and sections read,
so "no matching items" describes a file actually read.
append <ref> <note> — §5 in one command. Places a dated note INSIDE the
matched item, atomically, keeping one rolling backup;
REFUSES on an ambiguous or unknown ref rather than
guessing, because a note filed against the wrong item
corrupts the record it exists to improve.
path — where it resolved to, and whether it is here.
A NAMED ABSENCE, NEVER A SILENT ONE. On a fresh clone, a runner, or a second
instance there is no log; every command then names what is missing, where it
would be, and both env vars that control it, and exits 2. Honest-empty is exit 1
and says so in words. A reason-less empty result is indistinguishable from "no
matches", which is this repo's founding defect.
THE POINTER is what actually fixes the invisibility: nothing in a worktree even
hinted the log existed. IMPROVEMENT_BACKLOG.md is now TRACKED and holds a short
pointer, so all thirteen citations of that filename still resolve and `ls` still
shows it. It is removed from .gitignore deliberately: left ignored, the 481 KB
file could reappear at that path silently — tracked, it announces itself in
`git status`. The real log lives outside the repository entirely, so `git add`
on it fails with "outside repository", which is stronger than an ignore rule.
test_improvement_log.py fails if the pointer grows into a log.
capability_admission.SKIP_NAMES drops its IMPROVEMENT_BACKLOG.md entry: the
allowlist reason ("the backlog narrates history and cites closed records") no
longer describes anything in the tree, so the gate now scans one more file.
DEDUP FINDING (CLAUDE.md §0), recorded before writing code and again in the
module docstring since plans are not durable: grepped by concept, not name.
`git grep IMPROVEMENT_BACKLOG` returns 13 references, every one prose or a
comment — no reader, no writer, no accessor. `improvement_backlog|
improvement-backlog|backlog_notes` over *.py returns nothing. The only machinery
touching the filename was capability_admission.SKIP_NAMES, an exclusion.
Genuinely absent, so new. NO LEDGER ROW, deliberately: this is documentation
access with no dispatch path, outcome or learning sink, so the admission gate
does not bind on it — the same reasoning env_prereq.py records for itself — and
registering one would turn every sibling worktree's verify.py red for a module
they cannot see (§1, ledger shared per machine / code per worktree).
Not an ARCHITECTURE.md change: this is repo tooling like verify.py and
env_prereq.py, not a loop stage, rail, role, feedback surface or registry entry.
VERIFIED on the MERGE RESULT, after rebasing onto af6654d (PR #42, 143 files):
391 passed, 0 failed, 0 skipped, 84/84 selftests, 43/43 can-fire, 5/5 gates.
391 = main's 387 + exactly these four tests. Under the CI condition (both
ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME at empty dirs): 383 passed + 8 skipped =
391, exit 0, and all four new tests RUN there — none reads a ledger, an agent
CLI or ~/.codex, so NO skip ceiling moved. Floor 387 -> 391 for exactly those
four tests, measured on the merge result rather than on the branch base.
Break->revert, nine cases, each confirmed to fail while broken and pass after
revert: absence-as-empty (under --selftest and under pytest separately);
ambiguous append guessing hits[0]; append writing at EOF instead of inside the
item; the two nothing-exit-codes collapsed into one; the pointer size gate; the
pointer no longer naming the accessor; CLAUDE.md §0 step 3 reverted to a bare
path; CLAUDE.md §5 reverted to "edit the file". The append-placement case did
NOT discriminate on the first attempt — the fixture appended to the LAST section,
where "section end" and "end of file" are the same position — so the fixture was
rewritten to target a middle item and assert the note lands above the next
heading. That hole is recorded in the selftest so it cannot come back.
Co-authored-by: Tim Stranske <tim@stranskemo.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself: #42 merged underneath it, moved the floor 368 -> 387 and the ceiling 24 -> 26, and the resulting conflict forced the rebase that produced the 387 recorded here. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant -- a skipped test is still collected, measured the same day at 368 on both CI and this machine with pass/skip splits of 344/24 against 368/0. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and the break demo is what caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified: 387 passed, 0 failed, 0/26 skipped, 387 collected (floor 387), 83/83 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
13 tasks
Owner
Author
Correction: the four replies above did NOT ship with this mergeThis PR merged at 12:55:21Z at The four fixes are unchanged and now go to main on their own, cherry-picked onto Also stranded on the same branch, from another session and not mine to move: |
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself TWICE inside an hour: #42 landed underneath it (floor 368 -> 387, ceiling 24 -> 26) and then #59 did (387 -> 391), and each conflict forced a rebase and a fresh measurement. Under the old one-directional rule both would have merged green with a floor below reality. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant -- a skipped test is still collected, measured the same day at 368 on both CI and this machine with pass/skip splits of 344/24 against 368/0. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and the break demo is what caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified on the merge result: 391 passed, 0 failed, 0/26 skipped, 391 collected (floor 391), 84/84 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2; floor stays at main's 391, since this adds selftest assertions rather than pytest tests. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… lint debt
Every Python PR failed six checks — lint-ruff, lint-format, typecheck-mypy,
python 3.12, python 3.13, summary — and `ruff check .` on main reported 915
findings with 181 files needing reformatting. That reads as debt no single PR can
clear. It was not. All five upstream jobs died at the SAME shared install step,
before any tool ran:
Error: .../.github/workflows/autofix-versions.env is required;
refusing to install unpinned tooling.
identical in jobs 97151941071 / 97151941074 / 97151941041 / 97151952706 /
97151952767. Ruff, Black and mypy had never executed on this repository, not
once, so every prior statement about its lint debt was inferred.
The cause is a contradiction between two lines of one upstream file.
`Workflows/.github/sync-manifest.yml:33` syncs `pr-00-gate.yml` to consumers;
line 967 of the same file lists `autofix-versions.env` under EXCLUDED —
"consumers copy or override per docs/ci/WORKFLOWS.md. Intentionally not synced".
So the sync delivered a Gate that hard-requires a file the sync deliberately
never delivers, and this repo never did the documented copy
(`Workflows/docs/ci/WORKFLOWS.md:83`). This commit does it.
Neither option in the brief would have worked alone. Setting lint/format_check/
typecheck/run-mypy to false leaves python 3.12, python 3.13 and summary red,
because `require_exact_pin` demands pytest and pytest-xdist pins
unconditionally. And a second latch sat behind the first: with coverage on, the
reusable appends `--cov-config=pyproject.toml` unconditionally, so pytest dies
at startup here (`ConfigError: Couldn't read 'pyproject.toml'`) — while adding a
pyproject.toml makes the same workflow append `-e '.[app,dev]'`, which 129 flat
root modules with no build backend cannot satisfy. That is why the Ruff and mypy
config land as ruff.toml + mypy.ini, and why coverage is off.
THE REAL LATCH: two windows that could never agree. With no config present the
two CI surfaces resolved "no config" differently — the Gate pinned the pre-0.16
family (`--select E4,E7,E9,F`) while Autofix took Ruff 0.16's own much wider
default plus `--select I`. On today's main that is 37 findings versus 733; before
#42's Autofix commits landed it was 79 versus 915, which is where the "915, no PR
can drain it" reading came from. The gap is the defect, not its size on any given
day. Autofix rewrote the tree to satisfy rules the Gate never checked, on every
Gate failure, and the Gate stayed red anyway: four `chore(autofix)` commits on
#42 and one on #51, ~143 files and +20,562/-11,089 each. Reverting one got it
re-pushed.
ruff.toml collapses them into one window — Ruff reads it from either surface, so
what Autofix fixes is exactly what the Gate checks — and this commit brings the
tree TO that canon, which makes the loop structurally dead rather than dormant.
Verified by replaying Autofix's three commands verbatim: `ruff check --select I
--fix`, `ruff check --fix`, `black -l 100 .` → zero changes, 196 files
unchanged. Deferring format instead would have left the loop armed;
`.autofix-exclude`, the only repo-owned lever, cannot express "do not format
this repo", and `autofix.yml` carries no `sync_mode`, so an edit there would be
overwritten by the next template sync.
Measured 2026-08-23 at the pinned versions, blocking / drainable:
lint-ruff 39 -> 0 findings drainable 39 ON, green
lint-format 2 -> 0 files drainable 2 ON, green
python 3.12/3.13 0 failures ON, green
typecheck-mypy 601 errors in 89/189 drainable 0 OFF, annotated
coverage 1 startup error drainable 0 OFF, annotated
Because #42's Autofix commits reached main first, the drain here is 39 findings
and 2 files rather than the 141 and 126 it would have been a day earlier.
Both OFF toggles state blocking AND drainable AND "drains by" at the single
place the toggles are computed, and a test fails if any of the three fields goes
missing — the runtime rule enforced rather than written down. One literal per
toggle: the `with:` block and the `summary` job's coverage branch both read
`needs.detect.outputs.*`, and a test rejects a second hardcoded value. mypy.ini
is committed although the check is off, because without it `mypy .` aborts on a
duplicate-module setup error and the 601 would be unverifiable prose; it
silences no error code, by test — fifteen `disable_error_code` entries would
cover 597 of the 601 and make the job green while checking nothing.
docs/CI_LINT_BASELINE.md records the baseline and
`scripts/ci_lint_baseline.py` regenerates it, refusing to print numbers measured
with unpinned tools. A `measured-with` line is asserted against the pin file, so
bumping a tool version without re-measuring goes red — the drain and the
measurement are wired to move together.
E501 is deliberately not selected: 1068 lines still exceed 100 columns after
`black -l 100`, all long strings, URLs and comment prose. Selecting it would be
selecting a rule with no drain. Three E402 findings are exempted per site with a
reason, because each import follows the `sys.path.insert` that makes it
resolvable; the rule stays on everywhere else. RUF100 is not selected, because
72 `# noqa` comments name rules the narrow set does not check and deleting them
would have to be undone the moment the selection widens.
Verified: `python3 verify.py` on the owner's machine — 402 passed, 0 failed, 0
skipped, 84/84 selftests, 43/43 CAN FIRE, 5/5 gates. Fresh-machine simulation
with empty ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME: 394 passed, 8 skipped, 402
collected. CI-equivalent parallel run (`pytest -n auto --dist=loadgroup`): 393
passed, 9 skipped. Floor 391 -> 402 for the 11 new tests; no ceiling moved.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
pushed a commit
that referenced
this pull request
Aug 23, 2026
… silent `.verify-floor.json` has been found BELOW reality four times -- 21 low at the worst, then 8, then 1, then 2 -- and each was hand-raised after the fact by whoever happened to look. The stale number was never the defect. `_floor_problems` fired only on `collected < floor`, so a PR could add tests and never touch the file: silently green, with the floor left permissive by exactly the number added. #34 and #37 each did precisely that, which is what PR #50 then had to clean up by hand. `collected` is now an equality. Too few tests still fails; too many fails as well, printing the two integers to write and telling you to rebase first. That also makes the concurrent case self-enforcing, which is the part discipline could not fix. Once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT -- the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. This change demonstrated that on itself repeatedly: #42 landed underneath it (floor 368 -> 387, ceiling 24 -> 26, verify.py reformatted), then #59 (387 -> 391), then #61, all in one afternoon. Each conflict forced a rebase and a fresh measurement; under the old one-directional rule each would have merged green with a floor below reality. Git's own conflict detection is what enforces "measure on the merge result, not on the branch", the rule the note in that file had to restate three times with nothing behind it. `passed` deliberately stays a MINIMUM on passed+skipped. Only collection is machine-invariant, measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. Making that one strict too would fail a machine for honestly naming a missing prerequisite. Two further fixes in the same change: * `--update-floor` no longer REPLACES the note. It appends. The note is the only record of which prerequisite justifies each ceiling, so overwriting it destroyed the rationale on every use -- the file had to carry a warning about its own tool. * Drift does not block `--update-floor`. The first draft made it a latched gate: a floor behind reality became a problem, and the guard was `not problems`, so the one remedy the error message named was refused for the existence of the very condition it clears. `_blocks_floor_update` lets drift through while real failures still block, sharing `DRIFT_PREFIX` so message and predicate cannot diverge. Selftests cover both directions, skip-invariance, the unset-floor case, the unblock predicate and note preservation, each with a deliberate-break demonstration. Two of those tests were themselves defective and only the break demo caught it: the latched-gate assert exercised `_blocks_floor_update` in isolation and stayed green when the CALL SITE was reverted (built-but-not-wired), and the source-text assert written to fix that searched for a literal that appears in its own line, so it could never fail. The needle is now built from two fragments and the wiring is asserted at the guard. Verified on the merge result: 391 passed, 0 failed, 0/26 skipped, 391 collected (floor 391), 84/84 selftests, 43/43 can-fire, 5/5 gates. Ceilings untouched at 26/7/2; the floor stays at main's 391, since this adds selftest assertions rather than pytest tests. black clean at line-length 100; ruff at main's baseline of 6 pre-existing findings, none added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
fix(review): the four deferred #42 findings — causal finding attribution and a member-aware v2 lifecycle
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
…empts (#63) CodeRabbit finding on #42 (thread 3837879039), verified real against the real code path before patching — its own analysis chain had used a synthetic five-column table, not the real 25-column schema. `resolve_unresolved_worker_attempts` selected on worker role + NULL resolved_model + non-NULL profile_id with no status predicate. The profile attempt row is written `started` BEFORE the subprocess spawns (dispatcher.py:1511, exp_abcd.py:412), and `adapters.cli_reported_model` reads the first model in the session log within a 2h window of `started_ts` — a log that exists from the moment the CLI starts. So a run still executing probed clean and `--apply` stamped it `complete` with a resolved model and a `completed_ts` off the sweep's own clock: the one row shape CLAUDE.md §2 allows to support an exact-model claim, minted for a worker that had not finished and could still fall back, retry onto another model, or fail outright. The filter is `status='unresolved'`, which also excludes `failed` (dispatcher's `profile_process_start_failed`) — terminal, but it never ran, so there is no served model to recover. Exclusions are counted, not silently narrowed: `excluded_not_terminal` reports them keyed by status beside `candidates`. Not a starved drain: a `started` row is excluded only while in flight; its own completion closes it to `complete` (resolved) or `unresolved` (eligible next pass). Measured read-only on the live ledger: 56 candidates before, 56 after. Coverage pins all three cases — terminal `unresolved` IS swept, in-flight `started` and never-ran `failed` are left alone — with all three sharing one workspace so the probe resolves for every one and the status filter is the only protection. The in-test deliberate break is self-diagnosing on both stale modes (clause moved; query signature unrecognisable), after a second CodeRabbit thread (3838730241); its first proposed guard was tautological and was not used. verify.py on the merge result: 411 passed, 0 failed, 0 skipped, 84/84 selftests, 43/43 can-fire, 5/5 gates.
Merged
11 tasks
stranske
added a commit
that referenced
this pull request
Aug 23, 2026
…both stranded post-merge (#84) Both branches were held back from the branch cleanup because their tips carried commits pushed AFTER their PR merged, so "PR merged" did not mean "work landed". Verified per-symbol rather than by diff size — both branches are thousands of lines behind main, so a raw diff conflates stale with unlanded. #42 / commit 4e0d6ae — `adapters.py` catalog resolution. Main has `advertised_models` and NONE of the generalisation around it: `advertised_catalog`, `_advertised_catalog`, `_cached_catalog`, `agy_log_for`, `AGY_LOG_SUFFIX`, `CATALOG_ROUTING_TAGS`, `_catalog_model_id` were all absent. This is learning-loop provenance code (CLAUDE.md 2: "never treat a generic trace model as provider resolution"), and its whole point is that THE CATALOG IS THE AUTHORITY — a label resolves against the ids the CLI actually advertises, with routing TAGS (`auto`, `default`, `cli-default`) refused as non-identities. The commit's own note records that `VENDOR_MODEL_RE` rejects 42 of 204 real cursor ids, so shape-matching an id the CLI itself advertised is both redundant and wrong. Cherry-picked; `adapters.py`, `dispatcher.py` and `ledger_reconcile.py` applied clean. Two conflicts: * `.verify-floor.json` — took main's. A floor is a property of the MERGE RESULT, never carried in from a branch, so it is re-measured below. * `test_feedback_model_provenance.py` — TWO DIFFERENT tests in one region: main's `test_late_sweep_completes_terminal_attempts_never_one_in_flight` (from #63) and the branch's `test_gemini_provenance_reads_the_per_run_log_before_the_conversation_store`. Kept BOTH; they are independent. #34 / commit c1dc9a7 — README item 11 for `evidence_acquisition.py`, which main documented nowhere (zero occurrences). Every factual claim was re-verified against main's code rather than trusted: `capabilities.unblock()` exists; `ORCH_EVIDENCE_ACQUISITION_MAX_FEEDS`/`_MAX_ITEMS` default to 1 and 3; `LIVE_FLAG = "ORCH_EVIDENCE_ACQUISITION"` with SHADOW as the documented default; and the quoted summary line matches the format string verbatim (`feedable {n} / capped {n} / candidates {n} / fed {n}`). It is the drainable-vs-blocking line the latched-gate rule asks for, and it was the only place that reported it. DELIBERATE-BREAK -> REVERT: emptying `CATALOG_ROUTING_TAGS` fires `assert model_id_for_label("cursor", "Auto (default)") is None` in adapters' OWN selftest; reverted clean. Worth recording that `pytest test_feedback_model_provenance.py` did NOT catch that break — the guard is covered by a `--selftest`, not by a test_*.py, which is precisely why `verify.py` is the gate and a pytest subset is not. A redundant pytest test written before checking was dropped. FLOOR 427 -> 428, one new test, note appended not replaced. Verified FRESH-STATE (both ORCH_STATE_DIR and ORCH_LOCAL_RUNTIME at empty dirs, reproducing CI): VERIFIED — 420 passed, 0 failed, 79 selftests, 3/5 gates green, 8 tests + 5 selftests + 2 gates skipped for named prerequisites; 420 + 8 = 428 = floor. ruff + black -l 100 clean. NOT FIXED HERE, and not caused here: on this machine `test_capabilities.py`'s `test_gate_blocks_execution_is_opt_in_and_narrow` and `test_evidence_gate_kind_is_not_blanket_observer` fail on PRISTINE main too — the hourly fleet tick mutated the machine-local ledger and range-lane-rollout now classifies `matched_not_invoked` instead of `deliberately_gated`. Ledger STATE, not code; they skip with a named reason under a fresh ledger, which is what CI uses. Co-authored-by: Tim Stranske <tim@stranskemo.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
22 commits that were sitting on a pushed branch with no PR. Machine-evaluated as mergeable; see the
verdict below.
What it is
One coherent theme in three layers:
612ac1e,b1e8543,8788a2b) — a profile per agent, every seat's model named, andper-agent mining coverage so a one-seat miner is visible instead of averaging into an ensemble.
8788a2brestores the mid rung: registering only Pro had removed the choice.14520c7,fe59bc7,6a62ebb,7b07b93,f65299e,8ddeb25,cfd8895,0a7db33) — the chain that makes a per-agent claim substantiable. Stops four seats writing workerattempts they can never substantiate; resolves the model from what the CLI itself reported rather
than a store scrape; carries a profile on the ledger row (
f65299efinds the actual root cause: anull profile recorded beside propensity 1.0).
1ccdd2f,d5543b7,33b2f9f,053d0c5,40ad84d,eaf2f7b,9bc9bd0) — a corpus review is ONE subject with real arms, not N unrelated offloads;a subjectless declaration must not latch;
eaf2f7bfixes the second launcher still writinglegacy manifests, which is why
evaluations_v2stayed empty.Plus legibility fixes where a verdict contradicted its own counts (
3a3a0d6: cursor held 22 of whatit denied).
This implements provenance rules already codified in CLAUDE.md §2 — causal
operation_role=workerattribution, no
--modelcopied intoreported_model, arm+member+profile identity never collapsed toa provider name. The branch does not edit those rules; it makes them true in code.
Machine verdict
Corrected. This section first claimed green on the strength of a local run. CI disproved that:
the first run was RED with two problems, both invisible on the owner's machine. Both are now fixed
in
88bcb7c/69edf74, verified in a runner sandbox built to the five documented differences thatreproduces CI exactly (361 passed, 26 skipped, 387 collected).
1. A stub leak, fixed by isolation rather than a skip
adapters.advertised_modelsshells out toagy modelswhen its disk cache is cold. That cache iswarm on the owner's machine and cold everywhere else, so only elsewhere did the probe land inside a
monkeypatched
subprocess.runand overwrite the captured command — the per-run agy-log assertionthen compared against
['agy', 'models']:CLAUDE.md names this exact case and says the fix is isolation, because isolation makes CI run more.
The double now records only the run under test. Selftests on a bare runner go 75 → 76 modules —
dispatcher speaks instead of failing. The assertion is also self-describing now: its bare dump was
truncated out of the CI log, and naming type/len/head identified the cause on the first run.
2. Ceiling 24 → 26, floor 366 → 387
Two genuinely new skips, both drift detectors against a real installed agent runtime, so neither
can be moved below the ceiling (the preferred way to lower one): the agy advertised-models cache, and
~/.vibe/config.tomlforactive_model. A fixture would exercise the comparison while detecting noreal drift. Both name their missing prerequisite, so a green run still states what it did not check.
Floor measured on the merge result, per
.verify-floor.json's own rule.origin/mainis also merged in (edae357) — the branch was 19 commits stale.Relationship to #34
#34 cherry-picked two commits out of this branch (
e73b47d, plus thecapabilities.pyhalf of69a1c38) to un-redmain. This branch is a superset — it also carries69a1c38'sadapters.pyauth-probe half, which #34 deliberately left behind.
Recommended order: #34 first (small, green, and it carries the README gated-features entry for
ORCH_EVIDENCE_ACQUISITIONthat this branch does not have), then this. Verified clean in that order.If #34 is closed as superseded instead, port its README item 11 forward — otherwise a default-off
safety switch ships undocumented.
What was NOT verified
The machine gates and the added tests are the evidence here; I did not line-by-line review 3,578
lines of learning-loop code. Worth a reviewer's eye on the
feedback.py/adapters.pyprovenanceedges specifically, since CLAUDE.md §2 makes a wrong call there corrupt the Brain rather than fail
loudly.
The branch is based 19 commits back and
mainmoved 4 times during this evaluation — the drift riskargues for landing it as one unit rather than splitting it into themed PRs.
🤖 Generated with Claude Code
On the failing
python cichecks — pre-existing, and not fixable from this PRpr-00-gate.ymlarrived in a workflow-template sync and calls the fleet's sharedreusable-10-ci-python.yml. It triggers only onpull_request, never on push tomain, and thisPR is the first whose head contains it (because
mainwas merged in here). So the checks are new,and
mainhas never been held to them. There is noruff,mypy, or format config in the repo at all.Measured on the two trees:
mainThis PR is 187 violations better than
main. The failures are repo-wide debt, not a regressionhere, and no single PR can drain 915 violations.
mainis not branch-protected — no required statuschecks — so these are advisory. The repo's own gate,
verify.py, passes on this head.Fixing it belongs in its own PR against
main: either add a ruff/mypy config with a baseline, or setthe gate's
lint/format_check/typecheckinputs off until the debt is paid.The autofix commit (
52ab885)autofix.ymlfires on every Gate failure and pushed 142 files (+20,562 / −11,089) labelled"formatting/lint". I checked whether it changed behaviour rather than trusting the label: comparing
ast.unparseoutput before and after, the only changes arere.I→re.IGNORECASE(same value),typing→collections.abcimport moves, docstring reflowing, and one genuinely-unusedimport sys.It is behaviour-preserving, and
verify.pypassing on this SHA confirms it empirically.I did not revert it. It cannot loop-free be reverted — autofix re-triggers on the next Gate
failure and would push it again — and it lowers lint debt rather than raising it. The real cost is
reviewability: 142 files of reformatting now sit on top of the 19 substantive commits, so a reviewer
should read this PR commit-by-commit, skipping
52ab885.Summary by CodeRabbit
New Features
Bug Fixes