From 80920c5c7a6d33f8f5f4e5100b20820a9a9bc8a4 Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 08:56:31 -0500 Subject: [PATCH 1/7] fix(provenance): the late sweep may only complete TERMINAL worker attempts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit finding on PR #42 (thread 3837879039), verified real against the real code path before patching. `resolve_unresolved_worker_attempts` selected on `operation_role='worker' AND resolved_model IS NULL AND profile_id IS NOT NULL` with no status predicate. The profile attempt row is written `started` BEFORE the subprocess is spawned (dispatcher.py:1511, exp_abcd.py:412), so an IN-FLIGHT attempt matched every clause. `adapters.cli_reported_model` reads the FIRST model in the session log within a 2h window of `started_ts`, and that log exists from the moment the CLI starts -- so a run still executing probes clean, and `--apply` stamped it `complete` with a resolved model and a `completed_ts` off the sweep's own clock. Reproduced on a real tmp ledger: a `started` row with completed_ts NULL became ('complete', 'gpt-5.6-terra', ). That is the one row shape CLAUDE.md §2 allows to support an exact-model claim -- a successful `operation_role=worker` attempt with a provider-resolved model -- minted for a worker that had not finished and could still fall back, retry onto another model, or fail outright. It corrupts the Brain silently rather than failing loudly. The filter is `status='unresolved'`, which also excludes `failed` (dispatcher's `profile_process_start_failed`): terminal, but it never ran, so there is no served model to recover and resolving it from a neighbouring session in the window would be invention. CodeRabbit's rationale named only `started`. Not a starved drain: a `started` row is excluded only while in flight -- its own completion closes it to `complete` (resolved, no sweep needed) or `unresolved` (eligible next pass), so the exclusion clears itself without the sweep's help. Measured read-only on the live ledger: 56 candidates before the filter, 56 after; every genuinely drainable row is already `unresolved`. Exclusions are counted, never silently narrowed: `excluded_not_terminal` reports them keyed by status, beside `candidates`, matching the existing `excluded_unreportable` convention. `candidates: 0` next to `{started: 3}` reads as "wait for those runs"; `candidates: 0` alone reads as "the sweep is broken". Coverage: one new test pins all three cases -- terminal `unresolved` IS swept, in-flight `started` is left alone, never-ran `failed` is left alone -- with all three sharing one workspace so the probe resolves for every one of them and the status filter is the only thing that can protect the two ineligible rows. Includes an in-test deliberate break (a connection proxy that strips the status clause, restoring the pre-fix query) asserting the corruption reappears. Separately demonstrated break->revert at the source: deleting the clause from the query fails the test on the in-flight assertion; restored, green. FLOOR 387 -> 388, hand-edited with rationale (--update-floor clobbers the note). No ceiling moved and nothing new is skipped: the test builds its own tmp ledger and rollout fixture, so it runs on a bare runner. verify.py: 388 passed, 0 failed, 0 skipped, 83/83 selftests, 5/5 gates -- in the worktree and from a mirror-shaped copy at a different path. Ruff unchanged from the HEAD baseline (9 pre-existing findings, none added). Co-Authored-By: Claude Opus 5 --- .verify-floor.json | 6 +- ledger_reconcile.py | 40 ++++++++- test_feedback_model_provenance.py | 129 ++++++++++++++++++++++++++++++ 3 files changed, 171 insertions(+), 4 deletions(-) diff --git a/.verify-floor.json b/.verify-floor.json index eb70c9b..fcfddd3 100644 --- a/.verify-floor.json +++ b/.verify-floor.json @@ -1,8 +1,8 @@ { - "collected": 387, - "passed": 387, + "collected": 388, + "passed": 388, "skipped_max": 26, "selftest_skipped_max": 7, "gate_skipped_max": 2, - "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE." + "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 388 on 2026-08-23 (CodeRabbit follow-up on PR #42, thread 3837879039): exactly one new test, test_feedback_model_provenance.test_late_sweep_completes_terminal_attempts_never_one_in_flight, which pins that ledger_reconcile.resolve_unresolved_worker_attempts completes only TERMINAL unresolved worker attempts and never one still in flight. No ceiling moved and nothing new is skipped -- the test builds its own tmp ledger and codex rollout fixture and monkeypatches adapters.CODEX_SESSIONS, so it needs no agent CLI and no populated capability ledger and runs on a bare runner. Measured on the merge result (this branch is at af6654d, the #42 merge, which is current origin/main) per the rule above, NOT on a branch cut earlier: 388 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. Three sibling follow-up branches are in flight against this same main (CI/ruff config, arm-attribution + durability, adapters label->ID); if this file conflicts with one of them, resolve as the UNION and RE-MEASURE on the new merge result rather than taking either number -- that is what #42 and #50 did, and taking a side is what put the floor 8 below reality earlier." } diff --git a/ledger_reconcile.py b/ledger_reconcile.py index 3ae4452..6886c70 100644 --- a/ledger_reconcile.py +++ b/ledger_reconcile.py @@ -270,15 +270,50 @@ def resolve_unresolved_worker_attempts(*, apply: bool = False, limit: int = 5000 keeps no per-session log stays unresolved with its reason NAMED, and nothing is ever inferred from the requested model. Dry-run by default; reports the per-agent breakdown either way, so "resolved 37" always arrives next to "16 cannot report and here is why". + + Only TERMINAL attempts are eligible: `status='unresolved'`. An in-flight (`started`) + attempt is never completed here, and neither is one that never ran (`failed`) — see the + eligibility comment below for why each exclusion is required and why neither starves the + drain. """ + # TERMINAL ROWS ONLY -- `status='unresolved'` is the whole eligibility rule, and it is doing two + # jobs. A worker attempt's profile row is written `started` BEFORE the subprocess is spawned + # (`dispatcher`/`exp_abcd` pre-dispatch), so an IN-FLIGHT attempt matches every other clause + # here: role worker, profile set, resolved_model still NULL. `cli_reported_model` reads the + # FIRST model in the session log within a 2h window of `started_ts`, and that log exists as soon + # as the CLI starts -- so a run still executing probes clean, and `--apply` stamped it + # `complete` with a resolved model and a `completed_ts` of the sweep's own clock. That is the + # one row shape allowed to support an exact-model claim, manufactured for a worker that had not + # finished and could still fall back, retry onto another model, or fail outright. + # It also excludes `failed` (dispatcher's `profile_process_start_failed`), which is terminal but + # never ran: there is no served model to recover, so resolving it from a neighbouring session in + # the window would be pure invention. + # Not a starved drain (the trap this repo keeps falling into): a `started` row is excluded only + # while it is in flight. Its own completion closes it to `complete` (resolved -- no sweep + # needed) or `unresolved` (eligible on the next pass), so the exclusion clears itself without + # the sweep's help. Measured on the live ledger when this filter landed: 56 candidates before, + # 56 after -- every genuinely drainable row is already `unresolved`. with feedback._conn() as c: rows = c.execute( "SELECT ea.run_id, ea.profile_id, r.agent, r.target, r.ts " "FROM execution_attempts ea JOIN runs r ON r.run_id=ea.run_id " - "WHERE ea.operation_role='worker' AND ea.resolved_model IS NULL " + "WHERE ea.operation_role='worker' AND ea.status='unresolved' " + "AND ea.resolved_model IS NULL " "AND ea.profile_id IS NOT NULL ORDER BY r.ts DESC LIMIT ?", (int(limit),), ).fetchall() + # Counted, not silently narrowed. `candidates: 0` beside `not_terminal: {started: 3}` reads + # as "wait for those runs to finish"; `candidates: 0` alone reads as "the sweep is broken". + not_terminal = { + str(status or ""): int(count) + for status, count in c.execute( + "SELECT ea.status, COUNT(*) " + "FROM execution_attempts ea JOIN runs r ON r.run_id=ea.run_id " + "WHERE ea.operation_role='worker' AND ea.resolved_model IS NULL " + "AND ea.profile_id IS NOT NULL " + "AND (ea.status IS NULL OR ea.status<>'unresolved') GROUP BY ea.status" + ).fetchall() + } resolved: dict[str, int] = {} blocked: dict[str, int] = {} failed: dict[str, str] = {} @@ -349,6 +384,9 @@ def resolve_unresolved_worker_attempts(*, apply: bool = False, limit: int = 5000 # Reported beside it, never inside it: rows excluded because the seat can never report. # Naming them keeps the exclusion auditable, and keeps `candidates` an honest backlog. "excluded_unreportable": unreportable, + # Rows excluded as not-terminal, keyed by the status that excluded them. `started` clears + # itself when the run completes; `failed` never ran and is permanently and correctly out. + "excluded_not_terminal": not_terminal, "failed": failed, } diff --git a/test_feedback_model_provenance.py b/test_feedback_model_provenance.py index 23e9ab6..db5fc59 100644 --- a/test_feedback_model_provenance.py +++ b/test_feedback_model_provenance.py @@ -837,3 +837,132 @@ def test_every_seat_with_a_session_store_can_report_and_is_read_from_its_own_sto ) == "Gemini 3.6 Flash (High)" ) + + +def test_late_sweep_completes_terminal_attempts_never_one_in_flight(tmp_path, monkeypatch): + """The sweep may finish a TERMINAL unresolved attempt; it must not finish a running one. + + The profile attempt row is written `started` BEFORE the subprocess is spawned, so an in-flight + attempt matches every other clause of the sweep's query -- worker role, profile set, + `resolved_model` still NULL. `cli_reported_model` reads the first model in the session log + within a 2h window of `started_ts`, and that log exists from the moment the CLI starts, so a + run that is still executing probes CLEAN. Before the status filter, `--apply` stamped it + `complete` with a resolved model and a `completed_ts` taken from the sweep's own clock: the one + row shape allowed to support an exact-model claim, minted for a worker that had not finished and + could still fall back, retry onto another model, or fail outright. + + All three runs below share one workspace, so the probe resolves for ALL of them. The status + filter is therefore the only thing that can protect the two ineligible rows. + """ + import json + + import adapters + import ledger_reconcile + + old_db = feedback.DB_PATH + feedback.DB_PATH = tmp_path / "feedback.db" + try: + workspace = tmp_path / "offloads" / "ws-sweep" + workspace.mkdir(parents=True) + sessions = tmp_path / "sessions" + (sessions / "2026").mkdir(parents=True) + started = int(time.time()) + stamp = time.strftime("%Y-%m-%dT%H-%M-%S", time.localtime(started)) + (sessions / "2026" / f"rollout-{stamp}-x.jsonl").write_text( + json.dumps({"type": "session_meta", "payload": {"cwd": str(workspace.resolve())}}) + + "\n" + + json.dumps({"type": "turn_context", "payload": {"model": "gpt-5.6-terra"}}) + + "\n" + ) + monkeypatch.setattr(adapters, "CODEX_SESSIONS", sessions) + + def _pending(run_id): + """A pre-dispatch worker attempt: exactly what the dispatcher writes before spawning.""" + feedback.record_run(run_id, f"offload:{workspace}", "offload", "codex") + feedback.record_execution_attempt( + run_id, + attempt_id=f"attempt:profile:{run_id}", + operation_role="worker", + profile_id="codex-5.6-terra-high", + requested_provider="openai", + requested_model="gpt-5.6-terra", + status="started", + source="orchestrator-profile-decision", + started_ts=started, + ) + + def _row(run_id): + return ( + sqlite3.connect(feedback.DB_PATH) + .execute( + "SELECT status, resolved_model, completed_ts FROM execution_attempts " + "WHERE run_id=?", + (run_id,), + ) + .fetchone() + ) + + # TERMINAL: the completion path closed it unresolved. This is the sweep's whole purpose. + _pending("sweep-terminal") + feedback.complete_profile_attempt_unresolved( + "sweep-terminal", + selected_profile_id="codex-5.6-terra-high", + fallback_reason="resolved_model_not_reported_by_completion", + ) + # IN FLIGHT: the worker is still running, so nothing has closed it. + _pending("sweep-inflight") + # TERMINAL BUT NEVER RAN: the process failed to start, so no model ever served it. + _pending("sweep-failed") + feedback.complete_profile_attempt_unresolved( + "sweep-failed", + selected_profile_id="codex-5.6-terra-high", + fallback_reason="profile_process_start_failed", + status="failed", + ) + + report = ledger_reconcile.resolve_unresolved_worker_attempts(apply=True) + + # The terminal unresolved row gains the identity its log always carried. + assert _row("sweep-terminal")[:2] == ("complete", "gpt-5.6-terra") + # The in-flight row is untouched: no resolved model, no invented completion timestamp. + assert _row("sweep-inflight") == ("started", None, None) + # The never-ran row keeps its terminal failure rather than borrowing a neighbour's model. + assert _row("sweep-failed")[:2] == ("failed", None) + assert report["resolved_by_agent"] == {"codex": 1} + # `candidates` counts only what a reader could still drain. + assert report["candidates"] == 1 + # The exclusions are NAMED, not silently narrowed away: `candidates: 0` next to + # `{"started": 1}` reads as "wait for that run"; `candidates: 0` alone reads as "broken". + assert report["excluded_not_terminal"] == {"started": 1, "failed": 1} + + # DELIBERATE BREAK: drop the status clause, restoring the pre-fix query exactly. + real_conn = feedback._conn + + class _Unfiltered: + def __init__(self, c): + self._c = c + + def execute(self, sql, *a): + return self._c.execute(sql.replace("AND ea.status='unresolved' ", ""), *a) + + def __getattr__(self, name): + return getattr(self._c, name) + + def __enter__(self): + self._c.__enter__() + return self + + def __exit__(self, *exc): + return self._c.__exit__(*exc) + + monkeypatch.setattr(feedback, "_conn", lambda: _Unfiltered(real_conn())) + ledger_reconcile.resolve_unresolved_worker_attempts(apply=True) + broken_status, broken_model, broken_completed = _row("sweep-inflight") + assert (broken_status, broken_model) == ("complete", "gpt-5.6-terra"), ( + "the break must reproduce the corruption: a running worker stamped with an " + "exact resolved model" + ) + assert broken_completed is not None, "and with a completion timestamp it never earned" + # REVERTED by monkeypatch teardown; the filtered assertions above are the guard. + finally: + feedback.DB_PATH = old_db From e8d870601221066cddcf60a4a24b7140f334a975 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 23 Aug 2026 14:06:57 +0000 Subject: [PATCH 2/7] chore(codex-autofix): apply updates (PR #63) --- langsmith-fleet-worker-attempt.json | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 langsmith-fleet-worker-attempt.json diff --git a/langsmith-fleet-worker-attempt.json b/langsmith-fleet-worker-attempt.json new file mode 100644 index 0000000..62f46e1 --- /dev/null +++ b/langsmith-fleet-worker-attempt.json @@ -0,0 +1,16 @@ +{ + "agent": "codex", + "cli_version": "0.144.1", + "emitted_at": "2026-08-23T14:06:54.626218Z", + "execution_profile": "codex-default", + "fallback_models": [ + "gpt-5.5" + ], + "operation_role": "worker", + "pr_number": "63", + "requested_model": "gpt-5.6-terra", + "runner": "reusable-codex-run", + "schema": "langsmith-fleet/v1", + "selected_model": "gpt-5.6-terra", + "selection_reason": "input" +} From df1992970ecc8b7db80060a5bbb3638d32eb6547 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Sun, 23 Aug 2026 14:07:45 +0000 Subject: [PATCH 3/7] chore(autofix): formatting/lint --- improvement_log.py | 239 ++++++++++++++++++++++++++++------------ test_improvement_log.py | 45 +++++--- 2 files changed, 196 insertions(+), 88 deletions(-) diff --git a/improvement_log.py b/improvement_log.py index df037c3..7ecbbee 100644 --- a/improvement_log.py +++ b/improvement_log.py @@ -52,6 +52,7 @@ python3 improvement_log.py append python3 improvement_log.py --selftest """ + from __future__ import annotations import argparse @@ -72,9 +73,9 @@ # Exit codes, so a caller can distinguish the two kinds of nothing. EXIT_OK = 0 -EXIT_NO_MATCH = 1 # ran, read the whole log, found nothing — an HONEST empty -EXIT_ABSENT = 2 # the log itself is not on this machine — a NAMED absence -EXIT_REFUSED = 3 # the caller's item-ref matched zero or many items; nothing was written +EXIT_NO_MATCH = 1 # ran, read the whole log, found nothing — an HONEST empty +EXIT_ABSENT = 2 # the log itself is not on this machine — a NAMED absence +EXIT_REFUSED = 3 # the caller's item-ref matched zero or many items; nothing was written HEADING_RE = re.compile(r"^(#{2,3})\s+(.*?)\s*$") # `## 🟢 8. GitHub API rate-limit awareness` -> item number 8. The status emoji and any other leading @@ -109,6 +110,7 @@ def absence_note(path: Path) -> str: # Parsing: sections, so a hit can be reported under the item that owns it. # -------------------------------------------------------------------------------------------- + def parse_sections(text: str) -> list[dict]: """Every `##`/`###` heading with its line span and item number, in file order. @@ -122,12 +124,18 @@ def parse_sections(text: str) -> list[dict]: continue level, title = len(m.group(1)), m.group(2) num_m = ITEM_NUM_RE.match(title) - heads.append({"level": level, "title": title, "line": idx + 1, - "item": num_m.group(1) if num_m and level == 2 else None, - "end": len(lines)}) + heads.append( + { + "level": level, + "title": title, + "line": idx + 1, + "item": num_m.group(1) if num_m and level == 2 else None, + "end": len(lines), + } + ) # A section ends where the next heading of the same-or-shallower level begins. for i, head in enumerate(heads): - for nxt in heads[i + 1:]: + for nxt in heads[i + 1 :]: if nxt["level"] <= head["level"]: head["end"] = nxt["line"] - 1 break @@ -148,8 +156,8 @@ def _owning(heads: list[dict], line_no: int) -> dict | None: # search — CLAUDE.md §0 step 3, in one command. # -------------------------------------------------------------------------------------------- -def search(term: str, *, path: Path | None = None, limit: int = 40, - context: int = 0) -> dict: + +def search(term: str, *, path: Path | None = None, limit: int = 40, context: int = 0) -> dict: """Case-insensitive hits, each reported under the item heading that owns it. Returns the DENOMINATOR too (lines and sections searched), because "no matches" only means @@ -157,9 +165,15 @@ def search(term: str, *, path: Path | None = None, limit: int = 40, """ path = path or log_path() if not path.is_file(): - return {"present": False, "path": str(path), "term": term, - "absent_reason": absence_note(path), "matches": [], - "lines": 0, "sections": 0} + return { + "present": False, + "path": str(path), + "term": term, + "absent_reason": absence_note(path), + "matches": [], + "lines": 0, + "sections": 0, + } text = path.read_text(encoding="utf-8", errors="ignore") lines = text.splitlines() heads = parse_sections(text) @@ -169,37 +183,58 @@ def search(term: str, *, path: Path | None = None, limit: int = 40, if needle not in line.lower(): continue own = _owning(heads, idx + 1) - row = {"line": idx + 1, "text": line.strip(), - "section": own["title"] if own else "(before the first heading)", - "section_line": own["line"] if own else 0, - "item": (own or {}).get("item")} + row = { + "line": idx + 1, + "text": line.strip(), + "section": own["title"] if own else "(before the first heading)", + "section_line": own["line"] if own else 0, + "item": (own or {}).get("item"), + } if context: lo, hi = max(0, idx - context), min(len(lines), idx + context + 1) row["context"] = [ln.rstrip() for ln in lines[lo:hi]] matches.append(row) truncated = len(matches) > limit - return {"present": True, "path": str(path), "term": term, "absent_reason": None, - "matches": matches[:limit], "total_matches": len(matches), - "truncated": truncated, "lines": len(lines), "sections": len(heads)} + return { + "present": True, + "path": str(path), + "term": term, + "absent_reason": None, + "matches": matches[:limit], + "total_matches": len(matches), + "truncated": truncated, + "lines": len(lines), + "sections": len(heads), + } def render_search(rep: dict) -> str: if not rep["present"]: return rep["absent_reason"] - head = (f"improvement log: {rep['path']}\n" - f"searched {rep['lines']} lines / {rep['sections']} sections for {rep['term']!r}") + head = ( + f"improvement log: {rep['path']}\n" + f"searched {rep['lines']} lines / {rep['sections']} sections for {rep['term']!r}" + ) if not rep["matches"]: # An honest empty, and it says so in words as well as in the exit code. - return (f"{head}\nNO MATCHING ITEMS. The log was read in full and nothing mentions " - f"{rep['term']!r} — this is a real absence of matches, not a missing file.") - out = [head, f"{rep['total_matches']} match(es)" - + (f", showing the first {len(rep['matches'])}" if rep["truncated"] else "")] + return ( + f"{head}\nNO MATCHING ITEMS. The log was read in full and nothing mentions " + f"{rep['term']!r} — this is a real absence of matches, not a missing file." + ) + out = [ + head, + f"{rep['total_matches']} match(es)" + + (f", showing the first {len(rep['matches'])}" if rep["truncated"] else ""), + ] last = None for m in rep["matches"]: if m["section"] != last: out.append("") - out.append(f"## {m['section']} (line {m['section_line']}" - + (f", item {m['item']}" if m["item"] else "") + ")") + out.append( + f"## {m['section']} (line {m['section_line']}" + + (f", item {m['item']}" if m["item"] else "") + + ")" + ) last = m["section"] out.append(f" {m['line']}: {m['text']}") for ctx in m.get("context") or []: @@ -211,6 +246,7 @@ def render_search(rep: dict) -> str: # append — CLAUDE.md §5, in one command. # -------------------------------------------------------------------------------------------- + def find_section(ref: str, heads: list[dict]) -> dict: """Resolve an item-ref to exactly ONE section, or refuse and say which candidates it saw. @@ -229,29 +265,46 @@ def find_section(ref: str, heads: list[dict]) -> dict: kind = f"heading containing {ref!r}" if len(hits) == 1: return {"ok": True, "section": hits[0]} - return {"ok": False, "section": None, "candidates": hits, "kind": kind, - "reason": ("no section matches" if not hits - else f"{len(hits)} sections match — the ref is ambiguous")} - - -def append_note(ref: str, note: str, *, path: Path | None = None, - today: str | None = None) -> dict: + return { + "ok": False, + "section": None, + "candidates": hits, + "kind": kind, + "reason": ( + "no section matches" + if not hits + else f"{len(hits)} sections match — the ref is ambiguous" + ), + } + + +def append_note(ref: str, note: str, *, path: Path | None = None, today: str | None = None) -> dict: """Append one dated status note at the END of the matched section. Atomic, with one backup.""" path = path or log_path() if not path.is_file(): - return {"ok": False, "absent": True, "path": str(path), - "reason": absence_note(path)} + return {"ok": False, "absent": True, "path": str(path), "reason": absence_note(path)} if not note.strip(): - return {"ok": False, "absent": False, "path": str(path), - "reason": "refusing to append an empty note"} + return { + "ok": False, + "absent": False, + "path": str(path), + "reason": "refusing to append an empty note", + } text = path.read_text(encoding="utf-8") heads = parse_sections(text) found = find_section(ref, heads) if not found["ok"]: - return {"ok": False, "absent": False, "path": str(path), "ref": ref, - "candidates": [{"title": h["title"], "line": h["line"], "item": h["item"]} - for h in found["candidates"]], - "reason": f"{found['reason']} for {found['kind']}"} + return { + "ok": False, + "absent": False, + "path": str(path), + "ref": ref, + "candidates": [ + {"title": h["title"], "line": h["line"], "item": h["item"]} + for h in found["candidates"] + ], + "reason": f"{found['reason']} for {found['kind']}", + } sec = found["section"] lines = text.splitlines() stamp = today or datetime.datetime.now(datetime.timezone.utc).date().isoformat() @@ -264,7 +317,7 @@ def append_note(ref: str, note: str, *, path: Path | None = None, new = lines[:at] + ["", entry] + lines[at:] body = "\n".join(new) + "\n" backup = path.with_suffix(path.suffix + ".prev") - shutil.copy2(path, backup) # ONE rolling backup: this history is unversioned. + shutil.copy2(path, backup) # ONE rolling backup: this history is unversioned. fd, tmp = tempfile.mkstemp(dir=str(path.parent), prefix=".improvement-log-", suffix=".tmp") try: with os.fdopen(fd, "w", encoding="utf-8") as fh: @@ -273,9 +326,16 @@ def append_note(ref: str, note: str, *, path: Path | None = None, except BaseException: Path(tmp).unlink(missing_ok=True) raise - return {"ok": True, "absent": False, "path": str(path), "section": sec["title"], - "section_line": sec["line"], "inserted_at": at + 2, "entry": entry, - "backup": str(backup)} + return { + "ok": True, + "absent": False, + "path": str(path), + "section": sec["title"], + "section_line": sec["line"], + "inserted_at": at + 2, + "entry": entry, + "backup": str(backup), + } def render_append(rep: dict) -> str: @@ -288,11 +348,15 @@ def render_append(rep: dict) -> str: out.append("candidates (give a longer, unambiguous ref):") out += [f" line {c['line']}: {c['title']}" for c in cands[:10]] else: - out.append(f"run `improvement_log.py search ` against {rep['path']} to find the " - f"item, then use its number or a distinctive phrase from its heading.") + out.append( + f"run `improvement_log.py search ` against {rep['path']} to find the " + f"item, then use its number or a distinctive phrase from its heading." + ) return "\n".join(out) - return (f"appended to {rep['path']}\n section: {rep['section']} (line {rep['section_line']})\n" - f" line {rep['inserted_at']}: {rep['entry']}\n backup: {rep['backup']}") + return ( + f"appended to {rep['path']}\n section: {rep['section']} (line {rep['section_line']})\n" + f" line {rep['inserted_at']}: {rep['entry']}\n backup: {rep['backup']}" + ) # -------------------------------------------------------------------------------------------- @@ -338,8 +402,12 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: # A SUBPROCESS on purpose: the assertions below are about what a CALLER receives from the # CLI — text and exit code — not about what an internal helper returns. env = dict(os.environ, **{ENV_DIRECT: log}) - return subprocess.run([sys.executable, str(Path(__file__).resolve()), *argv], - capture_output=True, text=True, env=env) + return subprocess.run( + [sys.executable, str(Path(__file__).resolve()), *argv], + capture_output=True, + text=True, + env=env, + ) with tempfile.TemporaryDirectory() as td: log = Path(td) / LOG_NAME @@ -351,10 +419,12 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: # 1. search finds a term and reports the ITEM that owns it. p = run(["search", "thompson"], str(log)) check("search exits 0 on a hit", p.returncode == EXIT_OK, f"rc={p.returncode}") - check("search names the owning section", "7a. Cost-aware scoring" in p.stdout, - p.stdout[-200:]) - check("search reports the denominator", "sections for 'thompson'" in p.stdout, - p.stdout[:200]) + check( + "search names the owning section", "7a. Cost-aware scoring" in p.stdout, p.stdout[-200:] + ) + check( + "search reports the denominator", "sections for 'thompson'" in p.stdout, p.stdout[:200] + ) check("search finds every occurrence", "3 match(es)" in p.stdout, p.stdout[:200]) # 2. An HONEST empty: different exit code, and it says the file WAS read. @@ -367,8 +437,11 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: p = run(["search", "thompson"], str(missing)) check("absent log exits 2", p.returncode == EXIT_ABSENT, f"rc={p.returncode}") check("absence names the path", str(missing) in p.stdout, p.stdout[:300]) - check("absence names both env vars", - ENV_DIRECT in p.stdout and ENV_RUNTIME in p.stdout, p.stdout[:300]) + check( + "absence names both env vars", + ENV_DIRECT in p.stdout and ENV_RUNTIME in p.stdout, + p.stdout[:300], + ) check("absence is not an empty result", p.stdout.strip() != "", "empty stdout") # 4. append lands the note INSIDE the referenced item, dated. The target is item 4, which @@ -378,14 +451,23 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: p = run(["append", "4", "wired and verified"], str(log)) check("append exits 0", p.returncode == EXIT_OK, f"rc={p.returncode}\n{p.stderr[-300:]}") after = log.read_text(encoding="utf-8") - check("append wrote a dated note", "**STATUS " in after and "wired and verified" in after, - after[-200:]) + check( + "append wrote a dated note", + "**STATUS " in after and "wired and verified" in after, + after[-200:], + ) head, _, rest = after.partition("## ✅ 7. Telemetry integrity") - check("note is INSIDE item 4, above the next heading", "wired and verified" in head, - f"landed after item 4: {rest[-160:]!r}") + check( + "note is INSIDE item 4, above the next heading", + "wired and verified" in head, + f"landed after item 4: {rest[-160:]!r}", + ) check("note did not land at end of file", "wired and verified" not in rest, rest[-160:]) - check("append did not touch other items", - after.count("Thompson sampling was wired here and is DONE.") == 1, "duplicated") + check( + "append did not touch other items", + after.count("Thompson sampling was wired here and is DONE.") == 1, + "duplicated", + ) check("append left one backup", (Path(td) / f"{LOG_NAME}.prev").is_file(), "no .prev") # 5. A ref that matches many REFUSES and changes nothing. @@ -399,8 +481,9 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: p = run(["append", "99", "no such item"], str(log)) check("unknown ref refuses", p.returncode == EXIT_REFUSED, f"rc={p.returncode}") check("unknown ref suggests search", "search" in p.stdout, p.stdout[:300]) - check("unknown ref wrote nothing", log.read_text(encoding="utf-8") == before, - "file changed") + check( + "unknown ref wrote nothing", log.read_text(encoding="utf-8") == before, "file changed" + ) # 7. append to an absent log names the absence and does NOT create the file. p = run(["append", "9", "note"], str(missing)) @@ -412,10 +495,13 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: check("path exits 2 when absent", p.returncode == EXIT_ABSENT, f"rc={p.returncode}") try: doc = json.loads(p.stdout) - except Exception: # noqa: BLE001 + except Exception: # noqa: BLE001 doc = {} - check("path reports resolved + present", doc.get("path") == str(missing) - and doc.get("present") is False, p.stdout[:200]) + check( + "path reports resolved + present", + doc.get("path") == str(missing) and doc.get("present") is False, + p.stdout[:200], + ) print(f"improvement_log selftest: {len(failures)} failure(s)") for f in failures: @@ -427,9 +513,11 @@ def run(argv: list[str], log: str) -> subprocess.CompletedProcess: # CLI # -------------------------------------------------------------------------------------------- + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser( - description="Search and append this instance's improvement log (machine-local evidence).") + description="Search and append this instance's improvement log (machine-local evidence)." + ) ap.add_argument("--selftest", action="store_true") sub = ap.add_subparsers(dest="cmd") @@ -459,9 +547,14 @@ def main(argv: list[str] | None = None) -> int: if args.cmd == "path": target = log_path() present = target.is_file() - doc = {"path": str(target), "present": present, - "env": {ENV_DIRECT: os.environ.get(ENV_DIRECT), - ENV_RUNTIME: os.environ.get(ENV_RUNTIME)}} + doc = { + "path": str(target), + "present": present, + "env": { + ENV_DIRECT: os.environ.get(ENV_DIRECT), + ENV_RUNTIME: os.environ.get(ENV_RUNTIME), + }, + } if args.json: print(json.dumps(doc, indent=2)) else: diff --git a/test_improvement_log.py b/test_improvement_log.py index ba7b2bf..a82d03c 100644 --- a/test_improvement_log.py +++ b/test_improvement_log.py @@ -19,6 +19,7 @@ These are cheap, and they are the only thing that survives the next session. """ + from __future__ import annotations import pathlib @@ -40,22 +41,28 @@ def test_tracked_pointer_stays_a_pointer_and_never_becomes_the_log(): """The 481 KB of machine-local evidence must never arrive at this tracked path.""" - assert POINTER.is_file(), f"{POINTER.name} must be tracked in the tree — it is what makes the " \ - f"machine-local log discoverable from a worktree" + assert POINTER.is_file(), ( + f"{POINTER.name} must be tracked in the tree — it is what makes the " + f"machine-local log discoverable from a worktree" + ) size = POINTER.stat().st_size assert size <= POINTER_MAX_BYTES, ( f"{POINTER.name} is {size} bytes, over the {POINTER_MAX_BYTES}-byte pointer limit. This file " f"is a POINTER; the log itself is machine-local evidence and must not be committed. Append " - f"with `python3 {ACCESSOR} append \"\"` instead.") + f'with `python3 {ACCESSOR} append ""` instead.' + ) sections = len(re.findall(r"(?m)^##\s", POINTER.read_text(encoding="utf-8"))) - assert sections <= POINTER_MAX_SECTIONS, ( - f"{POINTER.name} has {sections} `##` sections — it is turning into the log it points at.") + assert ( + sections <= POINTER_MAX_SECTIONS + ), f"{POINTER.name} has {sections} `##` sections — it is turning into the log it points at." def test_tracked_pointer_names_the_accessor_and_both_rules(): """A pointer that does not name the accessor leaves a worktree as blind as before.""" text = POINTER.read_text(encoding="utf-8") - assert ACCESSOR in text, f"the pointer must name {ACCESSOR} — it is the only way to reach the log" + assert ( + ACCESSOR in text + ), f"the pointer must name {ACCESSOR} — it is the only way to reach the log" for cmd in ("search", "append"): assert f"{ACCESSOR} {cmd}" in text, f"the pointer must show `{ACCESSOR} {cmd}`" assert "ORCH_LOCAL_RUNTIME" in text, "the pointer must say WHERE the log lives" @@ -65,17 +72,22 @@ def test_claude_md_rules_name_the_accessor_not_a_bare_path(): """§0 step 3 and §5 are the two rules the accessor exists to make followable.""" text = CLAUDE_MD.read_text(encoding="utf-8") # §0 step 3 — the dedup check. - step3 = [ln for ln in text.splitlines() if ln.lstrip().startswith("3. ") - and "improvement log" in ln.lower()] + step3 = [ + ln + for ln in text.splitlines() + if ln.lstrip().startswith("3. ") and "improvement log" in ln.lower() + ] assert step3, "CLAUDE.md §0 step 3 must tell the reader to search the improvement log" dedup = text.split("## 0. Dedup-before-develop", 1)[-1].split("## 1. Editing", 1)[0] assert f"{ACCESSOR} search" in dedup, ( f"CLAUDE.md §0 must name `{ACCESSOR} search` — a bare path is unreadable from a worktree, " - f"which is what made this mandatory step unfollowable") + f"which is what made this mandatory step unfollowable" + ) # §5 — the status note. keep_true = text.split("## 5. Keep the docs true", 1)[-1] - assert f"{ACCESSOR} append" in keep_true, ( - f"CLAUDE.md §5 must name `{ACCESSOR} append` rather than telling the reader to edit a file") + assert ( + f"{ACCESSOR} append" in keep_true + ), f"CLAUDE.md §5 must name `{ACCESSOR} append` rather than telling the reader to edit a file" def test_accessor_reports_a_named_absence_to_a_caller(): @@ -88,10 +100,13 @@ def test_accessor_reports_a_named_absence_to_a_caller(): """ missing = HERE / "no-such-dir-for-tests" / "IMPROVEMENT_BACKLOG.md" assert not missing.exists() - proc = subprocess.run([sys.executable, str(HERE / ACCESSOR), "search", "anything"], - capture_output=True, text=True, cwd=str(HERE), - env={"PATH": "/usr/bin:/bin", "HOME": str(HERE), - "ORCH_IMPROVEMENT_LOG": str(missing)}) + proc = subprocess.run( + [sys.executable, str(HERE / ACCESSOR), "search", "anything"], + capture_output=True, + text=True, + cwd=str(HERE), + env={"PATH": "/usr/bin:/bin", "HOME": str(HERE), "ORCH_IMPROVEMENT_LOG": str(missing)}, + ) out = proc.stdout + proc.stderr assert proc.returncode == 2, f"an absent log must exit 2, not {proc.returncode}: {out[:300]}" assert str(missing) in out, "the absence must name the path it looked for" From 596a62215bab72447e7188b40d4d677c931508d2 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 23 Aug 2026 14:21:31 +0000 Subject: [PATCH 4/7] chore(codex-autofix): apply updates (PR #63) --- langsmith-fleet-worker-attempt.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/langsmith-fleet-worker-attempt.json b/langsmith-fleet-worker-attempt.json index 18b24e5..0682811 100644 --- a/langsmith-fleet-worker-attempt.json +++ b/langsmith-fleet-worker-attempt.json @@ -1,13 +1,13 @@ { "agent": "codex", "cli_version": "0.144.1", - "emitted_at": "2026-08-23T14:07:18.355766Z", + "emitted_at": "2026-08-23T14:21:29.115032Z", "execution_profile": "codex-default", "fallback_models": [ "gpt-5.5" ], "operation_role": "worker", - "pr_number": "61", + "pr_number": "63", "requested_model": "gpt-5.6-terra", "runner": "reusable-codex-run", "schema": "langsmith-fleet/v1", From e1175b5143fdfe3ef70d4d4bdcf960d01afe87d9 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 23 Aug 2026 14:25:18 +0000 Subject: [PATCH 5/7] chore(codex-autofix): apply updates (PR #63) --- langsmith-fleet-worker-attempt.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/langsmith-fleet-worker-attempt.json b/langsmith-fleet-worker-attempt.json index 0682811..48aa55d 100644 --- a/langsmith-fleet-worker-attempt.json +++ b/langsmith-fleet-worker-attempt.json @@ -1,7 +1,7 @@ { "agent": "codex", "cli_version": "0.144.1", - "emitted_at": "2026-08-23T14:21:29.115032Z", + "emitted_at": "2026-08-23T14:25:15.902470Z", "execution_profile": "codex-default", "fallback_models": [ "gpt-5.5" From f37d432cd1dd924de6eac0a31617129f7a4f462f Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 09:34:24 -0500 Subject: [PATCH 6/7] test(provenance): make the deliberate break self-diagnosing CodeRabbit thread 3838730241 on #63, verified: the break keyed on an exact substring including the trailing space, so a reformatted query would make `str.replace` a silent no-op. The filter would keep protecting the row, the corruption assertion would fail, and its message would blame the fix rather than the stale fixture. It fails RED either way -- no false green -- but it misdiagnoses, and in this repo a check has to name its own cause. Two guards, because there are two ways to go stale, and only one was proposed: 1. The clause MOVES within a still-recognisable query -- asserted per statement, naming the clause it expected and printing the actual SQL. 2. The query itself becomes UNRECOGNISABLE, so nothing is ever stripped -- caught after the run by `stripped`. The review's guarded snippet asserts only WHEN the FROM/JOIN + ORDER BY signature matches, so a rewrite that changes the signature leaves the break silently inert and its assert never runs. That is the same hole one level up. Not the review's first proposal, which is tautological: `broken != sql or clause not in sql` cannot fail, since `clause in sql` makes `broken != sql` necessarily true and the other branch covers the rest. Its second snippet is the right shape and is what guard 1 implements. The candidate query is identified by its FROM/JOIN plus ORDER BY fragments; the not-terminal count query shares the FROM/JOIN but ends in GROUP BY, so it is not mistaken for the candidate and other SQL passes through untouched. Both guards demonstrated then reverted: clause "unresolved" -> " = 'unresolved' " => "deliberate break is STALE: the candidate query no longer contains ... Update this fixture" + the real SQL ORDER BY r.ts DESC -> ASC (valid rewrite) => "deliberate break never fired: nothing matched the candidate query's ... the assertions below prove nothing" No new test, so the floor stays 392 (re-measured: 392 collected). verify.py: 392 passed, 0 failed, 0 skipped, 84/84 selftests, 5/5 gates. Clean under #60's incoming ruff config and black -l 100. Co-Authored-By: Claude Opus 5 --- test_feedback_model_provenance.py | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/test_feedback_model_provenance.py b/test_feedback_model_provenance.py index db5fc59..55da99c 100644 --- a/test_feedback_model_provenance.py +++ b/test_feedback_model_provenance.py @@ -936,14 +936,36 @@ def _row(run_id): assert report["excluded_not_terminal"] == {"started": 1, "failed": 1} # DELIBERATE BREAK: drop the status clause, restoring the pre-fix query exactly. + # + # This is keyed on an EXACT substring, trailing space included, so it has to fail loudly + # when that substring stops matching. A reformatted query would make `replace` a silent + # no-op: the filter would keep protecting the row, and the corruption assertion below would + # fail while blaming the fix rather than this fixture. Two guards, because there are two + # ways to go stale -- the clause moves within a query still recognisable (caught per + # statement), or the query itself becomes unrecognisable so nothing is ever stripped + # (caught by `stripped` after the run). Only the second covers an aliased or re-ordered + # rewrite, which is why the per-statement assert alone is not enough. real_conn = feedback._conn + clause = "AND ea.status='unresolved' " + stripped = [] class _Unfiltered: def __init__(self, c): self._c = c def execute(self, sql, *a): - return self._c.execute(sql.replace("AND ea.status='unresolved' ", ""), *a) + # The candidate query is the only statement carrying the clause. Identify it by + # the FROM/JOIN and ORDER BY fragments -- the not-terminal count query shares the + # FROM/JOIN but ends in GROUP BY, so it is not mistaken for the candidate. + if "FROM execution_attempts ea JOIN runs r" in sql and "ORDER BY r.ts DESC" in sql: + assert clause in sql, ( + f"deliberate break is STALE: the candidate query no longer contains " + f"{clause!r}, so stripping it does nothing and the corruption assertion " + f"below would blame the fix. Update this fixture. SQL: {sql!r}" + ) + stripped.append(sql) + return self._c.execute(sql.replace(clause, ""), *a) + return self._c.execute(sql, *a) def __getattr__(self, name): return getattr(self._c, name) @@ -957,6 +979,11 @@ def __exit__(self, *exc): monkeypatch.setattr(feedback, "_conn", lambda: _Unfiltered(real_conn())) ledger_reconcile.resolve_unresolved_worker_attempts(apply=True) + assert stripped, ( + "deliberate break never fired: nothing matched the candidate query's FROM/JOIN + " + "ORDER BY signature, so no clause was stripped and the assertions below prove " + "nothing. Update this fixture to match the current query." + ) broken_status, broken_model, broken_completed = _row("sweep-inflight") assert (broken_status, broken_model) == ("complete", "gpt-5.6-terra"), ( "the break must reproduce the corruption: a running worker stamped with an " From ff17a28b707dc921618747c42302edd9fc0d4ca8 Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 14:11:34 -0500 Subject: [PATCH 7/7] fix(floor): repair a garbled note fragment from the concurrent merge The automated merge of origin/main spliced this branch's floor entry mid-token: "FLOOR 407 -> 408 on 2026-08-23 (CodeRabbit follow-up ..." became "3 on 2026-08-23 (CodeRabbit follow-up ...", losing which transition the entry records. The counts were already correct and agree with my own resolution of the same conflict (408/408, re-measured independently: `pytest --collect-only` reports 408 = main fc1fd42's 407 plus this branch's one test). Only the prose was damaged, but this note is the sole record of WHY each floor moved, so a fragment that no longer names its transition is exactly the kind of unreadable evidence this file exists to prevent. Kept the automated merge's #68 entry rather than my own wording: it documents the union more fully and gives the artifact resolution a provenance rationale -- main's NEWER langsmith-fleet worker-attempt record is retained, because discarding a newer provenance observation to win a merge would corrupt exactly the causal-provenance evidence CLAUDE.md section 2 protects. That is a better reason than the one I used ("it is meaningless either way"). Co-Authored-By: Claude Opus 5 --- .verify-floor.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.verify-floor.json b/.verify-floor.json index 9703a86..b5362b9 100644 --- a/.verify-floor.json +++ b/.verify-floor.json @@ -4,5 +4,5 @@ "skipped_max": 26, "selftest_skipped_max": 7, "gate_skipped_max": 2, - "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that). FLOOR 402 -> 407 on 2026-08-23 (findability admission requirement). (findability admission requirement). (findability admission requirement). (findability admission requirement). Exactly five new pytest tests, all in test_capability_admission.py: test_findability_distinguishes_its_three_sub_causes, test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt, test_unreadable_reach_is_not_evaluated_and_never_a_failure, test_findability_exemption_is_declared_in_code_not_in_a_live_ledger, test_consult_sites_are_falsifiable_claims_about_real_callers. NO CEILING MOVED and nothing new skips: all five build synthetic ledgers in a tempdir or read committed tables, so none needs a populated capability ledger, an agent CLI or ~/.claude/skills. The one machine-dependent thing they touch -- an external consult site declared in capability_advisor.CONSULT_SITES whose skill prompt is not on this machine -- is reported as UNVERIFIED rather than skipped, because absence of the caller is not refutation of the claim; the in-tree site (tick) is asserted verified on every machine so the check can never degrade into 'everything unverified, nothing tested'. Measured on the merge result per the rule above: this file CONFLICTED three times while the branch was open, as main went 387 -> 391 -> 402 (#61, #64, #65, #60). Each time it was resolved as the UNION rather than by taking a side, and the count was RE-MEASURED on the new merge result rather than either number being carried forward: 402 (main at bd6da2e) + 5 (this branch's new tests) = 407. That is the rule this file already states -- measure the floor on the merge result, not on the branch -- and it mattered here, because #60 both deleted test_ci_gate_config.py and added more than it removed, so guessing in either direction would have been wrong. 3 on 2026-08-23 (CodeRabbit follow-up on PR #42, thread 3837879039; re-measured again after #56 made `collected` an EQUALITY, which makes an assumed number a hard RED rather than a quiet pass -- main stayed at 402 across #56, and the merge result measures 403, so #56 added no collected tests and this is main's 402 plus this branch's one): exactly one new test, test_feedback_model_provenance.test_late_sweep_completes_terminal_attempts_never_one_in_flight, which pins that ledger_reconcile.resolve_unresolved_worker_attempts completes only TERMINAL unresolved worker attempts and never one still in flight. No ceiling moved and nothing new is skipped -- the test builds its own tmp ledger and codex rollout fixture and monkeypatches adapters.CODEX_SESSIONS, so it needs no agent CLI and no populated capability ledger and runs on a bare runner. RESOLVED AGAINST #59 (improvement-log accessor), which raised the floor 387 -> 391 on main while this branch raised it to 388: taken as the UNION -- #59's rationale is retained above and the count was RE-MEASURED on the new merge result rather than keeping either number, which is the rule this file states and the mistake that once put the floor 8 below reality. 391 (main, incl. #59's four tests) + 1 (this branch's one new test) = 392 measured, not assumed: 392 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. Three sibling follow-up branches are in flight against this same main (CI/ruff config, arm-attribution + durability, adapters label->ID); if this file conflicts with one of them, resolve as the UNION and RE-MEASURE on the new merge result rather than taking either number -- that is what #42 and #50 did, and taking a side is what put the floor 8 below reality earlier. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's one-test entry beside it -- and the count RE-MEASURED on the merge result. main fc1fd42 collects 407; this branch adds 1; 408 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped -- the one new test builds its own tmp ledger and codex rollout fixture, so it runs on a bare runner. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record differing only in `emitted_at` and `pr_number`; main's NEWER record was kept, since discarding a newer provenance observation to win a merge would corrupt the causal-provenance evidence CLAUDE.md 2 protects." + "note": "Recorded by verify.py --update-floor, except the *_max ceilings, which are edited BY HAND and never re-measured. `collected` catches tests that stopped being collected; `passed` is compared against passed+skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. The *_max ceilings bound the skipped side: 24/7/2 is exactly what a machine with none of this instance's local prerequisites skips (a GitHub runner: no agent CLIs, no ~/.codex/skills, no /Applications/ChatGPT.app, no populated capability ledger), measured 2026-08-21. On the owner's machine all prerequisites exist and nothing skips at all. Raising a ceiling is a deliberate act: it means agreeing that one more thing is allowed to go unchecked, so say which and why in the commit. LOWERED 26 -> 24 on 2026-08-22, reverting the raise made earlier the same day. The two kill-switch exemption tests no longer need to skip on a bare runner: their declarations moved out of the running instance's ledger and into capabilities.KNOWN_DECLARATIONS, so they assert code-derived truth and run everywhere. Moving a test back below the ceiling is the preferred way to lower it -- fix what made it machine-dependent, rather than agreeing to check less. FLOOR 345 -> 353 on 2026-08-22: 345 was measured on a branch cut before #13 (research panels/rounds/domain studies) merged, so the recorded floor sat 8 tests BELOW what main actually collects. A floor below reality is the permissive direction -- those 8 could have silently stopped being collected and still cleared the check, which is exactly the hole this file exists to close. Measure the floor on the merge result, not on the branch. Raised again on 2026-08-22 by the producer-identity-scope branch, which adds tests on top of the 353 recorded by #15; re-measured after rebasing rather than assumed. NOTE: `verify.py --update-floor` REPLACES this note with a generic one, so it must be restored by hand after every use \u2014 the ceiling rationale is the only record of which prerequisite justifies each skip. FLOOR 365 -> 366 on 2026-08-22 (heartbeat-ordering work, PR #18): exactly one new test, test_capabilities.test_no_tick_producer_runs_above_the_heartbeat_export. No ceiling moved and nothing new is skipped -- it reads source files rather than a populated ledger, so it runs on any machine. The branch recorded 354 because it was cut before #16 merged; re-measured on the MERGE RESULT per the rule above, which is exactly the mistake that put the floor 8 below reality last time. FLOOR 366 -> 368 on 2026-08-23: main collected 368 while this file recorded 366, drift left by #34 (evidence-acquisition landed, +1) and #37 (tick capability evidence, +1) whose authors each measured against a branch cut before the other merged. A floor BELOW reality is the permissive direction this file exists to close -- those two could have silently stopped being collected and still cleared the check. Measured on the merge result per the rule above: 368 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. CEILING 24 -> 26 and FLOOR 368 -> 387 on 2026-08-23 (profiles/provenance branch, PR #42). This file CONFLICTED with #50, which raised the floor 366 -> 368 on main while this branch raised it to 387; resolved as the UNION rather than by taking a side -- #50's rationale is retained above and the count was RE-MEASURED on the new merge result instead of keeping either number. 368 (main) + 19 (this branch's net new tests) = 387; #50 corrected recorded drift rather than adding coverage, which is why 387 is unchanged from the pre-conflict measurement. Measured in a runner sandbox reproducing CI exactly (361 passed, 26 skipped, 387 collected) AND on the owner's machine (387 passed, 0 skipped, 5/5 gates). The two new skips are drift detectors against a REAL installed agent runtime, so neither can be moved below the ceiling -- the preferred way to lower one: (1) agy advertised-models cache absent, since comparing declared model ids against the catalogue agy actually advertises needs that catalogue, and a fixture would exercise the comparison while detecting no real drift; (2) vibe config absent (~/.vibe/config.toml), since active_model cannot be read to check for drift when there is no config to read. Both name their missing prerequisite, so a green run still states what it did not check. A third candidate skip was REFUSED: dispatcher's per-run agy-log assertion failed on a bare runner because adapters.advertised_models shells out to `agy models` when its disk cache is cold, and that probe landed inside a monkeypatched subprocess.run and overwrote the captured command. That is a stub leak, so it was fixed by ISOLATING the double rather than by skipping -- which makes CI run MORE. FLOOR 387 -> 391 on 2026-08-23 (improvement-log accessor, PR #59): exactly four new tests, all in test_improvement_log.py -- three read tracked files in the tree (the pointer's size and content, and that CLAUDE.md 0 step 3 and 5 name the accessor rather than a bare path) and one runs the accessor as a subprocess against a path that cannot exist. None reads a populated ledger, an agent CLI or ~/.codex, so all four RUN on a bare runner and NO ceiling moved: nothing new is skipped. Measured on the MERGE RESULT after rebasing onto origin/main af6654d, which collected 387 -- not on the branch base, per the rule above. FLOOR 391 -> 402 on 2026-08-23 (Gate python-ci configuration, the PR that adds the missing .github/workflows/autofix-versions.env): exactly 11 new tests, all in test_ci_gate_config.py, which read committed files only -- the pin file, ruff.toml, mypy.ini, pr-00-gate.yml's toggle annotations and docs/CI_LINT_BASELINE.md. NO ceiling moved. On any CHECKOUT -- CI, the owner's tree, a second instance -- all 11 run: they need no installed linter and no populated ledger. In the EXEC-MIRROR layout all 11 skip with one named reason, because orch-sync-mirror.sh copies root-level *.py only, so .github/workflows, docs/ and scripts/ are genuinely absent there (env_prereq.repo_files_absent). That lands at 11/26 on a machine that otherwise skips nothing, and CI stays at 26/26, so no ceiling needed raising. The skip gate is the presence of those DIRECTORIES, never of the pin file itself -- gating on the file would have made the test that checks for it unable to fail. Measured on the merge result, twice: the branch was rebuilt on origin/main after #42 and #59 merged, and re-measured after #61 merged and was merged in -- 393 passed + 9 skipped = 402 collected both times, so #61 added no collected tests and this floor is not sitting below reality. #61 itself left main's floor at 391, which is exactly main-without-these-11, so there is no inherited drift to correct. RULE CHANGE 2026-08-23: `collected` is now an EQUALITY, not a minimum. Every floor entry above this one records the number being found BELOW reality and hand-raised after the fact -- 21 low at the worst, then 8, then 1, then 2 -- because nothing ever required a test-adding PR to touch this file, so the permissive direction was silent by construction and the rule 'measure on the merge result' had to be restated three times with nothing enforcing it. verify.py now FAILS when collected exceeds the floor, printing the two integers to write. That also makes the concurrency case self-enforcing: once every test-adding branch must edit these same two lines, two concurrent branches CONFLICT IN GIT, so the second cannot merge without rebasing onto the first and re-measuring on the actual merge result. Demonstrated repeatedly on the change itself: six merges landed on main in the two hours it took to write, moving this file 368 -> 387 -> 391 -> 402, and every one would have left the floor below reality under the old one-directional rule. `passed` deliberately stays a MINIMUM on passed+skipped: only collection is machine-invariant (a skipped test is still collected), measured across machines at 391 collected on both, with pass/skip splits of 365/26 on CI against 391/0 locally. The *_max ceilings are untouched by this change and nothing new is skipped. `--update-floor` also stops REPLACING this note -- it appends -- so the warning above about restoring it by hand no longer applies; and drift does NOT block --update-floor, since a gate that forbade its own only remedy would be a deadlock (the first draft was exactly that). FLOOR 402 -> 407 on 2026-08-23 (findability admission requirement). (findability admission requirement). (findability admission requirement). (findability admission requirement). Exactly five new pytest tests, all in test_capability_admission.py: test_findability_distinguishes_its_three_sub_causes, test_findability_blocks_new_capabilities_and_reports_older_ones_as_debt, test_unreadable_reach_is_not_evaluated_and_never_a_failure, test_findability_exemption_is_declared_in_code_not_in_a_live_ledger, test_consult_sites_are_falsifiable_claims_about_real_callers. NO CEILING MOVED and nothing new skips: all five build synthetic ledgers in a tempdir or read committed tables, so none needs a populated capability ledger, an agent CLI or ~/.claude/skills. The one machine-dependent thing they touch -- an external consult site declared in capability_advisor.CONSULT_SITES whose skill prompt is not on this machine -- is reported as UNVERIFIED rather than skipped, because absence of the caller is not refutation of the claim; the in-tree site (tick) is asserted verified on every machine so the check can never degrade into 'everything unverified, nothing tested'. Measured on the merge result per the rule above: this file CONFLICTED three times while the branch was open, as main went 387 -> 391 -> 402 (#61, #64, #65, #60). Each time it was resolved as the UNION rather than by taking a side, and the count was RE-MEASURED on the new merge result rather than either number being carried forward: 402 (main at bd6da2e) + 5 (this branch's new tests) = 407. That is the rule this file already states -- measure the floor on the merge result, not on the branch -- and it mattered here, because #60 both deleted test_ci_gate_config.py and added more than it removed, so guessing in either direction would have been wrong. FLOOR 407 -> 408 on 2026-08-23 (CodeRabbit follow-up on PR #42, thread 3837879039; re-measured again after #56 made `collected` an EQUALITY, which makes an assumed number a hard RED rather than a quiet pass -- main stayed at 402 across #56, and the merge result measures 403, so #56 added no collected tests and this is main's 402 plus this branch's one): exactly one new test, test_feedback_model_provenance.test_late_sweep_completes_terminal_attempts_never_one_in_flight, which pins that ledger_reconcile.resolve_unresolved_worker_attempts completes only TERMINAL unresolved worker attempts and never one still in flight. No ceiling moved and nothing new is skipped -- the test builds its own tmp ledger and codex rollout fixture and monkeypatches adapters.CODEX_SESSIONS, so it needs no agent CLI and no populated capability ledger and runs on a bare runner. RESOLVED AGAINST #59 (improvement-log accessor), which raised the floor 387 -> 391 on main while this branch raised it to 388: taken as the UNION -- #59's rationale is retained above and the count was RE-MEASURED on the new merge result rather than keeping either number, which is the rule this file states and the mistake that once put the floor 8 below reality. 391 (main, incl. #59's four tests) + 1 (this branch's one new test) = 392 measured, not assumed: 392 passed, 0 failed, 0 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates. Three sibling follow-up branches are in flight against this same main (CI/ruff config, arm-attribution + durability, adapters label->ID); if this file conflicts with one of them, resolve as the UNION and RE-MEASURE on the new merge result rather than taking either number -- that is what #42 and #50 did, and taking a side is what put the floor 8 below reality earlier. RESOLVED AGAINST #68 (findability admission requirement) on 2026-08-23: taken as the UNION per the rule this file states -- #68's five-test entry is retained above and this branch's one-test entry beside it -- and the count RE-MEASURED on the merge result. main fc1fd42 collects 407; this branch adds 1; 408 measured with `pytest --collect-only -q` on the merge result, not assumed. Ceilings untouched at 26/7/2 and nothing new is skipped -- the one new test builds its own tmp ledger and codex rollout fixture, so it runs on a bare runner. Also resolved in the same merge: langsmith-fleet-worker-attempt.json, a CI-emitted `langsmith-fleet/v1` worker-attempt record differing only in `emitted_at` and `pr_number`; main's NEWER record was kept, since discarding a newer provenance observation to win a merge would corrupt the causal-provenance evidence CLAUDE.md 2 protects." }