From ca829e8d4db17cd9abfe5766b788d6d4c346d3df Mon Sep 17 00:00:00 2001 From: Tim Stranske Date: Sun, 23 Aug 2026 08:57:33 -0500 Subject: [PATCH 1/2] =?UTF-8?q?test(admission):=20restore=20the=20dropped?= =?UTF-8?q?=20commitments=20assertion=20=E2=80=94=20as=20the=20one=20it=20?= =?UTF-8?q?was=20protecting?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `capability_admission._selftest` computed `cited = {d["record"] for d in com["dangling_citations"]}` and asserted nothing on it, under a section header claiming "the real historical failure must be detected, not hypothetically detectable" and a comment saying the assertion "must never be quietly dropped". A value computed for a check that no longer checks is this repo's founding defect in miniature, so the binding was right to go — but the header's claim went unenforced with it. WHAT THE ASSERTION SHOULD BE. Not the one that was there. Two findings settle it: * `assert "2026-07-15-range-lane-trial-review.md" in cited` cannot be restored: the record has since been WRITTEN, so the live dangling set is empty wherever the ledger exists. This is exactly the "if someone fixes that line" branch the old comment anticipated. * `assert not cited` is not worth restoring either: `test_dated_promises_left_an_artifact` already asserts the strongest available claim on the live set — that BOTH lists are empty. A weaker copy in the selftest adds nothing. ...and the synthetic-file check below it is NOT sufficient, which is the real hole. `_probe_commitments` always passes an explicit `root=`, so NOTHING in the suite exercises `root or HERE`. A default that stopped resolving to the checkout returns `clean: True` over ZERO files — and the selftest's `isinstance(com["clean"], bool)` and the pytest emptiness assertion BOTH stay green on a scan that examined nothing. That is verify.py's vacuous zero-exit one level up. So `_probe_live_root` asserts what the binding was reaching for, on the real tree, off the same synthetic-empty-`AUDITS` harness that already makes `_probe_commitments` machine-independent (with no records in existence, every dated citation dangles, so the verdict is identical on the owner's machine and a bare runner): * NON-VACUITY — the default-root scan must surface the tree's own dated citations (4 today: CLAUDE.md, README.md, completion_event_adapter.py, and orchestrate.sh:109, the motivating line). * PROVENANCE — each report's file:line must actually contain the record it names. The probe only ever checks `record`, so a right-record/wrong-line report passes it, and "go look here" is this gate's entire output. * THE SKIP — this file must never appear in its own report. It cites two dated records in its own docstring because it documents the detector; a finding clearable only by deleting that documentation is a permanently-red gate, and a permanently-red gate gets switched off. * NO LEDGER, NO VERDICT — the fail-open branch must return "nothing found", never a verdict it could not compute. Pointed at a nonexistent path so it runs on the ledger machine too. DELIBERATE-BREAK -> REVERT, all four reverted clean: 1. `root or HERE` -> `root or (HERE / "config")`: non-vacuity fires. Under the SAME break, `_probe_commitments` PASSES and `test_dated_promises_left_an_artifact` PASSES (1 passed) — which is the proof the hole was real and unguarded. 2. `enumerate(splitlines(), 1)` -> `, 0)`: provenance fires, naming CLAUDE.md:143. 3. `SKIP_NAMES` emptied: the self-report check fires. FIRST DRAFT DID NOT — it asserted `not reported & SKIP_NAMES`, comparing the report against the very set whose failure it was meant to catch, so emptying the set made it vacuously true. Caught only by the break test, and rewritten as a literal. Worth recording: the first attempt at the fix reproduced the defect. 4. the `skipped` key removed from the no-ledger return: the fail-open check fires. Dedup (CLAUDE.md 0), recorded in the `capability-admission-gate` ledger notes: grepped by concept for scan-reaches-the-tree / vacuity / provenance coverage. The only three things that assert on `commitments()` are `_probe_commitments` (synthetic root), the selftest shape check, and the pytest emptiness test. Not present; extended the existing gate rather than adding one. No new capability, no rail/role change, so ARCHITECTURE.md and the diagram are unaffected. Verified with `python3 verify.py`: 387 passed, 0 failed, 0/26 skipped, 83/83 selftests, 43/43 can-fire, 5/5 gates, commitments clean. Floor and ceilings untouched — no new pytest test and no new skip, so nothing moved. Adds 0.13s to the selftest. `ruff check --select E4,E7,E9,F` and `black --check --line-length 100` clean, so this does not re-introduce the F841 that removed the binding. Co-Authored-By: Claude Opus 5 --- capability_admission.py | 89 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 84 insertions(+), 5 deletions(-) diff --git a/capability_admission.py b/capability_admission.py index dc60147..38d0287 100644 --- a/capability_admission.py +++ b/capability_admission.py @@ -627,6 +627,76 @@ def _probe_commitments(probe: pathlib.Path) -> None: assert not commitments(root=probe)["overdue_without_record"] +def _probe_live_root(probe: pathlib.Path) -> None: + """Prove the DEFAULT-root scan reaches THIS TREE, and reports what it finds accurately. + + Called with `AUDITS` already pointed at the synthetic empty record set, so every dated record + cited in the tree dangles by construction and the verdicts are identical on every machine. + + WHY THIS EXISTS, and why it is not a second copy of `_probe_commitments`. `_selftest` used to + compute `cited = {d["record"] for d in commitments()["dangling_citations"]}` and assert nothing + on it; the dead binding was removed as an unused local, leaving the section header's claim — + "the real historical failure must be detected, not hypothetically detectable" — unenforced. + The assertion it once carried named the range-lane record cited by `orchestrate.sh`, and it + CANNOT be restored as it stood: that record was eventually written, so the live dangling set is + now empty where the ledger exists. Nor is `assert not cited` worth restoring — that is a weaker + copy of `test_capability_admission.test_dated_promises_left_an_artifact`, which already asserts + the strongest available claim on the live set, that BOTH lists are empty. + + What neither of those covers is the hole this closes. `_probe_commitments` always passes an + explicit `root=`, so nothing in the suite exercises `root or HERE`. A default that stopped + resolving to the checkout — module relocated, `SCAN_SUFFIXES` narrowed, `SKIP_NAMES` widened, + `iterdir` over the wrong directory — returns `clean: True` over ZERO files. The live selftest + assertion (`isinstance(com["clean"], bool)`) and the pytest emptiness assertion would BOTH stay + green forever on a scan that examined nothing, which is `verify.py`'s vacuous zero-exit one + level up: the check runs, reads green, and looks at nothing. So what has to surface here is the + real tree's own citations — `orchestrate.sh`'s among them — not a synthetic `fake.sh`. + """ + live = commitments()["dangling_citations"] + # NON-VACUITY. Against a record set where nothing exists, every dated citation in the tree + # dangles, so this set is empty only if the scan read no files. Should the tree ever + # legitimately stop citing dated records, the live check has genuinely become vacuous and this + # anchor needs re-pointing — that is the loud failure, and it is the intended one. + assert live, ( + "the default-root scan surfaced no dated-record citations, against a record set in which " + f"every one of them dangles — so `commitments()` examined nothing under {HERE}. Check " + "`root or HERE`, SCAN_SUFFIXES and SKIP_NAMES before touching this assertion." + ) + # PROVENANCE. `_probe_commitments` only ever checks `record`, so a report that names the right + # record at the wrong file:line passes it. That report is unactionable — the whole output of + # this gate is "go look here" — and misattribution is invisible from the record name alone. + for d in live: + src = HERE / d["file"] + assert src.is_file(), f"dangling report names a file that is not in this tree: {d}" + lines = src.read_text(encoding="utf-8", errors="ignore").splitlines() + assert 1 <= d["line"] <= len(lines), f"line number falls outside {d['file']}: {d}" + assert d["record"] in lines[d["line"] - 1], f"citation misattributed to a line: {d}" + # THE SKIP MUST STILL HOLD on the live path, stated as a LITERAL rather than as + # `reported & SKIP_NAMES` — that first draft compared the report against the very set whose + # failure it was meant to catch, so emptying SKIP_NAMES made it vacuously true and the break + # test caught nothing. This file cites two dated records in its own docstring because it + # documents the detector; were it ever scanned it would report ITSELF, and a finding that can + # only be cleared by deleting the detector's documentation is a permanently-red gate, which + # gets switched off. So the concrete fact is asserted, independent of the mechanism. + reported = {d["file"] for d in live} + assert pathlib.Path(__file__).name not in reported, ( + f"the detector reported its own documented examples: {sorted(reported)}. " + f"SKIP_NAMES must keep {pathlib.Path(__file__).name} out of its own scan." + ) + # NO LEDGER, NO VERDICT — the other reason a live-set assertion cannot be restored, pinned so + # it stays a deliberate fail-open rather than an accident. Absence must return "nothing found", + # never a verdict it could not compute. Pointed at a path that does not exist, so this runs on + # the ledger machine too instead of only where the ledger happens to be missing. + saved = globals()["AUDITS"] + globals()["AUDITS"] = probe / "no-such-ledger" + try: + absent = commitments() + finally: + globals()["AUDITS"] = saved + assert absent.get("skipped") and absent["clean"], absent + assert not absent["dangling_citations"] and not absent["overdue_without_record"], absent + + def _selftest() -> None: ledger = capabilities.load_declared(capabilities.REG) assert ledger, "ledger must load" @@ -688,12 +758,16 @@ def _selftest() -> None: assert cid in ledger, f"waiver names unknown capability {cid}" # COMMITMENTS: the real historical failure must be detected, not hypothetically detectable. + # The LIVE verdict is asserted in test_capability_admission.test_dated_promises_left_an_artifact + # (both lists empty); here the live call only has to answer at all. com = commitments() - cited = {d["record"] for d in com["dangling_citations"]} assert isinstance(com["clean"], bool) - # orchestrate.sh cites the range-lane review record that was never written. If someone fixes - # that line, this assertion should be updated — but it must never be quietly dropped, so the - # check below proves the DETECTOR works using a synthetic file either way. + # `orchestrate.sh` cites the range-lane review record. That record HAS since been written, so + # the assertion that once named it here is retired on purpose rather than quietly dropped — + # `_probe_live_root` carries what it was actually protecting: that the default-root scan reads + # the real tree and attributes what it finds correctly. Read its docstring before changing + # either block; between them they cover the detector (synthetic input) and the live wiring + # (real input), and dropping one leaves the other passing over nothing. import tempfile with tempfile.TemporaryDirectory(prefix="cap-adm-") as td: @@ -706,6 +780,9 @@ def _selftest() -> None: # has no audit ledger: the first CI run died right here. # 2. Even where the ledger exists, the verdicts below depended on which records happen # to be in it. A synthetic empty set makes all four deterministic everywhere. + # `_probe_live_root` runs off the SAME swap for the same reason, one input further out: it + # scans the real tree, and only an empty record set makes "every dated citation dangles" + # true on the owner's machine and a bare runner alike. # This is the harness, not an assertion: every assert below is unchanged, and now runs on # any machine instead of only on this one. audits_probe = probe / "audits" @@ -714,6 +791,7 @@ def _selftest() -> None: globals()["AUDITS"] = audits_probe try: _probe_commitments(probe) + _probe_live_root(probe) finally: globals()["AUDITS"] = saved_audits @@ -739,7 +817,8 @@ def _selftest() -> None: env_prereq.report_gaps("capability_admission.py", gaps) print( "capability_admission.py selftest: OK (every requirement can fail and can pass, " - "grandfathering visible, waivers expire, dangling + overdue commitments detected)" + "grandfathering visible, waivers expire, dangling + overdue commitments detected, " + "live-tree scan proven non-vacuous and correctly attributed)" + (f" — {len(set(gaps))} section(s) skipped, see above" if gaps else "") ) From 6ea7a1c51f4815070f5e0cb47ce28627dd7ef9b6 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sun, 23 Aug 2026 14:21:31 +0000 Subject: [PATCH 2/2] chore(codex-autofix): apply updates (PR #67) --- langsmith-fleet-worker-attempt.json | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 langsmith-fleet-worker-attempt.json diff --git a/langsmith-fleet-worker-attempt.json b/langsmith-fleet-worker-attempt.json new file mode 100644 index 0000000..d584f41 --- /dev/null +++ b/langsmith-fleet-worker-attempt.json @@ -0,0 +1,16 @@ +{ + "agent": "codex", + "cli_version": "0.144.1", + "emitted_at": "2026-08-23T14:21:27.477082Z", + "execution_profile": "codex-default", + "fallback_models": [ + "gpt-5.5" + ], + "operation_role": "worker", + "pr_number": "67", + "requested_model": "gpt-5.6-terra", + "runner": "reusable-codex-run", + "schema": "langsmith-fleet/v1", + "selected_model": "gpt-5.6-terra", + "selection_reason": "input" +}