From 9c6cae4aac8737c236d6a3b28c8de12792558773 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:20:48 +1200 Subject: [PATCH 01/16] docs(corpus): plan for issue #636 -- launchpad/docs/corpus/AGENTS.md Reviewed by an independent cross-model review-plan pass (Codex, gpt-5.6-sol, high effort): eleven findings, all applied. Three more were found afterwards, the load-bearing one being that the plan blocked its own STEP 2 on an OPEN item -- an OPEN item a step must consume is a missing input, not an open question. Refs #636 Signed-off-by: Serina Mcfall --- .../2026-08-26-issue-636-corpus-agents-md.md | 245 ++++++++++++++++++ 1 file changed, 245 insertions(+) create mode 100644 launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md diff --git a/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md b/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md new file mode 100644 index 00000000000..20e9f919b4e --- /dev/null +++ b/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md @@ -0,0 +1,245 @@ +Issue #636 — task: document AGENTS.md +Stated size: none given (no `Size` line/label) → asked Serina; she chose 30–60 minutes → cap: 8 steps + +Revised 2026-08-26 after an independent cross-model review-plan pass (Codex, gpt-5.6-sol, +high effort) against the first draft. Eleven findings; all applied. That pass also refuted +one finding from the author's own self-review — see OPEN item 4 for what was withdrawn and +why, because the withdrawn reasoning is the kind that comes back. + +ALREADY TRUE (verified against git at 0052f5a7820ca4ca261efa233feb8bb53858ade6) + #622 is merged: launchpad/docs/corpus/schema/node.schema.json is the frontmatter + contract this document must describe rather than restate. Required fields are + id, type, status, origin, audiences, evidence; relationships is optional; and + `additionalProperties` is false, so the frontmatter has no field outside that set. + The schema's own description of `evidence` calls it "The node's provenance ledger." + There is no separate `provenance` property and adding one is a schema violation — + verified: injecting `provenance:` yields "Additional properties are not allowed + ('provenance' was unexpected)". The ledger IS the provenance mechanism. + #623 is merged (PR #1422): launchpad/project-intelligence/corpus/validate.py exists + and `just corpus-validate` runs it (Justfile:1004-1005). DEFAULT_ROOT is + "launchpad/docs/corpus". `just corpus-validate` passes NO `--root`, so it always + validates the real corpus and can never be pointed at a scratch tree. + The scan excludes exactly one top-level directory by name, `schema/`. A new + launchpad/docs/corpus/AGENTS.md WILL therefore be scanned and must carry + schema-valid frontmatter. Not optional for this file. + launchpad/docs/corpus/ today contains only `schema/` — ZERO real corpus content nodes. + AGENTS.md is the first, so it can cite no sibling node as an example and can declare + no `relationships` target: find_unresolved_relationship_targets rejects any target + no loaded node's id matches, and today no such id exists. + The validator does not read the Markdown body at all — `_, frontmatter, _body = + text.split("---\n", 2)`. Every claim about body content, links, scope or one-concept + discipline is enforced by a human or not at all. + Citation checking is STRUCTURAL, not evidential. A bare path is checked to resolve to a + real file inside the repo (is_file(), so a directory is rejected). A `path:line` + citation is NOT checked against the file's length — `Justfile:999999` returns `ok` + against a 1005-line file; filed as #1459, and this task avoids the positional forms + because of it. A GitHub blob/raw URL must be pinned to a full 40-char SHA, but a + pinned link to a real file proves only that the URL is well-formed. A non-GitHub URL + and a `commit ` reference both land on the non-fatal UNVERIFIED channel. + Nothing the validator reports means a citation SUPPORTS the statement it sits under. + find_ownership_violations fails closed on every non-.md file under the corpus root, + including files under `generated/`, until #1316 defines the provenance contract. It + says nothing about how MANY .md nodes exist — five valid nodes validate together — + so the DoD's "exactly one hand-authored document" is not a property any tool checks. + .github/workflows/launchpad-corpus-validate.yml triggers on `launchpad/docs/corpus/**` + for pull_request and push-to-launchpad. This change is CI-gated with no workflow edit. + This file will not only be a corpus node. `launchpad/scripts/preflight_core.py` sets + `RULES_FILENAMES = ("AGENTS.md", "CLAUDE.md")` (line 164) and resolves the NEAREST one + per changed path into `nearest_rules`, which is a FATAL_FIELD. Creating + launchpad/docs/corpus/AGENTS.md therefore makes it the governing rules file surfaced + for every future change under launchpad/docs/corpus/. That is what #605's acceptance + criterion wants, but it means the document is read as instructions as well as + validated as a node — so it must actually instruct, not merely describe. + There is no mkdocs.yml anywhere in the repository — checked, so that despite + launchpad/AGENTS.md §3 describing launchpad/docs/ as "MkDocs knowledge layer" there is + no navigation to register and no docs build this file could break. The only workflows + touching launchpad/docs/ are the two corpus ones above. + `just` is NOT on PATH without `. ./bin/activate-hermit`, which must run in the SAME + command as the recipe. Every done-when below therefore calls the interpreter directly. + The 45 standards/template issues (#1307–#1351) are all OPEN, as are #639, #1316, #1410. + Baseline before this plan existed: no branch, worktree, PR or plan for #636 or #639. + That is a statement about the pre-plan baseline, not about the worktree now — this + plan file is itself untracked in it. + +STEP 1 Record the evidence base the issue demands before drafting. [independent] + The issue requires it BEFORE drafting: repository ref, inspected source paths and + symbols, tests, specifications, migrations and configuration, relevant Git + history/PRs/issues, and anything expected but NOT verified. Write it to the + session scratchpad as a structured checklist with one section per required + category — not a flat path list, which is satisfiable by writing nothing. + Every category gets either at least one inspected item or an explicit + "not verified, because …" line. Path candidates are checked with `test -f`, not + `test -e`: the validator requires is_file(), so a directory passes a `-e` gate + and fails two steps later as a citation error. + done when: the note names commit 0052f5a7820ca4ca261efa233feb8bb53858ade6; + every required category has an entry or an explicit not-verified reason; and + every filesystem path it lists satisfies `test -f` from the worktree root. + +STEP 2 Create AGENTS.md with schema-valid frontmatter and a skeleton. [needs 1] + id: `corpus-agents` — mirrors the file's own name so a reader maps node id to + path without a lookup table. Chosen HERE rather than left to the builder because + ids are never renamed: an id picked ad hoc mid-build is permanent. #1317 (the + identifier standard) is still open, so there is no scheme to conform to yet; if + it later prescribes a different one, that is a migration, not an edit. + type: `agent` — decided by Serina 2026-08-26, and precedent for the remaining 45 + nodes. Reasoning: the enum's values are PRD #602's in-scope surfaces, this + document's audience and subject are both agents, and it becomes the resolved + AGENTS.md for the subtree (see ALREADY TRUE), which is an agent-facing surface + in the most literal sense. `governance` was the rejected alternative — it would + eventually absorb every process document and stop discriminating. + status: active. origin: launchpad. + audiences: [agent] at minimum. evidence: one entry per substantive claim. No + `relationships` block — no other node exists to target, and inventing one is a + hard validator error. + The repository revision goes HERE, in the evidence ledger, because the ledger is + the schema's provenance mechanism and no other schema-legal slot exists. It lands + on the UNVERIFIED channel (a commit reference names no openable file), which is + correct and non-fatal — but it means nothing enforces it, so STEP 8 checks it. + done when: `python3` validates the file's frontmatter against + launchpad/docs/corpus/schema/node.schema.json with zero errors — the schema + itself, not a YAML parse, which would accept frontmatter with no `evidence` key + at all; AND the ledger contains an entry recording + 0052f5a7820ca4ca261efa233feb8bb53858ade6. + +STEP 3 Prove the validator actually scans the new node. [needs 2] ← RUNS HERE + A pass that skipped the file looks identical to a pass that checked it — the + failure this step rules out, and the class of defect (`schema/` exclusion, + symlink escape) that cost #623 two review rounds. + done when: `python3 launchpad/project-intelligence/corpus/validate.py` exits 0 + with the file present; AND after temporarily setting `type:` to a value outside + the schema enum the same command exits non-zero AND its output names this node's + id; AND the file is restored and the command exits 0 again. Asserting on the + named id, not merely on a non-zero status: a non-zero status is produced by any + failure, including the command not existing. + +STEP 4 Write the node contract section — what a corpus node IS. [needs 3] + Where the file goes, the one-node-per-document rule, id stability, the closed + enums, and the canonical-vs-generated boundary (non-.md files fail closed today; + #1316 owns the contract that will change that). LINK to + launchpad/docs/corpus/schema/README.md and node.schema.json rather than restating + field lists — a second copy of an enum is a second thing to drift, and the + validator never reads the body, so a stale copy stays green forever. + done when: the section names the schema files by path, reproduces no enum member + list and no field-combination matrix, and the validator still exits 0. + +STEP 5 Write the evidence section — the ledger, and what validation proves. [needs 4] + NOT a restatement of the three classes and their field rules: those are canonical + in node.schema.json and schema/README.md, and copying them here contradicts both + STEP 4's anti-drift rule and the DoD's "without duplicating canonical content". + Link them. What this section adds is the part living nowhere else: which citation + forms the validator can check, that checking is STRUCTURAL only, that a `PASS` + never means a citation supports its statement, that a GitHub file link needs a + full 40-char SHA or CI fails, and that the positional `path:line` form does not + verify the line exists (#1459) so a bare path is the safer citation today. + done when: the section reproduces no field matrix or enum list; states the + structural-vs-evidential distinction explicitly; states the full-SHA pin rule; + and cites launchpad/project-intelligence/CONTRACT.md and validate.py by path. + +STEP 6 Write the create / update / retire procedures. [needs 5] + Three numbered procedures, each ending in the validator command. Retire is the + one most likely to be got wrong: `status: retired`, id NEVER reused or renamed, + inbound relationships from other nodes considered before retiring. #605's + acceptance criterion for this file is exactly "create/update/retire one node + without oral guidance" — this step is that criterion, and STEP 8's QA charter is + what tests it. + done when: all three procedures are present; each is a numbered sequence a reader + could follow without asking a question; and each terminates in a command whose + exit status decides success. + +STEP 7 Write the scope-and-omissions section and the outbound links. [needs 6] + State what this document does NOT cover and who owns it: the 45 standards and + templates (#1307–#1351), generated-content provenance (#1316), claim-type and + flagged state (#1410), the human-facing entry point (#639). Link ADR-0028 and + ADR-0029 by repo path. Say plainly that no sibling corpus node exists yet, so the + frontmatter carries no `relationships` — an absence with a reason, not an + oversight a reviewer has to guess at. + done when: the section names each deferred area with its owning issue number, and + every repo path it cites satisfies `test -f`. Body links are checked by hand + here because the validator never reads the body. + +STEP 8 Audit the ledger, the scope, and the changed-file set. [needs 7] + Four checks, none of which any tool performs: + (a) Every substantive body claim maps to a ledger entry of the right class, and + for each FACT the cited source is OPENED and read to confirm it actually + supports the statement. A citation that only passes structurally, or that + lands on the UNVERIFIED channel, does not support a FACT — either open the + source and keep the class, or reclassify to INFERENCE with a confidence, or + to TEAM_KNOWLEDGE with provided_by. + (b) The revision recorded in STEP 2's ledger matches the revision actually + inspected in STEP 1. + (c) Scope: the document still represents ONE independently maintainable concept. + Any second concept found while drafting is removed and filed as its own task, + per the DoD — it is not folded in because it happened to be well-sourced. + (d) Changed-file inventory against the merge base: `launchpad/docs/corpus/` + contains exactly one added hand-authored .md file. Five valid nodes validate + together, so "exactly one document" is a property only this check enforces. + done when: all four pass; the validator exits 0; and the build report lists every + UNVERIFIED notice with the classification decision taken for it, or states there + were none. + +PARALLEL Nothing here may fan out. STEPs 2 and 4–8 all edit the single file + launchpad/docs/corpus/AGENTS.md, which makes them strictly sequential + regardless of how independent their subject matter looks. STEP 1 is the only + step touching no repository file, and everything else needs its output. + #639 (README.md) is genuinely independent and could run as a parallel worktree + — but it is a different issue and a different PR, not a step of this plan. + +GATES review-plan: DONE — an independent Codex pass reviewed the first draft and its + eleven findings are applied above. review-code after STEP 8: applies despite + this being docs-only, because the frontmatter is machine-validated input to a + checked-in validator, not prose. review-tests does NOT apply — no tests added, + and #623's suite already covers the validator. review-a11y does NOT apply — no + UI. review-adjudicate then review-final before merge, and review-final MUST be + cross-model: three same-model passes on #120 missed what one Codex pass caught + immediately, and this plan is a fresh data point for the same rule. + qa explore mode APPLIES. Charter: in a scratch corpus root OUTSIDE the repo, + follow STEP 6's procedures literally, as written, supplying nothing the document + did not state. Exercise ALL THREE — create, update and retire — because #605's + criterion names all three and a create-only charter cannot detect an update + procedure that forgets provenance. Use at least two scratch nodes so retirement + has an inbound relationship to handle. Validate with + `python3 launchpad/project-intelligence/corpus/validate.py --root "$SCRATCH"`, + never `just corpus-validate`, which takes no `--root` and would validate the + real corpus while appearing to test the scratch one. Include a negative control: + a deliberately invalid scratch node the run must reject, proving that root was + the one scanned. Nothing from the scratch root is committed. + +BUDGET STEP 8(a). Opening every cited source and confirming it supports its statement + is the only step whose cost scales with the ledger's length, and it is the one + with no tool behind it — the validator reports structure and stops. It is also + the step most likely to be quietly downgraded to "the validator passed", which + is precisely the substitution it exists to prevent. + +OPEN 1. RESOLVED 2026-08-26 by Serina: `type: agent`, and `id: corpus-agents`. Both + are now stated in STEP 2 with their reasoning. They were listed here in an + earlier revision, which was a defect: STEP 2 cannot write frontmatter without + them, so a plan that both required them and forbade the builder deciding them + blocked its own second step. An OPEN item that a step must consume is not an + open question — it is a missing input. + 2. Forward references. The standards this document would defer to (#1307–#1351) + do not exist. Naming them as "owned by #NNNN, not yet written" is honest but + ages into a stale list; omitting them leaves a cold-start agent unable to + tell a gap from an omission. STEP 7 assumes naming them is right. + 3. Whether YAML frontmatter at the top of a file that IS a rules document + degrades it as instructions. No longer hypothetical: preflight_core.py will + resolve this file as the nearest AGENTS.md for the subtree (see ALREADY + TRUE), so it is handed to agents as governing rules, frontmatter first. + Harmless to the validator and to preflight, which resolves the path without + parsing content. Unverified against any agent harness actually reading it. + 4. WITHDRAWN, recorded because the reasoning will recur. The first draft claimed + STEP 3's mutation clause could be satisfied by `just: command not found` + (exit 127). The Codex pass refuted it: STEP 3's first and third clauses + require exit 0, which 127 fails, so a missing `just` is fail-CLOSED. The + underlying operational fact is real and kept — `just` needs Hermit activated + in the same command — which is why every done-when now calls `python3` + directly. The severity claim was wrong; the PATH observation was not. + +LEFT OUT launchpad/docs/corpus/README.md — that is #639, a separate PR; #636's own + out-of-scope list forbids materially editing a second hand-authored node here. + The generated-content provenance contract (#1316) — the validator deliberately + refuses to guess at it and so does this document. ADR-0029's claim-type + classification and flagged state (#1410) — that issue's work; `status: flagged` + is simply not used. Fixing #1459 (positional citations not checked against file + length) — a defect in merged code, filed separately; this task routes around it + by preferring bare-path citations rather than changing the validator. Any change + to the schema or the validator — both are merged and this task consumes them. From a9f3e01e4101a6853a671f9b110d0a0e7fac987e Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:24:55 +1200 Subject: [PATCH 02/16] feat(corpus): AGENTS.md frontmatter and skeleton (#636 STEPs 2-3) The corpus's first hand-authored node. Frontmatter validates against node.schema.json with zero errors; 11 evidence entries; no relationships, because no sibling node exists to target and an unresolved target is a hard validator error. The repository revision is recorded in the evidence ledger -- the schema calls that array the provenance ledger and permits no other field, so it is the only schema-legal place for it. STEP 3 (RUNS HERE) proved the validator actually scans the node rather than skipping it: mutating type: to a value outside the enum produced 'FAIL corpus-agents: schema violation at type' and exit 1, naming the node's id; restoring it returned exit 0. Also replaces test_real_corpus_root_currently_has_no_content_outside_schema, which asserted the corpus root is EMPTY and therefore could not survive the first authored node. Authorised by Serina as a deliberate widening of the plan's LEFT OUT scope, recorded in the plan. The replacement asserts what the original's own comment said it was for -- content present AND nothing from schema/ -- and both halves were negative-controlled before being trusted. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 68 +++++++++++++++++++ .../2026-08-26-issue-636-corpus-agents-md.md | 14 ++++ .../corpus/tests/test_validate.py | 37 +++++++--- 3 files changed, 109 insertions(+), 10 deletions(-) create mode 100644 launchpad/docs/corpus/AGENTS.md diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md new file mode 100644 index 00000000000..06629c942a0 --- /dev/null +++ b/launchpad/docs/corpus/AGENTS.md @@ -0,0 +1,68 @@ +--- +id: corpus-agents +type: agent +status: active +origin: launchpad +audiences: + - agent +evidence: + - statement: "This node was authored and checked against repository revision 0052f5a7820ca4ca261efa233feb8bb53858ade6." + entry_class: FACT + evidence: + - "commit 0052f5a7820ca4ca261efa233feb8bb53858ade6" + - statement: "Markdown with YAML front matter is the one canonical authored representation of a corpus node; every other serialization is a generated derived view." + entry_class: FACT + evidence: + - "launchpad/decisions/ADR-0028-corpus-canonical-representation.md" + - statement: "A node's front matter is validated against node.schema.json, which requires id, type, status, origin, audiences and evidence, and permits no field outside that set." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - statement: "The corpus root is launchpad/docs/corpus, and validate.py is the deterministic check that governs it." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - "Justfile" + - statement: "The schema/ subtree is excluded from validation because it is the schema's own testing infrastructure rather than corpus content." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "Evidence precedence is contextual by claim type, and two authoritative sources of the same claim type in conflict leave the node flagged for a human rather than silently resolved." + entry_class: FACT + evidence: + - "launchpad/decisions/ADR-0029-corpus-evidence-precedence.md" + - statement: "Citations take six shapes and only three of them name a file that can be opened." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/CONTRACT.md" + - statement: "Citation checking is structural: the validator confirms a cited path resolves to a real file inside the repository, never that the file supports the statement it sits under." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "A relationship whose target matches no loaded node's id is a hard validation error." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "This file is also resolved as the nearest AGENTS.md for every change under launchpad/docs/corpus, so it is read as governing instructions and not only as a corpus node." + entry_class: FACT + evidence: + - "launchpad/scripts/preflight_core.py" + - statement: "Every change under launchpad/docs/corpus is gated in CI by the corpus validate workflow." + entry_class: FACT + evidence: + - ".github/workflows/launchpad-corpus-validate.yml" +--- + +# Working with the documentation corpus + +## What a corpus node is + +## Evidence, citations, and what validation proves + +## Creating a node + +## Updating a node + +## Retiring a node + +## Scope and omissions diff --git a/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md b/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md index 20e9f919b4e..65d3570b82e 100644 --- a/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md +++ b/launchpad/plans/2026-08-26-issue-636-corpus-agents-md.md @@ -243,3 +243,17 @@ LEFT OUT launchpad/docs/corpus/README.md — that is #639, a separate PR; #636' length) — a defect in merged code, filed separately; this task routes around it by preferring bare-path citations rather than changing the validator. Any change to the schema or the validator — both are merged and this task consumes them. + + AMENDED during build, 2026-08-26, authorised by Serina. ONE exception to the + line above: `test_real_corpus_root_currently_has_no_content_outside_schema` in + launchpad/project-intelligence/corpus/tests/test_validate.py asserted the corpus + root is EMPTY. #636 authors the first node, so that test cannot survive this + issue — it fails in CI via launchpad-corpus-validate.yml, not merely locally. + It is replaced (not deleted) by an assertion of what its author's own comment + says it was for: the root holds authored content AND none of it comes from + schema/. Both halves are needed; the second alone is satisfied by an exclusion + that rejects everything. Neither the review-plan self-pass nor the independent + Codex pass caught this — the plan asserted "#623's suite already covers the + validator" without running it against a corpus containing a node. + CONSEQUENCE: the diff now touches tests, so review-tests JOINS the roster that + the GATES line above says does not apply. Treat the GATES line as amended. diff --git a/launchpad/project-intelligence/corpus/tests/test_validate.py b/launchpad/project-intelligence/corpus/tests/test_validate.py index 2d08fcc504d..66966552126 100644 --- a/launchpad/project-intelligence/corpus/tests/test_validate.py +++ b/launchpad/project-intelligence/corpus/tests/test_validate.py @@ -982,11 +982,13 @@ class SchemaDirExclusionTest(unittest.TestCase): """schema/ is #622's own infrastructure, never scanned as corpus content. Proven against a purpose-built fixture tree containing BOTH a schema/ file - and a real sibling, not merely against today's real launchpad/docs/corpus/ - root -- that root currently has zero non-schema content, so a test asserting - only "no file under schema/ leaked" against it would pass vacuously even if - exclusion were broadened to reject everything. An independent review-tests - pass found this. + and a real sibling, rather than against the real launchpad/docs/corpus/ root + alone. When this suite was written that root held zero non-schema content, so + a test asserting only "no file under schema/ leaked" against it would have + passed vacuously even if exclusion were broadened to reject everything. An + independent review-tests pass found that. The fixture test below stays the + primary proof for the same reason: it controls both sides of the comparison, + where the real root only ever shows whatever happens to be committed. """ def test_sibling_discovered_schema_dir_excluded(self) -> None: @@ -1003,13 +1005,28 @@ def test_sibling_discovered_schema_dir_excluded(self) -> None: self.assertIn(real_sibling, files) self.assertNotIn(inside_schema, files) - def test_real_corpus_root_currently_has_no_content_outside_schema(self) -> None: - # A documentation-style sanity check on today's real state, not the - # primary proof of exclusion (see test_sibling_discovered_schema_dir_ - # excluded above, which is the one that can actually fail on regression). + def test_real_corpus_root_has_content_and_excludes_schema(self) -> None: + """The real root holds authored nodes, and none of them come from schema/. + + This replaces `test_real_corpus_root_currently_has_no_content_outside_ + schema`, which asserted the root was EMPTY. That assertion was true when + #623 wrote it and had a shelf life ending at the first authored node -- + #636's launchpad/docs/corpus/AGENTS.md, which is what broke it. A test + encoding a temporary state as a permanent assertion fails on the change + it was supposed to permit, and says nothing about the behaviour under + test when it does. + + Both halves are asserted deliberately. Without the non-empty check this + degrades to the vacuous form the class docstring warns about: an + exclusion broadened to reject EVERYTHING would satisfy "nothing from + schema/ was discovered" perfectly. + """ root = validate.repo_root() / validate.DEFAULT_ROOT files = validate.discover_markdown_files(root) - self.assertEqual(files, []) + self.assertNotEqual(files, []) + self.assertFalse( + any("schema" in f.relative_to(root).parts[:1] for f in files) + ) if __name__ == "__main__": From 178790b7e5194c642d06063fa22e918c6d83ab9b Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:26:18 +1200 Subject: [PATCH 03/16] docs(corpus): AGENTS.md node-contract section (#636 STEP 4) Adds the lookup table of authoritative sources and the 'what a corpus node is' section: one file per node, one idea per node, where nodes go, id permanence, relationship resolution, and the authored-vs-generated boundary. Reproduces no enum member list and no field-combination matrix -- those stay canonical in node.schema.json and schema/README.md and are linked instead. The checker never reads body prose, so a copy here would stay green while drifting. Verified: no type-enum member and none of entry_class/provided_by/ confidence appear in the body. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 51 +++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 06629c942a0..00b801a37a3 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -55,8 +55,59 @@ evidence: # Working with the documentation corpus +Instructions for creating, updating and retiring one node in +`launchpad/docs/corpus/`. Look up the section you need; this is a reference, not a +tutorial. + +**Authoritative sources — this file duplicates none of them:** + +| For | Read | +|---|---| +| The front-matter contract (fields, enums, conditional rules) | `launchpad/docs/corpus/schema/node.schema.json` | +| Prose explanation of those fields | `launchpad/docs/corpus/schema/README.md` | +| Adding a value to a closed enum | `launchpad/docs/corpus/schema/COMPATIBILITY.md` | +| Relationship types and their directionality | `launchpad/docs/corpus/schema/relationships.schema.json` | +| Why Markdown + front matter is canonical | `launchpad/decisions/ADR-0028-corpus-canonical-representation.md` | +| How to rank conflicting evidence | `launchpad/decisions/ADR-0029-corpus-evidence-precedence.md` | +| The six citation shapes | `launchpad/project-intelligence/CONTRACT.md` §3 | +| What the checker actually enforces | `launchpad/project-intelligence/corpus/validate.py` | + +If this file and any of those disagree, **they win** — this one has drifted and +should be fixed. + ## What a corpus node is +**One file is one node.** A node is a Markdown file with YAML front matter, and that +is the single canonical authored representation; anything else — JSON, an index, a +graph serialization — is a generated derived view, never hand-authored. + +**One node is one independently maintainable idea.** If a second concept, contract or +procedure turns up while you are writing, it does not get folded in. File it as its +own task and link to it. + +**Where it goes.** Anywhere under `launchpad/docs/corpus/`, except `schema/` — that +subtree is the schema's own testing infrastructure and is deliberately skipped by the +checker, so a node placed there is never validated at all. + +**Front matter.** Validated against `node.schema.json`. Field names, which fields are +required, the closed enums and the conditional rules between fields all live in that +file and in `schema/README.md`; they are not repeated here, because a second copy +drifts silently — the checker never reads this document's prose, so a stale copy +would stay green forever. + +**`id` is permanent.** Kebab-case, assigned once, never renamed. Generated views +derive from it reproducibly, so renaming an id is a migration, not an edit. + +**Relationships are optional and must resolve.** A `relationships[].target` naming an +id no node in the corpus carries is a hard error. A node with no sibling to point at +correctly declares none. + +**Authored versus generated.** Every non-`.md` file under the corpus root must live +in a `generated/` directory. Today the checker rejects such files even there, because +no generator exists yet to reproduce them from canonical Markdown, and a hand-written +file in `generated/` is indistinguishable from a real projection. That contract is +owned by #1316; until it lands, a corpus change adds Markdown only. + ## Evidence, citations, and what validation proves ## Creating a node From 8ca658e0c1ef8e74efaa0f114b590f50994e41c1 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:27:19 +1200 Subject: [PATCH 04/16] docs(corpus): AGENTS.md evidence and citation section (#636 STEP 5) The part that lives nowhere else: what the checker actually does with each of CONTRACT.md's six citation shapes, and three things a passing run does not mean -- that a citation supports its claim (checking is structural, the file is never opened and compared), that UNVERIFIED is a pass, or that a line number was verified (#1459). Names the three evidence classes and what each is FOR, but not which fields the schema requires per class -- that matrix stays in node.schema.json. Verified absent from the body: entry_class, provided_by, confidence. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 68 +++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 00b801a37a3..6fa3aeed95e 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -110,6 +110,74 @@ owned by #1316; until it lands, a corpus change adds Markdown only. ## Evidence, citations, and what validation proves +Every substantive claim in a node's body needs an entry in its front-matter +`evidence` array. That array is also the node's **provenance ledger** — there is no +separate provenance field, so the revision a node was written against belongs in +there too, as a commit citation. + +### Choosing a class + +Three classes exist: `FACT`, `INFERENCE`, `TEAM_KNOWLEDGE`. Which one you choose +decides which additional fields the schema then requires or forbids — those rules are +in `node.schema.json` and `schema/README.md`, and are not restated here. + +What the classes are *for* is the part that is easy to get wrong: + +- **`FACT`** — you opened the cited source and it says so. Not "a source exists that + probably says so." +- **`INFERENCE`** — you reasoned to it from evidence. Reasoning is not fact, however + good it is. +- **`TEAM_KNOWLEDGE`** — a person told you, and nothing corroborates it. This is the + class that exists for uncorroborated statements; using it honestly is better than + promoting a recollection to `FACT`. + +When two sources disagree, do not average them and do not pick the newer one. For how +the system **currently behaves**, executable evidence — code, config, schema, passing +tests — outranks documentation and history. For **intended or authorized** behaviour, +accepted decisions outrank code that has drifted from them. When two sources of the +*same* claim type conflict, stop: record the conflict and leave the node flagged for a +human rather than resolving it yourself. `ADR-0029` is the full rule. + +### What the checker does with each citation shape + +`CONTRACT.md` §3 defines the six shapes. What `validate.py` does with them is not +documented anywhere else, so it is here: + +| Shape | Checker's verdict | +|---|---| +| Bare repository path | Resolved; must be a real **file** inside the repo. A directory fails. | +| Path with a line or line range | Resolved as a path. **The line number is not checked** — see below. | +| GitHub file link | Must be pinned to a full 40-character commit SHA, and must name a file. | +| Commit reference | Reported `UNVERIFIED`. Nothing on disk to open. | +| Graph edge | Reported `UNVERIFIED`. | +| Tool result | Reported `UNVERIFIED`. | + +Anything matching **no** known shape is a hard error, not an `UNVERIFIED` notice. + +### Three things a passing run does not mean + +**1. It does not mean a citation supports its claim.** Checking is *structural*. The +checker confirms a path resolves to a real file; it never opens that file and compares +it against your `statement`. A `FACT` citing a real file that says nothing on the +subject passes cleanly. Only a human reading the source establishes a `FACT`. + +**2. `UNVERIFIED` is not a pass.** Those notices are printed, never fatal, and they +mean the checker recognised the shape and could not open it. A `FACT` resting only on +`UNVERIFIED` citations has not been checked by anything — open the source and keep the +class, or change the class. + +**3. A line number is not verified.** `Justfile:999999` is accepted against a +1005-line file (#1459). Prefer a bare path until that is fixed; a position that has +silently drifted is worse than no position, because it looks precise. + +### Pinning + +A GitHub link to a repository file must use the full 40-character commit SHA. +`blob/main` is rejected, and correctly: evidence that can change underneath a green +validation run is the exact staleness provenance exists to catch. A link pinned but +naming no file is also rejected — it cites a repository at a commit, not the source +of the claim. + ## Creating a node ## Updating a node From f6abebccd83fdfadc19c274f0600b7ddd62b05be Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:28:17 +1200 Subject: [PATCH 05/16] docs(corpus): AGENTS.md create/update/retire procedures (#636 STEP 6) This is #605's acceptance criterion for the file -- a cold-start agent creating, updating or retiring one node without oral guidance. Three numbered procedures, each ending in the same check command. Adds a 'Running the check' section giving the interpreter form (works without Hermit; the just recipe needs activation first) and documenting --root, since without it the command always validates the real corpus whatever directory you are standing in. Retire covers the three things most likely to be got wrong: inbound relationships resolved before retirement, status set rather than the file deleted, and the id never reused or renamed. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 62 +++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 6fa3aeed95e..a8ece1e967c 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -178,10 +178,72 @@ validation run is the exact staleness provenance exists to catch. A link pinned naming no file is also rejected — it cites a repository at a commit, not the source of the claim. +## Running the check + +All three procedures below end with the same command, run from the repository root: + +```bash +python3 launchpad/project-intelligence/corpus/validate.py +``` + +Exit status 0 is a pass; 1 means at least one error, and every error names the node it +came from. `just corpus-validate` runs exactly this, but needs the Hermit environment +activated first (`. ./bin/activate-hermit`) — the direct form above does not. The same +command runs in CI on every change under `launchpad/docs/corpus/`, so a local failure +is a CI failure. + +To check a corpus tree somewhere other than the real one, pass `--root `. Without +it the command always validates `launchpad/docs/corpus/`, whatever directory you are +standing in. + ## Creating a node +1. **Confirm it is one idea.** If you are describing two contracts, or a concept and + the procedure that uses it, that is two nodes. File the second as its own task now. +2. **Check nothing already covers it.** Read the existing nodes under + `launchpad/docs/corpus/`. If one is close, you are updating, not creating. +3. **Record what you inspected, before drafting.** The repository revision + (`git rev-parse HEAD`), the source paths and symbols you read, the tests, + specifications and configuration you consulted, and — explicitly — anything you + expected to verify and could not. +4. **Choose the `id`.** Kebab-case, and permanent from this moment. Pick something that + describes the idea, not where the file currently sits. +5. **Create the file** anywhere under `launchpad/docs/corpus/` except `schema/`. +6. **Write the front matter** against `node.schema.json`. Include a commit citation for + the revision from step 3 — the ledger is the only schema-legal place for it. +7. **Write one `evidence` entry per substantive claim** you intend to make. Classify + honestly; open every source you call a `FACT`. +8. **Write the body**, structured for lookup. State what the node does not cover. +9. **Add relationships only to nodes that exist.** A target no node carries is a hard + error. None is a valid answer. +10. **Run the check.** Fix what it names, and re-run until it exits 0. + ## Updating a node +1. **Confirm the change belongs in this node.** New idea, not new detail about the + existing one? That is a new node. +2. **Re-record the revision you are checking against.** `git rev-parse HEAD` now, not + the one already in the ledger. +3. **Re-verify the claims you are touching** against sources at that revision. A claim + whose source moved is not still a `FACT` because it used to be. +4. **Update the ledger in the same edit as the body.** A new claim without an entry, or + an entry left behind by a deleted claim, are the two ways these drift apart. +5. **Update the recorded revision** to the one from step 2. +6. **Leave the `id` alone.** Always. +7. **Run the check.** + ## Retiring a node +1. **Find what points at it.** Search the corpus for the node's `id`; any node with a + relationship targeting it needs handling first, or the corpus is left with a + relationship that no longer resolves. +2. **Decide what replaces it.** If another node takes over the subject, say so in that + node — a reader arriving at a retired node needs somewhere to go. +3. **Set `status` to the retired value** defined in `node.schema.json`. Do not delete + the file: generated views and inbound links resolve through the id, and deleting it + breaks them silently. +4. **Never reuse or rename the `id`.** A retired id stays spent. +5. **Record why**, in the body and in the ledger, at the revision you checked. +6. **Run the check.** + ## Scope and omissions From 9c7facc48223d759e7019f7757326bd958eeee02 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:29:49 +1200 Subject: [PATCH 06/16] docs(corpus): AGENTS.md scope and omissions (#636 STEP 7) Names each deferred area with its owning issue rather than leaving a reader unable to tell a gap from an omission: the standards and templates (#1307-#1351, none merged), generated-artifact provenance (#1316), claim-type and flagged state (#1410), the human entry point (#639), and unverified line numbers in citations (#1459). Records why this node carries no relationships -- at its recorded revision it was the only authored node, and a target no node carries is a hard error -- so the absence reads as deliberate rather than forgotten. Every repo path cited in the body verified to resolve to a real file. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 37 +++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index a8ece1e967c..afec7c545ae 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -91,9 +91,9 @@ checker, so a node placed there is never validated at all. **Front matter.** Validated against `node.schema.json`. Field names, which fields are required, the closed enums and the conditional rules between fields all live in that -file and in `schema/README.md`; they are not repeated here, because a second copy -drifts silently — the checker never reads this document's prose, so a stale copy -would stay green forever. +file and in `launchpad/docs/corpus/schema/README.md`; they are not repeated here, +because a second copy drifts silently — the checker never reads this document's +prose, so a stale copy would stay green forever. **`id` is permanent.** Kebab-case, assigned once, never renamed. Generated views derive from it reproducibly, so renaming an id is a migration, not an edit. @@ -119,7 +119,8 @@ there too, as a commit citation. Three classes exist: `FACT`, `INFERENCE`, `TEAM_KNOWLEDGE`. Which one you choose decides which additional fields the schema then requires or forbids — those rules are -in `node.schema.json` and `schema/README.md`, and are not restated here. +in `node.schema.json` and `launchpad/docs/corpus/schema/README.md`, and are not +restated here. What the classes are *for* is the part that is easy to get wrong: @@ -247,3 +248,31 @@ standing in. 6. **Run the check.** ## Scope and omissions + +**This document covers** how to create, update and retire one corpus node, what the +front-matter contract is and where it lives, how to classify and cite evidence, and +what the deterministic check does and does not establish. + +**It does not cover, and these are gaps rather than silence:** + +| Not covered here | Owned by | +|---|---| +| Per-type standards — naming, identifiers, linking, provenance, status, taxonomy, diagrams, evidence | #1307–#1351, none merged yet | +| Templates for each node type — concept, component, capability, interface, flow, policy, procedure, runbook, reference, specification, and the rest | #1307–#1351, none merged yet | +| How generated artifacts prove their provenance, and the exception process for them | #1316 | +| Encoding ADR-0029's claim-type classification and the flagged state in the schema and checker | #1410 | +| The human-facing entry point to the corpus | #639 | +| Line numbers in citations not being verified against file length | #1459 | + +Until the standards land there is no per-type template to follow: write the node +against `node.schema.json` and the rules above, and expect a later task to reshape it. + +**No `relationships` in this node's own front matter.** At the revision recorded in its +ledger this was the only authored node in the corpus, so there was nothing to point at +— and a `relationships[].target` naming an id no node carries is a hard error. The +absence is deliberate, not an oversight. + +**This file is read twice.** It is a corpus node, validated like any other; it is also +resolved as the nearest `AGENTS.md` for every change under `launchpad/docs/corpus/`, +so an agent working anywhere in this subtree is handed it as governing instructions. +Write it to be followed, not merely to be accurate. From 21afc79f31b92ba75d61f082a605a80a7e40f0bc Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:31:48 +1200 Subject: [PATCH 07/16] docs(corpus): audit AGENTS.md against its own ledger (#636 STEP 8) The audit found two things the earlier steps had missed. The document's own rule -- 'a FACT resting only on UNVERIFIED citations has not been checked by anything' -- condemned the node's own ledger entry 1, the provenance record, whose only citation is a commit reference. Resolved by stating the one carve-out narrowly and giving it a real check (git cat-file -e , run, exit 0) rather than by quietly exempting it. A commit citation on a claim about repository CONTENT is explicitly still not covered. Ledger entry 11 overreached: 'every change under launchpad/docs/corpus is gated in CI' where the workflow triggers on pull requests and pushes to launchpad. Narrowed to what the workflow actually does. Also records a scope judgement the DoD asks for: the citation-shape table is reference material rather than instruction and belongs in the evidence standard (#1314) when that lands. Noted in place rather than folded in silently. Verified: all 11 ledger entries resolve, every body path is a real file, merge-base equals the recorded revision, and exactly one hand-authored .md was added under the corpus root. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index afec7c545ae..71b3b50ed0b 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -47,7 +47,7 @@ evidence: entry_class: FACT evidence: - "launchpad/scripts/preflight_core.py" - - statement: "Every change under launchpad/docs/corpus is gated in CI by the corpus validate workflow." + - statement: "Changes under launchpad/docs/corpus are validated in CI on pull requests and on pushes to the launchpad branch." entry_class: FACT evidence: - ".github/workflows/launchpad-corpus-validate.yml" @@ -142,7 +142,9 @@ human rather than resolving it yourself. `ADR-0029` is the full rule. ### What the checker does with each citation shape `CONTRACT.md` §3 defines the six shapes. What `validate.py` does with them is not -documented anywhere else, so it is here: +documented anywhere else, so it is here — provisionally. This table is reference +material rather than instruction, and belongs in the evidence standard once that +lands (#1314); when it moves, this section links to it instead. | Shape | Checker's verdict | |---|---| @@ -167,6 +169,18 @@ mean the checker recognised the shape and could not open it. A `FACT` resting on `UNVERIFIED` citations has not been checked by anything — open the source and keep the class, or change the class. +There is exactly one carve-out, and widening it defeats the rule. The **provenance +entry recording the revision** cites a commit id, which no file can corroborate +because the citation *is* the claim. It is still checkable, just not by this checker: + +```bash +git cat-file -e # exit 0 means that revision exists in this repository +``` + +Run that, and the entry is a `FACT`. Every other claim needs a source you opened. +A commit citation attached to a claim *about repository content* is not covered here +— that claim needs the file, at that revision. + **3. A line number is not verified.** `Justfile:999999` is accepted against a 1005-line file (#1459). Prefer a bare path until that is fixed; a position that has silently drifted is worse than no position, because it looks precise. From cb70ae024c06939404224faef4055319a707f0bd Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:50:41 +1200 Subject: [PATCH 08/16] test(corpus): strengthen real-root discovery test (#636, review-tests finding) An independent review-tests pass defeated the first replacement: patching discover_markdown_files to a hardcoded 'return [root / "AGENTS.md"]' -- a constant that never touches the filesystem -- satisfied both 'non-empty' and 'nothing from schema/' while proving neither discovery nor exclusion. My own negative control had broken the exclusion set and emptied the corpus, but never tried a constant. Showing that assertions CAN fail is not the same as showing they can only pass for the right reason. The expectation is now derived from an independent filesystem walk and compared for equality, which catches partial walks and both exclusion directions. It does NOT catch the constant today, and the docstring says so rather than claiming otherwise: the corpus holds exactly one node, so [root/'AGENTS.md'] IS the correct answer and no assertion against the real tree can tell a constant from a real walk. That resolves itself at the second node. test_sibling_discovered_schema_dir_excluded catches it now -- verified FAIL under that mutant -- and is the real proof of discovery behaviour. Measured, after fixing a mutation harness that had patched a different module instance than the test uses and so reported every mutant as caught: returns nothing -> FAIL (caught) exclusion disabled -> FAIL (caught) constant return -> PASS (not caught; sibling catches it) Refs #636 Signed-off-by: Serina Mcfall --- .../corpus/tests/test_validate.py | 56 +++++++++++++++---- 1 file changed, 46 insertions(+), 10 deletions(-) diff --git a/launchpad/project-intelligence/corpus/tests/test_validate.py b/launchpad/project-intelligence/corpus/tests/test_validate.py index 66966552126..49fea94cfe2 100644 --- a/launchpad/project-intelligence/corpus/tests/test_validate.py +++ b/launchpad/project-intelligence/corpus/tests/test_validate.py @@ -1005,8 +1005,8 @@ def test_sibling_discovered_schema_dir_excluded(self) -> None: self.assertIn(real_sibling, files) self.assertNotIn(inside_schema, files) - def test_real_corpus_root_has_content_and_excludes_schema(self) -> None: - """The real root holds authored nodes, and none of them come from schema/. + def test_real_corpus_root_discovery_matches_an_independent_walk(self) -> None: + """Discovery over the real root returns EXACTLY the nodes that are there. This replaces `test_real_corpus_root_currently_has_no_content_outside_ schema`, which asserted the root was EMPTY. That assertion was true when @@ -1016,18 +1016,54 @@ def test_real_corpus_root_has_content_and_excludes_schema(self) -> None: it was supposed to permit, and says nothing about the behaviour under test when it does. - Both halves are asserted deliberately. Without the non-empty check this - degrades to the vacuous form the class docstring warns about: an - exclusion broadened to reject EVERYTHING would satisfy "nothing from - schema/ was discovered" perfectly. + The first replacement asserted only "non-empty AND nothing from schema/", + which an independent review-tests pass defeated immediately: replacing + discover_markdown_files with a hardcoded `return [root / "AGENTS.md"]` + -- a constant that never touches the filesystem -- satisfied both halves + while proving neither discovery nor exclusion. Asserting that a check CAN + fail is not the same as asserting it can only pass for the right reason. + + So the expectation is now derived from the filesystem independently of + the function under test, and compared for equality. Measured against + mutants of discover_markdown_files, this test catches: + + returns nothing -> FAIL (caught) + exclusion disabled -> FAIL (caught) + hardcoded constant -> PASS (NOT caught today) + + The constant is not caught, and cannot be by any assertion made here + while the real corpus holds exactly ONE node: `[root / "AGENTS.md"]` is + the correct answer today, so a constant and a real walk are + indistinguishable from outside. That is a property of the real tree, not + of this assertion -- and it resolves itself the moment a second node + lands, at which point the equality form catches constants and partial + walks for free. `test_sibling_discovered_schema_dir_excluded` catches the + constant NOW, because its fixture tree holds names no constant predicts. + That test, not this one, is the proof of discovery behaviour. + + `assertNotEqual(expected, [])` is the guard against inheriting the + original sin: with an empty corpus both sides would be [] and equality + would pass vacuously. If that ever fires, the corpus is empty and this + test should be read as reporting that, not as a discovery bug. + + The walk mirrors the canonical-location rule as well as the schema/ + exclusion, because discover_markdown_files drops symlinks resolving + outside the root. A node symlinked in from elsewhere therefore fails + here loudly rather than silently widening what counts as corpus content. """ root = validate.repo_root() / validate.DEFAULT_ROOT - files = validate.discover_markdown_files(root) - self.assertNotEqual(files, []) - self.assertFalse( - any("schema" in f.relative_to(root).parts[:1] for f in files) + resolved_root = root.resolve() + + expected = sorted( + path + for path in root.rglob("*.md") + if path.relative_to(root).parts[0] != "schema" + and path.resolve().is_relative_to(resolved_root) ) + self.assertNotEqual(expected, []) + self.assertEqual(validate.discover_markdown_files(root), expected) + if __name__ == "__main__": unittest.main() From 1304a657be2803166a0d34ac9b5bc20b9c2776a1 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 15:55:57 +1200 Subject: [PATCH 09/16] docs(corpus): address review-code findings on AGENTS.md (#636) FINDING A -- the carve-out claimed enforcement it does not have. 'There is exactly one carve-out, and widening it defeats the rule' read as though some check held the line. Nothing does: a second, third or tenth commit-only FACT produces extra non-fatal UNVERIFIED notices and still exits 0, which the reviewer proved by adding one. Now says plainly that this is a convention a REVIEWER holds, names the signal to look for (more than one commit-only FACT in a ledger), and states that no check will raise it. FINDING B -- create-node step 3 asked the author to record four categories of evidence and gave a destination to exactly one of them. Adds the mapping: revision to the ledger, sources to the citations of the claims they support, and expected-but-unverified to the body's scope section. Plus the rule that anything reaching none of the three was either a missing claim or a stale note. Applying B to this node exposed that it did not follow its own new rule -- the same self-contradiction class STEP 8 caught. Its scope section now names the two things unverified when it was written: no agent harness was tested reading it as a resolved AGENTS.md, and relationships.schema.json's enum was never read directly. Re-audited after the change: exactly one commit-only FACT remains, the enforcement claim is gone, validator exits 0, 79 tests pass. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 44 +++++++++++++++++++++++++++------ 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 71b3b50ed0b..5c54e047f3f 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -169,17 +169,24 @@ mean the checker recognised the shape and could not open it. A `FACT` resting on `UNVERIFIED` citations has not been checked by anything — open the source and keep the class, or change the class. -There is exactly one carve-out, and widening it defeats the rule. The **provenance -entry recording the revision** cites a commit id, which no file can corroborate -because the citation *is* the claim. It is still checkable, just not by this checker: +One conventional exception: the **provenance entry recording the revision** cites a +commit id, which no file can corroborate because the citation *is* the claim. It is +still checkable, just not by this checker: ```bash git cat-file -e # exit 0 means that revision exists in this repository ``` -Run that, and the entry is a `FACT`. Every other claim needs a source you opened. -A commit citation attached to a claim *about repository content* is not covered here -— that claim needs the file, at that revision. +Run that, and the entry is a `FACT`. Every other claim needs a source you opened. A +commit citation attached to a claim *about repository content* is not covered — that +claim needs the file, at that revision. + +**Nothing enforces this.** The checker treats every commit citation identically: a +second, third or tenth `FACT` resting only on `commit ` produces nothing but +extra non-fatal `UNVERIFIED` notices and still exits 0. Verified by adding one and +watching the run pass. So this is a convention a **reviewer** has to hold, not a rule +the tooling holds — if a node's ledger shows more than one commit-only `FACT`, that is +the signal, and no check will raise it for you. **3. A line number is not verified.** `Justfile:999999` is accepted against a 1005-line file (#1459). Prefer a bare path until that is fixed; a position that has @@ -220,7 +227,18 @@ standing in. 3. **Record what you inspected, before drafting.** The repository revision (`git rev-parse HEAD`), the source paths and symbols you read, the tests, specifications and configuration you consulted, and — explicitly — anything you - expected to verify and could not. + expected to verify and could not. Working notes need not be committed, but every + category has a destination in the finished node, and they are not the same one: + + | What you recorded | Where it ends up | + |---|---| + | The revision | A commit citation in the `evidence` ledger (step 6) | + | Paths, symbols, tests, specs, configuration you read | Citations on the `evidence` entries whose claims they support (step 7) | + | Expected but could not verify | The body's scope-and-omissions section (step 8), named as a gap | + + Anything that reaches none of those three was not needed. If you inspected a source + that backs no claim, you have either a missing claim or a stale note — decide which + rather than leaving it in a file nobody reads. 4. **Choose the `id`.** Kebab-case, and permanent from this moment. Pick something that describes the idea, not where the file currently sits. 5. **Create the file** anywhere under `launchpad/docs/corpus/` except `schema/`. @@ -286,6 +304,18 @@ ledger this was the only authored node in the corpus, so there was nothing to po — and a `relationships[].target` naming an id no node carries is a hard error. The absence is deliberate, not an oversight. +**Expected but not verified when this node was written**, per the rule in *Creating a +node* step 3: + +- **No agent harness was tested reading this file as its resolved `AGENTS.md`.** The + front matter is harmless to the checker, and to `preflight_core.py`, which resolves + the path without parsing content. Whether it degrades the file for a harness that + *reads* it as instructions is unknown. +- **`relationships.schema.json` was not read directly.** `node.schema.json` states that + a test guards the two relationship enums against drifting apart; that test was not + run. Immaterial to this node, which declares no relationships, but a reader relying + on the linked file for the enum is relying on that guard, not on a check made here. + **This file is read twice.** It is a corpus node, validated like any other; it is also resolved as the nearest `AGENTS.md` for every change under `launchpad/docs/corpus/`, so an agent working anywhere in this subtree is handed it as governing instructions. From 60d4947b7145a6ef25f185b9c25d43e43d99de3c Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 16:48:59 +1200 Subject: [PATCH 10/16] docs(corpus): address the cross-model final review (#636) Codex review-final returned REQUEST_CHANGES with seven conditions, then a second pass confirmed five fixed and refused two. Both refusals were correct. This commit is the whole of that work. 1. LEDGER COMPLETENESS. Seven entries added (11 -> 18) for claims the body taught with no classified entry behind them: the evidence classes, the "supersedes" vocabulary, the syntax-only GitHub check, the external-URL verdict, and the delete-vs-retire distinction. The recheck caught two of these still wrong. An entry said the schema "permits no field outside that set" after listing the six required fields -- it also permits "relationships", so the entry was wrong about the file it cited. And the "supersedes" entry cited node.schema.json, which carries only the enum; the directionality it asserts ("source replaces target; target becomes historical") lives in relationships.schema.json:30, a file this node had listed as read-but-unverified and cited anyway. Read it, cited it. 2. RETIREMENT WAS BUILT ON A FALSE PREMISE. It claimed inbound relationships "no longer resolve" after retirement. Measured: with the file retained -- which the same procedure requires -- they resolve fine; only deletion breaks them. Rewritten so the reason matches the mechanism, the danger is stated correctly (edges keep pointing at a node that stopped being current, and nothing warns you), and a replacement declares "supersedes", which the procedure never mentioned. 3. CITATION TABLE OVERSTATED VERIFICATION. A GitHub link pinned to a real SHA naming a file that never existed returns ok -- the checker never contacts GitHub. The table now carries a "does it prove the target exists?" column, gains the external-URL row it was missing, and names the trap. All seven rows independently confirmed against the classifier. 4. PROVENANCE SEMANTICS -- and the defect the recheck was right to refuse. The update procedure said to bump the revision on every edit; this node kept its original across seven. The first attempt defined the field as an evidence snapshot and classified that as INFERENCE citing node.schema.json. That schema says nothing whatever about revisions, so the citation could not support the statement: a policy choice had been given a class that made it look derived. It is now TEAM_KNOWLEDGE attributed to #636's own definition of done -- "the draft is checked against the repository revision recorded in provenance" -- which is where the meaning comes from. ADR-0029 requires exactly that: history and team knowledge stay attributed rather than being promoted to fact. The node also claimed it was "the only authored node in the corpus" at the recorded revision. It did not exist at that revision and neither did any other node: git cat-file -e on that path at that sha exits 128, and the corpus held zero .md files outside schema/. Corrected. The revision still does not move, and the reason is checkable rather than asserted: every cited source is byte-identical between that revision and now, so verifying a claim at HEAD and at the recorded revision were the same act. git diff --name-only -- returns nothing. 5. AUDIENCE. Added "reviewer": the document assigns reviewers an enforcement duty no tool performs, so they are addressed by it. The adjudicator had declined this finding (nothing consumes the field today); the final pass argued audience describes who is addressed, not who parses it. Went with the latter. Whether "developer" also belongs is left undecided. 6. The test module docstring claimed absolute fixture isolation that was false before this branch and more visibly false after it. States the rule with its one named exception instead. Flagged independently by the adjudicator and the final pass. Pre-existing, fixed while in the file: the same docstring advertised "python3 -m unittest launchpad.project_intelligence.corpus.tests.test_validate", which has never worked -- the directory is "project-intelligence" with a hyphen, not a legal package name. Confirmed ModuleNotFoundError, removed. 7. Full suites re-run in a writable environment: 79 validator tests OK, 27 schema tests OK, real corpus exit 0. The plan's two-node create/update/ retire charter ran with its negative control -- a deliberately invalid node failed and named itself, proving the scratch root was the one scanned; retirement kept beta's edge resolving; deleting alpha instead broke it exactly as the corrected text now says. Final state: 18 ledger entries, schema errors 0, every citation resolves, no cited source changed since the recorded revision, exactly one commit-only FACT, 79 + 27 tests green, real corpus exit 0. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 145 ++++++++++++++---- .../corpus/tests/test_validate.py | 29 +++- 2 files changed, 137 insertions(+), 37 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 5c54e047f3f..aa85d41444f 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -5,6 +5,7 @@ status: active origin: launchpad audiences: - agent + - reviewer evidence: - statement: "This node was authored and checked against repository revision 0052f5a7820ca4ca261efa233feb8bb53858ade6." entry_class: FACT @@ -14,7 +15,7 @@ evidence: entry_class: FACT evidence: - "launchpad/decisions/ADR-0028-corpus-canonical-representation.md" - - statement: "A node's front matter is validated against node.schema.json, which requires id, type, status, origin, audiences and evidence, and permits no field outside that set." + - statement: "A node's front matter is validated against node.schema.json, which requires id, type, status, origin, audiences and evidence, additionally permits relationships, and rejects any field beyond those seven." entry_class: FACT evidence: - "launchpad/docs/corpus/schema/node.schema.json" @@ -51,6 +52,36 @@ evidence: entry_class: FACT evidence: - ".github/workflows/launchpad-corpus-validate.yml" + - statement: "Evidence entries are classified FACT, INFERENCE or TEAM_KNOWLEDGE, and the class chosen decides which further fields the schema requires or forbids." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - "launchpad/project-intelligence/CONTRACT.md" + - statement: "supersedes is a typed relationship whose declared directionality is that the source replaces the target and the target becomes historical." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/relationships.schema.json" + - statement: "A non-GitHub external URL is reported UNVERIFIED, because the validator can neither pin it to a commit nor open it." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "A GitHub file link is checked for a full-SHA pin and a non-empty path segment only; the validator never establishes that the named file exists." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "Deleting a node breaks every relationship targeting it, while retiring it by status change leaves those relationships resolving." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "Retirement is therefore a status change that keeps the file and spends the id permanently, rather than a deletion." + entry_class: INFERENCE + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - "launchpad/decisions/ADR-0028-corpus-canonical-representation.md" + confidence: 0.8 + - statement: "The revision recorded in this ledger is the revision the node's claims were checked against, not a record of when the file was last edited." + entry_class: TEAM_KNOWLEDGE + provided_by: "launchpad-26/buzz#636 definition of done: 'The draft is checked against the repository revision recorded in provenance'" --- # Working with the documentation corpus @@ -146,17 +177,32 @@ documented anywhere else, so it is here — provisionally. This table is referen material rather than instruction, and belongs in the evidence standard once that lands (#1314); when it moves, this section links to it instead. -| Shape | Checker's verdict | -|---|---| -| Bare repository path | Resolved; must be a real **file** inside the repo. A directory fails. | -| Path with a line or line range | Resolved as a path. **The line number is not checked** — see below. | -| GitHub file link | Must be pinned to a full 40-character commit SHA, and must name a file. | -| Commit reference | Reported `UNVERIFIED`. Nothing on disk to open. | -| Graph edge | Reported `UNVERIFIED`. | -| Tool result | Reported `UNVERIFIED`. | +Read the middle column carefully: only two rows involve opening anything. + +| Shape | Checker's verdict | Does it prove the target exists? | +|---|---|---| +| Bare repository path | Opened on disk; must be a real **file** inside the repo. A directory fails. | **Yes** | +| Path with a line or line range | The path is opened. The line number is **not** checked at all. | File yes, line **no** | +| GitHub file link | **Syntax only.** Must be pinned to a full 40-character SHA and have a non-empty path segment after it. | **No** | +| External (non-GitHub) URL | Reported `UNVERIFIED`. Nothing to pin, nothing to open. | No | +| Commit reference | Reported `UNVERIFIED`. Nothing on disk to open. | No | +| Graph edge | Reported `UNVERIFIED`. | No | +| Tool result | Reported `UNVERIFIED`. | No | Anything matching **no** known shape is a hard error, not an `UNVERIFIED` notice. +**The GitHub row is the trap.** The checker never contacts GitHub. It reads the URL as +a string, and a link pinned to a real commit but naming a file that has never existed +passes as cleanly as a correct one: + +``` +https://github.com/launchpad-26/buzz/blob//does-not-exist.md -> ok +``` + +So a typo in a remote path ships silently. A repo-relative path is checked against the +filesystem and a GitHub link is not — prefer the former for anything in this +repository, and treat a GitHub link as a *pin*, not as evidence the target is there. + ### Three things a passing run does not mean **1. It does not mean a citation supports its claim.** Checking is *structural*. The @@ -253,31 +299,58 @@ standing in. ## Updating a node +**What the recorded revision means.** It is the revision the node's claims were +*checked against* — the wording comes from #636's definition of done, which requires +that "the draft is checked against the repository revision recorded in provenance", and +is attributed to that source in this node's ledger rather than inferred from the schema, +which says nothing about revisions. It is **not** a last-touched timestamp. Edit +prose without re-checking a source and the revision stays where it is; re-check the +sources and it moves, whether or not the body changed. Bumping it on every edit would +assert a verification that never happened, which is the one thing provenance exists to +prevent. + 1. **Confirm the change belongs in this node.** New idea, not new detail about the existing one? That is a new node. -2. **Re-record the revision you are checking against.** `git rev-parse HEAD` now, not - the one already in the ledger. -3. **Re-verify the claims you are touching** against sources at that revision. A claim - whose source moved is not still a `FACT` because it used to be. -4. **Update the ledger in the same edit as the body.** A new claim without an entry, or +2. **Re-verify the claims you are touching**, against those sources at current `HEAD` + (`git rev-parse HEAD`). A claim whose source moved is not still a `FACT` because it + used to be. +3. **Update the ledger in the same edit as the body.** A new claim without an entry, or an entry left behind by a deleted claim, are the two ways these drift apart. -5. **Update the recorded revision** to the one from step 2. -6. **Leave the `id` alone.** Always. -7. **Run the check.** +4. **Move the recorded revision to that `HEAD` only if you re-verified against it.** + If you re-verified every claim, move it. If you re-verified some, move it and be + sure the rest still hold at that revision too — a single node carries one snapshot, + so moving it makes a statement about the whole ledger. If you re-verified nothing, + leave it alone. +5. **Leave the `id` alone.** Always. +6. **Run the check.** ## Retiring a node -1. **Find what points at it.** Search the corpus for the node's `id`; any node with a - relationship targeting it needs handling first, or the corpus is left with a - relationship that no longer resolves. -2. **Decide what replaces it.** If another node takes over the subject, say so in that - node — a reader arriving at a retired node needs somewhere to go. -3. **Set `status` to the retired value** defined in `node.schema.json`. Do not delete - the file: generated views and inbound links resolve through the id, and deleting it - breaks them silently. -4. **Never reuse or rename the `id`.** A retired id stays spent. -5. **Record why**, in the body and in the ledger, at the revision you checked. -6. **Run the check.** +Retiring is a **status change, not a deletion**. The file stays, so the checker keeps +loading the node and inbound relationships keep resolving. Nothing here is enforced by +tooling — a retired node with stale inbound edges validates exactly like a healthy one. + +1. **Set `status` to the retired value** defined in `node.schema.json`. Do not delete + the file. Deleting it is what breaks inbound relationships: a + `relationships[].target` naming an id nothing carries is a hard error, and every + node pointing at the deleted one starts failing. +2. **Find what points at it.** Search the corpus for the node's `id`. Those edges will + still resolve — that is the problem, not the safety net. Readers and generated + views will keep being sent to a node that has stopped being current, and no check + will ever mention it. +3. **Decide what replaces it, and say so in the vocabulary.** If another node takes + over the subject, that node declares `supersedes` targeting the retired id — the + type exists in `relationships.schema.json` for exactly this. Repointing the inbound + edges from step 2 at the replacement is a judgement call: repoint the ones that + wanted the subject, leave the ones that genuinely meant the retired node. +4. **If nothing replaces it**, say that in the retired node's body. A reader arriving + from an old link needs to be told the subject is gone, not left guessing. +5. **Never reuse or rename the `id`.** A retired id stays spent — renaming breaks + generated views that resolve through it, and reuse silently points old references + at new content. +6. **Record why**, in the body and in the ledger, at the revision you checked. +7. **Run the check.** It will pass whether or not you did steps 2-4 correctly. That is + the point of doing them deliberately. ## Scope and omissions @@ -300,9 +373,19 @@ Until the standards land there is no per-type template to follow: write the node against `node.schema.json` and the rules above, and expect a later task to reshape it. **No `relationships` in this node's own front matter.** At the revision recorded in its -ledger this was the only authored node in the corpus, so there was nothing to point at -— and a `relationships[].target` naming an id no node carries is a hard error. The -absence is deliberate, not an oversight. +ledger the corpus contained **no** authored nodes at all — this one did not exist yet +either — and it is still the only one, so there is nothing to point at. A +`relationships[].target` naming an id no node carries is a hard error. The absence is +deliberate, not an oversight, and the first sibling node is the moment to revisit it. + +**Why this node's recorded revision has not moved.** It was edited many times after that +revision, and claims were added along the way — but every source it cites is +byte-identical between that revision and now, so checking a claim "at HEAD" and checking +it at the recorded revision were the same act. The snapshot covers the later claims for +that reason, not by assumption; it is checkable with `git diff --name-only -- `, which returns nothing. Per *Updating a node*, the revision tracks +verification rather than editing, so moving it would say a re-check happened where none +was needed. **Expected but not verified when this node was written**, per the rule in *Creating a node* step 3: diff --git a/launchpad/project-intelligence/corpus/tests/test_validate.py b/launchpad/project-intelligence/corpus/tests/test_validate.py index 49fea94cfe2..ffdff6ffc01 100644 --- a/launchpad/project-intelligence/corpus/tests/test_validate.py +++ b/launchpad/project-intelligence/corpus/tests/test_validate.py @@ -1,11 +1,28 @@ """Unit tests for the deterministic corpus validator -- issue #623. -Run: python3 -m unittest launchpad.project_intelligence.corpus.tests.test_validate - or: python3 -m unittest discover -s launchpad/project-intelligence/corpus/tests -p "test_*.py" - -These tests only ever point --root at fixtures under this directory, never at the -real launchpad/docs/corpus/ -- that root's own content (or lack of it) must never -change what this suite asserts. +Run: python3 -m unittest discover -s launchpad/project-intelligence/corpus/tests -p "test_*.py" + +(The dotted-module form this docstring used to advertise never worked: the directory is +`project-intelligence`, with a hyphen, which is not a legal Python package name, so it +raised ModuleNotFoundError. Broken since #623; found by a cross-model review pass.) + +Almost every test points --root at fixtures under this directory, so the real +launchpad/docs/corpus/ cannot change what they assert. That is the rule, and new +tests should follow it. + +ONE test is deliberately outside it: +`SchemaDirExclusionTest.test_real_corpus_root_discovery_matches_an_independent_walk` +walks the real corpus root and requires it to be non-empty. It exists to catch a +discovery regression against real committed content, which no fixture can stand in +for, and it will legitimately fail if the corpus is ever emptied or relocated -- at +which point it is reporting that, not a discovery bug. Its own docstring says what it +can and cannot catch. + +An earlier revision of this docstring claimed the never-touch-the-real-root rule was +absolute. It was not true even then: the test this one replaced also read the real +root, asserting it was EMPTY. Two independent reviews (an adjudicator and a +cross-model final pass) flagged the contradiction separately, which is why the rule is +now stated with its exception rather than as an absolute nobody was keeping. """ from __future__ import annotations From 64060a49a102a27f9f964ee138951fcd8727a21b Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 17:22:22 +1200 Subject: [PATCH 11/16] docs(corpus): standard for the confidence field (#1309) Adds launchpad/docs/corpus/standards/confidence.md, the canonical policy node for the confidence key on an evidence entry. The field is a number in a closed interval, required for INFERENCE only and forbidden on FACT and TEAM_KNOWLEDGE. The schema holds that much. What it cannot hold is what the number means, and an uninterpreted number between 0 and 1 is decoration -- an author picks 0.8 because it feels about right and a reader treats it as calibrated. The standard states that no calibration exists or is collected, that values are not comparable between authors, and that no confidence however high promotes an INFERENCE to a FACT. Its load-bearing section separates reasoning from evidence from dressing up a decision: an entry whose citation supports the subject of a claim but not the choice the claim makes is a decision in disguise and belongs in TEAM_KNOWLEDGE. That move was caught by cross-model review on #636 and no check can catch it, so it is documented with both worked examples. Records a measured divergence: a NaN confidence satisfies node.schema.json and passes corpus validation while memory.py rejects the identical value, because every comparison against NaN is false. Filed as #1463; the standard carries a MUST for finite values to cover it until that lands. Declares no relationships -- every sibling standard is unmerged, and a target naming an id no loaded node carries is a hard validation error. Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/standards/confidence.md | 352 ++++++++++++++++++ ...6-issue-1309-corpus-standard-confidence.md | 121 ++++++ 2 files changed, 473 insertions(+) create mode 100644 launchpad/docs/corpus/standards/confidence.md create mode 100644 launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md diff --git a/launchpad/docs/corpus/standards/confidence.md b/launchpad/docs/corpus/standards/confidence.md new file mode 100644 index 00000000000..bbf07ff3860 --- /dev/null +++ b/launchpad/docs/corpus/standards/confidence.md @@ -0,0 +1,352 @@ +--- +id: corpus-standard-confidence +type: governance +status: active +origin: launchpad +audiences: + - agent + - reviewer +evidence: + - statement: "This node was authored and checked against repository revision 60d4947b7145a6ef25f185b9c25d43e43d99de3c." + entry_class: FACT + evidence: + - "commit 60d4947b7145a6ef25f185b9c25d43e43d99de3c" + - statement: "confidence is a number between 0.0 and 1.0, required for INFERENCE entries only, and forbidden on FACT and TEAM_KNOWLEDGE entries." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - statement: "Both bounds are inclusive, an integer is accepted as a number, and a boolean or a quoted string is rejected on type before the range is ever considered." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - statement: "memory.py's __post_init__ raises when confidence is missing from an INFERENCE, present on any other class, not a real number, or outside the closed interval." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/memory.py" + - statement: "The corpus check enforces the confidence rule through node.schema.json alone; validate.py never imports memory.py, so the two are independent enforcement paths rather than one calling the other." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - "launchpad/docs/corpus/schema/node.schema.json" + - statement: "A confidence of NaN satisfies node.schema.json and passes corpus validation, while memory.py rejects the same value as outside the interval." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - "launchpad/project-intelligence/memory.py" + - statement: "Inference and team knowledge may supply context but are never treated as fact on their own, and stay attributed to their source and distinguishable from FACT claims." + entry_class: FACT + evidence: + - "launchpad/decisions/ADR-0029-corpus-evidence-precedence.md" + - statement: "The flagged status names two same-claim-type authoritative sources contradicting each other with no human resolution yet, and is explicitly not a generic low-confidence marker." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - "launchpad/decisions/ADR-0029-corpus-evidence-precedence.md" + - statement: "Citation checking is structural: the check confirms a cited path resolves to a real file and never that the file supports the statement it sits under." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/AGENTS.md" + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "The only INFERENCE in the corpus at the recorded revision is AGENTS.md's claim that retirement is a status change rather than a deletion, carried at confidence 0.8." + entry_class: FACT + evidence: + - "launchpad/docs/corpus/AGENTS.md" + - statement: "No corpus mechanism records whether a past inference turned out to be correct, so a confidence value can only express the author's assessed strength of reasoning and never an observed frequency." + entry_class: INFERENCE + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + - "launchpad/project-intelligence/corpus/validate.py" + - "launchpad/project-intelligence/memory.py" + confidence: 0.7 + - statement: "Because the schema accepts any in-range value without recording how it was chosen, two entries carrying the same number need not represent comparable strength, so values are not safely comparable between authors or nodes." + entry_class: INFERENCE + evidence: + - "launchpad/docs/corpus/schema/node.schema.json" + confidence: 0.8 + - statement: "An INFERENCE whose citation supports the subject of the claim but not the choice the claim makes is a decision in disguise, and belongs in TEAM_KNOWLEDGE attributed to whoever decided it." + entry_class: TEAM_KNOWLEDGE + provided_by: "launchpad-26/buzz#636 cross-model final review, relayed in the #1309 task brief: a second INFERENCE on AGENTS.md was rejected as laundering an unsourced policy choice into a class that made it look derived, and was reclassified to TEAM_KNOWLEDGE attributed to the issue's definition of done" +--- + +# Standard: `confidence` + +What the `confidence` number on an evidence entry means, when it is required, how an +author picks one, and what a reader may conclude from it. + +This is a policy node. Look up the section you need. + +| For | Read | +|---|---| +| The field's machine contract — type, range, which class requires or forbids it | `launchpad/docs/corpus/schema/node.schema.json` | +| Prose walkthrough of the front-matter fields | `launchpad/docs/corpus/schema/README.md` | +| The same rule enforced at runtime on the in-process store | `launchpad/project-intelligence/memory.py` | +| How to rank conflicting evidence, and when to stop and escalate | `launchpad/decisions/ADR-0029-corpus-evidence-precedence.md` | +| Creating, updating and retiring a node | `launchpad/docs/corpus/AGENTS.md` | + +Those files are authoritative. Where this document and any of them disagree, **they +win** — this one has drifted and should be fixed. The field-combination rules and the +enum members are deliberately not copied here: the check never reads body prose, so a +second copy would stay green forever after it went stale. + +## Scope and authority + +**This standard governs** the `confidence` key on an entry in a corpus node's +`evidence` ledger — the one place the field exists. + +**Its authority is derived, not original.** The structural half of this standard is +already law: `node.schema.json` enforces it, `validate.py` runs that schema, and CI runs +`validate.py`. This document does not create those rules and cannot relax them. What it +adds is the half no schema can hold — what the number is *for*, how to choose one +honestly, and what a reader is entitled to conclude. That half is enforced by review. + +**On the interpretation of an INFERENCE, ADR-0029 outranks this document.** It is the +accepted decision; this is a standard written under it. + +## What the number is for + +An INFERENCE is a claim you reasoned to rather than read. `confidence` is **the +author's own rating of how strongly the cited evidence supports the statement** — a +declared strength of reasoning, published so a reader can weigh the claim and a reviewer +can challenge it. + +That is the whole of it. In particular: + +- It is **not a probability**, and not a frequency. Nothing in the corpus records + whether a past inference turned out to be right, so no number here has ever been + scored against an outcome. There is no calibration behind it and none is collected. +- It is **not a quality score** for the node, the author, or the citation. +- It is **not a promotion path**. A confidence of 0.99 does not make an INFERENCE into a + FACT. Class is decided by *how you came to know the thing*, never by how sure you feel + about it, and ADR-0029 is explicit that inference is never treated as fact on its own. +- It is **not a conflict marker**. Two authoritative sources of the same claim type + saying different things is `status: flagged`, not a low number. See *Exceptions and + escalation*. + +**The failure this field exists to prevent.** A number between 0 and 1 with no stated +meaning is decoration. The specific way that goes wrong: an author writes `0.8` because +it feels about right, and a reader — or a generated view, or a later agent — treats it +as calibrated and reasons onward from it. The number then carries more weight than +anything that produced it. Everything below is aimed at that. + +## Requirements + +These are MUSTs. The first three are enforced mechanically; the rest are enforced by +review, and a reviewer who lets one through has approved a defect. + +1. **Every INFERENCE entry MUST carry a `confidence`, and no FACT or TEAM_KNOWLEDGE + entry may carry one.** This is structural and there is no exception to seek — the + schema rejects the node either way. +2. **The value MUST be a number within the closed interval the schema defines.** Both + bounds are inclusive. A quoted string is not a number and is rejected on type. +3. **The value MUST be finite.** The schema does not currently enforce this — see + *Enforcement, and where it stops* — so this one is on the author and the reviewer. +4. **The reasoning the number rates MUST be visible.** The reader has to be able to see + what was reasoned from what: in the `statement`, in the body section the entry + supports, or in both. A number attached to reasoning nobody can inspect cannot be + challenged, and an unchallengeable claim is the thing this corpus exists to avoid. +5. **An entry MUST NOT be an INFERENCE if its citation supports the subject of the claim + but not the choice the claim makes.** That is a decision, not a derivation. Reclassify + it to TEAM_KNOWLEDGE and name who decided. See *Reasoning versus deciding*. +6. **A number MUST NOT be moved because someone pushed back on it.** Re-verify the claim + or reclassify the entry. Adjusting the number to settle an argument records agreement + where there was none. +7. **When a claim is re-verified at a new revision, its confidence MUST be re-considered + in the same edit.** A number that outlived the reasoning it rated is worse than no + number, because it still looks current. + +## Guidance + +These are SHOULDs. Depart from them with a reason. + +**Use a coarse scale.** Three bands carry everything this field can honestly express: + +| Band | Means | Typical shape | +|---|---|---| +| High | The cited sources constrain the conclusion; a competent reader would reach the same one. | Two independent sources agree, and the step from them to the claim is short. | +| Medium | The reasoning is sound but rests on a step the sources do not fully close. | One source, plus a general principle. Or an absence-of-evidence argument over a scope you actually checked. | +| Low | You believe it, and you can see how it could be wrong. | A single weak source, or a long inferential chain. | + +**Two decimal places are not warranted.** The scale has no calibration behind it, so +`0.83` claims a precision that nothing supports. Pick one number per band and reuse it. + +**Prefer removing the inference to rating it.** If a source would settle the claim, open +the source and make it a FACT. High confidence is not a substitute for five minutes of +reading, and it is the cheaper-looking of the two only until someone relies on it. + +**Split compound claims rather than averaging.** If one half is solid and the other is a +guess, a middling number describes neither. Two entries, two numbers. + +**Do not publish a claim you would rate very low.** At that point the honest artefact is +not a weak INFERENCE but a named gap in the node's scope-and-omissions section — "this +was expected and could not be verified." A gap tells a reader to go and find out; a +low-confidence claim invites them to use it anyway. + +**Keep the number stable.** It moves when the reasoning or the evidence moves, and at no +other time. + +## Reasoning versus deciding + +**This is the distinction most worth getting right, and the one nothing will catch for +you.** + +Two entries can look identical in front matter — an `INFERENCE`, some citations, a +number — while being completely different objects: + +- **Reasoning from evidence.** The sources constrain the conclusion. Someone else, + handed only those sources, could get there. +- **Dressing up a decision.** Someone chose something. A citation was then attached that + is *about the same subject* but does not compel the choice. The class makes it look + derived. It was not derived; it was decided. + +The second is the more dangerous artefact, because `INFERENCE` implies the claim came +from the evidence, and a reader who trusts that will not go looking for the person who +actually made the call. The choice becomes unattributable — and an unattributable +decision cannot be revisited, because nobody can be asked why. + +**The test.** Read the statement and the citations, and nothing else. *Could a competent +reader, without knowing what the team wanted, arrive at this statement from these +sources?* + +- Yes → INFERENCE. Rate it. +- No, because the statement contains a choice the sources leave open → TEAM_KNOWLEDGE, + with `provided_by` naming the person, issue or decision that made it. + +**The tell** is a citation that supports the *topic* of the claim while saying nothing +about the *specific* thing being asserted. Read your own citation as an adversary would: +does it say this, or does it merely concern this? + +**Worked example — a legitimate INFERENCE.** `AGENTS.md` claims that retiring a node is +a status change rather than a deletion, at confidence 0.8, citing the checker and +ADR-0028. Both citations do real work: the checker's behaviour establishes that deleting +a node breaks every relationship targeting it, and ADR-0028's requirement that generated +projections derive reproducibly from a stable id establishes that the id cannot be +released. The conclusion follows from the two together. Neither source says "retirement +is a status change" — that step is the author's, which is exactly why it is an INFERENCE +and not a FACT, and why 0.8 rather than higher. + +**Worked example — the move to avoid.** That same node originally carried a second +INFERENCE, which a cross-model review rejected: it stated a policy choice and cited a +file that did not discuss the policy. It was reclassified to TEAM_KNOWLEDGE and +attributed to the issue's definition of done, which is where the choice had actually come +from. Nothing about the front matter had been invalid. The schema was satisfied, the +citation resolved, the check passed. Only a reader comparing the statement against the +source caught it. + +**This is the load-bearing point of the whole standard.** No number, however carefully +chosen, repairs a misclassified entry — a decision at confidence 0.4 is still a decision +wearing the wrong clothes. Get the class right first; the number is the smaller question. + +## What a reader may conclude + +**You may** read it as the author's rating of how strongly the cited evidence supports +the statement, and use it to decide what to check first. + +**You may not:** + +- Treat it as a probability that the claim is true. +- **Compare it across entries, nodes or authors.** The schema accepts any in-range value + and records nothing about how it was chosen, so one author's high band and another's + need not mean the same thing. Two entries at the same number are not thereby equally + strong. +- Aggregate, average or multiply values. They are not measurements and the arithmetic + means nothing. +- Read a high value as "nearly a FACT". The class is the load-bearing signal; the number + ranks entries within a class, and even that only against the same author's other work. +- Rely on it having been reviewed. A number that no reviewer questioned is a number + nobody objected to, which is not the same as a number anyone confirmed. + +**What a passing validation tells you about a confidence value:** that it is present +where required, absent where forbidden, and numerically in range. Nothing else. Checking +is structural — a citation is confirmed to resolve to a real file, never to support the +statement it sits under. + +## Enforcement, and where it stops + +**Enforced mechanically**, by `node.schema.json` through `validate.py`, and in CI on +every change under the corpus root: presence on INFERENCE, absence on the other two +classes, numeric type, and the closed range. A node violating any of these does not +merge. + +**Enforced separately at runtime**, by `memory.py`'s `__post_init__`, for the in-process +store. These are two independent paths to the same rule, not one calling the other — +`validate.py` does not import `memory.py`. Both must be satisfied by anything that +travels between them, and where they differ the stricter one is the safe assumption. + +**Not enforced by anything:** + +| Gap | Consequence | +|---|---| +| Whether the number is justified, or was reasoned at all | An arbitrary value passes cleanly. This standard's judgement rules are review-enforced only. | +| Whether the citation supports the claim | Checking is structural. A FACT or an INFERENCE citing a real file that says nothing on the subject passes. | +| Whether an INFERENCE is really a decision | The move described in *Reasoning versus deciding* is invisible to every check that exists. | +| **A non-finite value** | `confidence: .nan` satisfies the schema and passes corpus validation, while `memory.py` rejects the identical value as out of range. The schema's own description points at `memory.py` as the enforced rule, so this is a real divergence between the two paths and not a deliberate relaxation. Requirement 3 exists to cover it. | + +The pattern across that table: everything a schema can hold is held, and everything that +requires reading is not. Reviewing a confidence value means reading the sources. There +is no cheaper check, and a green run is not one. + +## Exceptions and escalation + +**There is no exception process for the structural requirements.** They are enforced +before merge and cannot be waived by agreement. Changing them means changing +`node.schema.json` under `launchpad/docs/corpus/schema/COMPATIBILITY.md`, which is a +schema change, not an exception. + +**When you cannot pick a number honestly**, that is a signal about the entry, not a +reason to guess. In order: + +1. **Can a source settle it?** Open it. The entry becomes a FACT and the question goes + away. +2. **Is it two claims?** Split it, and rate each. +3. **Is it actually a decision?** Reclassify to TEAM_KNOWLEDGE and attribute it. +4. **Is it none of those, and still not something you would stand behind?** Do not + publish it as a claim. Record it as a gap in the node's scope-and-omissions section. + +**When two authoritative sources of the same claim type contradict each other**, do not +express the disagreement as a middling confidence. That hides a conflict inside a number +and produces a node that looks merely tentative when it is actually unresolved. Record +the contradiction, set the node's `status` to `flagged`, and leave it for a human. +ADR-0029 is the governing rule and its escalation is deliberate: a flagged node is the +accepted safer failure mode, not a defect to be tidied away. + +**When a reviewer disputes a value**, the author re-verifies and then either moves the +number with a stated reason or reclassifies the entry. Requirement 6 forbids splitting +the difference. If it stays unresolved, it escalates as a conflict rather than settling +at an average nobody believes. + +## Scope and omissions + +**This document covers** what `confidence` means, its requirements and guidance, how to +choose a value, how to tell reasoning from a disguised decision, what a reader may +conclude, and what enforcement does and does not reach. + +**It does not cover, and these are gaps rather than silence:** + +| Not covered here | Owned by | +|---|---| +| The other evidence-entry fields — `entry_class`, `evidence`, `provided_by` — and the citation shapes | The evidence standard, #1314 | +| The field-combination matrix and the enum members themselves | `launchpad/docs/corpus/schema/node.schema.json` | +| Encoding ADR-0029's claim-type classification and the flagged state in the schema and checker | #1410 | +| Whether a node's classification is per-node or per-claim | Settled as per-entry by the schema; the wider question sits with #605 | +| Any numeric scale with real calibration behind it | Nothing. No such thing exists here, and this document does not invent one. | + +**No `relationships` in this node's front matter.** Every sibling standard is unmerged +at the recorded revision, and a `relationships[].target` naming an id no loaded node +carries is a hard validation error — so there is nothing this node could legally point +at. `AGENTS.md` is the only other authored node and is the natural first edge. The +absence is deliberate, and the moment the sibling standards land is the moment to +revisit it. + +**Expected but not verified when this node was written:** + +- **The bands in *Guidance* are not derived from anything.** They are a proposed + convention for keeping values coarse, offered because unbounded precision is the + observed failure. No study, no sample, and only one INFERENCE existed in the corpus to + look at. Treat them as a starting convention that practice should correct. +- **No generated view was tested consuming a `confidence` value.** No generator exists + yet, so how a projection renders or ranks these numbers is unknown, and the + "not comparable" rule above has not been tested against a consumer that might assume + otherwise. +- **The NaN divergence was measured against the current schema and `memory.py` only.** + Whether any other consumer of the field shares the schema's permissiveness or + `memory.py`'s strictness was not established. diff --git a/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md b/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md new file mode 100644 index 00000000000..6fef1d9451e --- /dev/null +++ b/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md @@ -0,0 +1,121 @@ +Issue #1309 — task: document corpus standard for confidence +Stated size: no `Size` line → cap: 5 steps (set by the feature #605 task brief, not asked per-issue) + +ALREADY TRUE (verified against git, not notes) + Worktree `__worktrees/task-1309-corpus-standard-confidence` is on branch + `task/1309-corpus-standard-confidence`, based on `origin/task/636-corpus-agents-md`, + HEAD `60d4947b7145a6ef25f185b9c25d43e43d99de3c`, working tree clean. + `launchpad/docs/corpus/AGENTS.md` exists on this base (the instruction node, #636, unmerged as PR #1462). + `launchpad/docs/corpus/standards/` does NOT exist — no sibling standard has landed. + The only authored corpus node today is `AGENTS.md`; everything else under + `launchpad/docs/corpus/` is `schema/`, which `validate.py` deliberately skips. + `node.schema.json` already encodes the confidence rule in three `allOf` branches: + INFERENCE requires `evidence` + `confidence`; FACT forbids `confidence` and `provided_by`; + TEAM_KNOWLEDGE requires `provided_by` and forbids `confidence`. + `launchpad/project-intelligence/memory.py` `__post_init__` enforces the same rule at runtime, + plus two checks the schema does not have (bool rejection, NaN rejection). + `validate.py` imports `jsonschema` and `yaml` only — it never imports `memory.py`. + The corpus path is enforced by the schema, not by `memory.py`. + Measured by running `Draft202012Validator` over synthesised nodes (recorded in STEP 1): + `0.0` and `1.0` both pass (bounds inclusive); integer `1` passes; `1.1` and `-0.1` fail; + `true` and `"0.8"` fail on type; **`.nan` PASSES the schema** but `memory.py` rejects it. + ADR-0029 is accepted and states inference is "never treated as fact on their own". + No open or closed issue on `launchpad-26/buzz` matches "confidence NaN schema". + +STEP 1 [independent] Record the measured constraint set as a reproducible probe script + under the session scratchpad, covering: the three `allOf` branches, both inclusive + bounds, integer acceptance, bool/string rejection, and the NaN divergence between + `node.schema.json` and `memory.py`. This is evidence-gathering, not a + deliverable — nothing under `launchpad/` changes in this step. + done when: the probe script runs and prints one PASS/FAIL line per case, and its + output shows `INFERENCE conf nan -> PASS` against the schema while + `memory.py` raises `confidence must be within [0.0, 1.0]` for the same value. + +STEP 2 [needs 1] Create `launchpad/docs/corpus/standards/confidence.md` with schema-valid + front matter only (`id: corpus-standard-confidence`, `type: governance`, + `status: active`, `origin: launchpad`, `audiences`, an `evidence` ledger whose + first entry is the `commit 60d4947b…` provenance citation, and **no** + `relationships` key). + done when: `python3 launchpad/project-intelligence/corpus/validate.py` exits 0 with + the new file present, and a YAML parse of the front matter (not a grep — a grep + confuses "prints 0" with "exits 1") reports `relationships` absent from the + top-level keys and `id == corpus-standard-confidence`. + +STEP 3 [needs 2] ← RUNS HERE Write the body: scope and authority, what the number + means, MUST vs SHOULD split, how an author picks one, what a reader may and may + not conclude, the reasoning-from-evidence versus dressing-up-a-decision + distinction, enforcement and its gaps, and the exception/escalation route to + `status: flagged` per ADR-0029. Link the schema and ADR-0029 rather than + restating enum lists or the field-combination matrix. + done when: `validate.py` exits 0; every `##` section named in this step is present + in the file; and the body links `launchpad/docs/corpus/schema/node.schema.json` + and `launchpad/decisions/ADR-0029-corpus-evidence-precedence.md` at least once each. + NOTE: "does not restate the field-combination matrix" is deliberately NOT a + done-when here. A one-line grep cannot detect a matrix restated as a table or a + bulleted list — review-plan proved that by constructing one that slipped through. + It is a reading judgement, so it belongs to STEP 4's audit, not to a regex. + +STEP 4 [needs 3] Audit the finished node against its own ledger: every `##`-level + substantive claim has an `evidence` entry, every `FACT` cites a source that was + actually opened, exactly one commit-only FACT exists, every INFERENCE carries a + confidence the document's own rules would justify, and the body nowhere restates + the schema's field-combination matrix or an enum member list (the judgement STEP 3 + deliberately did not delegate to a regex). + done when: `validate.py` exits 0; a YAML parse of the ledger — not a grep, so + indentation and quote style cannot skew the count — reports exactly one entry + whose every citation matches `^commit [0-9a-f]{40}$`; every INFERENCE entry has a + `confidence` key; and a written audit note maps each ledger entry to the body claim + it supports, naming any entry that supports none. + +STEP 5 [needs 4] Run the full corpus test suite and the validator as the last command of + their own invocations (verify-gate stamp), file the NaN-divergence finding as a + GitHub issue on `launchpad-26/buzz` linked to parent #605, then commit with `-s` + via an `-F` message file. + done when: `python3 -m unittest discover -s launchpad/project-intelligence/corpus/tests -p "test_*.py"` + reports OK; `validate.py` exits 0; `gh issue view --repo launchpad-26/buzz` + resolves and its body references #605; and `git log --format=%B -1` shows a + `Signed-off-by:` trailer. + +PARALLEL None. Steps 2, 3 and 4 all edit the same single file + (`launchpad/docs/corpus/standards/confidence.md`), so they are strictly + sequential regardless of how separable the content looks. STEP 1 touches no + repository file but produces the measurements STEP 2's ledger cites, so it + still precedes them. No step may be dispatched as a parallel subagent. + +GATES `review-plan` on this plan before STEP 1 (self-review, not independent). + `review-code` after STEP 5. `review-tests` does NOT apply — the diff adds no + test file and changes none; if that stops being true, it applies. + `review-adjudicate` over every finding both raise. A mandatory Codex + cross-model final pass after adjudication. + `qa` explore mode does **not** apply: the deliverable is a Markdown document + with no runtime interface to exercise. The only executable surface touched is + `validate.py`, which this change calls but does not modify. + +BUDGET STEP 3. The hard part is not length, it is saying what a confidence number + licenses a reader to conclude without either overclaiming calibration the + repository cannot support or hedging into uselessness. Expect the + reasoning-versus-decision distinction to need more than one pass. + +OPEN Whether `developer` belongs in `audiences`. #636 carries `agent` and + `reviewer`, and the brief calls the question deliberately unsettled. Planned + choice: `agent`, `reviewer` — an author picking a number is acting as agent, + a person checking it is acting as reviewer, and a developer reading corpus + prose is not addressed by this document in any way the other two do not + already cover. Stated so a reviewer can overturn it cheaply. + Whether the NaN divergence is a schema defect or an accepted limit of + structural validation. Planned handling: report it, do not fix it here — + `schema/` is out of this issue's scope and its DoD forbids a second authored + node. It becomes a filed issue, not a silent edit. + +LEFT OUT Any `relationships` edge. Every sibling standard (#1307–#1351) is unmerged, + and a `relationships[].target` naming an id no loaded node carries is a hard + validation error. The absence is stated in the body with its reason, as + `AGENTS.md` does for itself. + Editing `launchpad/docs/corpus/AGENTS.md`, even where this work suggests it + could be clearer — the brief forbids it and the DoD scopes this task to one + authored document. + Restating the enum members or the field-combination matrix in body prose. + `validate.py` never reads body prose, so a second copy would stay green + forever after it went stale. + Fixing `#1459` (line numbers unverified) or the NaN gap. Both are pre-existing + and owned elsewhere. From a1e8bbcd0846321c6f6684acfe551096da4d974a Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 17:26:24 +1200 Subject: [PATCH 12/16] docs(corpus): stop asserting provenance policy this node cannot source (#636) Third cross-model pass, third refusal on the same condition. It was right three times, and the third refusal finally identified why the first two fixes failed: the document was writing corpus-wide provenance policy it had no authority to write. Reclassifying it (INFERENCE, then TEAM_KNOWLEDGE) only changed the label on an unsourced decision. WHAT ACTUALLY CHANGED: the document stops making the claim. #1321 ("document corpus standard for provenance", unlanded) owns the question of whether a recorded revision may stay put across edits. This node now says so, states its own approach as working practice pending that standard rather than as a rule others must follow, and defers. The ledger entry no longer asserts the policy -- it records only what #636's definition of done actually requires, which is a fact about the issue and is attributed to it. Also fixes a self-contradiction the last fix opened, exactly the failure shape this repository keeps producing. The previous commit added a FACT citing relationships.schema.json for the supersedes directionality. The same node still said, in its unverified list, that relationships.schema.json had never been read. Both could not stand. The file WAS read; the stale disclaimer is now narrowed to what remains genuinely unchecked -- that the two enum lists agree, which rests on a test that was not run. Ledger completeness, third sweep: entries added for the generated-artifact rejection, the non-enforcement of the one-commit-only-FACT convention, and positional citations not being bounds-checked. Prose that asserted this node's own history -- node counts at the recorded revision, byte-identity of cited sources -- is removed rather than given entries: it told the reader to trust the document about a fact they can check themselves in one command, so the document now gives them the command instead. TEAM_KNOWLEDGE's description corrected. It said "a person told you", which the schema does not require (it asks who or what told the corpus) and which made the class look unavailable for an attributed issue or decision record. It now also says outright what it is not: a place to park a decision you made yourself. Attributing an extrapolation to the thing it started from does not make it something you were told. Updating step 4 restated as four explicit branches including the byte-identical case, which the earlier "bump on every edit" wording omitted while this very node relied on it -- the inconsistency the pass named. Verified: 21 ledger entries, schema errors 0, every citation resolves, exactly one commit-only FACT, no cited source changed since the recorded revision, 79 validator tests OK, real corpus exit 0. Refs #636 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 99 +++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 37 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index aa85d41444f..cbad36bf09e 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -79,9 +79,21 @@ evidence: - "launchpad/project-intelligence/corpus/validate.py" - "launchpad/decisions/ADR-0028-corpus-canonical-representation.md" confidence: 0.8 - - statement: "The revision recorded in this ledger is the revision the node's claims were checked against, not a record of when the file was last edited." + - statement: "Issue #636 requires that the draft is checked against the repository revision recorded in provenance." entry_class: TEAM_KNOWLEDGE - provided_by: "launchpad-26/buzz#636 definition of done: 'The draft is checked against the repository revision recorded in provenance'" + provided_by: "launchpad-26/buzz#636 definition of done" + - statement: "Every non-.md file under the corpus root is rejected today, including one placed under generated/, because no generator exists to reproduce it from canonical Markdown." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "A second or subsequent FACT resting only on a commit citation produces another non-fatal UNVERIFIED notice and does not fail the run." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" + - statement: "A citation naming a line or line range is checked for the path only; the line number is never compared against the file's length." + entry_class: FACT + evidence: + - "launchpad/project-intelligence/corpus/validate.py" --- # Working with the documentation corpus @@ -159,9 +171,12 @@ What the classes are *for* is the part that is easy to get wrong: probably says so." - **`INFERENCE`** — you reasoned to it from evidence. Reasoning is not fact, however good it is. -- **`TEAM_KNOWLEDGE`** — a person told you, and nothing corroborates it. This is the - class that exists for uncorroborated statements; using it honestly is better than - promoting a recollection to `FACT`. +- **`TEAM_KNOWLEDGE`** — something told to the corpus that no source corroborates, with + `provided_by` naming who or what said it: a person, an issue, a decision record. It + is the class for uncorroborated statements, and using it honestly beats promoting a + recollection to `FACT`. It is **not** a place to park a decision you made yourself — + attributing an extrapolation to the thing it started from does not make it something + you were told. When two sources disagree, do not average them and do not pick the newer one. For how the system **currently behaves**, executable evidence — code, config, schema, passing @@ -299,15 +314,13 @@ standing in. ## Updating a node -**What the recorded revision means.** It is the revision the node's claims were -*checked against* — the wording comes from #636's definition of done, which requires -that "the draft is checked against the repository revision recorded in provenance", and -is attributed to that source in this node's ledger rather than inferred from the schema, -which says nothing about revisions. It is **not** a last-touched timestamp. Edit -prose without re-checking a source and the revision stays where it is; re-check the -sources and it moves, whether or not the body changed. Bumping it on every edit would -assert a verification that never happened, which is the one thing provenance exists to -prevent. +**What the recorded revision means — working practice, not settled policy.** #636's +definition of done requires that "the draft is checked against the repository revision +recorded in provenance", so the revision is at minimum the one the claims were checked +against. Everything beyond that — whether it may stay put across edits, what to do when +only some claims are re-verified — is **#1321's** to settle and is not established here. +Until it does, this document works to the rule below, and says so rather than dressing +it up as a corpus-wide standard. 1. **Confirm the change belongs in this node.** New idea, not new detail about the existing one? That is a new node. @@ -316,11 +329,19 @@ prevent. used to be. 3. **Update the ledger in the same edit as the body.** A new claim without an entry, or an entry left behind by a deleted claim, are the two ways these drift apart. -4. **Move the recorded revision to that `HEAD` only if you re-verified against it.** - If you re-verified every claim, move it. If you re-verified some, move it and be - sure the rest still hold at that revision too — a single node carries one snapshot, - so moving it makes a statement about the whole ledger. If you re-verified nothing, - leave it alone. +4. **Decide whether the recorded revision moves.** A node carries one snapshot, so + moving it makes a statement about the whole ledger, not just the claims you touched. + - Re-verified every claim at `HEAD` → move it. + - Re-verified some → move it only if the rest still hold at `HEAD` too. Check, do + not assume. + - Re-verified nothing → leave it. + - **Every cited source byte-identical between the recorded revision and `HEAD`** → + leave it. Checking a claim at either point was the same act, so moving the + revision would assert a re-check that added nothing. + `git diff --name-only -- ` returning empty is + the test. This case is why the rule is stated as four branches and not as "bump on + every edit": that shorter rule was in an earlier draft and contradicted what this + very node does. 5. **Leave the `id` alone.** Always. 6. **Run the check.** @@ -372,20 +393,23 @@ what the deterministic check does and does not establish. Until the standards land there is no per-type template to follow: write the node against `node.schema.json` and the rules above, and expect a later task to reshape it. -**No `relationships` in this node's own front matter.** At the revision recorded in its -ledger the corpus contained **no** authored nodes at all — this one did not exist yet -either — and it is still the only one, so there is nothing to point at. A -`relationships[].target` naming an id no node carries is a hard error. The absence is -deliberate, not an oversight, and the first sibling node is the moment to revisit it. - -**Why this node's recorded revision has not moved.** It was edited many times after that -revision, and claims were added along the way — but every source it cites is -byte-identical between that revision and now, so checking a claim "at HEAD" and checking -it at the recorded revision were the same act. The snapshot covers the later claims for -that reason, not by assumption; it is checkable with `git diff --name-only -- `, which returns nothing. Per *Updating a node*, the revision tracks -verification rather than editing, so moving it would say a re-check happened where none -was needed. +**No `relationships` in this node's own front matter.** There is no other node to point +at, and a `relationships[].target` naming an id no node carries is a hard error. The +absence is deliberate; the first sibling node is the moment to revisit it. + +**How to check this node's own recorded revision.** Run +`git diff --name-only -- `. Empty output means +no cited source has moved since the revision was recorded, so every claim still stands +where it was checked. Do not take that on this document's word — the command is the +check. + +**What the corpus has NOT settled about revisions.** Whether a recorded revision may +stay put while a node is edited, and what an author must do when only some claims are +re-verified, is **#1321's** to decide (`document corpus standard for provenance`, +unlanded). Until it lands, *Updating a node* below states this document's working +practice, not a corpus-wide rule — three independent review passes rejected earlier +attempts to present it as one, on the grounds that no authorized source establishes it. +When #1321 lands, that section defers to it. **Expected but not verified when this node was written**, per the rule in *Creating a node* step 3: @@ -394,10 +418,11 @@ node* step 3: front matter is harmless to the checker, and to `preflight_core.py`, which resolves the path without parsing content. Whether it degrades the file for a harness that *reads* it as instructions is unknown. -- **`relationships.schema.json` was not read directly.** `node.schema.json` states that - a test guards the two relationship enums against drifting apart; that test was not - run. Immaterial to this node, which declares no relationships, but a reader relying - on the linked file for the enum is relying on that guard, not on a check made here. +- **The relationship enums in `node.schema.json` and `relationships.schema.json` were + not checked against each other.** `relationships.schema.json` was read for the + `supersedes` directionality this node cites. `node.schema.json` states that a test + guards the two enum lists against drifting apart; that test was not run, so a reader + relying on the two agreeing is relying on that guard rather than on a check made here. **This file is read twice.** It is a corpus node, validated like any other; it is also resolved as the nearest `AGENTS.md` for every change under `launchpad/docs/corpus/`, From 6b49a7c1e71d91bf5688b85420716e866b2974a2 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 17:38:36 +1200 Subject: [PATCH 13/16] docs(corpus): make the confidence standard obey itself (#1309) Two review findings, both adjudicated Medium, both defects in this document rather than in anything it describes. A standard that contradicts itself teaches the wrong lesson to every sibling that copies it, so both are fixed here rather than deferred. review-code, confirmed by review-adjudicate: the Guidance section said to pick one number per band and reuse it, but never said which number belonged to which band -- and then shipped two INFERENCE entries at two different values with no band label and no stated reason. The bands now carry explicit pegs of 0.8, 0.6 and 0.4, the criterial column is named so the illustrations stop reading as tests, and this node's own two entries are labelled to their bands as the nearest worked example. The scope-and-omissions entry now disclaims the specific values, not just the idea of banding: they are agreed pegs so two authors mean the same thing by the same numeral, and they measure nothing. review-plan, surviving into the document and caught by review-adjudicate: the header claimed the field-combination rules and enum members were deliberately not copied, while Requirement 1 restates the confidence row and names all three classes. A standard about this field cannot omit the field's own rule, so the claim was the wrong half to keep. The header now names that one restatement as a deliberate drift surface and points at the two places that must change if the schema's rule ever does. One neighbouring hole opened by the first fix and closed here: Requirement 6 forbids moving a number, and the second entry moved from 0.7 to 0.6. Snapping an unchanged assessment onto a band peg is not the move that rule prohibits, but from a diff the two are indistinguishable, so Requirement 6 now says so and requires the reason be recorded. This message is that record. Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/standards/confidence.md | 60 ++++++++++++++----- 1 file changed, 45 insertions(+), 15 deletions(-) diff --git a/launchpad/docs/corpus/standards/confidence.md b/launchpad/docs/corpus/standards/confidence.md index bbf07ff3860..6878835210e 100644 --- a/launchpad/docs/corpus/standards/confidence.md +++ b/launchpad/docs/corpus/standards/confidence.md @@ -57,7 +57,7 @@ evidence: - "launchpad/docs/corpus/schema/node.schema.json" - "launchpad/project-intelligence/corpus/validate.py" - "launchpad/project-intelligence/memory.py" - confidence: 0.7 + confidence: 0.6 - statement: "Because the schema accepts any in-range value without recording how it was chosen, two entries carrying the same number need not represent comparable strength, so values are not safely comparable between authors or nodes." entry_class: INFERENCE evidence: @@ -84,9 +84,17 @@ This is a policy node. Look up the section you need. | Creating, updating and retiring a node | `launchpad/docs/corpus/AGENTS.md` | Those files are authoritative. Where this document and any of them disagree, **they -win** — this one has drifted and should be fixed. The field-combination rules and the -enum members are deliberately not copied here: the check never reads body prose, so a -second copy would stay green forever after it went stale. +win** — this one has drifted and should be fixed. + +**One part of the schema's field-combination matrix is restated here and the rest is +not.** Requirement 1 below states the `confidence` row, because a standard about this +field cannot omit the field's own rule. Everything else is left to the schema +deliberately: the other entries' rules, the numeric bounds, and every enum's member list. +The check never reads body prose, so a copy that goes stale stays green forever — which +makes the one copy that does exist here a named drift surface rather than an accident. +**If the schema's confidence rule changes, Requirement 1 and the enforcement table in +*Enforcement, and where it stops* are the two places in this document that must change +with it.** ## Scope and authority @@ -149,7 +157,10 @@ review, and a reviewer who lets one through has approved a defect. it to TEAM_KNOWLEDGE and name who decided. See *Reasoning versus deciding*. 6. **A number MUST NOT be moved because someone pushed back on it.** Re-verify the claim or reclassify the entry. Adjusting the number to settle an argument records agreement - where there was none. + where there was none. Re-encoding an unchanged assessment onto the band values in + *Guidance* is not a move under this rule — the judgement is identical and only its + expression changed — but say so where the change is recorded, because from the diff + alone the two are indistinguishable. 7. **When a claim is re-verified at a new revision, its confidence MUST be re-considered in the same edit.** A number that outlived the reasoning it rated is worse than no number, because it still looks current. @@ -160,14 +171,29 @@ These are SHOULDs. Depart from them with a reason. **Use a coarse scale.** Three bands carry everything this field can honestly express: -| Band | Means | Typical shape | -|---|---|---| -| High | The cited sources constrain the conclusion; a competent reader would reach the same one. | Two independent sources agree, and the step from them to the claim is short. | -| Medium | The reasoning is sound but rests on a step the sources do not fully close. | One source, plus a general principle. Or an absence-of-evidence argument over a scope you actually checked. | -| Low | You believe it, and you can see how it could be wrong. | A single weak source, or a long inferential chain. | +| Band | Value | Means (this column decides the band) | Illustration, not a test | +|---|---|---|---| +| High | `0.8` | The cited sources constrain the conclusion; a competent reader given only them would reach the same one, and the step to it is short. | Sources that agree and leave little room to land elsewhere. | +| Medium | `0.6` | The reasoning is sound but rests on a step the sources do not fully close. | A general principle bridging a gap. An absence-of-evidence argument over a scope you actually checked. | +| Low | `0.4` | You believe it, and you can see how it could be wrong. | A single weak source, or a long inferential chain. | + +**Read the "Means" column, not the illustrations.** The fourth column is there to make the +bands concrete and is not criterial — a one-source claim can be High if that source really +does close the question, and a three-source claim can be Medium if all three leave the same +step open. Count what the sources *settle*, never how many there are. **Two decimal places are not warranted.** The scale has no calibration behind it, so -`0.83` claims a precision that nothing supports. Pick one number per band and reuse it. +`0.83` claims a precision that nothing supports. Use the band's value and nothing else. +The three values are evenly spaced, carry one decimal, and start at `0.8` because that is +what the corpus's only pre-existing INFERENCE already used — the convention continues +practice rather than inventing a fresh scale beside it. + +**This node's own ledger follows the convention**, and is the nearest worked example: + +| This node's INFERENCE | Band | Why | +|---|---|---| +| Values are not comparable between authors or nodes | High, `0.8` | One citation, but it settles the question: a schema that accepts any in-range value and records nothing about how it was chosen leaves no other conclusion available. | +| No mechanism records whether a past inference was correct, so the number is assessed strength and not observed frequency | Medium, `0.6` | Three citations, all of them silences. An absence-of-evidence argument over a scope that was checked — which is exactly as far as it can be pushed, because the scope is three files rather than the whole repository. | **Prefer removing the inference to rating it.** If a source would settle the claim, open the source and make it a FACT. High confidence is not a substitute for five minutes of @@ -339,10 +365,14 @@ revisit it. **Expected but not verified when this node was written:** -- **The bands in *Guidance* are not derived from anything.** They are a proposed - convention for keeping values coarse, offered because unbounded precision is the - observed failure. No study, no sample, and only one INFERENCE existed in the corpus to - look at. Treat them as a starting convention that practice should correct. +- **The bands in *Guidance*, and the values `0.8` / `0.6` / `0.4`, are not derived from + anything.** They are a proposed convention for keeping values coarse and comparable + within a node, offered because unbounded precision is the observed failure. No study, + no sample, and only one INFERENCE existed in the corpus to look at. `0.8` was chosen to + match that one entry and the other two spaced evenly from it — which is a reason, but + not evidence. A convention is all this is: three agreed pegs, so that two authors at + least mean the same thing by the same numeral. Treat them as a starting point that + practice should correct, and do not read the spacing as measuring anything. - **No generated view was tested consuming a `confidence` value.** No generator exists yet, so how a projection renders or ranks these numbers is unknown, and the "not comparable" rule above has not been tested against a consumer that might assume From bfb4885c048b6f9d990af2b500f9e5f4623b8ec3 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 18:03:51 +1200 Subject: [PATCH 14/16] docs(corpus): fix the standard's claims about itself (#1309) review-final found nine findings, and the cluster that mattered was not about confidence at all -- it was the document's claims about its own contents. For a node whose whole subject is not asserting more than the evidence supports, and which roughly 45 siblings are meant to copy, that is the report. All nine are defects in this document, all small, all inside the issue's scope, so they are fixed here rather than deferred. High. The stated reason for declaring no relationships was false. It said there was nothing this node could legally point at, and the next sentence named AGENTS.md. AGENTS.md carries id corpus-agents, is a loaded node, and a references edge to it validates today. The convention of declaring none is settled and unchanged; the justification is now the true one -- add the edge set in one pass once the siblings land -- and names corpus-agents as the first edge to make. The same false premise was in the plan's LEFT OUT and is corrected there too. Medium, four of them. The Requirements preamble claimed the first three MUSTs were machine-enforced while Requirement 3 said the schema does not reach it; it now says two. The header named two places tracking the schema when three do, and is now a table. The peg rationale asserted cross-author shared meaning that the reader rules and this node's own 0.8 INFERENCE both deny; adopting pegs is now stated as a hope about future practice, never as something a reader may sort on. The only-INFERENCE-in-the-corpus FACT cited AGENTS.md alone, which cannot establish a quantifier -- it now also cites validate.py for the schema/ exclusion and the fixture that carries the other one, which is exactly the merely-on-topic citation this document calls the tell. Medium, and the sharpest. Requirement 6's carve-out claimed re-encoding means the judgement is identical and only its expression changed. Measured against the document's own rules that is false: 0.7 to 0.6 moved the entry a full band from its neighbour, and ranking within a class is the one thing the number legitimately does. The carve-out now says re-encoding preserves the assessment but can change the rank, requires the record to name the Means row it matches, and admits there is no tie-break for a midpoint. Low, two. A number between 0 and 1 restated bounds the header said were left to the schema. And MUST be finite was overbroad -- .inf and -.inf are both caught on range, so NaN is the single value that escapes, because every comparison against it is false and the assertions never fire. Requirement 3 and the enforcement table both say so now. Per Requirement 6 as amended: the 0.6 entry matches the Medium band's Means row, an absence-of-evidence argument over a scope that was checked, because its scope is three files rather than the repository. Not fixed here: the NaN divergence itself stays filed as #1463, since the schema sits outside this issue's scope. Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/standards/confidence.md | 93 ++++++++++++------- ...6-issue-1309-corpus-standard-confidence.md | 12 ++- 2 files changed, 67 insertions(+), 38 deletions(-) diff --git a/launchpad/docs/corpus/standards/confidence.md b/launchpad/docs/corpus/standards/confidence.md index 6878835210e..820fbe4e159 100644 --- a/launchpad/docs/corpus/standards/confidence.md +++ b/launchpad/docs/corpus/standards/confidence.md @@ -47,10 +47,12 @@ evidence: evidence: - "launchpad/docs/corpus/AGENTS.md" - "launchpad/project-intelligence/corpus/validate.py" - - statement: "The only INFERENCE in the corpus at the recorded revision is AGENTS.md's claim that retirement is a status change rather than a deletion, carried at confidence 0.8." + - statement: "AGENTS.md carries an INFERENCE at confidence 0.8 claiming that retirement is a status change rather than a deletion, and it is the only INFERENCE in the validated corpus at the recorded revision because validate.py excludes the schema/ subtree, whose valid fixture carries one too." entry_class: FACT evidence: - "launchpad/docs/corpus/AGENTS.md" + - "launchpad/project-intelligence/corpus/validate.py" + - "launchpad/docs/corpus/schema/fixtures/valid/node-full.md" - statement: "No corpus mechanism records whether a past inference turned out to be correct, so a confidence value can only express the author's assessed strength of reasoning and never an observed frequency." entry_class: INFERENCE evidence: @@ -86,15 +88,22 @@ This is a policy node. Look up the section you need. Those files are authoritative. Where this document and any of them disagree, **they win** — this one has drifted and should be fixed. -**One part of the schema's field-combination matrix is restated here and the rest is -not.** Requirement 1 below states the `confidence` row, because a standard about this -field cannot omit the field's own rule. Everything else is left to the schema -deliberately: the other entries' rules, the numeric bounds, and every enum's member list. -The check never reads body prose, so a copy that goes stale stays green forever — which -makes the one copy that does exist here a named drift surface rather than an accident. -**If the schema's confidence rule changes, Requirement 1 and the enforcement table in -*Enforcement, and where it stops* are the two places in this document that must change -with it.** +**This document restates the schema only where its own subject forces it to.** A standard +about `confidence` cannot omit the rule for `confidence`, so Requirement 1 states that one +row of the field-combination matrix and Requirements 2 and 3 describe how the value itself +is bounded. What is *not* restated: the literal bound values, the rules for the other +entry fields, and every enum's member list. The check never reads body prose, so a copy +that goes stale stays green forever — which makes the copies that do exist here named +drift surfaces rather than accidents. + +**If the schema's rules change, these are the places in this document that must change +with them**, and there is nowhere else: + +| Restated here | Tracks | +|---|---| +| Requirement 1 | The `confidence` row of the field-combination matrix | +| Requirements 2 and 3, and the enforcement table in *Enforcement, and where it stops* | What the schema does and does not catch on the value itself — the NaN gap in particular, which #1463 is filed to close | +| The description of `flagged`, in *What the number is for* and in *Exceptions and escalation* | The schema's `status` description, which #1410 is filed to encode further | ## Scope and authority @@ -130,15 +139,16 @@ That is the whole of it. In particular: saying different things is `status: flagged`, not a low number. See *Exceptions and escalation*. -**The failure this field exists to prevent.** A number between 0 and 1 with no stated -meaning is decoration. The specific way that goes wrong: an author writes `0.8` because +**The failure this field exists to prevent.** A number with no stated meaning is +decoration. The specific way that goes wrong: an author writes `0.8` because it feels about right, and a reader — or a generated view, or a later agent — treats it as calibrated and reasons onward from it. The number then carries more weight than anything that produced it. Everything below is aimed at that. ## Requirements -These are MUSTs. The first three are enforced mechanically; the rest are enforced by +These are MUSTs. The first two are enforced mechanically. Requirement 3 is a MUST that +no check reaches — see *Enforcement, and where it stops*. The rest are enforced by review, and a reviewer who lets one through has approved a defect. 1. **Every INFERENCE entry MUST carry a `confidence`, and no FACT or TEAM_KNOWLEDGE @@ -146,8 +156,11 @@ review, and a reviewer who lets one through has approved a defect. schema rejects the node either way. 2. **The value MUST be a number within the closed interval the schema defines.** Both bounds are inclusive. A quoted string is not a number and is rejected on type. -3. **The value MUST be finite.** The schema does not currently enforce this — see - *Enforcement, and where it stops* — so this one is on the author and the reviewer. +3. **The value MUST be a real number, not NaN.** The infinities are already caught on + range — `.inf` exceeds the maximum and `-.inf` falls below the minimum — so NaN is + the single value that escapes, because every comparison against it is false and the + range assertions therefore never apply. See *Enforcement, and where it stops*. This + one is on the author and the reviewer alone. 4. **The reasoning the number rates MUST be visible.** The reader has to be able to see what was reasoned from what: in the `statement`, in the body section the entry supports, or in both. A number attached to reasoning nobody can inspect cannot be @@ -157,10 +170,13 @@ review, and a reviewer who lets one through has approved a defect. it to TEAM_KNOWLEDGE and name who decided. See *Reasoning versus deciding*. 6. **A number MUST NOT be moved because someone pushed back on it.** Re-verify the claim or reclassify the entry. Adjusting the number to settle an argument records agreement - where there was none. Re-encoding an unchanged assessment onto the band values in - *Guidance* is not a move under this rule — the judgement is identical and only its - expression changed — but say so where the change is recorded, because from the diff - alone the two are indistinguishable. + where there was none. Re-encoding an existing value onto the band values in *Guidance* + is not a move under this rule, but do not tell yourself nothing changed: snapping to a + peg preserves the assessment while it can and does alter the entry's **rank** against + its neighbours, which is the one thing the number legitimately does. So the record must + name the band's "Means" row the assessment matches and why — not merely report that a + re-encoding happened. A value equidistant from two pegs has no tie-break rule here; it + is decided by the "Means" column and by nothing else. 7. **When a claim is re-verified at a new revision, its confidence MUST be re-considered in the same edit.** A number that outlived the reasoning it rated is worse than no number, because it still looks current. @@ -185,7 +201,7 @@ step open. Count what the sources *settle*, never how many there are. **Two decimal places are not warranted.** The scale has no calibration behind it, so `0.83` claims a precision that nothing supports. Use the band's value and nothing else. The three values are evenly spaced, carry one decimal, and start at `0.8` because that is -what the corpus's only pre-existing INFERENCE already used — the convention continues +what the only INFERENCE in the validated corpus already used — the convention continues practice rather than inventing a fresh scale beside it. **This node's own ledger follows the convention**, and is the nearest worked example: @@ -305,7 +321,7 @@ travels between them, and where they differ the stricter one is the safe assumpt | Whether the number is justified, or was reasoned at all | An arbitrary value passes cleanly. This standard's judgement rules are review-enforced only. | | Whether the citation supports the claim | Checking is structural. A FACT or an INFERENCE citing a real file that says nothing on the subject passes. | | Whether an INFERENCE is really a decision | The move described in *Reasoning versus deciding* is invisible to every check that exists. | -| **A non-finite value** | `confidence: .nan` satisfies the schema and passes corpus validation, while `memory.py` rejects the identical value as out of range. The schema's own description points at `memory.py` as the enforced rule, so this is a real divergence between the two paths and not a deliberate relaxation. Requirement 3 exists to cover it. | +| **A NaN value — and only NaN** | `confidence: .nan` satisfies the schema and passes corpus validation, while `memory.py` rejects the identical value as out of range. The infinities do not escape: `.inf` and `-.inf` are both caught on the bounds. NaN slips through because every comparison against it is false, so the range assertions never fire rather than failing. The schema's own description points at `memory.py` as the enforced rule, so this is a real divergence between the two paths and not a deliberate relaxation. Requirement 3 covers it until #1463 closes it. | The pattern across that table: everything a schema can hold is held, and everything that requires reading is not. Reviewing a confidence value means reading the sources. There @@ -356,23 +372,32 @@ conclude, and what enforcement does and does not reach. | Whether a node's classification is per-node or per-claim | Settled as per-entry by the schema; the wider question sits with #605 | | Any numeric scale with real calibration behind it | Nothing. No such thing exists here, and this document does not invent one. | -**No `relationships` in this node's front matter.** Every sibling standard is unmerged -at the recorded revision, and a `relationships[].target` naming an id no loaded node -carries is a hard validation error — so there is nothing this node could legally point -at. `AGENTS.md` is the only other authored node and is the natural first edge. The -absence is deliberate, and the moment the sibling standards land is the moment to -revisit it. +**No `relationships` in this node's front matter, and the reason is narrower than it +looks.** Every sibling standard is unmerged at the recorded revision, so no edge to one +would resolve — a `relationships[].target` naming an id no loaded node carries is a hard +validation error. But that is not the same as *nothing* being linkable: `AGENTS.md` +carries `id: corpus-agents`, is a loaded node, and a `references` edge to it would +validate today. The absence is a deliberate choice to add the whole edge set in one pass +once the siblings land, not a claim that the corpus offers nothing to point at. The +first sibling standard to merge is the moment to revisit it, and `corpus-agents` is the +edge to make first. **Expected but not verified when this node was written:** - **The bands in *Guidance*, and the values `0.8` / `0.6` / `0.4`, are not derived from - anything.** They are a proposed convention for keeping values coarse and comparable - within a node, offered because unbounded precision is the observed failure. No study, - no sample, and only one INFERENCE existed in the corpus to look at. `0.8` was chosen to - match that one entry and the other two spaced evenly from it — which is a reason, but - not evidence. A convention is all this is: three agreed pegs, so that two authors at - least mean the same thing by the same numeral. Treat them as a starting point that - practice should correct, and do not read the spacing as measuring anything. + anything.** They are a proposed convention for keeping values coarse, offered because + unbounded precision is the observed failure. No study, no sample, and one authored + INFERENCE in the corpus to look at. `0.8` was chosen to match that entry and the other + two spaced evenly from it — which is a reason, but not evidence. Treat them as a + starting point that practice should correct, and do not read the spacing as measuring + anything. +- **Adopting the pegs does not make values comparable, and this document does not claim + it does.** The prohibition in *What a reader may conclude* is unconditional and stands: + the schema guarantees nothing about how any value was chosen, so no reader may compare + across authors or nodes. What a shared convention can do is give *future* authors who + followed it a common vocabulary — which is a hope about practice, not a property a + reader may rely on, and certainly not licence to sort. Until something records that an + author followed the pegs, a reader cannot tell whether they did. - **No generated view was tested consuming a `confidence` value.** No generator exists yet, so how a projection renders or ranks these numbers is unknown, and the "not comparable" rule above has not been tested against a consumer that might assume diff --git a/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md b/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md index 6fef1d9451e..7971a732906 100644 --- a/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md +++ b/launchpad/plans/2026-08-26-issue-1309-corpus-standard-confidence.md @@ -107,10 +107,14 @@ OPEN Whether `developer` belongs in `audiences`. #636 carries `agent` and `schema/` is out of this issue's scope and its DoD forbids a second authored node. It becomes a filed issue, not a silent edit. -LEFT OUT Any `relationships` edge. Every sibling standard (#1307–#1351) is unmerged, - and a `relationships[].target` naming an id no loaded node carries is a hard - validation error. The absence is stated in the body with its reason, as - `AGENTS.md` does for itself. +LEFT OUT Any `relationships` edge. Every sibling standard (#1307–#1351) is unmerged, and + a `relationships[].target` naming an id no loaded node carries is a hard + validation error, so no edge to a sibling would resolve. CORRECTION, from + review-final: that does not mean nothing is linkable — `AGENTS.md` carries + `id: corpus-agents`, is a loaded node, and a `references` edge to it would + validate today. Declaring none is a deliberate choice to add the whole edge set + in one pass once the siblings land, and the body must say that rather than + claiming nothing is legal to point at. Editing `launchpad/docs/corpus/AGENTS.md`, even where this work suggests it could be clearer — the brief forbids it and the DoD scopes this task to one authored document. From 806af4e4153b438ed287173e09aa8926b2d40a5a Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 18:13:35 +1200 Subject: [PATCH 15/16] docs(corpus): fix four AGENTS.md defects found by authoring against it (#636) Three sibling agents authored corpus nodes following this document literally, and reported what broke. These four are the defects that produced real false claims in their work, not stylistic complaints. Filed as #1471; fixed here because #1462 is still open and 45 further nodes would inherit them. 1. THE ONLY-NODE JUSTIFICATION WENT FALSE, AND TWO AGENTS COPIED IT. "A node with no sibling to point at correctly declares none" read as a general rule. It was a fact about one moment -- when this was the corpus's only node -- and stopped holding as soon as a second existed. Both the #1309 and #1307 agents reproduced the reasoning and wrote a justification that was false at the time they wrote it; their adjudicators caught it, not their own reviews. Now says to enumerate what exists and give the real reason, and records why. 2. AN ISSUE-ONLY SOURCE HAD NO HONEST CLASS. The citation table lists tool results as legitimate while "Three things a passing run does not mean" says a FACT resting only on UNVERIFIED citations "has not been checked by anything", with one named exception. A claim whose only source is a GitHub issue has no openable form and cannot be SHA-pinned, so both passages applied and disagreed. The #1307 agent hit this with nothing honest to choose. Now states it: TEAM_KNOWLEDGE with provided_by naming the issue, which is what ADR-0029 requires of GitHub history anyway. 3. A RANGE WAS READ AS A MAPPING. The not-covered table paired eight subjects with "#1307-#1351", which is 45 issues and no subject-to-issue pairing at all. The #1307 agent cited this table for nine invented mappings -- exactly the "FACT cited to a file that does not discuss the claim" failure the document warns about two sections earlier. Now says outright that it is a range, not a mapping, and gives the search command. 4. CREATE STEP 3 ROUTED SOMETHING STEP 8 DID NOT ACCEPT. Step 3 sends expected-but-unverified items to the scope section; step 8 said only "state what the node does not cover". A boundary and a confidence disclosure are different things, so working the steps in order left that category homeless. Both the #1307 and #1309 agents reported it independently. Step 8 now names both. Verified: schema errors 0, 21 ledger entries, validator exit 0, 79 tests OK. Refs #636 #1471 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 35 ++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index cbad36bf09e..4025b334a74 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -142,8 +142,14 @@ prose, so a stale copy would stay green forever. derive from it reproducibly, so renaming an id is a migration, not an edit. **Relationships are optional and must resolve.** A `relationships[].target` naming an -id no node in the corpus carries is a hard error. A node with no sibling to point at -correctly declares none. +id no node in the corpus carries is a hard error, so an edge may only name a node that +is already merged. Declaring none is always valid — but **check before you justify it**. +"There was nothing to point at" was true when this was the corpus's only node and stops +being true the moment a second one merges. Enumerate what exists +(`ls launchpad/docs/corpus/**/*.md`) and give the real reason, which may simply be that +the edges are being added in one pass later. Two independent agents authoring sibling +nodes copied an earlier version of this paragraph and produced a **false** justification +from it, because it read as a general rule rather than a fact about one moment. **Authored versus generated.** Every non-`.md` file under the corpus root must live in a `generated/` directory. Today the checker rejects such files even there, because @@ -242,6 +248,14 @@ Run that, and the entry is a `FACT`. Every other claim needs a source you opened commit citation attached to a claim *about repository content* is not covered — that claim needs the file, at that revision. +**When the only source is an issue, a PR or a discussion**, you have no openable file and +no way to pin one: the validator's repository-link check matches only file and tree views, +so an issue URL is an external URL and lands on `UNVERIFIED`. Do not force it into a +`FACT` on a tool-result or URL citation. Use `TEAM_KNOWLEDGE` with `provided_by` naming +the issue — that is what the class is for, and ADR-0029 requires GitHub history to stay +attributed rather than be promoted to fact. An earlier draft of this section left that +case with no honest class at all, which an agent authoring a sibling node hit directly. + **Nothing enforces this.** The checker treats every commit citation identically: a second, third or tenth `FACT` resting only on `commit ` produces nothing but extra non-fatal `UNVERIFIED` notices and still exits 0. Verified by adding one and @@ -307,7 +321,11 @@ standing in. the revision from step 3 — the ledger is the only schema-legal place for it. 7. **Write one `evidence` entry per substantive claim** you intend to make. Classify honestly; open every source you call a `FACT`. -8. **Write the body**, structured for lookup. State what the node does not cover. +8. **Write the body**, structured for lookup, with a scope section carrying **two + distinct things**: what the node does not cover and who owns it, and — separately — + what you expected to verify from step 3 and could not. A boundary and a confidence + disclosure are different, and an earlier version of this step named only the first, + leaving step 3's third category with nowhere to go. 9. **Add relationships only to nodes that exist.** A target no node carries is a hard error. None is a valid answer. 10. **Run the check.** Fix what it names, and re-run until it exits 0. @@ -383,8 +401,15 @@ what the deterministic check does and does not establish. | Not covered here | Owned by | |---|---| -| Per-type standards — naming, identifiers, linking, provenance, status, taxonomy, diagrams, evidence | #1307–#1351, none merged yet | -| Templates for each node type — concept, component, capability, interface, flow, policy, procedure, runbook, reference, specification, and the rest | #1307–#1351, none merged yet | +| Per-type standards — naming, identifiers, linking, provenance, status, taxonomy, diagrams, evidence | somewhere in #1307–#1351 | +| Templates for each node type — concept, component, capability, interface, flow, policy, procedure, runbook, reference, specification, and the rest | somewhere in #1307–#1351 | + +**That range is a range, not a mapping.** It is 45 issues across those subjects, and this +table does **not** say which number owns which subject. Look the subject up +(`gh issue list --repo launchpad-26/buzz --search "corpus standard for "`) rather +than citing this table for a subject-to-issue pairing — an agent authoring a sibling node +did exactly that and produced nine invented mappings, which is the "FACT cited to a file +that does not discuss the claim" failure this document warns about two sections up. | How generated artifacts prove their provenance, and the exception process for them | #1316 | | Encoding ADR-0029's claim-type classification and the flagged state in the schema and checker | #1410 | | The human-facing entry point to the corpus | #639 | From ebe2daf721c7d7a96fdd84eba0a0a5d37eefa109 Mon Sep 17 00:00:00 2001 From: Serina Mcfall Date: Wed, 26 Aug 2026 18:19:05 +1200 Subject: [PATCH 16/16] docs(corpus): fix a false attribution and the relationship-target trap (#636) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two more defects reported by the #1308 agent after authoring against this document. Both are mine, and the first is embarrassing in context. 1. THE CITATION TABLE MISATTRIBUTED ITSELF, IN THE SECTION ABOUT NOT DOING THAT. It said "CONTRACT.md §3 defines the six shapes" above a SEVEN-row table whose GitHub-link and external-URL rows appear nowhere in §3. Measured: grepping that section for http/url/github returns zero hits. So the sentence cited a file that does not discuss two of the rows beneath it -- the exact failure this document warns about, committed by this document. The #1308 agent took the table as an accurate summary and built a scope argument on "four of six shapes name code" before their plan review caught it. Now states what §3 actually enumerates, that the two URL forms are the validator's and not §3's, and that the table is seven rows and not a summary. 2. "ADD RELATIONSHIPS ONLY TO NODES THAT EXIST" WAS UNFOLLOWABLE, AND DANGEROUSLY SO. "Exist" was ambiguous between the branch you are working on and the branch you merge into, and the checker loads whatever is present where it runs. An agent branched off this unmerged node could therefore target `corpus-agents`, validate clean locally, and break CI on `launchpad`, where the node is absent. The #1308 agent hit exactly that and their review-code raised it as a Blocker. Step 9 now says "exists on the branch you are merging INTO", explains why the distinction is the trap, and gives the command to check the merge base rather than the local tree. This is the second round of AGENTS.md fixes sourced from agents authoring against it, and the pattern is worth naming: every defect found this way was invisible to three review passes over the document itself, and obvious within minutes to someone trying to follow it. Reviewing a procedure is not the same as executing one. Verified: validator exit 0, 79 tests OK, schema clean. Refs #636 #1471 #1478 Signed-off-by: Serina Mcfall --- launchpad/docs/corpus/AGENTS.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/launchpad/docs/corpus/AGENTS.md b/launchpad/docs/corpus/AGENTS.md index 4025b334a74..e42228305d9 100644 --- a/launchpad/docs/corpus/AGENTS.md +++ b/launchpad/docs/corpus/AGENTS.md @@ -193,7 +193,13 @@ human rather than resolving it yourself. `ADR-0029` is the full rule. ### What the checker does with each citation shape -`CONTRACT.md` §3 defines the six shapes. What `validate.py` does with them is not +`CONTRACT.md` §3 enumerates six shapes — file range, file line, bare path, graph edge, +tool result, commit. It contains **no URL form at all** (grep it for `http`: zero hits). +The two URL rows below are forms `validate.py` recognises and §3 does not enumerate, so +this table is **seven** rows and is not a summary of §3. An earlier version of this +sentence claimed it was, and an agent authoring a sibling node built a scope argument on +the miscount before their plan review caught it. What `validate.py` does with any of them +is not documented anywhere else, so it is here — provisionally. This table is reference material rather than instruction, and belongs in the evidence standard once that lands (#1314); when it moves, this section links to it instead. @@ -326,8 +332,15 @@ standing in. what you expected to verify from step 3 and could not. A boundary and a confidence disclosure are different, and an earlier version of this step named only the first, leaving step 3's third category with nowhere to go. -9. **Add relationships only to nodes that exist.** A target no node carries is a hard - error. None is a valid answer. +9. **Add relationships only to nodes that exist on the branch you are merging INTO.** + Not the branch you are working on — that distinction is the whole trap. The checker + loads whatever is present where it runs, so a target that resolves in your worktree + can be a hard error in CI: an agent branched off an unmerged node targeted it, + validated clean locally, and would have broken the run on `launchpad` where that node + does not exist yet. Check against the merge base + (`git ls-tree -r --name-only origin/launchpad -- launchpad/docs/corpus`), not your own + tree. None is a valid answer, and while the corpus is being built out it is usually + the right one. 10. **Run the check.** Fix what it names, and re-run until it exits 0. ## Updating a node