docs(corpus): evidence standard (#1314) - #1491
Conversation
Creates launchpad/docs/corpus/standards/evidence.md with its complete evidence
ledger, the twelve section headings, and the scope-and-authority section.
The ledger is written first and in full because it is the contract the remaining
steps fill in: a body claim added later without a matching entry is invisible to
every check that exists, so predicting the claims up front is cheaper than
auditing for orphans afterwards.
Scope decisions taken here rather than left implicit. This node is the batch's
most-deferred-into one, so it says outright what it accepts and what it hands
back:
- AGENTS.md says its citation-shape table belongs in the evidence standard
once that lands. This node takes the half that names no openable file --
commit, graph edge, tool result, non-GitHub URL -- and leaves the
code-naming forms with the code-references standard, which already treats
them in more detail. A second copy here would be the exact stale-duplicate
failure these conventions exist to prevent.
- The confidence number stays with #1309. This node decides which class;
that one decides what value.
No relationships, and the reason is merge order rather than an empty corpus.
origin/launchpad carries no node outside schema/, so any edge declared here
would validate locally and be a hard error in CI.
Claims about the two unmerged sibling branches are TEAM_KNOWLEDGE attributed to
the issues that record them, not FACTs: their files do not exist on this branch,
so no citation to them could resolve, and a pinned link would prove nothing.
Verified: validator exit 0 with exactly 2 UNVERIFIED notices (this node's
provenance entry and corpus-agents'), 12 section headings, 31 non-blank lines
under the scope section, 79 validator tests OK.
Refs #1314
Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
Writes the ledger section and the class section. The load-bearing sentence is that the class records HOW YOU CAME TO KNOW the thing -- not how sure you are, not how important the claim is, and not how good the source looks. Two things stated that no other document states: Class is not a ranking. FACT is not the good one and TEAM_KNOWLEDGE is not the weak one. A FACT cited to a file whose author was wrong is a faithfully-recorded error; a TEAM_KNOWLEDGE entry attributed to the person who decided is the most authoritative record of why that will ever exist. Reaching for FACT because it sounds stronger converts an honest attribution into an unattributable assertion. Class has exactly one mechanical consequence, and it is in memory.py rather than in the corpus: code contradicting a FACT or INFERENCE supersedes it with a new FACT, while the same observation never supersedes a TEAM_KNOWLEDGE entry -- only a person's later statement can. That asymmetry is the clearest operational statement of what the classes mean, and it is labelled as memory.py's behaviour rather than the corpus's, because the node schema defines no supersession or temporal fields at all. A corpus ledger records claims; git records their history. Neither the field-combination matrix nor any enum member list is restated. Two properties of the matrix are named in prose without reproducing it, because authors get both wrong: INFERENCE needs citations just as FACT does, and TEAM_KNOWLEDGE needs none. Verified: validator exit 0 with 2 UNVERIFIED notices, zero inline copies of the status, type or audiences enums, 79 validator tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
…1314 STEP 3) The centrepiece. The distinction nothing can check for you, and the section this node exists to carry. THE ADDITION THAT MATTERS: three outcomes, not two. The usual treatment is a binary -- if it is not really an INFERENCE, make it TEAM_KNOWLEDGE and attribute it. That assumes there is always somebody to attribute it to. When the sources leave the choice open and the author made it while writing, no class is honest and the claim is withdrawn: a named gap in scope-and-omissions, not a relabelled assertion. The worked example is this branch's own history, given as a command to run (git log -1 a1e8bbc) rather than as something to take on trust. AGENTS.md classified an authored policy INFERENCE citing a silent schema, then TEAM_KNOWLEDGE attributed to an issue's definition of done, and three cross-model passes refused both on the same condition. The third refusal found why the first two fixes failed: reclassifying only changed the label on an unsourced decision. The front matter was never invalid -- schema satisfied, citations resolved, checker exit 0 on all three rounds. The test is framed at the ledger level -- does the source SAY this, or merely CONCERN this -- and hands the INFERENCE-rating half to #1309 explicitly rather than restating it, since that node owns the number. When sources disagree defers to ADR-0029 for the rule and adds only the part an author gets wrong: decide which KIND of claim you are making first, because behaviour claims and intent claims have opposite tiebreakers, and most apparent conflicts are two separate FACTs rather than one conflict. A real conflict sets status flagged and stops; it is never expressed as a hedge. Verified: validator exit 0 with 2 UNVERIFIED notices; the section names all three outcomes (INFERENCE 4, TEAM_KNOWLEDGE 3, withdraw 1); a1e8bbc named twice in the document; 79 validator tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
…#1314 STEP 4) Writes the remaining six sections. What a citation establishes leads with the fact the whole node turns on: checking is STRUCTURAL. A FACT citing a real file that says nothing on its subject passes with no notice at all -- not an error, not an UNVERIFIED line, nothing. The table covers only this node's half of the vocabulary, the forms that name no openable file, and points at #1308 for the code-naming forms. Measured, not read out of another document: a commit citation is never checked against the object store, so a commit id that has never existed in this repository is reported identically to one that has. The one permitted commit-only FACT rests on git cat-file -e, which the validator itself never runs. #1478 is stated rather than inherited. CONTRACT.md section 3 enumerates six shapes with zero occurrences of http; validate.py implements a URL branch anyway; AGENTS.md presents seven rows as six. This node does not resolve that -- it takes a side about DERIVATION and says every verdict here was measured against validate.py, because the program is what decides whether a node merges. #1476 is answered. This node claims class assignment and ledger composition (MUSTs 2-11) and says so, and then says the harder thing: prose is not a mechanism. Neither node declares relationships, so nothing machine-readable links them, and naming an owner reduces the risk #1476 identifies rather than eliminating it. DEVIATION FROM THE PLAN'S OWN CHECK, recorded rather than worked around. STEP 4's done-when enumerated the issue numbers permitted in the document and omitted #1314 -- this node's own issue -- which appears in a direct quotation of AGENTS.md's deferral pointer. The document is right and the check's list was incomplete. Every other reference is on the list. Verified: validator exit 0 with 2 UNVERIFIED notices; 12 numbered MUSTs and 7 numbered SHOULDs under their own headings; issue references are exactly 605 636 639 1308 1309 1314 1316 1321 1410 1459 1463 1476 1478; 79 tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
…ITY (#1314) Sharpens seven ledger statements to say what their cited sources actually say, and adds two entries: ADR-0029's bar on copying private evidence into the public corpus, and COMPATIBILITY.md's rule that removing a field or enum value, or narrowing a type, is breaking and needs a dated entry plus a re-validation pass. The enforcement section no longer implies COMPATIBILITY.md governs the corpus root generally; it governs schema changes, and the text now says so. Saved mid-pipeline: the authoring agent stopped at the session rate limit before finishing its review stages. Gates re-run clean at this tree by the recovering session -- validator PASS (2 expected UNVERIFIED commit-only FACTs), 27/27 schema tests, 79/79 corpus tests. Review stages, findings triage and the PR are still outstanding. Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
A source-verification audit opened every cited file and compared it against the statement above it -- the one check nothing automates. It found three Blockers, all of them this document's own thesis failing on this document. Every one passed validation the whole time. BLOCKER 1+2 -- two ledger entries asserted a verdict validate.py does not give. They claimed a GitHub link not pinned to a file is UNVERIFIED. Measured by calling _classify_citation directly, four recognised-but-malformed GitHub URL shapes are HARD ERRORS: a mutable ref, a tree view, a blame view, and a pinned link naming no file. The entries cited the URL branch, which is ABOUT URLs and does not say what they claimed. Split into three entries per this document's own SHOULD 4, and the verdict table gains the four error rows plus the ok row, so a reader can predict whether their node merges. BLOCKER 3 -- the ledger asserted AGENTS.md "introduces its citation table as CONTRACT.md's shapes". AGENTS.md says the opposite in the paragraph introducing that table: it is seven rows and "is not a summary of section 3". It also records that an earlier version of its own sentence claimed otherwise, and that an agent authoring a sibling node built a scope argument on the miscount. This node is that sibling node and it re-introduced the miscount. That is left IN the document rather than quietly corrected, because it is the best worked example the corpus has: the citation resolved, AGENTS.md genuinely discusses the subject, every check passed, and only opening the file and reading the sentence caught it. The divergence is restated correctly as two-way -- CONTRACT.md vs validate.py -- with AGENTS.md describing both accurately, and #1478's AGENTS.md limb reported as stale rather than inherited. Also closed: the developer-audience entry had no body home; claims about the #1308 node's front matter (relationships, status) contradicted this node's own statement that those files cannot be cited from here; "the checker exited 0 on all three rounds" was unsupportable and is now argued from what a schema can see; "measured by running it" now ships the reproduction snippet and admits the run is not in the ledger, because no citation form expresses an execution. Eleven ledger entries added or corrected, including the two self-correction entries recording this failure. 37 entries, still exactly one commit-only FACT. Verified: validator exit 0 with 2 expected UNVERIFIED notices; verdict rows re-measured against validate.py at ebe2daf; 79 corpus tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
Codex reviewed this branch and returned a do-not-merge verdict with two Blockers. All six findings verified by execution before acting; five confirmed, one partly refuted by its own cited source. BLOCKER 1 -- CONFIRMED. A FACT claimed the validator accepts a pinned link "naming a file within the repository". It does not: the check is on URL shape alone. Measured -- a link with a nonexistent owner, repository AND file returns ok. AGENTS.md warns of exactly this. Restated as shape-only, saying outright that neither repository nor file existence is established. An on-topic citation that did not support the specific wording: this document's own thesis, again. BLOCKER 2 -- PARTLY REFUTED, and the refutation is in the issue Codex cited. Codex held that claiming ownership of the class rules was an author decision with nobody to attribute, which by MUST 7 must be withdrawn. But #1476's Expected section says "whichever #1314 chooses, the two documents should end up with one owner for class rules" -- it delegates the choice. And #1308's own section 1, quoted in #1476, already allocates the code-naming forms to itself and the tool-output forms here, so the split was not invented either. Both are now carried as TEAM_KNOWLEDGE attributed to #1476 rather than asserted. What Codex was right about stands: the claim was unattributed, and calling AGENTS.md "half wrong" overstated it. AGENTS.md's pointer predates #1308's allocation; that is now reported as a discrepancy this node did not cause. HIGH 3 -- CONFIRMED in substance. MUST 1 did not distinguish a factual claim from a normative rule, so it appeared to require a citation for every MUST -- which would mean manufacturing sources for rules, the laundering this document warns against. MUST 1 now covers claims of fact only, and the node's authority to issue rules is a single attributed entry naming #1314's definition of done. HIGH 4 -- CONFIRMED. Enforcement grouped four bullets under "enforced again independently by memory.py". Measured -- MemoryEntry accepts free prose and a path to a file that does not exist; it checks non-empty strings and stops. Split into two lists, and the two are now called overlapping rather than equivalent. MEDIUM 5 -- CONFIRMED. "Four hard-error variants" was wrong: commits and edit are errors too, raw.githubusercontent.com is a separate accepted host, and an uppercase full SHA is rejected under a message that says "mutable ref". The example-count is replaced by the three actual decision rules, with the misleading message called out. MEDIUM 6 -- CONFIRMED. "No check that exists or could exist" was indefensible and self-contradicted two sections later. Now scoped to the schema and the deterministic validator, naming review as the thing that does catch it. 41 entries, still exactly one commit-only FACT. A duplicate-key YAML error my own splice introduced was caught by the validator and fixed before commit. Verified: validator exit 0 with 2 expected UNVERIFIED notices; all twelve URL verdicts re-asserted here re-measured against the classifier; 79 corpus tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
review-code found a second instance of the defect the cross-model pass found first, still live in the ledger. Called Blocker-grade and fixed here rather than filed, because it is a self-contradiction inside the artefact under construction: the document states MUST 9 and then breaches it three hundred lines above. The entry classified the audience selection TEAM_KNOWLEDGE and attributed it to #605's outcome. That outcome says a developer can CREATE a node. It says nothing about who this standard is written FOR. The "because" clause was the author's extrapolation wearing the source's name -- precisely MUST 9: attributing an extrapolation to the thing it started from does not make it something you were told. Reattributed to #1314's definition of done, which asks for "audiences appropriate to the node" and so delegates the choice. Same remedy as the ownership finding: the honest source is the instruction that told the author to choose, not the material the author reasoned from. The body now offers the reasoning AS reasoning rather than as a sourced claim, and records the breach with its mechanism -- found by review, invisible to the checker, because the schema only requires provided_by to be a non-empty string. That is the third time this document has caught itself doing the thing it exists to warn against, and the third time it has been left on the page. Also closes the Low: the "reviewers" half of the audience claim had no ledger entry; one entry now covers all three audiences. Verified: validator exit 0 with 2 expected UNVERIFIED notices; 79 corpus tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
…1314) review-adjudicate and a same-vendor review-final left four live findings, all Medium or Low and none blocking. Fixed anyway: three of the four are this document contradicting itself, which in a standard about evidence honesty costs more than the severity suggests. N1 (Medium, introduced by the previous commit's own fix) -- the body presented "audiences appropriate to the node" as a quotation from #1314. The definition of done actually reads "audiences, provenance/evidence and typed relationships appropriate to the node"; the qualifier attaches to typed relationships. A stitched fragment in quotation marks, in the document whose central test is whether the source says this or merely concerns this. The claim itself was true and is now stated without the manufactured quote. Finding A (Medium) -- "four hard-error variants" survived the earlier fix untouched and contradicted this node's own ledger entry, which enumerates three conditions. Body and ledger disagreeing is exactly what MUST 1 exists to prevent, in the document that issues MUST 1. Now three, matching the entry. Finding B (Medium) -- the three pinning rules added last commit are #1308's, and restating them breached this node's own SHOULD 6 with no stated departure reason, while contradicting the boundary set in its own scope section. SHOULD 6 permits departure WITH a reason, so the reason is now stated: the verdict table is unusable without them. They are marked provisional, #1308 wins on conflict, and the block is named as the stale copy to delete once #1308 merges. Finding C (Low) -- the plan left open whether a commit message is a legitimate provided_by value; MUST 8 answered it corpus-wide without the question being reopened. Now named in scope-and-omissions as a position taken without a source, issued under this node's authority rather than presented as a finding, and revisable if #605 rules otherwise. Verified: the DoD fragment re-read from the issue; zero occurrences of "four hard-error" remain; validator exit 0 with 2 expected UNVERIFIED notices; 79 corpus tests OK. Refs #1314 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
…dger (#1314) Two defects from #1488, filed by the #1312 sibling, both of which applied here. DEFECT B -- MERGE-ORDER HAZARD IN BARE-PATH CITATIONS. AGENTS.md step 9 forbids a RELATIONSHIP target absent from the merge target and gives a git ls-tree check for it. It says nothing about CITATIONS, and the hazard is identical: a citation resolves against whatever tree the validator runs in, so a path present only on the authoring branch passes locally and is a hard error after merge. This node had it. Nine ledger entries cite launchpad/docs/corpus/AGENTS.md, which is absent from origin/launchpad. Measured, with the negative control that makes the proof mean anything: TEST 1 node on a clean origin/launchpad tree -> FAIL, 9 errors, exit 1 TEST 2 same node, AGENTS.md citations swapped out -> PASS, exit 0 Test 2 is what proves those citations are the cause rather than something else in the node. Without it test 1 shows only that something is wrong. The citations are NOT removed -- nine of this node's claims are about what AGENTS.md says, and it is based on #1462 precisely because it cannot exist without that file. The fix is to state the constraint with its measurement and gate the merge: this node must never reach launchpad ahead of #1462. The general rule is now written down, because step 9 does not carry it, citing #1488. DEFECT C -- LEDGER/BODY RECONCILIATION. A full both-directions pass found 40 body claims with no ledger entry, concentrated in the merge-order section I had just written -- which is exactly the failure mode #1488 describes, claims shifting as the document is written. Added seven entries covering AGENTS.md step 9, AGENTS.md as the lifecycle and provided_by source, memory.py's narrower field contract, COMPATIBILITY.md's additive case, #605's outcome text, and the #1488/#1462 gating. Extended three entries whose body text had outgrown them: ADR-0029's two claim-type rankings, memory.py's stale-flagging, and the schema README. Trimmed rather than sourced where the body had quietly extended a sourced entry: five cells in the scope-and-omissions table carried clauses AGENTS.md does not record. The table now says outright that its mappings are AGENTS.md's and not this node's inferences. The reverse direction was clean -- no entry lacked a body home -- except that the authority entry was pointed at rather than claimed. The body now states where the MUSTs and SHOULDs get their authority. Caught while re-verifying: adding entries moved the AGENTS.md citation count from seven to nine, making the sentence stating it false. Fixed, and the number re-derived from the ledger rather than edited by hand. Verified: TEST 1 and TEST 2 both re-run against the final node; validator exit 0 with 2 expected UNVERIFIED notices; 79 corpus tests OK; 27 schema tests OK. Refs #1314. Cites #1488. Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
tucktuck101
left a comment
There was a problem hiding this comment.
Request changes
Medium: unsupported empirical assertions violate the standard's own ledger rule
launchpad/docs/corpus/standards/evidence.md:350-352,684-685,690-691,752-753 assert unattributed prevalence/effectiveness claims ("single most common," "almost never chosen," "single highest-yield"). No front-matter ledger entry supports them. That conflicts with the standard's own MUST 1 and #1314's traceability requirement. Recast as non-factual normative guidance or add honest ledger evidence.
Medium: duplicates citation rules owned by #1308 and embeds merge-order prose
:512-547 reproduces the citation-shape material assigned to the code-references standard (#1308) as a "provisional copy," and :855-892 embeds temporary #1462 branch/merge-order state into an active standard, which also forces the relationships omission (:894-908). Remove the duplication, defer to #1308 once it lands, and add the appropriate relationships.
What looks good
The class matrix, structural-only characterization, CONTRACT-vs-validator URL distinction, and ownership boundary are otherwise accurate.
[panel: fable+codex]
…-standard-evidence Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
, declare the corpus-agents edge (#1314) Four sentences asserted unattributed prevalence/effectiveness ("single most common," "almost never chosen," "single highest-yield," "nearly always available and nearly never taken") with no ledger entry backing them, conflicting with this node's own MUST 1. Recast each as normative guidance without the frequency claim, since no measurement exists to cite honestly. The citation-form table carried a "provisional copy" of #1308's three shape rules, which SHOULD 6 -- link another document's rules rather than restating them -- already forbids doing. Removed the copy; the table now distinguishes the verdicts without asserting what #1308 currently says. The merge-order section claimed AGENTS.md had not merged and that nine citations would fail on launchpad -- true when authored, false now: AGENTS.md landed via #1468/#1469/#1477 before this review. Re-ran the section's own reproduction command against current origin/launchpad (PASS, 0 errors, was FAIL/9 errors) and rewrote it as history rather than a live constraint. corpus-agents is therefore loadable on the merge target, so this node now declares relationships: [depends-on: corpus-agents] instead of stating merge order as the reason for none. corpus-standard-code-references (#1308, still unmerged) stays deferred. review-final finding (fable+codex panel), verified against branch head before fixing: none of this was closed, and the merge-order staleness extended past what the review flagged. Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
Round-2 findings addressedBoth verified against branch head (
While verifying the merge-order section, found it had gone stale independent of the review: it claimed AGENTS.md hadn't merged and that 9 citations would fail on Merged Fixed at |
Fixed since this review by follow-up commit cb1bfd4: all three unattributed prevalence claims reworded into non-empirical normative prose, the duplicated #1308 citation-shape rules replaced with a pointer to PR #1480, stale merge-order prose corrected, and the missing corpus-agents relationship added. Verified directly against current text. Dismissing as stale/resolved.
There was a problem hiding this comment.
Reviewed commit 612e62f6811f2e6deba04be43184882161d53d35 against merge base dea96f837690703b1298ab3f0e49e1b82fecb8cf.
Incomplete
This review is INCOMPLETE and must not be read as a full pass:
- no dimension was actually reviewed: the pipeline ran the 'default_reviewer' stub reviewer, which reports every dimension clean without reading it (a real dimension reviewer is #116)
Containment
No containment findings.
Fetched and empty: pr_review_comments.
Automated containment covers the delimiter boundary and unambiguous injection tells only. It does not cover injection phrased as ordinary, unremarkable prose. The absence of a containment finding is not evidence that this pull request contains no injection attempt.
Closes #1314.
Issue type
Task
claude-opus-5[1m]Adds
launchpad/docs/corpus/standards/evidence.md— the canonical policy node for the corpusevidenceledger: what it is, how a claim is classified, what a citation establishes, and what a passing validation run does and does not mean.Based on
origin/task/636-corpus-agents-md(#1462), notlaunchpad, because it is authored againstAGENTS.md, which has not merged.What it covers
The three classes and what each is for; the distinction between reasoning from evidence and dressing up a decision; how conflicting evidence is ranked; the citation forms that name no openable file; 12 MUSTs and 7 SHOULDs; enforcement and where it stops; exceptions and escalation.
The load-bearing claim, measured rather than inherited: citation checking is structural. The validator confirms a cited path resolves to a real file and never opens it, so a
FACTciting a real file that says nothing on its subject passes with no notice at all.The document caught itself three times
Each was a real defect, each passed every automated check, and each is left recorded on the page rather than quietly corrected — they are the best worked examples the corpus has.
AGENTS.md"introduces its citation table asCONTRACT.md's shapes".AGENTS.mdsays the opposite in that very paragraph — and records that an agent authoring a sibling node built a scope argument on that miscount. This node is that sibling node, and it re-introduced the identical error.unverified. Measured: three hard-error conditions reject recognised-but-malformed links.Verification
python3 launchpad/project-intelligence/corpus/validate.pyPASS, 2UNVERIFIED(the two permitted provenance commit citations)_classify_citationatebe2daf72origin/task/636-corpus-agents-mdunmoved atebe2daf72FACTReview gates
review-planreview-codereview-finalreview-adjudicateNot verified
review-finalstood in and is same-vendor, so it shares this author's blind spots. Re-run Codex against84ee5a336before merge.review-adjudicateread task: settle whether code-references or evidence (#1314) owns the FACT and ledger-composition rules #1476 independently and confirmed the refutation.validate.pyrecords that no corpus generator exists.memory.py's supersession behaviour was read, not executed. Its field contract was executed.Departure from the definition of done
"Typed relationships appropriate to the node" — the node declares none, deliberately.
corpus-agentsis loadable from this branch, so an edge would validate here and become a hard error the moment this node reachedlaunchpadahead of #1462. The merge target carries no corpus node outsideschema/:Edges get declared in one pass once the set has landed. This is a stated tension, not a silent omission.
Escalations
CONTRACT.md§3 andvalidate.pydisagree on the citation forms. Restated accurately here: it is two-way, not three-way, and bug: CONTRACT.md, AGENTS.md and validate.py disagree on the corpus citation forms and their count #1478's ownAGENTS.mdlimb is now stale — it quotes a revision since corrected. Reported, not edited.relationshipsuntil both merge.AGENTS.md's pointer sends its whole citation table here, but task: document corpus standard for code references #1308's §1 already claims the code-naming half. Two destinations, one pointer. This node may not editAGENTS.md; reported.AGENTS.mdstep 9 guardsrelationshipsagainst the merge target but is silent on bare-path citations, which carry the identical hazard. Filed by the task: document corpus standard for diagrams #1312 sibling; this node had the defect. Not re-filed. The document now states the general rule as an interim guard, citing bug: three more defects in the corpus AGENTS.md create procedure, found by following it literally on #1312 #1488, because step 9 does not carry it.🤖 Generated with Claude Code