docs(corpus): standard for test references (#1325)

[serina-mcfall]

Summary

Adds launchpad/docs/corpus/standards/test-references.md, the canonical corpus policy node for citing a test as evidence — which claim a test citation supports, which citation shape fits it, and how this repository's conditional and flaky tests bear on trusting a cited pass as current behavior.

Related issue

Closes #1325

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-sonnet-5
Session reference	N/A
Initiating human	N/A - dispatched as part of a batch of 5 sibling corpus-standard tasks under PRD #605; no specific human handle given at dispatch

Objective

One hand-authored corpus node, launchpad/docs/corpus/standards/test-references.md (id: corpus-standard-test-references), stating the policy for citing a test — its existence, a specific test case, or a test run's observed result — as evidence in a corpus node's evidence ledger.

Impacted components

• launchpad/docs/corpus/standards/test-references.md

Approach and rejected alternatives

Scoped narrowly to "citing a test as evidence" and deliberately excluded two adjacent areas already owned elsewhere: the generic repository-path/GitHub-link citation mechanics (issue #1308, open as unmerged PR #1480 — read for context only, not treated as settled per this task's dispatch note) and the general evidence-classification/graph-edge/tool-result contract (issue #1314, unmerged). What remained, and what this node actually adds, is the test-specific slice: which of the shared citation shapes fits which test-shaped claim (existence vs. a run's result vs. current-behavior), and a flakiness/staleness discussion grounded in this repository's real testing conventions (ADR-0020, the #[ignore] convention, desktop/scripts/summarize-flaky-tests.mjs).

Rejected alternative: folding the tool-result shape's general syntax/enforcement rules into this node. Rejected because code-references.md (PR #1480) explicitly disclaims that scope as #1314's, and this node's own MUST 2 only needed the slice of that grammar relevant to wrapping a real test invocation — duplicating the rest here would fight #1314 over the same ground.

Verification

Command run:

python3 launchpad/project-intelligence/corpus/validate.py

Raw output:

UNVERIFIED  corpus-agents: evidence entry 1, citation 1: is a commit reference, which names no openable file
UNVERIFIED  corpus-agents: evidence entry 20, citation 1: is a graph-edge or tool-result citation, which names no openable file
UNVERIFIED  corpus-readme: evidence entry 1, citation 1: is a commit reference, which names no openable file
UNVERIFIED  corpus-standard-confidence: evidence entry 1, citation 1: is a commit reference, which names no openable file
UNVERIFIED  corpus-standard-decision-references: evidence entry 1, citation 1: is a commit reference, which names no openable file
UNVERIFIED  corpus-standard-test-references: evidence entry 1, citation 1: is a commit reference, which names no openable file
UNVERIFIED  corpus-standard-test-references: evidence entry 6, citation 2: is a graph-edge or tool-result citation, which names no openable file
UNVERIFIED  corpus-standard-test-references: evidence entry 9, citation 2: is an external URL this validator can neither pin nor open
UNVERIFIED  corpus-standard-test-references: evidence entry 16, citation 3: is a graph-edge or tool-result citation, which names no openable file
PASS  corpus validation found no errors; 9 item(s) reported unverified

Exit status: 0.

Second command run, the legitimate local pre-commit gate (validate.py run directly is not recognized by this session's verify-gate hook; this is the confirmed working route):

python3 -m unittest discover -s launchpad/docs/corpus/schema/tests -p "test_*.py" -v

Raw output (tail):

test_full_fixture_passes (test_schema.ValidFixtureTest.test_full_fixture_passes) ... ok
test_minimal_fixture_passes (test_schema.ValidFixtureTest.test_minimal_fixture_passes) ... ok

----------------------------------------------------------------------
Ran 27 tests in 0.040s

OK

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

• The wrapped tool-result citation form used in MUST 2 (run_command('cargo test -p buzz-core --lib kind::tests::') -> 3 passed; 0 failed) was checked by running the _SYMBOL/_TOOL_RESULT_RE/_GRAPH_EDGE_RE/_COMMIT_CITATION_RE/_FILE_POSITION_RE patterns copied verbatim from validate.py directly in Python against both the wrapped and unwrapped strings — not by constructing a full fixture node and running validate.py end-to-end against that exact string. The two are expected to agree since the copied patterns are the same objects _classify_citation calls, but that exact end-to-end path was not exercised.
• No corpus node yet cites a test as evidence of any claim, so this standard is written from the validator's own regular expressions and from real testing conventions documented elsewhere in the repository (ADR-0020, TESTING.md, ARCHITECTURE.md, the flaky-test summarizer script), not from a worked example inside the corpus.
• Whether cargo nextest's own retry semantics (not currently used in this repository's CI configuration) would, if adopted, produce a signal comparable to summarize-flaky-tests.mjs's "flaky" label was not researched — the claim made in the node is only that no such signal exists today, not that none could exist.
• Did not run just corpus-validate (needs Hermit activated with a different working-directory assumption than this worktree check used); ran the direct python3 launchpad/project-intelligence/corpus/validate.py form instead, which the corpus AGENTS.md states is equivalent and does not require Hermit.

Security implications

None. This is a documentation-only change to a Markdown policy node under launchpad/docs/corpus/standards/; it does not touch runtime code, credentials, or CI trust boundaries. No secret-shaped path or credential pattern was cited (the validator's own credential-citation checks also ran clean, per the validation output above).

Escalations

• Issue bug: four defects in the corpus AGENTS.md, found by following it literally on #1307 #1471 (finding 2) documents an unresolved tension in AGENTS.md between its citation-shape table (which permits a tool-result FACT) and its "three things a passing run does not mean" passage (which names only one conventional UNVERIFIED-only-FACT exception). This node's MUST 4 and its Exceptions and escalation section describe that tension honestly, as it applies specifically to citing an observed test run, rather than inventing a resolution — the actual resolution is task: document corpus standard for evidence #1314's, per bug: four defects in the corpus AGENTS.md, found by following it literally on #1307 #1471's own text. No new issue filed; this defers to the existing one.
• No other out-of-scope defects were found while authoring this node. The sibling PR docs(corpus): the code-references standard (#1308) #1480 (code-references.md) was read for context only, per this task's dispatch note, and no defect was found in it worth raising independently of what bug: four defects in the corpus AGENTS.md, found by following it literally on #1307 #1471 already documents.

[tucktuck101]

Review — standard for test references (#1325)

The relationship vocabulary and frontmatter are correct and the validator passes. But the
document's central table publishes an invented file path with a ok verdict — in a standard
whose own MUST 1 forbids exactly that.

Blocking — the worked example cites a test file that does not exist, and records it as passing

test-references.md:172-173, in The two shapes in play, and what a pass proves:

Form	Example	Checker's verdict	What it proves
Bare repository path to a test file	crates/buzz-core/tests/kind_registry.rs	ok	The file exists.
Path with a line, naming a test function	crates/buzz-core/tests/kind_registry.rs:42	ok	The file exists.

Verified against launchpad: crates/buzz-core/tests/ is not a directory, and
find . -name 'kind_registry*' returns nothing anywhere in the repo. So both rows are false,
and the recorded verdict is the opposite of the truth — the real classifier returns
status='error', detail='does not resolve to a real file in the repository' for both forms.

An author copying either documented-as-ok row into a node gets a hard validation error and
exit 1. And the document's own MUST 1 (:185-192) requires a test citation to "rest on the
author having opened the file and located the test, not on a plausible-looking path" — this
table is a plausible-looking path. CI cannot catch it because the string lives in the body, and
validate.py discards everything after the frontmatter.

crates/buzz-core/src/kind.rs and crates/buzz-core/src/kind.rs:899 both classify ok and
would serve as real substitutes.

The other three rows of the same table are empirically correct (prose invocation -> error,
wrapped tool result -> unverified, graph edge -> unverified), so this is two outlier rows in an
otherwise accurate table — not a systemic failure. But they are the two rows an author is most
likely to copy.

Medium — the "citing a test as evidence" slice is claimed by two standards

This document claims that slice while sibling #1523's linking.md routes it entirely to #1308.
Both cannot own it. One should defer to the other explicitly, in one sentence each.

Batch correction — the AGENTS.md hunk is not an index registration

launchpad/docs/corpus/AGENTS.md at this head is byte-identical to the launchpad tip
(verified with git diff --quiet). The hunk is the task/636 change inherited through a stale
merge base (919886b41), not a registration this PR adds — merging produces no net AGENTS.md
change and there is no cross-standard conflict there. No standards index exists to register in.

What is correct

• Relationship vocabulary matches relationships.schema.json and the inlined
  node.schema.json copy exactly — five members, no extras, none missing, no hand-written
  generated inverse.
• The distinction between what a citation's shape proves and what its contents prove is
  the right framing, and MUST 2's warning about graph edges reading like tool results is a real
  observed failure mode.
• Three of five table rows verified correct against the real classifier.
• CI green at head (latest run per check); node validates clean.

Reviewed by tucktuck101's review lane. Every failing claim reported above was reproduced by me against this PR head before posting.