docs(corpus): add test-contract template - #1540
Conversation
…o's own conformance precedent Defines launchpad/docs/corpus/templates/test-contract.md per #605's real acceptance criterion (purpose, required sections, evidence expectations, industry model adapted). Verified Pact's consumer-driven model against docs.pact.io and found it does not fit Buzz's single-repo, single-deployment relay-plus-in-repo-clients architecture; the template instead adapts two real, already-verified precedents already in this repository: docs/multi-tenant-conformance.md paired with crates/buzz-test-client/tests/conformance_multitenant.rs, and crates/buzz-conformance/TRACE_SCHEMA.md, which calls itself "the contract between the relay's emitter and the independent replay checker." Closes #1349 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
serina-mcfall
added
type:task
tucktuck101
left a comment
tucktuck101
left a comment
There was a problem hiding this comment.
Review — test-contract template
Accurate and well-sourced — the Pact, ISTQB and Google SWE citations hold, and the required
sections (named verifying test plus a runnable command) are the right ones for this node kind.
One finding, and it is one bullet.
Medium — the boundary against the invariant template is unnamed, and the two overlap
test-contract.md:132 defines the covered subject as "a single testable statement about
behaviour (an invariant, an isolation guarantee, a compatibility promise) plus the concrete
test(s) that currently exercise it" — naming invariant as a covered obligation kind.
Sibling PR #1549's invariant.md:255-291 requires an instance to carry "Invariant statement",
"Scope", "Enforcement today" (whose five tiers include "test-enforced (a test would fail)") and
"Consequence of violation".
I read both Boundary sections in full. test-contract.md:202-217 excludes test-suite docs,
#1325's citation standard, conformance tables, and ADRs/PRDs — it never mentions
#1343/invariant. invariant.md:229-253 excludes #1320, #1344 and #1342 — it never mentions
#1349/test-contract, which is also absent from its "It does not cover" table at :436-445.
So an author holding a test-backed invariant — e.g. "kind 40002 replies must bump reply_count
on the thread root", which is invariant.md's own flagship example — is pointed at both
templates by both templates, and neither yields.
The distinguishing axis exists but is unstated: test-contract requires a named verifying
test and a runnable command and fixes instances at type: verification; invariant makes a
test one of five acceptable enforcement tiers and lets instances pick type by subject matter.
My ruling: test-contract owns obligation-plus-named-test; invariant owns
always-holds-property-plus-enforcement-tier.
One bullet in each Boundary section naming the other template and that axis. Both PRs are in
this wave and can be fixed together — this is the only finding I have on this PR, and I would
have approved it otherwise.
What is correct
- Pact docs and Pact Specification v4 attributions verified; ISTQB and Google SWE ch11
quotations hold. - The
type: governancereasoning for the template node, and the correct disjunctive statement
aboutAGENTS.mdbeingtype: agent, are both right — sibling #1550 gets that second point
wrong and this PR gets it right. - The frontmatter skeleton is complete and schema-valid, so an author can copy it directly.
- Required sections 3-4 (named test, runnable command) make the node's central claim checkable
rather than aspirational. - CI green at head (latest run per check); node validates clean.
Reviewed by tucktuck101's review lane. Every failing claim reported above was reproduced by me against this PR head before posting.
2 participants
Summary
Adds
launchpad/docs/corpus/templates/test-contract.md, the corpus template for a node type that documents one testable obligation plus the specific test(s) that verify it. Verified that Pact (the obvious industry candidate) does not fit this repo's architecture, and grounded the template instead in two real, already-built precedents already in this codebase:docs/multi-tenant-conformance.md/conformance_multitenant.rsandcrates/buzz-conformance/TRACE_SCHEMA.md.Related issue
Closes #1349
Issue type
Task
Agent provenance
Objective
Create
launchpad/docs/corpus/templates/test-contract.mdas the canonical template a realtest-contractcorpus node (a node type carryingtype: verification) is built from.Impacted components
Approach and rejected alternatives
The issue's own DoD text is stale boilerplate copied from the standards-track issues (MUST/SHOULD/enforcement/policy checklist) — every one of #1326-#1351 carries it verbatim. This PR builds against parent Feature #605's real acceptance criterion instead: "every template states its purpose, required sections, evidence expectations and the industry model/standard it adapts." The template's own "Note on this document's structure" section states this mismatch explicitly, following the precedent set by #1528 (architecture-context).
Rejected alternative: using Pact (
docs.pact.io) as the adapted industry model, which is the obvious candidate the batch brief flagged for verification. ReadTESTING.mdandcrates/buzz-test-client/'s actual test structure first, then fetched Pact's own docs. Pact's mechanism is consumer-driven contract testing: a consumer's own tests generate a contract file, and a separately deployed provider is verified against it independently, usually via a Pact Broker. Buzz doesn't have that shape — one relay binary tested end-to-end against CLI/desktop/mobile clients built from the same source tree, same commit, same CI run (confirmed viaTESTING.mdandAGENTS.md's "Nostr-first HTTP surface" section). Forcing Pact's consumer/provider/broker vocabulary onto that would misdescribe it, so it was rejected as the primary model rather than cited anyway for the sake of having a named standard.What the template adapts instead, found while investigating: this repository already has two home-grown "test contract" precedents.
docs/multi-tenant-conformance.mdis a prose obligation table mirrored one row per module incrates/buzz-test-client/tests/conformance_multitenant.rs, which calls itself, in its own doc-comment, "the executable form of the conformance contract" — not-yet-landed obligations are#[ignore]d and stubbed with apending_lane(...)helper that panics viatodo!(), so an empty test body can never pass as done. Separately,crates/buzz-conformance/TRACE_SCHEMA.mdopens by calling itself, verbatim, "the contract between the relay's emitter and the independent replay checker," and its siblingLIMITS.mdstates plainly what a green run does and does not prove. Both are real, checked-in, and directly on point — a stronger primary source than an external standard that doesn't structurally fit.Verification
Command run:
Raw output:
Command run:
Raw output:
Not verified
Codex is currently down (#1467), so this PR has same-model (Claude) adversarial self-review rather than a genuine cross-model review pass — a real cross-model review has not happened. No real
test-contractnode has been built from this template yet, so whether the required-sections list actually holds up in practice is untested. Pact's Pact Broker workflow and its newer bi-directional contract-testing mode were not researched in depth, since the consumer/provider mismatch already ruled Pact out as the primary model before those details would matter.crates/buzz-conformance's property tests (tests/proptest_checker.rs) and fixture-replay tests (tests/replay_fixtures.rs) were read for structure but not executed. These gaps are also recorded in the node's own "Expected but not verified" section.Security implications
None. This is a documentation-only change adding one Markdown file under
launchpad/docs/corpus/templates/; it changes no runtime code, no CI configuration beyond what already validates the corpus tree, and introduces no new secrets or external calls.Escalations
None. No blocking or non-blocking findings surfaced while researching or drafting this node that warranted a separate issue.