docs(corpus): add test-strategy corpus template - #1550
Conversation
Add launchpad/docs/corpus/templates/test-strategy.md, defining the shape of a corpus node documenting a system or component's overall multi-tier testing approach. Grounded in this repository's own ADR-0020 (adopted testing methodology) and TESTING.md, cross-referenced against ISTQB's test-strategy definition and Google's test-size framework, with an explicit boundary against #1349/test-contract (one obligation) and #1325/test-references (citation mechanics). Closes #1350 Signed-off-by: Serina Mcfall <serina.mcfall@gmail.com>
tucktuck101
left a comment
tucktuck101
left a comment
There was a problem hiding this comment.
Review — test-strategy corpus template
Unusually well-cited: this template uses this repository's own test-level vocabulary rather
than inventing a parallel taxonomy, and its ADR-0020 quotations, TESTING.md references and
ISTQB definitions check out. One false universal in the ledger.
High — a FACT asserts a universal its own counterexample refutes
The evidence entry states: "Every existing corpus node about how to author corpus content,
rather than about a piece of verification/architecture/capability content itself, uses
type: governance (README.md, standards/confidence.md, standards/decision-references.md, and
the already-drafted templates/test-contract.md, templates/capability.md)."
launchpad/docs/corpus/AGENTS.md — id corpus-agents, the node whose entire subject is the
procedure for creating, updating and retiring corpus content — carries type: agent. Verified
in its frontmatter on launchpad.
The enumerated list is entirely correct; only the universal quantifier fails. And no validator
can catch it, because the entry's evidence array lists only the three governance nodes — the
counterexample is excluded from the citation. Your own siblings in this wave state the correct
disjunctive form: test-contract.md:14-20 ("...or type: agent (AGENTS.md, the one node that
is itself an agent-facing procedure)") and invariant.md:22-28. Adopt that wording; the
template's conclusion that governance is right for a template document is unaffected.
Medium — the required sections omit ADR-0020's retry and flakiness machinery
The levels table asks for purpose, infrastructure, invoking command and gating status, and
section 3 asks for enforcement status. Nothing asks about ADR-0020 items 5-6: the retry policy
(retries: process.env.CI ? 2 : 0 — zero locally so a developer feels their own flakiness, two
in CI so pass-on-retry is measurable) or flakiness visibility
(desktop/scripts/summarize-flaky-tests.mjs surfacing anything that passed on retry). Grepping
the template for just ci|retries|flaky|shard returns nothing outside one quoted sentence.
That matters given the template's own stated purpose — that a node "claim[ing] enforcement it
does not have would repeat exactly the gap ADR-0020 was written to make visible". Retry-masked
greenness is that failure mode, and the template never asks about it. just ci as the
repository-level contract command is also missing.
Low — the body says five levels and lists four
"five levels separated by the infrastructure they need" is followed by four groupings, with
desktop E2E smoke and desktop E2E integration collapsed. The ledger enumerates all five
correctly, so this is body-level compression — but a reader counting finds four against a
stated five, in the section whose job is to state the taxonomy precisely.
Low — one FACT bundles three ISTQB definitions behind one URL
https://istqb-glossary.page/test-strategy/ carries only the first of the three definitions
cited; analytical and methodical test strategy live at their own URLs. Both of those support the
template's wording, so the substance is sound and the citation is one page too shallow. The
gloss "analyzes the test basis (e.g. product risk)" also adds a parenthetical ISTQB's
definition does not contain.
Not findings, ruled out on evidence
- Index registration: the corpus
AGENTS.mdforbids a hand-authored node index, so not
modifying it is correct. - Test-level vocabulary: checked against ADR-0020 and
TESTING.md— this template uses the
repo's own names, no parallel taxonomy. - CI green at head (latest run per check); node validates clean.
Reviewed by tucktuck101's review lane. Every failing claim reported above was reproduced by me against this PR head before posting.
2 participants
Summary
Adds
launchpad/docs/corpus/templates/test-strategy.md, the template governing how a corpus node documents a system or component's overall multi-tier testing approach. Grounded in this repository's own ADR-0020 and TESTING.md rather than importing an industry model wholesale, with an explicit boundary against#1349/test-contract and#1325/test-references.Related issue
Closes #1350
Issue type
Task
Agent provenance
Objective
Create
launchpad/docs/corpus/templates/test-strategy.mdas the corpus template for atest-strategy-shaped node (a system or component's overall, multi-tier testing approach).Impacted components
Approach and rejected alternatives
Issue #1350's Definition of Done is the same standards-track policy boilerplate copied across #1326-#1351 (MUST/SHOULD, enforcement/escalation, "single canonical policy node"). Per the batch brief and #605's real acceptance criterion ("every template states its purpose, required sections, evidence expectations and the industry model/standard it adapts"), I built this as a template, not a policy, and named the mismatch explicitly in the document's own "Note on this document's structure" section — the same move
templates/test-contract.mdandtemplates/capability.md(already-drafted siblings) made for the identical boilerplate.Two industry models were considered: ISTQB's
test strategyglossary definition (levels + what happens within them) and Google'stest sizeframework (small/medium/large, classified by infrastructure footprint, from Software Engineering at Google). Neither was imported wholesale. This repository already has a real, accepted, whole-repository test strategy — ADR-0020 (adopt-upstream-testing-methodology) — with five levels separated by the infrastructure they need, an#[ignore]convention, and an honest finding thatrequired_status_checksonlaunchpadreturns 404 (not currently enforced). The template adapts ADR-0020's own shape (level, infrastructure, command, gating status) rather than either industry model's full taxonomy, while citing both models as the vocabulary a reader would recognize.Rejected alternative: importing ISTQB's full strategy-type taxonomy (analytical/methodical/process-compliant/etc.) as required template content. Rejected because this repository's own real precedent (ADR-0020) does not organize itself around those types, and forcing an unused vocabulary onto the template would misdescribe the one worked example the corpus actually has.
Boundary against #1349 (test-contract, unmerged PR #1540) and #1325 (test-references, unmerged PR #1524): both diffs were read directly (
git show origin/task/1349-...:...test-contract.mdandorigin/task/1325-...:...test-references.md) before drafting. test-contract is the corpus's smallest verification unit — one testable obligation plus its verifying test. test-references is a corpus-wide citation-mechanics standard, not a node type. test-strategy is neither: it is the map across levels that a test-contract node would eventually instantiate one obligation of, stated explicitly in this document's "Boundary against neighboring corpus content" section.Verification
Command run:
Raw output:
Command run:
Raw output:
Not verified
test-strategynode has been built from this template yet, so whether the required-sections list holds up for a component much smaller (a single crate) or much larger (the whole repository, ADR-0020's own scale) than the worked example is untested.glossary.istqb.org) returned no readable content when fetched directly for this task; the ISTQB definition cited in the node was read via a mirror (istqb-glossary.page) that attributes it to ISTQB, not cross-checked against the original site directly.abseil.io), not from Google's original 2010 Testing Blog post, which returned only its title and comment thread when fetched — the two were not diffed against each other for wording drift.INFERENCEwith confidence 0.75, not aFACT.Security implications
None. This is a documentation-only change adding one Markdown file under
launchpad/docs/corpus/templates/; it changes no runtime code, no CI configuration, and no access control.Escalations
None. No blocking or deferred findings surfaced while drafting this node — it is a straightforward template with no code, no schema change, and no relationship targets available yet (the two most relevant future edges, to
#1349/test-contract and#1325/test-references, both name unmerged nodes and are recorded as future work directly in the document's own Relationships section rather than filed as separate issues).