From 5de2c03f23d13a60cd11c1f5f1ba2a55b6ab8295 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 12:16:27 -0400 Subject: [PATCH 01/10] Update DEMA agent definitions from template Sync the .github/agents agent definitions and AGENTS.md with the latest DEMA Agents template, and add the new planning agent. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/agents/developer.agent.md | 5 +- .github/agents/formal-review.agent.md | 1 + .github/agents/implementation.agent.md | 54 +++++----- .github/agents/lint-fix.agent.md | 3 +- .github/agents/planning.agent.md | 134 +++++++++++++++++++++++++ .github/agents/quality.agent.md | 33 ++++-- .github/agents/template-sync.agent.md | 43 ++++++-- AGENTS.md | 17 ++-- 8 files changed, 233 insertions(+), 57 deletions(-) create mode 100644 .github/agents/planning.agent.md diff --git a/.github/agents/developer.agent.md b/.github/agents/developer.agent.md index 5f208eb..5f2b988 100644 --- a/.github/agents/developer.agent.md +++ b/.github/agents/developer.agent.md @@ -19,7 +19,7 @@ Perform software development tasks by determining and applying appropriate stand If one exists, fetch it as the starting point; adjust placeholder names and heading depth to match the target path before writing the file - For each modified file, identify which companion artifacts need updating - (requirements, design docs, tests, review-sets) + (requirements, design docs, verification docs, tests, review-sets, README.md, user guides) - Include companion artifact updates in the work plan 4. **Execute work** following standards requirements and quality checks 5. **Formatting**: Run `pwsh ./fix.ps1` to silently apply all @@ -35,8 +35,7 @@ Perform software development tasks by determining and applying appropriate stand # Developer Agent Report **Result**: (SUCCEEDED|FAILED) - -## Work Summary +**Report**: `.agent-logs/developer-{subject}-{unique-id}.md` - **Files Modified**: {List of files created/modified/deleted} - **Languages Detected**: {Languages identified} diff --git a/.github/agents/formal-review.agent.md b/.github/agents/formal-review.agent.md index 7dd8e84..14e487a 100644 --- a/.github/agents/formal-review.agent.md +++ b/.github/agents/formal-review.agent.md @@ -44,6 +44,7 @@ standards from the selection matrix in AGENTS.md. # Formal Review Report **Result**: (SUCCEEDED|FAILED) +**Report**: `.agent-logs/formal-review-{subject}-{unique-id}.md` ## Review Summary diff --git a/.github/agents/implementation.agent.md b/.github/agents/implementation.agent.md index 7cc0352..d94e880 100644 --- a/.github/agents/implementation.agent.md +++ b/.github/agents/implementation.agent.md @@ -28,36 +28,18 @@ The state-transitions include retrying a limited number of times: ## PLANNING State (start) -Call the **explore** agent as a sub-agent (built-in agent type) with: +Call the **planning** agent as a sub-agent (custom agent from `.github/agents/`) with: - **context**: the user's request + any previous quality findings + retry context -- **goal**: produce a verified implementation plan through these steps: - - 1. Investigate the codebase and develop a concrete implementation plan that - addresses the request - 2. **Identify companion artifact deliverables**: for every code change in the - plan, list the requirements files, design documents, and review-set entries - that must be created or updated - traceability must flow requirements → - design → code, so these are mandatory deliverables, not optional extras - 3. Review the plan for assumptions, weaknesses, and gaps - identify up to 5 - key assumptions and rate each as: - - **VERIFIED**: confirmed by codebase evidence - - **LIKELY**: consistent with codebase patterns but not directly confirmed - - **UNVERIFIED**: not confirmed by any evidence - 4. For any assumption rated UNVERIFIED or LIKELY, attempt to resolve it - through additional investigation and revise the plan to address identified - weaknesses - repeat the critique-and-strengthen cycle up to 2 additional - times if unresolved issues remain, but stop as soon as the plan is stable - 5. List up to 5 risks to the implementation - 6. Assess feasibility: can this be implemented in a single development pass? - 7. State a **recommendation**: GO or INCOMPLETE - GO if the plan is sound, or - INCOMPLETE if critical unknowns remain that only the user can resolve - -Once the explore sub-agent finishes: - -- IF recommendation is INCOMPLETE: Transition to REPORT with Result: INCOMPLETE, +- **goal**: produce a verified implementation plan, or a targeted plan to address + the identified quality issues if this is a retry + +Once the planning sub-agent finishes: + +- IF Result is FAILED: Transition to REPORT with Result: FAILED +- IF Result is INCOMPLETE: Transition to REPORT with Result: INCOMPLETE, listing the unknowns and what CAN be implemented once they are resolved -- OTHERWISE (GO): Transition to DEVELOPMENT +- OTHERWISE (SUCCEEDED): Transition to DEVELOPMENT ## DEVELOPMENT State @@ -76,7 +58,8 @@ Once the developer sub-agent finishes: Call the **quality** agent as a sub-agent (custom agent from `.github/agents/`) with: -- **context**: the user's request + development summary + files changed + previous issues (if any) +- **context**: the user's request + development summary + files changed + planning companion artifact table + + previous issues (if any) - **goal**: check the quality of the work performed for any issues Once the quality sub-agent finishes: @@ -92,6 +75,9 @@ Once the quality sub-agent finishes: this agent may report INCOMPLETE when the request cannot be implemented without information only the user can provide. +For full planning details (assumptions, risks, feasibility), read the planning +report file referenced in the planning agent's response. + Generate the completion report using the template below, then save it to `.agent-logs/{agent-name}-{subject}-{unique-id}.md` per the AGENTS.md reporting requirements, and return the summary to the caller. @@ -102,19 +88,20 @@ requirements, and return the summary to the caller. # Implementation Orchestration Report **Result**: (SUCCEEDED|FAILED|INCOMPLETE) -**Final State**: (PLANNING|DEVELOPMENT|QUALITY|REPORT) +**Report**: `.agent-logs/implementation-{subject}-{unique-id}.md` +**Last Active State**: (PLANNING|DEVELOPMENT|QUALITY) **Retry Count**: ## State Machine Execution -- **Planning Results**: {Implementation plan, assumption ratings, risks, and recommendation} +- **Planning Results**: {Planning report path; plan summary and SUCCEEDED/INCOMPLETE/FAILED result} - **Development Results**: {Summary of developer agent results} - **Quality Results**: {Summary of quality agent results} - **State Transitions**: {Log of state changes and decisions} ## Sub-Agent Coordination -- **Explore Agent (Planning)**: {Plan, assumption verdicts, top risks, GO/INCOMPLETE recommendation} +- **Planning Agent**: {Report file path, SUCCEEDED/INCOMPLETE/FAILED result, plan summary} - **Developer Agent**: {Development status and files modified} - **Quality Agent**: {Validation results and compliance status} @@ -123,4 +110,9 @@ requirements, and return the summary to the caller. - **Implementation Success**: {Overall completion status} - **Quality Compliance**: {Final quality validation status} - **Issues Resolved**: {Problems encountered and resolution attempts} + +## Unknowns (only when Result is INCOMPLETE) + +- **Unresolved Questions**: {List each question the user must answer} +- **What Can Proceed**: {Work that can be done without the missing information} ``` diff --git a/.github/agents/lint-fix.agent.md b/.github/agents/lint-fix.agent.md index 549e751..36d3ca1 100644 --- a/.github/agents/lint-fix.agent.md +++ b/.github/agents/lint-fix.agent.md @@ -68,8 +68,7 @@ submission, not during normal development. # Lint Fix Report **Result**: (SUCCEEDED|FAILED) - -## Summary +**Report**: `.agent-logs/lint-fix-{subject}-{unique-id}.md` - **Iterations**: {Number of fix-loop iterations performed} - **Files Modified**: {List of all files changed} diff --git a/.github/agents/planning.agent.md b/.github/agents/planning.agent.md new file mode 100644 index 0000000..20e75ee --- /dev/null +++ b/.github/agents/planning.agent.md @@ -0,0 +1,134 @@ +--- +name: planning +description: Planning agent that investigates the codebase, develops a verified implementation plan, and identifies all companion artifact deliverables. +user-invocable: true +--- + +# Planning Agent + +Investigate the codebase and produce a verified implementation plan with all +companion artifact deliverables. + +## Step 1 — Load Standards + +Read the relevant standards from `.github/standards/` using the selection matrix +in `AGENTS.md` based on the artifact types in scope for the request (requirements, +design, verification, documentation, code). + +## Step 2 — Investigate and Plan + +Read `docs/design/introduction.md` first (if present), then investigate the +codebase to develop a concrete implementation plan: + +- Identify all files to create, modify, or delete +- Describe the change required for each file + +## Step 3 — Identify Companion Artifact Deliverables + +For each planned change, assess the mandatory companion artifacts below (create/update/N/A +with justification): + +- **Requirements** — functional changes require a requirement entry +- **Design Documentation** — new or changed components require design docs +- **Verification Documentation** — new or changed components require verification docs +- **Tests** — functional changes require test coverage +- **Review Sets** — changes to the software item hierarchy (units or subsystems + added, removed, or reorganized) require review-set updates +- **README.md** — user-facing changes require README updates +- **User Guide** — user-facing features require user guide updates + +## Step 4 — Critique and Strengthen + +Identify up to 5 key assumptions and rate each: + +- **VERIFIED**: confirmed by codebase evidence +- **LIKELY**: consistent with codebase patterns but not directly confirmed +- **UNVERIFIED**: not confirmed by any evidence + +For UNVERIFIED or LIKELY assumptions, investigate further and revise the plan. +Repeat up to 2 more times, stopping when the plan is stable. + +## Step 5 — Risk Assessment + +List up to 5 risks with a brief mitigation for each. + +## Step 6 — Feasibility Assessment + +State whether this can be implemented in a single development pass and any +preconditions that affect feasibility. + +## Step 7 — Recommendation + +- **SUCCEEDED** — the plan is sound and the developer agent can proceed +- **INCOMPLETE** — critical unknowns remain that only the user can resolve; + list each unknown explicitly +- **FAILED** — investigation could not produce a viable plan + +# REPORT Phase + +Save the full analysis to `.agent-logs/planning-{subject}-{unique-id}.md` per +the AGENTS.md reporting requirements. + +Then respond to the caller with ONLY the lean structured summary below. + +# Report Template + +```markdown +# Planning Report + +**Result**: (SUCCEEDED|INCOMPLETE|FAILED) +**Request Summary**: {Brief restatement of the task as understood} +**Report**: `.agent-logs/planning-{subject}-{unique-id}.md` + +## Implementation Plan + +| File | Action | Description | +|------|--------|-------------| +| {path} | create/modify/delete | {what changes and why} | + +## Companion Artifact Deliverables + +| Category | File | Action | +|----------|------|--------| +| Requirements | {path} | create/update/N/A — {justification} | +| Design Documentation | {path} | create/update/N/A — {justification} | +| Verification Documentation | {path} | create/update/N/A — {justification} | +| Tests | {path} | create/update/N/A — {justification} | +| Review Sets | {path} | create/update/N/A — {justification} | +| README.md | {path} | create/update/N/A — {justification} | +| User Guide | {path} | create/update/N/A — {justification} | + +## Assumption Analysis + +| # | Assumption | Rating | Resolution | +|---|-----------|--------|------------| +| 1 | {assumption} | VERIFIED/LIKELY/UNVERIFIED | {resolution or N/A} | + +## Risk Assessment + +1. **[severity]** {risk} — {mitigation} + +## Feasibility Assessment + +{Single-pass or not, and why. Any preconditions.} + +## Unknowns + +{Only present when Result is INCOMPLETE. List each question the user must +resolve before implementation can proceed.} +``` + +# Lean Structured Response (returned to caller) + +```markdown +**Result**: (SUCCEEDED|INCOMPLETE|FAILED) +**Report**: `.agent-logs/planning-{subject}-{unique-id}.md` + +**Plan**: +{Repeat the Implementation Plan table} + +**Companion Artifacts**: +{Repeat the Companion Artifact Deliverables table} + +**Unknowns**: {Only if INCOMPLETE — list questions for the user} +``` diff --git a/.github/agents/quality.agent.md b/.github/agents/quality.agent.md index 380d11f..26fd251 100644 --- a/.github/agents/quality.agent.md +++ b/.github/agents/quality.agent.md @@ -13,14 +13,23 @@ Grade and validate software development work by ensuring compliance with project 1. **Analyze the task request AND completed work** to determine scope: identify which artifact categories were changed, and which *should have been changed* given the task - new user-visible features always require requirements, - design, and review-set coverage regardless of whether those files were touched; - test-only additions (corner-case tests, defensive boundary tests, regression - tests) do not require a corresponding requirement + design, verification docs, and README/user guide updates regardless of + whether those files were touched; Review Sets are always in scope when + the software item hierarchy changes (units or subsystems added, removed, or + reorganized); test-only additions (corner-case tests, defensive boundary + tests, regression tests) do not require a corresponding requirement; if a + planning companion artifact table is provided in context, cross-reference it + — any artifact listed as create/update must be covered in the evaluation and + FAIL if the artifact was not produced 2. **Read relevant standards** using the selection matrix in AGENTS.md 3. **Evaluate all in-scope categories** - N/A only when the task genuinely cannot affect a category; if the task introduces new user-visible features or - structural changes then Requirements, Design Documentation, and Review - Management are always in scope and FAIL if the artifacts were not updated + structural changes then Requirements, Design Documentation, and Verification + Documentation are always in scope and FAIL if the artifacts were not updated; + Documentation (README/user guide) is always in scope for user-facing changes + and FAIL if not updated; Review Sets are always in scope when the + software item hierarchy changes (units or subsystems added, removed, or + reorganized) and FAIL if review-sets were not updated 4. **Validate tool compliance** using ReqStream, ReviewMark, and build tools 5. **Generate focused quality report** per the AGENTS.md reporting requirements - save to `.agent-logs/{agent-name}-{subject}-{unique-id}.md` and return the summary to the caller @@ -36,6 +45,7 @@ For each checklist item in the template below, record as `(PASS|FAIL|N/A) - {one # Quality Assessment Report **Result**: (SUCCEEDED|FAILED) +**Report**: `.agent-logs/quality-{subject}-{unique-id}.md` **Overall Grade**: (PASS|FAIL) ## Required Fixes (only when Result is FAILED) @@ -50,13 +60,13 @@ Priority-ordered list of issues that MUST be resolved for the next retry: - **Evaluated**: {List sections assessed and why} - **Skipped**: {One-line per skipped section with reason, e.g., "Design - Documentation: N/A - no design files modified"} + Documentation: N/A - no component behavior, structure, or interface changed"} ## Requirements Compliance: (PASS|FAIL|N/A) - Were requirements created/updated for all functional changes? - Were source filters applied for platform-specific requirements? -- Is requirements traceability maintained to tests? +- Is forward traceability from requirements to verification artifacts preserved? ## Design Documentation Compliance: (PASS|FAIL|N/A) @@ -80,7 +90,14 @@ Priority-ordered list of issues that MUST be resolved for the next retry: - Are cross-hierarchy test dependencies documented in design docs? - Do all tests pass? -## Review Management Compliance: (PASS|FAIL|N/A) +## Verification Documentation Compliance: (PASS|FAIL|N/A) + +- Were verification documents created/updated for all new or changed components? +- Do verification documents include all mandatory sections (Verification Approach, Test Environment, + Acceptance Criteria, Test Scenarios)? +- Is requirements-to-test coverage tracked via the ReqStream trace matrix (not embedded in verification docs)? + +## Review Sets Compliance: (PASS|FAIL|N/A) - Were review-sets updated for structural changes? - Is review scope appropriate for change magnitude? diff --git a/.github/agents/template-sync.agent.md b/.github/agents/template-sync.agent.md index c013503..df4d488 100644 --- a/.github/agents/template-sync.agent.md +++ b/.github/agents/template-sync.agent.md @@ -22,12 +22,18 @@ Delegate each group to a sub-agent. - **Root config files** - all non-collection files at the repository root - **One group per flat `docs/` folder** - e.g. `docs/build_notes/`, `docs/user_guide/` +- **One group for root files in each of `docs/design/`, `docs/verification/`, + `docs/reqstream/`** - e.g. `docs/design/introduction.md` — separate from the + system subtrees beneath them - **One group per system subtree** in `docs/design/`, `docs/verification/`, `docs/reqstream/` - each subtree and all its descendants is one group # Orchestration -For each group intersecting the requested scope, call a sub-agent with: +For Audit mode, call an **explore** sub-agent (built-in) per group. +For Sync, Scaffold, and Recreate modes, call a **general-purpose** sub-agent (built-in) per group. + +For each group intersecting the requested scope, call the appropriate sub-agent with: - **context**: - Group scope and template URL from the `# Reference Template` section in `AGENTS.md` @@ -40,7 +46,10 @@ For each group intersecting the requested scope, call a sub-agent with: each item is a subsystem or unit, then select the appropriate template (`subsystem-name.*` or `unit-name.*`) regardless of the item's folder depth — do not infer item type from path depth alone - - If a template counterpart cannot be fetched, skip the file and report it + - If a file has no template counterpart, skip it and report it as + "No template found" — this is not a failure + - If a file appears in `repository-map.md` but its template cannot be fetched, + report Result: FAILED and list the affected files - **goal**: - Based on the given mode: - **Audit** - fetch each template counterpart; compare headings; report missing @@ -78,9 +87,25 @@ For each group intersecting the requested scope, call a sub-agent with: `TODO:` placeholders in YAML string values (e.g. `title:`, `justification:`) are content placeholders — always resolve them to real content; infer from README, related files, sibling docs, and path; if confident write directly; - if ambiguous offer 2–3 concrete options and ask the user; keep asking until - they answer - never leave a TODO or TEMPLATE-DIRECTIVE in the output unless - the user explicitly requests it + if ambiguous, **do not ask interactively** — return the unresolved questions + in the result so the orchestrator can ask the user and re-invoke; never leave + a TODO or TEMPLATE-DIRECTIVE in the output unless the user explicitly requests it + - Return results in this format for each file in the group: + + ```markdown + ### {file-path} + + - **Template**: {template path or "not found"} + - **Missing sections**: {list or "none"} + - **Heading depth issues**: {list or "none"} + - **Content format issues**: {list or "none"} *(Recreate only)* + - **Action**: (Reported | Sections added | Created | Rebuilt | No template found) + - **Unresolved Questions**: {list or "none"} + ``` + +If any sub-agent returns unresolved questions, collect them, ask the user, then +re-invoke the affected sub-agent(s) with the answers before assembling the final report. +If questions remain unresolved after asking the user, report Result: INCOMPLETE. Collect sub-agent results and assemble the final report. @@ -89,7 +114,8 @@ Collect sub-agent results and assemble the final report. ```markdown # Template Sync Report -**Result**: (SUCCEEDED|FAILED) +**Result**: (SUCCEEDED|FAILED|INCOMPLETE) +**Report**: `.agent-logs/template-sync-{subject}-{unique-id}.md` **Mode**: (Audit|Sync|Scaffold|Recreate) ## Files @@ -102,8 +128,13 @@ Collect sub-agent results and assemble the final report. - **Content format issues**: {list of sections where intra-section content did not match the template comment's prescribed format, or "none"} *(Recreate only)* - **Action**: (Reported | Sections added | Created | Rebuilt | No template found) +- **Unresolved Questions**: {list or "none"} ## Summary - **Conformant**: {count} | **Deviations**: {count} | **Updated**: {count} + +## Unknowns (only when Result is INCOMPLETE) + +- **Unresolved Questions**: {List each placeholder or ambiguity the user must resolve} ``` diff --git a/AGENTS.md b/AGENTS.md index 8dd5bb5..7fbbb4f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -38,7 +38,7 @@ commit messages, and reports). This repository follows a reference template for structure and file conventions. -- **Template URL**: `https://github.com/demaconsulting/Agents/raw/refs/heads/template` +- **template-url**: `https://github.com/demaconsulting/Agents/raw/refs/heads/template` - **Repository map**: `{template-url}/repository-map.md` - **Template files**: `{template-url}/{file-path}` for files described in the map @@ -88,12 +88,15 @@ The default agent should handle simple, straightforward tasks directly. Delegate to specialized agents only for specific scenarios: - **Pre-PR lint cleanup** (fix all lint issues before pull request) → Call the lint-fix agent -- **Light development work** (small fixes, simple features) → Call the developer agent +- **Scoped fixes with no new user-visible behavior** (PR review comments, doc + corrections, known bug fixes with defined root cause) → Call the developer agent - **Light quality checking** (basic validation) → Call the quality agent -- **Formal feature implementation** (complex, multi-step) → Call the implementation agent -- **Formal bug resolution** (complex debugging, systematic fixes) → Call the implementation agent +- **Any change introducing new user-visible behavior** (features, enhancements, + new commands or options) → Call the implementation agent +- **Formal bug resolution** (complex debugging, unknown root cause) → Call the implementation agent - **Formal reviews** (compliance verification, detailed analysis) → Call the formal-review agent - **Structural audit**: (repository layout vs. template) → Call the template-sync agent +- **Implementation planning only** (review a plan before committing to implementation) → Call the planning agent # Agent Reporting (Specialized Agents Must Follow) @@ -102,16 +105,16 @@ Specialized agents MUST generate a completion report: 1. Save to `.agent-logs/{agent-name}-{subject}-{unique-id}.md` where `{subject}` is a kebab-case task summary (max 5 words) and `{unique-id}` is a short unique suffix (e.g., 8-char hex or timestamp) -2. Start with `**Result**: (SUCCEEDED|FAILED)` as the first metadata field +2. Start with `**Result**: (SUCCEEDED|FAILED|INCOMPLETE)` as the first metadata field 3. Include the agent-specific report sections defined in each agent's prompt 4. Return the summary to the caller Result semantics for orchestrator decision-making: -- **SUCCEEDED**: Work completed and all applicable quality gates met +- **SUCCEEDED**: Work completed and all quality gates applicable to that agent's scope met - **FAILED**: Work could not be completed or quality gates not met - **INCOMPLETE**: Work cannot proceed without information only the user can - provide (implementation agent only) + provide (implementation, planning, and template-sync agents) # Formatting (After Making Changes) From 4dff33ff6a1fc4a4adc50ea56e9722ebf35dddc7 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 12:39:24 -0400 Subject: [PATCH 02/10] Sync repository with updated DEMA template Bring all tracked files in line with the latest canonical DEMA template: - docs/design/introduction.md: convert Software Structure section from ASCII code-block tree to bullet-list with (System)/(Subsystem)/(Unit)/(OTS) labels - docs/design/ots.md: rename title, add Selection Criteria, Version Management Policy, General Integration Approach, Qualification Strategy - docs/design/ots/*.md: remove ' Integration Design' suffix from headings - docs/design/test-results-library/io/*.md: promote headings from ### to ## - docs/verification/ots.md: rename title and headings, add Qualification Evidence and Regression Approach sections - docs/verification/ots/*.md: remove ' Verification' suffix from headings - .fileassert.yaml: add verification tag and Pandoc/WeasyPrint verification document tests - fix.ps1, lint.ps1: add MODIFICATION POLICY comment blocks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .fileassert.yaml | 35 ++++++++++++- docs/design/introduction.md | 47 +++++++++-------- docs/design/ots.md | 50 ++++++++++++++++++- docs/design/ots/apimark.md | 2 +- docs/design/ots/buildmark.md | 2 +- docs/design/ots/fileassert.md | 2 +- docs/design/ots/pandoc.md | 2 +- docs/design/ots/reqstream.md | 2 +- docs/design/ots/reviewmark.md | 2 +- docs/design/ots/sarifmark.md | 2 +- docs/design/ots/sonarmark.md | 2 +- docs/design/ots/versionmark.md | 2 +- docs/design/ots/weasyprint.md | 2 +- docs/design/ots/xunit.md | 2 +- .../io/junit-serializer.md | 14 +++--- .../io/serializer-helpers.md | 14 +++--- .../test-results-library/io/serializer.md | 14 +++--- .../test-results-library/io/trx-serializer.md | 14 +++--- docs/verification/ots.md | 36 ++++++++++++- docs/verification/ots/apimark.md | 2 +- docs/verification/ots/buildmark.md | 2 +- docs/verification/ots/fileassert.md | 2 +- docs/verification/ots/pandoc.md | 2 +- docs/verification/ots/reqstream.md | 2 +- docs/verification/ots/reviewmark.md | 2 +- docs/verification/ots/sarifmark.md | 2 +- docs/verification/ots/sonarmark.md | 2 +- docs/verification/ots/versionmark.md | 2 +- docs/verification/ots/weasyprint.md | 2 +- docs/verification/ots/xunit.md | 2 +- fix.ps1 | 7 ++- lint.ps1 | 9 +++- 32 files changed, 203 insertions(+), 81 deletions(-) diff --git a/.fileassert.yaml b/.fileassert.yaml index 3efb67d..8bdaab6 100644 --- a/.fileassert.yaml +++ b/.fileassert.yaml @@ -1,7 +1,7 @@ --- # FileAssert document validation tests for TestResults. # Tests are tagged by document group to allow per-group execution during the build pipeline. -# Tags: build-notes, code-quality, code-review, design, user-guide, requirements. +# Tags: build-notes, code-quality, code-review, design, verification, user-guide, requirements. # # NOTE: build-notes through user-guide tests provide OTS evidence for Pandoc and WeasyPrint # and run before ReqStream. The requirements tests run after ReqStream and validate the @@ -193,6 +193,39 @@ tests: text: - contains: Design + # --- VERIFICATION DOCUMENT --- + + - name: Pandoc_VerificationHtml + description: Verification HTML was generated by Pandoc + tags: [verification] + files: + - pattern: docs/verification/generated/verification.html + count: 1 + html: + - query: //head/title + count: 1 + text: + - contains: Verification + + - name: WeasyPrint_VerificationPdf + description: Verification PDF was generated by WeasyPrint + tags: [verification] + files: + - pattern: "docs/generated/TestResults Software Verification Design.pdf" + count: 1 + pdf: + metadata: + - field: Title + contains: Verification + - field: Author + contains: DEMA Consulting + - field: Subject + contains: Verification + pages: + min: 3 + text: + - contains: Verification + # --- USER GUIDE --- - name: Pandoc_UserGuideHtml diff --git a/docs/design/introduction.md b/docs/design/introduction.md index d386da5..9398957 100644 --- a/docs/design/introduction.md +++ b/docs/design/introduction.md @@ -32,30 +32,29 @@ OTS software items used in this repository: ## Software Structure -```text -TestResults Library (System) -├── IO (Subsystem) -│ ├── Serializer (Unit) -│ ├── SerializerHelpers (Unit) -│ ├── TrxSerializer (Unit) -│ └── JUnitSerializer (Unit) -├── TestOutcome (Unit) -├── TestResult (Unit) -└── TestResults (Unit) - -OTS Software Items -├── BuildMark -├── FileAssert -├── Pandoc -├── ReqStream -├── ReviewMark -├── SarifMark -├── SonarMark -├── VersionMark -├── WeasyPrint -├── xUnit -└── ApiMark -``` +- **TestResultsLibrary** (System) - .NET library for reading and writing test result files in TRX and JUnit XML formats + - **IO** (Subsystem) - translates between the in-memory model and XML result formats + - Serializer (Unit) - identifies XML result formats and delegates deserialization + - SerializerHelpers (Unit) - provides Utf8StringWriter for XML declarations + - TrxSerializer (Unit) - serializes and deserializes TRX test result documents + - JUnitSerializer (Unit) - serializes and deserializes JUnit XML test result documents + - TestOutcome (Unit) - defines supported test outcomes and classification helpers + - TestResult (Unit) - represents one test case result with metadata and output + - TestResults (Unit) - represents a complete test run and its ordered results + +**OTS Dependencies:** + +- BuildMark (OTS) +- FileAssert (OTS) +- Pandoc (OTS) +- ReqStream (OTS) +- ReviewMark (OTS) +- SarifMark (OTS) +- SonarMark (OTS) +- VersionMark (OTS) +- WeasyPrint (OTS) +- xUnit (OTS) +- ApiMark (OTS) ## Folder Layout diff --git a/docs/design/ots.md b/docs/design/ots.md index c05bb26..e622a59 100644 --- a/docs/design/ots.md +++ b/docs/design/ots.md @@ -1,11 +1,38 @@ -# OTS Software Items +# OTS Dependencies This repository uses Off-The-Shelf (OTS) software tools as integral parts of the CI pipeline and documentation build process. Each tool satisfies a specific requirement and is integrated as a `dotnet` tool or NuGet package. The OTS items are listed below with their integration roles; detailed integration design for each is in the corresponding `docs/design/ots/` file. -## OTS Integration Strategy +## Selection Criteria + +OTS items are selected based on the following criteria: + +- **License compatibility**: only items published under a permissive open-source license (MIT, Apache 2.0, or equivalent) are + accepted, ensuring compliance with the project's distribution requirements. +- **Ecosystem alignment**: items must be available as a `dotnet` tool or NuGet package so they integrate naturally with the + .NET SDK toolchain and do not require separate runtime environments. +- **Maturity and community support**: items should have an active release history, publicly accessible source, and + demonstrated use in comparable software projects. +- **Security track record**: no unresolved critical CVEs at the time of adoption; items from the DemaConsulting organisation + are under the same source-control and review processes as the host project. +- **Qualification feasibility**: the item must be verifiable through CI pipeline execution, document output assertions, or + library test exercise without requiring access to vendor internal test suites. + +## Version Management Policy + +Pinned versions are recorded in `.config/dotnet-tools.json` (for `dotnet` tool items) and in project NuGet package +references. This ensures reproducible builds across all developer workstations and CI runners. + +Dependabot is configured with a weekly NuGet scan; it raises pull requests automatically for any tool or package version +upgrade. All upgrade pull requests follow the standard review and CI gate process before merging. + +Major version upgrades (where the tool's public API or CLI interface changes) trigger a design review to assess whether +the affected integration design documents require updating. Minor and patch upgrades that pass CI without changes are +merged without a design review. + +## General Integration Approach Most tool-type OTS items (BuildMark, FileAssert, Pandoc, ReqStream, ReviewMark, SarifMark, SonarMark, VersionMark, and WeasyPrint) are installed as `dotnet` tools and invoked from the CI @@ -16,6 +43,25 @@ The pipeline execution model treats a successful CI run as primary integration e pipeline tools. Document-output tools (Pandoc, WeasyPrint, and ApiMark) additionally produce artifacts that FileAssert validates to prove correct execution and meaningful output. +## Qualification Strategy + +OTS items are qualified for use through integration evidence collected during CI pipeline execution rather than internal +unit testing of the OTS source, because their source is external and not under local development control: + +- **CI pipeline execution**: successful pipeline runs prove that pipeline-tool items (BuildMark, ReqStream, ReviewMark, + SarifMark, SonarMark, and VersionMark) are installed, invoked correctly, and produce the expected outputs. +- **Document output validation**: FileAssert assertions on generated HTML and PDF files qualify document-generation + items (Pandoc and WeasyPrint) by confirming correct content, structure, and minimum size requirements. +- **Artifact content validation**: FileAssert assertions on the NuGet package archive qualify ApiMark by confirming + expected documentation files are present. +- **Self-validation**: FileAssert is qualified by invoking it with `--version` and `--help` flags to confirm installation + and operational status. +- **Library test exercise**: xUnit is qualified indirectly; if the framework were non-functional, no library tests could + pass and the CI gate would fail. + +Qualification evidence is referenced by test name in each OTS requirements YAML file and linked to requirements by +ReqStream. A successful CI pipeline run with `--enforce` confirms that all referenced evidence exists and is current. + ## OTS Items | OTS Item | Role | Integration | diff --git a/docs/design/ots/apimark.md b/docs/design/ots/apimark.md index 674227a..b1605a1 100644 --- a/docs/design/ots/apimark.md +++ b/docs/design/ots/apimark.md @@ -1,4 +1,4 @@ -## ApiMark Integration Design +## ApiMark ### Purpose diff --git a/docs/design/ots/buildmark.md b/docs/design/ots/buildmark.md index f5dc39d..e37d567 100644 --- a/docs/design/ots/buildmark.md +++ b/docs/design/ots/buildmark.md @@ -1,4 +1,4 @@ -## BuildMark Integration Design +## BuildMark ### Purpose diff --git a/docs/design/ots/fileassert.md b/docs/design/ots/fileassert.md index 1f9e477..3990297 100644 --- a/docs/design/ots/fileassert.md +++ b/docs/design/ots/fileassert.md @@ -1,4 +1,4 @@ -## FileAssert Integration Design +## FileAssert ### Purpose diff --git a/docs/design/ots/pandoc.md b/docs/design/ots/pandoc.md index bec804b..b962419 100644 --- a/docs/design/ots/pandoc.md +++ b/docs/design/ots/pandoc.md @@ -1,4 +1,4 @@ -## Pandoc Integration Design +## Pandoc ### Purpose diff --git a/docs/design/ots/reqstream.md b/docs/design/ots/reqstream.md index a661470..5db883e 100644 --- a/docs/design/ots/reqstream.md +++ b/docs/design/ots/reqstream.md @@ -1,4 +1,4 @@ -## ReqStream Integration Design +## ReqStream ### Purpose diff --git a/docs/design/ots/reviewmark.md b/docs/design/ots/reviewmark.md index 5334abd..708482e 100644 --- a/docs/design/ots/reviewmark.md +++ b/docs/design/ots/reviewmark.md @@ -1,4 +1,4 @@ -## ReviewMark Integration Design +## ReviewMark ### Purpose diff --git a/docs/design/ots/sarifmark.md b/docs/design/ots/sarifmark.md index dbc576c..a276078 100644 --- a/docs/design/ots/sarifmark.md +++ b/docs/design/ots/sarifmark.md @@ -1,4 +1,4 @@ -## SarifMark Integration Design +## SarifMark ### Purpose diff --git a/docs/design/ots/sonarmark.md b/docs/design/ots/sonarmark.md index 9bb0a24..13dc10c 100644 --- a/docs/design/ots/sonarmark.md +++ b/docs/design/ots/sonarmark.md @@ -1,4 +1,4 @@ -## SonarMark Integration Design +## SonarMark ### Purpose diff --git a/docs/design/ots/versionmark.md b/docs/design/ots/versionmark.md index 09fb88e..2769107 100644 --- a/docs/design/ots/versionmark.md +++ b/docs/design/ots/versionmark.md @@ -1,4 +1,4 @@ -## VersionMark Integration Design +## VersionMark ### Purpose diff --git a/docs/design/ots/weasyprint.md b/docs/design/ots/weasyprint.md index 2b24c22..92ac60b 100644 --- a/docs/design/ots/weasyprint.md +++ b/docs/design/ots/weasyprint.md @@ -1,4 +1,4 @@ -## WeasyPrint Integration Design +## WeasyPrint ### Purpose diff --git a/docs/design/ots/xunit.md b/docs/design/ots/xunit.md index 6b30cca..aa89941 100644 --- a/docs/design/ots/xunit.md +++ b/docs/design/ots/xunit.md @@ -1,4 +1,4 @@ -## xUnit Integration Design +## xUnit ### Purpose diff --git a/docs/design/test-results-library/io/junit-serializer.md b/docs/design/test-results-library/io/junit-serializer.md index 28a7add..b4bd596 100644 --- a/docs/design/test-results-library/io/junit-serializer.md +++ b/docs/design/test-results-library/io/junit-serializer.md @@ -1,13 +1,13 @@ -### JUnitSerializer +## JUnitSerializer -#### Purpose +### Purpose The JUnitSerializer unit converts between the shared in-memory model and JUnit XML test result documents. It adapts the richer local model to the narrower JUnit schema by grouping results into suites, projecting outcomes onto JUnit elements, and recovering as much metadata as the format provides when reading JUnit input. -#### Data Model +### Data Model **DefaultSuiteName**: `string` - Sentinel suite name used when a `TestResult` has an empty `ClassName`. @@ -21,7 +21,7 @@ root element. **MessageAttributeName**: `string` - Attribute name reused for `failure`, `error`, and `skipped` message text. -#### Key Methods +### Key Methods **Serialize**: Writes a `TestResults` model as JUnit XML. @@ -87,7 +87,7 @@ represents the earliest test start, matching the convention used by common JUnit `TestResult` default; the `DefaultSuiteName` sentinel classname is mapped back to an empty string so round-trips produce the original `ClassName` value. -#### Error Handling +### Error Handling `Serialize()` throws `ArgumentNullException` when `results` is null. `Deserialize()` throws `ArgumentNullException` or `ArgumentException` for null or whitespace input and throws @@ -100,7 +100,7 @@ to an empty class name. Because JUnit has no inconclusive element, `Inconclusive serialize as plain `testcase` elements, making them indistinguishable from `Passed` outcomes on deserialization; this round-trip loss is a known limitation of the JUnit format. -#### Dependencies +### Dependencies - **TestResults** - provides run metadata for the `testsuites` root and receives deserialized runs. @@ -111,7 +111,7 @@ on deserialization; this round-trip loss is a known limitation of the JUnit form - **System.Globalization** - formats and parses timestamps and durations using invariant culture. -#### Callers +### Callers - **Serializer** - delegates JUnit deserialization after format identification. - **Library consumers** - call the static serialize and deserialize entry points directly when diff --git a/docs/design/test-results-library/io/serializer-helpers.md b/docs/design/test-results-library/io/serializer-helpers.md index 6664d1e..b8409f8 100644 --- a/docs/design/test-results-library/io/serializer-helpers.md +++ b/docs/design/test-results-library/io/serializer-helpers.md @@ -1,12 +1,12 @@ -### SerializerHelpers +## SerializerHelpers -#### Purpose +### Purpose The SerializerHelpers unit provides shared infrastructure used by both XML serializers. Its single responsibility is to make string-based XML serialization report UTF-8 rather than the default UTF-16 declaration produced by `StringWriter`. -#### Data Model +### Data Model **Utf8StringWriter**: `internal sealed class` - Specialized `StringWriter` implementation used by both serializer units when saving XML to a string. @@ -14,7 +14,7 @@ by both serializer units when saving XML to a string. **Encoding**: `Encoding` - Overridden property that always returns `Encoding.UTF8` so the XML prolog declares `encoding="utf-8"`. -#### Key Properties +### Key Properties **Encoding**: Read-only property override that reports UTF-8 as the writer encoding. @@ -26,17 +26,17 @@ prolog declares `encoding="utf-8"`. The property exists because .NET `StringWriter` defaults to UTF-16, which would advertise the wrong encoding for XML strings that callers later persist as UTF-8. -#### Error Handling +### Error Handling N/A - the unit exposes a deterministic property override with no inputs, branching, or local error recovery. -#### Dependencies +### Dependencies - **System.IO.StringWriter** - base class for string-backed text writing. - **System.Text.Encoding** - provides the UTF-8 encoding instance returned by the override. -#### Callers +### Callers - **TrxSerializer** - uses `Utf8StringWriter` when saving TRX XML to a string. - **JUnitSerializer** - uses `Utf8StringWriter` when saving JUnit XML to a string. diff --git a/docs/design/test-results-library/io/serializer.md b/docs/design/test-results-library/io/serializer.md index 6bad4f7..8e47bb4 100644 --- a/docs/design/test-results-library/io/serializer.md +++ b/docs/design/test-results-library/io/serializer.md @@ -1,12 +1,12 @@ -### Serializer +## Serializer -#### Purpose +### Purpose The Serializer unit is the IO subsystem facade for callers that do not know the incoming format. It identifies whether XML content is TRX or JUnit and delegates deserialization to the corresponding format-specific serializer. -#### Data Model +### Data Model **TestResultFormat**: `enum` - Defines the three public detection outcomes: `Unknown`, `Trx`, and `JUnit`. @@ -14,7 +14,7 @@ and `JUnit`. **TrxNamespaceUri**: `string` - Hard-coded XML namespace used to distinguish genuine TRX `TestRun` roots from unrelated XML documents. -#### Key Methods +### Key Methods **Identify**: Detects the serialized test-result format from XML content. @@ -39,7 +39,7 @@ and checks `testsuites` or `testsuite` root names for JUnit. The method keeps format selection in one place so callers can deserialize mixed result files through a single entry point. -#### Error Handling +### Error Handling `Identify()` catches `XmlException` and converts invalid XML to `TestResultFormat.Unknown`. `Deserialize()` validates `contents` before any XML parsing: passing `null` throws @@ -47,14 +47,14 @@ through a single entry point. `ArgumentException`. It throws `InvalidOperationException` when `Identify()` cannot classify the document. Exceptions raised by the delegated serializer propagate to the caller. -#### Dependencies +### Dependencies - **TrxSerializer** - performs format-specific TRX deserialization. - **JUnitSerializer** - performs format-specific JUnit deserialization. - **TestResults** - common return type for deserialization. - **System.Xml.Linq.XDocument** - parses XML roots for format identification. -#### Callers +### Callers N/A - entry-point unit, called directly by library consumers and exercised by system and subsystem tests. diff --git a/docs/design/test-results-library/io/trx-serializer.md b/docs/design/test-results-library/io/trx-serializer.md index 67689da..e0c35f7 100644 --- a/docs/design/test-results-library/io/trx-serializer.md +++ b/docs/design/test-results-library/io/trx-serializer.md @@ -1,12 +1,12 @@ -### TrxSerializer +## TrxSerializer -#### Purpose +### Purpose The TrxSerializer unit converts between the shared in-memory model and Microsoft TRX test result documents. It is responsible for preserving the richer TRX metadata model, including run metadata, test-definition cross-references, output streams, and summary counters. -#### Data Model +### Data Model **TrxNamespace**: `XNamespace` - Fixed TRX namespace used for element construction and lookup. @@ -27,7 +27,7 @@ and parse TRX timestamps. **DurationFormatString**: `string` - Constant/invariant format string (`"c"`) used to format and parse TRX durations as TimeSpan values. -#### Key Methods +### Key Methods **Serialize**: Writes a `TestResults` model as TRX XML. @@ -67,7 +67,7 @@ resolution is linear rather than repeatedly scanning the XML document. The method accepts named values and defined numeric values, then falls back to `Failed` so callers never receive an undefined `TestOutcome` from TRX deserialization. -#### Error Handling +### Error Handling `Serialize()` throws `ArgumentNullException` when `results` is null. `Deserialize()` throws `ArgumentNullException` or `ArgumentException` for null or whitespace input, and throws @@ -77,7 +77,7 @@ Malformed or missing GUIDs fall back to newly generated GUIDs, malformed duratio to `TimeSpan.Zero`, malformed timestamps fall back to `DateTime.UtcNow`, and unrecognized outcome values fall back to `TestOutcome.Failed`. -#### Dependencies +### Dependencies - **TestResults** - provides run-level metadata for `TestRun` and receives deserialized runs. - **TestResult** - provides per-test execution data and receives deserialized test cases. @@ -86,7 +86,7 @@ outcome values fall back to `TestOutcome.Failed`. - **System.Xml.Linq** and **System.Xml.XPath** - construct and query TRX XML. - **System.Globalization** - formats and parses invariant timestamps and durations. -#### Callers +### Callers - **Serializer** - delegates TRX deserialization after format identification. - **Library consumers** - call the static serialize and deserialize entry points directly when diff --git a/docs/verification/ots.md b/docs/verification/ots.md index b0c1ff3..2537bcd 100644 --- a/docs/verification/ots.md +++ b/docs/verification/ots.md @@ -1,10 +1,10 @@ -# OTS Software Item Verification +# OTS Verification This document describes the overall verification strategy for the OTS software tools used in the TestResults repository. Detailed verification evidence for each OTS item is in the corresponding `docs/verification/ots/` file. -## OTS Verification Strategy +## Verification Strategy OTS items are verified through integration evidence rather than internal unit tests, because their source is external and not under local development control. Three evidence categories are @@ -28,6 +28,38 @@ Test evidence is referenced by test name in each OTS requirements YAML file and requirements by ReqStream. A successful CI pipeline run with `--enforce` proves all referenced tests exist and are passing. +## Qualification Evidence + +The following evidence categories are used to demonstrate that each OTS item is fit for purpose: + +- **CI pipeline execution success**: a passing pipeline run with no errors or warnings constitutes evidence that the tool + is installed, correctly configured, and producing the expected outputs. This evidence is captured once per release + pipeline run and is referenced by test name in the OTS requirements YAML files. +- **Document output assertions**: FileAssert checks that each generated HTML and PDF document exists, is non-empty, + contains expected metadata strings, and meets minimum size and page-count thresholds. These assertions are recorded as + named tests and linked to OTS requirements through ReqStream. +- **Artifact content assertions**: FileAssert checks that the NuGet package archive produced by ApiMark contains the + expected Markdown documentation files. +- **Self-validation invocations**: `fileassert --version` and `fileassert --help` are captured as named tests to confirm + the tool is installed and operational. A `reviewmark --version` invocation provides equivalent evidence for ReviewMark. +- **Library test pass**: all passing library tests constitute evidence that xUnit is functional. + +All evidence items are named, linked to requirements, and verified complete by ReqStream on each CI run. + +## Regression Approach + +When an OTS item is upgraded to a new version, the following steps are performed to ensure the upgrade does not +introduce regressions: + +- The full CI pipeline is executed against the new version. Any failure in pipeline execution, document output + assertions, or library tests is treated as a regression and blocks the upgrade. +- For major version upgrades, the integration design documentation is reviewed to confirm the described features and + integration patterns remain accurate. Any discrepancies are resolved before the upgrade is accepted. +- ReqStream enforces that all OTS requirement links are satisfied by current test evidence; an upgrade that removes + previously passing evidence cannot merge until the evidence is restored. +- Version changes are recorded automatically in the SBOM artifacts rather than in design documentation, ensuring the + audit trail is always current without manual document updates. + ## OTS Items | OTS Item | Verification Category | Evidence Source | diff --git a/docs/verification/ots/apimark.md b/docs/verification/ots/apimark.md index 3bb1a1f..6c1b3ca 100644 --- a/docs/verification/ots/apimark.md +++ b/docs/verification/ots/apimark.md @@ -1,4 +1,4 @@ -## ApiMark Verification +## ApiMark ### Verification Approach diff --git a/docs/verification/ots/buildmark.md b/docs/verification/ots/buildmark.md index 513932c..5dd1af4 100644 --- a/docs/verification/ots/buildmark.md +++ b/docs/verification/ots/buildmark.md @@ -1,4 +1,4 @@ -## BuildMark Verification +## BuildMark ### Verification Approach diff --git a/docs/verification/ots/fileassert.md b/docs/verification/ots/fileassert.md index af9ff7a..aa542b5 100644 --- a/docs/verification/ots/fileassert.md +++ b/docs/verification/ots/fileassert.md @@ -1,4 +1,4 @@ -## FileAssert Verification +## FileAssert ### Verification Approach diff --git a/docs/verification/ots/pandoc.md b/docs/verification/ots/pandoc.md index f1da161..ca9238c 100644 --- a/docs/verification/ots/pandoc.md +++ b/docs/verification/ots/pandoc.md @@ -1,4 +1,4 @@ -## Pandoc Verification +## Pandoc ### Verification Approach diff --git a/docs/verification/ots/reqstream.md b/docs/verification/ots/reqstream.md index 2c7e367..f985212 100644 --- a/docs/verification/ots/reqstream.md +++ b/docs/verification/ots/reqstream.md @@ -1,4 +1,4 @@ -## ReqStream Verification +## ReqStream ### Verification Approach diff --git a/docs/verification/ots/reviewmark.md b/docs/verification/ots/reviewmark.md index 8a79bc7..540d2a7 100644 --- a/docs/verification/ots/reviewmark.md +++ b/docs/verification/ots/reviewmark.md @@ -1,4 +1,4 @@ -## ReviewMark Verification +## ReviewMark ### Verification Approach diff --git a/docs/verification/ots/sarifmark.md b/docs/verification/ots/sarifmark.md index b35e774..950b44e 100644 --- a/docs/verification/ots/sarifmark.md +++ b/docs/verification/ots/sarifmark.md @@ -1,4 +1,4 @@ -## SarifMark Verification +## SarifMark ### Verification Approach diff --git a/docs/verification/ots/sonarmark.md b/docs/verification/ots/sonarmark.md index b37a87d..11fad7a 100644 --- a/docs/verification/ots/sonarmark.md +++ b/docs/verification/ots/sonarmark.md @@ -1,4 +1,4 @@ -## SonarMark Verification +## SonarMark ### Verification Approach diff --git a/docs/verification/ots/versionmark.md b/docs/verification/ots/versionmark.md index 6c99425..ce91647 100644 --- a/docs/verification/ots/versionmark.md +++ b/docs/verification/ots/versionmark.md @@ -1,4 +1,4 @@ -## VersionMark Verification +## VersionMark ### Verification Approach diff --git a/docs/verification/ots/weasyprint.md b/docs/verification/ots/weasyprint.md index 47fa84d..27fb3a9 100644 --- a/docs/verification/ots/weasyprint.md +++ b/docs/verification/ots/weasyprint.md @@ -1,4 +1,4 @@ -## WeasyPrint Verification +## WeasyPrint ### Verification Approach diff --git a/docs/verification/ots/xunit.md b/docs/verification/ots/xunit.md index 4c57bf2..0728362 100644 --- a/docs/verification/ots/xunit.md +++ b/docs/verification/ots/xunit.md @@ -1,4 +1,4 @@ -## xUnit Verification +## xUnit ### Verification Approach diff --git a/fix.ps1 b/fix.ps1 index f012e9f..d52dc1a 100644 --- a/fix.ps1 +++ b/fix.ps1 @@ -2,11 +2,16 @@ # # PURPOSE: # Applies all available auto-fixers with progress output. Always exits 0. -# Run this after making changes to automatically handle formatting. +# Run this after making changes to automatically handle formatting +# so agents and developers do not need to respond to lint output. # Handles: dotnet format, markdownlint, yamlfix, YAML line endings. # # EXTENSION POINTS: # Search for "[PROJECT-SPECIFIC]" comments to add project-specific fixers. +# +# MODIFICATION POLICY: +# Only modify this file to add project-specific operations at the designated +# [PROJECT-SPECIFIC] extension points, or to update tool versions as needed. function Get-VenvActivateScript { if (Test-Path ".venv/Scripts/Activate.ps1") { return ".venv/Scripts/Activate.ps1" } diff --git a/lint.ps1 b/lint.ps1 index afadc2e..8959539 100644 --- a/lint.ps1 +++ b/lint.ps1 @@ -2,10 +2,17 @@ # # PURPOSE: # Runs all lint checks and reports failures. Exits 1 on error. -# Used by CI/CD as the merge gate. +# Used by CI/CD as the merge gate and by the lint-fix agent +# during pre-PR cleanup. +# +# To auto-fix formatting issues, run fix.ps1 instead. # # EXTENSION POINTS: # Search for "[PROJECT-SPECIFIC]" comments to add project-specific checks. +# +# MODIFICATION POLICY: +# Only modify this file to add project-specific operations at the designated +# [PROJECT-SPECIFIC] extension points, or to update tool versions as needed. function Get-VenvActivateScript { if (Test-Path ".venv/Scripts/Activate.ps1") { return ".venv/Scripts/Activate.ps1" } From 0c3033145b2483df4b76d42736cb639e776d54c3 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 12:43:51 -0400 Subject: [PATCH 03/10] Add TestResults-Verification review set and fix reviewmark gaps - Add docs/verification/**/*.md to needs-review patterns - Add verification introduction and system verification docs to TestResults-Architecture review set per standard - Add new TestResults-Verification review set covering verification introduction, ots.md, system and subsystem verification docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .reviewmark.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/.reviewmark.yaml b/.reviewmark.yaml index 4e1de0f..061852e 100644 --- a/.reviewmark.yaml +++ b/.reviewmark.yaml @@ -11,6 +11,7 @@ needs-review: - '**/*.cs' # All C# source and test files - docs/reqstream/**/*.yaml # Requirements files - docs/design/**/*.md # Design documentation files + - docs/verification/**/*.md # Verification documentation files - docs/user_guide/**/*.md # User guide documentation - '!**/obj/**' # Exclude build output - '!**/bin/**' # Exclude build output @@ -44,6 +45,8 @@ reviews: - docs/reqstream/test-results-library.yaml - docs/design/introduction.md - docs/design/test-results-library.md + - docs/verification/introduction.md + - docs/verification/test-results-library.md - test/DemaConsulting.TestResults.Tests/TestResultsLibraryTests.cs - id: TestResults-Design @@ -56,6 +59,15 @@ reviews: - docs/design/test-results-library.md - docs/design/test-results-library/**/*.md + - id: TestResults-Verification + title: Review that TestResults Library Verification is Consistent and Complete + paths: + - docs/reqstream/test-results-library.yaml + - docs/verification/introduction.md + - docs/verification/ots.md + - docs/verification/test-results-library.md + - docs/verification/test-results-library/**/*.md + - id: TestResults-AllRequirements title: Review that All TestResults Library Requirements are Complete paths: From 69c2d23c16734a1b6e1e9fc6f2b1ce6e482f29c5 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 12:49:42 -0400 Subject: [PATCH 04/10] Add review-set group comments to .reviewmark.yaml Follow the commenting style used across DEMA repositories: - '# Purpose' before the Purpose review - '# TestResults - Specials' before system-level reviews - '# TestResults - {Name}' before each unit/subsystem group - '# OTS Items' before OTS reviews Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .reviewmark.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.reviewmark.yaml b/.reviewmark.yaml index 061852e..61f4577 100644 --- a/.reviewmark.yaml +++ b/.reviewmark.yaml @@ -30,6 +30,7 @@ evidence-source: # Review sets following hierarchical scope principles. # Each review-set focuses on a single compliance question with manageable file counts. reviews: + # Purpose - id: Purpose title: Review that Advertised Features Match System Design paths: @@ -39,6 +40,7 @@ reviews: - docs/design/introduction.md - docs/design/test-results-library.md + # TestResults - Specials - id: TestResults-Architecture title: Review that TestResults Library Architecture Satisfies Requirements paths: @@ -76,6 +78,7 @@ reviews: - docs/reqstream/test-results-library/**/*.yaml - docs/reqstream/ots/**/*.yaml + # TestResults - IO - id: TestResults-IO title: Review that TestResults IO Satisfies Subsystem Requirements paths: @@ -84,6 +87,7 @@ reviews: - docs/verification/test-results-library/io.md - test/DemaConsulting.TestResults.Tests/IO/IOTests.cs + # TestResults - TestOutcome - id: TestResults-TestOutcome title: Review that TestResults TestOutcome Implementation is Correct paths: @@ -92,6 +96,7 @@ reviews: - src/DemaConsulting.TestResults/TestOutcome.cs - test/DemaConsulting.TestResults.Tests/TestOutcomeTests.cs + # TestResults - TestResult - id: TestResults-TestResult title: Review that TestResults TestResult Implementation is Correct paths: @@ -100,6 +105,7 @@ reviews: - src/DemaConsulting.TestResults/TestResult.cs - test/DemaConsulting.TestResults.Tests/TestResultTests.cs + # TestResults - TestResults - id: TestResults-TestResults title: Review that TestResults TestResults Implementation is Correct paths: @@ -145,6 +151,7 @@ reviews: - src/DemaConsulting.TestResults/IO/JUnitSerializer.cs - test/DemaConsulting.TestResults.Tests/IO/JUnitSerializerTests.cs + # OTS Items - id: OTS-BuildMark title: Review that BuildMark Provides Required Functionality paths: From f1eb97653a37720351701852379c83a3f9996374 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 12:58:32 -0400 Subject: [PATCH 05/10] Revert io/ unit heading depth to ### (correct for depth-2 folder) The previous template-sync fix incorrectly promoted these headings from ### to ##. Per the technical-documentation standard, files two subfolders deep under the collection root must use ### as their top-level heading. Restored the original ### depth for all four io/ unit design files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../test-results-library/io/junit-serializer.md | 14 +++++++------- .../test-results-library/io/serializer-helpers.md | 14 +++++++------- docs/design/test-results-library/io/serializer.md | 14 +++++++------- .../test-results-library/io/trx-serializer.md | 14 +++++++------- 4 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/design/test-results-library/io/junit-serializer.md b/docs/design/test-results-library/io/junit-serializer.md index b4bd596..28a7add 100644 --- a/docs/design/test-results-library/io/junit-serializer.md +++ b/docs/design/test-results-library/io/junit-serializer.md @@ -1,13 +1,13 @@ -## JUnitSerializer +### JUnitSerializer -### Purpose +#### Purpose The JUnitSerializer unit converts between the shared in-memory model and JUnit XML test result documents. It adapts the richer local model to the narrower JUnit schema by grouping results into suites, projecting outcomes onto JUnit elements, and recovering as much metadata as the format provides when reading JUnit input. -### Data Model +#### Data Model **DefaultSuiteName**: `string` - Sentinel suite name used when a `TestResult` has an empty `ClassName`. @@ -21,7 +21,7 @@ root element. **MessageAttributeName**: `string` - Attribute name reused for `failure`, `error`, and `skipped` message text. -### Key Methods +#### Key Methods **Serialize**: Writes a `TestResults` model as JUnit XML. @@ -87,7 +87,7 @@ represents the earliest test start, matching the convention used by common JUnit `TestResult` default; the `DefaultSuiteName` sentinel classname is mapped back to an empty string so round-trips produce the original `ClassName` value. -### Error Handling +#### Error Handling `Serialize()` throws `ArgumentNullException` when `results` is null. `Deserialize()` throws `ArgumentNullException` or `ArgumentException` for null or whitespace input and throws @@ -100,7 +100,7 @@ to an empty class name. Because JUnit has no inconclusive element, `Inconclusive serialize as plain `testcase` elements, making them indistinguishable from `Passed` outcomes on deserialization; this round-trip loss is a known limitation of the JUnit format. -### Dependencies +#### Dependencies - **TestResults** - provides run metadata for the `testsuites` root and receives deserialized runs. @@ -111,7 +111,7 @@ on deserialization; this round-trip loss is a known limitation of the JUnit form - **System.Globalization** - formats and parses timestamps and durations using invariant culture. -### Callers +#### Callers - **Serializer** - delegates JUnit deserialization after format identification. - **Library consumers** - call the static serialize and deserialize entry points directly when diff --git a/docs/design/test-results-library/io/serializer-helpers.md b/docs/design/test-results-library/io/serializer-helpers.md index b8409f8..6664d1e 100644 --- a/docs/design/test-results-library/io/serializer-helpers.md +++ b/docs/design/test-results-library/io/serializer-helpers.md @@ -1,12 +1,12 @@ -## SerializerHelpers +### SerializerHelpers -### Purpose +#### Purpose The SerializerHelpers unit provides shared infrastructure used by both XML serializers. Its single responsibility is to make string-based XML serialization report UTF-8 rather than the default UTF-16 declaration produced by `StringWriter`. -### Data Model +#### Data Model **Utf8StringWriter**: `internal sealed class` - Specialized `StringWriter` implementation used by both serializer units when saving XML to a string. @@ -14,7 +14,7 @@ by both serializer units when saving XML to a string. **Encoding**: `Encoding` - Overridden property that always returns `Encoding.UTF8` so the XML prolog declares `encoding="utf-8"`. -### Key Properties +#### Key Properties **Encoding**: Read-only property override that reports UTF-8 as the writer encoding. @@ -26,17 +26,17 @@ prolog declares `encoding="utf-8"`. The property exists because .NET `StringWriter` defaults to UTF-16, which would advertise the wrong encoding for XML strings that callers later persist as UTF-8. -### Error Handling +#### Error Handling N/A - the unit exposes a deterministic property override with no inputs, branching, or local error recovery. -### Dependencies +#### Dependencies - **System.IO.StringWriter** - base class for string-backed text writing. - **System.Text.Encoding** - provides the UTF-8 encoding instance returned by the override. -### Callers +#### Callers - **TrxSerializer** - uses `Utf8StringWriter` when saving TRX XML to a string. - **JUnitSerializer** - uses `Utf8StringWriter` when saving JUnit XML to a string. diff --git a/docs/design/test-results-library/io/serializer.md b/docs/design/test-results-library/io/serializer.md index 8e47bb4..6bad4f7 100644 --- a/docs/design/test-results-library/io/serializer.md +++ b/docs/design/test-results-library/io/serializer.md @@ -1,12 +1,12 @@ -## Serializer +### Serializer -### Purpose +#### Purpose The Serializer unit is the IO subsystem facade for callers that do not know the incoming format. It identifies whether XML content is TRX or JUnit and delegates deserialization to the corresponding format-specific serializer. -### Data Model +#### Data Model **TestResultFormat**: `enum` - Defines the three public detection outcomes: `Unknown`, `Trx`, and `JUnit`. @@ -14,7 +14,7 @@ and `JUnit`. **TrxNamespaceUri**: `string` - Hard-coded XML namespace used to distinguish genuine TRX `TestRun` roots from unrelated XML documents. -### Key Methods +#### Key Methods **Identify**: Detects the serialized test-result format from XML content. @@ -39,7 +39,7 @@ and checks `testsuites` or `testsuite` root names for JUnit. The method keeps format selection in one place so callers can deserialize mixed result files through a single entry point. -### Error Handling +#### Error Handling `Identify()` catches `XmlException` and converts invalid XML to `TestResultFormat.Unknown`. `Deserialize()` validates `contents` before any XML parsing: passing `null` throws @@ -47,14 +47,14 @@ through a single entry point. `ArgumentException`. It throws `InvalidOperationException` when `Identify()` cannot classify the document. Exceptions raised by the delegated serializer propagate to the caller. -### Dependencies +#### Dependencies - **TrxSerializer** - performs format-specific TRX deserialization. - **JUnitSerializer** - performs format-specific JUnit deserialization. - **TestResults** - common return type for deserialization. - **System.Xml.Linq.XDocument** - parses XML roots for format identification. -### Callers +#### Callers N/A - entry-point unit, called directly by library consumers and exercised by system and subsystem tests. diff --git a/docs/design/test-results-library/io/trx-serializer.md b/docs/design/test-results-library/io/trx-serializer.md index e0c35f7..67689da 100644 --- a/docs/design/test-results-library/io/trx-serializer.md +++ b/docs/design/test-results-library/io/trx-serializer.md @@ -1,12 +1,12 @@ -## TrxSerializer +### TrxSerializer -### Purpose +#### Purpose The TrxSerializer unit converts between the shared in-memory model and Microsoft TRX test result documents. It is responsible for preserving the richer TRX metadata model, including run metadata, test-definition cross-references, output streams, and summary counters. -### Data Model +#### Data Model **TrxNamespace**: `XNamespace` - Fixed TRX namespace used for element construction and lookup. @@ -27,7 +27,7 @@ and parse TRX timestamps. **DurationFormatString**: `string` - Constant/invariant format string (`"c"`) used to format and parse TRX durations as TimeSpan values. -### Key Methods +#### Key Methods **Serialize**: Writes a `TestResults` model as TRX XML. @@ -67,7 +67,7 @@ resolution is linear rather than repeatedly scanning the XML document. The method accepts named values and defined numeric values, then falls back to `Failed` so callers never receive an undefined `TestOutcome` from TRX deserialization. -### Error Handling +#### Error Handling `Serialize()` throws `ArgumentNullException` when `results` is null. `Deserialize()` throws `ArgumentNullException` or `ArgumentException` for null or whitespace input, and throws @@ -77,7 +77,7 @@ Malformed or missing GUIDs fall back to newly generated GUIDs, malformed duratio to `TimeSpan.Zero`, malformed timestamps fall back to `DateTime.UtcNow`, and unrecognized outcome values fall back to `TestOutcome.Failed`. -### Dependencies +#### Dependencies - **TestResults** - provides run-level metadata for `TestRun` and receives deserialized runs. - **TestResult** - provides per-test execution data and receives deserialized test cases. @@ -86,7 +86,7 @@ outcome values fall back to `TestOutcome.Failed`. - **System.Xml.Linq** and **System.Xml.XPath** - construct and query TRX XML. - **System.Globalization** - formats and parses invariant timestamps and durations. -### Callers +#### Callers - **Serializer** - delegates TRX deserialization after format identification. - **Library consumers** - call the static serialize and deserialize entry points directly when From bf20af24a47bcf8ba2277f39c6241cc74f04605c Mon Sep 17 00:00:00 2001 From: Malcolm Nixon Date: Fri, 19 Jun 2026 13:15:57 -0400 Subject: [PATCH 06/10] Address formal review findings across documentation and requirements Fixes based on formal review of all 24 review-sets: README and system design: - Remove Continuous Compliance from library feature list - Add supported target frameworks (.NET Standard 2.0, 8, 9, 10) to design - Clarify Dependencies section to distinguish runtime vs CI tooling OTS items Verification docs: - Fix stale requirement IDs in test-result.md and test-results.md - Add IdentifyFormat verification scenario to system-level verification IO subsystem: - Move round-trip requirement from Serializer unit to IO subsystem (io.yaml) - Update Serializer design to position round-trip as subsystem-level guarantee - Add complete JUnit outcome mapping table (all 14 TestOutcome values) - Fix obsolete requirement ID references in SerializerTests.cs OTS documentation: - BuildMark: fix GITHUB_TOKEN -> GH_TOKEN; improve acceptance criteria; fix stale req ID - SarifMark: add enforcement requirement; add to acceptance criteria - SonarMark: remove unsupported metrics claim - FileAssert: fix --tags -> positional args syntax - WeasyPrint: add Requirements and TraceMatrix PDF coverage - ReqStream: split compound requirement into 5 atomic requirements - ReviewMark: split compound requirement into 5; add --lint req/design/verification - Pandoc: add template/TOC/numbered-sections requirements and verification Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .fileassert.yaml | 8 +++ README.md | 3 - docs/design/ots/buildmark.md | 4 +- docs/design/ots/fileassert.md | 18 +++--- docs/design/ots/reviewmark.md | 3 + docs/design/ots/sonarmark.md | 4 +- docs/design/test-results-library.md | 10 +++- .../io/junit-serializer.md | 30 ++++++++++ .../test-results-library/io/serializer.md | 4 +- docs/reqstream/ots/pandoc.yaml | 43 ++++++++++++++ docs/reqstream/ots/reqstream.yaml | 59 ++++++++++++++++--- docs/reqstream/ots/reviewmark.yaml | 50 ++++++++++++++-- docs/reqstream/ots/sarifmark.yaml | 8 +++ docs/reqstream/ots/weasyprint.yaml | 2 + docs/reqstream/test-results-library/io.yaml | 14 ++++- .../test-results-library/io/serializer.yaml | 10 ---- docs/verification/ots/buildmark.md | 8 +-- docs/verification/ots/pandoc.md | 32 ++++++++++ docs/verification/ots/reqstream.md | 21 ++++--- docs/verification/ots/reviewmark.md | 16 +++-- docs/verification/ots/sarifmark.md | 8 ++- docs/verification/ots/weasyprint.md | 12 +++- docs/verification/test-results-library.md | 10 +++- .../test-results-library/test-result.md | 10 ++-- .../test-results-library/test-results.md | 30 ++++++---- .../IO/SerializerTests.cs | 4 +- 26 files changed, 338 insertions(+), 83 deletions(-) diff --git a/.fileassert.yaml b/.fileassert.yaml index 8bdaab6..ebcb83b 100644 --- a/.fileassert.yaml +++ b/.fileassert.yaml @@ -171,6 +171,10 @@ tests: html: - query: //head/title count: 1 + - query: //nav[@id='TOC'] + count: 1 + - query: (//span[@class='header-section-number'])[1] + count: 1 text: - contains: Design @@ -204,6 +208,10 @@ tests: html: - query: //head/title count: 1 + - query: //nav[@id='TOC'] + count: 1 + - query: (//span[@class='header-section-number'])[1] + count: 1 text: - contains: Verification diff --git a/README.md b/README.md index c2a0f88..c5a2baa 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,6 @@ test outcomes and results, supporting serialization to/from TRX (Visual Studio T - 📦 **NuGet Ready** - Easy integration via NuGet package - 📊 **Multiple Formats** - Supports both TRX and JUnit XML formats - ✅ **Compatible** - Works with Visual Studio, Azure DevOps, and CI/CD systems -- 🔒 **Continuous Compliance** - Compliance evidence generated automatically on every CI run, - following the [Continuous Compliance](https://github.com/demaconsulting/ContinuousCompliance) - methodology ## Installation diff --git a/docs/design/ots/buildmark.md b/docs/design/ots/buildmark.md index e37d567..8670c3d 100644 --- a/docs/design/ots/buildmark.md +++ b/docs/design/ots/buildmark.md @@ -10,7 +10,7 @@ chosen because it integrates directly with the GitHub Actions environment, captu ### Features Used - **GitHub Actions metadata query**: retrieves workflow run details, trigger information, and - linked issues from the GitHub API using the `GITHUB_TOKEN` environment variable. + linked issues from the GitHub API using the `GH_TOKEN` environment variable. - **Git integration**: records the commit SHA and branch name associated with the build. - **Issue tracking**: lists GitHub issues linked to the pull request or commit being built. - **Known-issues reporting**: collects open issues labeled as known defects and includes them in @@ -22,7 +22,7 @@ chosen because it integrates directly with the GitHub Actions environment, captu BuildMark is installed as a `dotnet` tool via the `dotnet-tools.json` manifest and restored before the pipeline documentation step. It is invoked once per release pipeline run after the -build and test steps succeed. The tool reads `GITHUB_TOKEN`, `GITHUB_REPOSITORY`, and +build and test steps succeed. The tool reads `GH_TOKEN`, `GITHUB_REPOSITORY`, and `GITHUB_RUN_ID` from the GitHub Actions environment and writes its output to `docs/build_notes/generated/build_notes.md`. That file is then consumed by Pandoc to produce the build-notes HTML, which WeasyPrint converts to the release PDF. No runtime configuration diff --git a/docs/design/ots/fileassert.md b/docs/design/ots/fileassert.md index 3990297..20876f2 100644 --- a/docs/design/ots/fileassert.md +++ b/docs/design/ots/fileassert.md @@ -17,15 +17,17 @@ expected content, acting as the primary OTS evidence mechanism for Pandoc and We and asserts minimum page count and expected text content in the rendered PDF body. - **Zip archive entry validation**: inspects NuGet package (`.nupkg`) zip archives to confirm that required documentation files are present with the expected path patterns. -- **Tag-based test filtering**: accepts a `--tags` argument so the pipeline can run only the - assertions relevant to a given build phase (for example, `design` or `user-guide`). +- **Tag-based test filtering**: accepts positional tag arguments so the pipeline can run only the + assertions relevant to a given build phase (for example, `fileassert design` or + `fileassert user-guide`). ### Integration Pattern FileAssert is installed as a `dotnet` tool via the `dotnet-tools.json` manifest. It is invoked -multiple times during the pipeline, once per document group, with `--tags` selecting the -relevant subset of assertions from `.fileassert.yaml`. It runs after the corresponding -Pandoc/WeasyPrint step and before ReqStream so that document-validation results are available -as OTS evidence. The assertion criteria are maintained in `.fileassert.yaml` at the repository -root. No initialization or disposal steps are required; the tool reads the configuration file, -evaluates each assertion, and exits with a non-zero code if any assertion fails. +multiple times during the pipeline, once per document group, with positional tag arguments +selecting the relevant subset of assertions from `.fileassert.yaml`. It runs after the +corresponding Pandoc/WeasyPrint step and before ReqStream so that document-validation results +are available as OTS evidence. The assertion criteria are maintained in `.fileassert.yaml` at +the repository root. No initialization or disposal steps are required; the tool reads the +configuration file, evaluates each assertion, and exits with a non-zero code if any assertion +fails. diff --git a/docs/design/ots/reviewmark.md b/docs/design/ots/reviewmark.md index 708482e..a21206f 100644 --- a/docs/design/ots/reviewmark.md +++ b/docs/design/ots/reviewmark.md @@ -21,6 +21,9 @@ produce the evidence PDFs that demonstrate formal review compliance. artifact used during the formal review session. - **Working directory override**: the `--working-dir` flag allows the tool to run from a directory other than the repository root, supporting CI matrix builds. +- **Configuration validation**: the `--lint` flag validates the `.reviewmark.yaml` configuration + against its schema and reports structural or semantic errors. It is invoked in `lint.ps1` as a + CI gate to prevent malformed configuration from reaching the pipeline. ### Integration Pattern diff --git a/docs/design/ots/sonarmark.md b/docs/design/ots/sonarmark.md index 13dc10c..8a1d2b0 100644 --- a/docs/design/ots/sonarmark.md +++ b/docs/design/ots/sonarmark.md @@ -2,8 +2,8 @@ ### Purpose -SonarMark is a quality reporting tool that retrieves quality-gate status and metrics data from -SonarCloud and renders them as a markdown document included in the release artifacts. It is +SonarMark is a quality reporting tool that retrieves quality-gate status, issues, and hotspots +from SonarCloud and renders them as a markdown document included in the release artifacts. It is chosen to surface static analysis quality data from SonarCloud in a format that can be reviewed alongside other compliance artifacts. diff --git a/docs/design/test-results-library.md b/docs/design/test-results-library.md index 043305b..0cf41de 100644 --- a/docs/design/test-results-library.md +++ b/docs/design/test-results-library.md @@ -89,7 +89,9 @@ flowchart TD N/A - the system relies only on the .NET runtime and base class library APIs such as LINQ to XML, XPath, culture-aware formatting, and text encoding. No separately documented OTS or -shared package design files are modeled in this design collection. +shared package design files are modeled in this design collection. CI and documentation +tooling OTS items are documented separately in the OTS design collection but are not runtime +dependencies of the library. ## Risk Control Measures @@ -108,8 +110,10 @@ N/A - not a safety-classified software item. ## Design Constraints -- Platform: the implementation is an in-process C#/.NET library with no built-in file, - network, or database transport layer. +- Platform: the implementation is an in-process C#/.NET library targeting .NET Standard 2.0, + .NET 8, .NET 9, and .NET 10, with no built-in file, network, or database transport layer. +- Runtime dependencies: zero — the library depends only on the .NET runtime and base class + library; no third-party packages are required at runtime. - Format neutrality: the model units store test semantics, not TRX-specific or JUnit-specific XML nodes, so conversion stays isolated in the IO subsystem. - XML compatibility: TRX detection requires the exact TRX namespace and `TestRun` root, diff --git a/docs/design/test-results-library/io/junit-serializer.md b/docs/design/test-results-library/io/junit-serializer.md index 28a7add..643dc29 100644 --- a/docs/design/test-results-library/io/junit-serializer.md +++ b/docs/design/test-results-library/io/junit-serializer.md @@ -87,6 +87,36 @@ represents the earliest test start, matching the convention used by common JUnit `TestResult` default; the `DefaultSuiteName` sentinel classname is mapped back to an empty string so round-trips produce the original `ClassName` value. +#### Outcome Mapping + +JUnit XML supports only three distinct outcome states — `failure`, `error`, and `skipped` child +elements, with absence of any child element meaning the test passed. The table below documents how +every `TestOutcome` value maps through JUnit serialization and deserialization. Entries marked +**lossy** do not survive a round-trip. + +| `TestOutcome` | Serializes to JUnit element | Deserializes back as | Fidelity | +| --- | --- | --- | --- | +| `Passed` | *(no child element)* | `Passed` | Preserved | +| `PassedButRunAborted` | *(no child element)* | `Passed` | **Lossy** | +| `Warning` | *(no child element)* | `Passed` | **Lossy** | +| `Inconclusive` | *(no child element)* | `Passed` | **Lossy** | +| `Disconnected` | *(no child element)* | `Passed` | **Lossy** | +| `Completed` | *(no child element)* | `Passed` | **Lossy** | +| `InProgress` | *(no child element)* | `Passed` | **Lossy** | +| `Failed` | `failure` element | `Failed` | Preserved | +| `Error` | `error` element | `Error` | Preserved | +| `Timeout` | `error` element | `Error` | **Lossy** | +| `Aborted` | `error` element | `Error` | **Lossy** | +| `NotExecuted` | `skipped` element | `NotExecuted` | Preserved | +| `NotRunnable` | `skipped` element | `NotExecuted` | **Lossy** | +| `Pending` | `skipped` element | `NotExecuted` | **Lossy** | + +The grouping logic mirrors `TestOutcomeExtensions`: `Failed` maps to a `failure` element; +`Error`, `Timeout`, and `Aborted` (the outcomes where `IsErrorOutcome` returns true) map to an +`error` element; outcomes where `IsExecuted()` returns false (`NotRunnable`, `NotExecuted`, +`Pending`) map to a `skipped` element; all remaining executed outcomes that are not `Failed` or +error-class produce a plain `testcase` with no child outcome element, which reads back as `Passed`. + #### Error Handling `Serialize()` throws `ArgumentNullException` when `results` is null. `Deserialize()` throws diff --git a/docs/design/test-results-library/io/serializer.md b/docs/design/test-results-library/io/serializer.md index 6bad4f7..bf65a8c 100644 --- a/docs/design/test-results-library/io/serializer.md +++ b/docs/design/test-results-library/io/serializer.md @@ -37,7 +37,9 @@ and checks `testsuites` or `testsuite` root names for JUnit. `JUnitSerializer.Deserialize()` and throws for unknown formats. The method keeps format selection in one place so callers can deserialize mixed result files -through a single entry point. +through a single entry point. Round-trip fidelity across TRX and JUnit is a subsystem-level +guarantee achieved through the combined behavior of TrxSerializer and JUnitSerializer; the +Serializer itself only provides the dispatch mechanism. #### Error Handling diff --git a/docs/reqstream/ots/pandoc.yaml b/docs/reqstream/ots/pandoc.yaml index fca8114..725ca80 100644 --- a/docs/reqstream/ots/pandoc.yaml +++ b/docs/reqstream/ots/pandoc.yaml @@ -24,3 +24,46 @@ sections: - Pandoc_ReviewReportHtml - Pandoc_DesignHtml - Pandoc_UserGuideHtml + + - id: TestResults-OTS-Pandoc-Template + title: Pandoc shall apply an HTML template to produce a standalone HTML document. + justification: | + The documentation build pipeline configures Pandoc with `template: template.html` + via each document collection's `definition.yaml`. FileAssert validates that the + generated design and verification HTML files each contain a `` element inside + a `<head>` element, which is only present when Pandoc processes the inputs through + the custom HTML template. Plain fragment output (no template) would contain no + `<head>` or `<title>`. Passing these assertions proves Pandoc correctly applies the + template to every document collection that requires it. + tags: [ots] + tests: + - Pandoc_DesignHtml + - Pandoc_VerificationHtml + + - id: TestResults-OTS-Pandoc-Toc + title: Pandoc shall generate a table of contents for document collections. + justification: | + The documentation build pipeline configures Pandoc with `table-of-contents: true` + via each document collection's `definition.yaml`. FileAssert validates that the + generated design and verification HTML files each contain a `<nav id="TOC">` element, + which the HTML template emits only when Pandoc has processed the + `table-of-contents: true` option and produced a TOC structure. Passing these + assertions proves Pandoc correctly generates the table of contents. + tags: [ots] + tests: + - Pandoc_DesignHtml + - Pandoc_VerificationHtml + + - id: TestResults-OTS-Pandoc-NumberSections + title: Pandoc shall apply sequential section numbers to document headings. + justification: | + The documentation build pipeline configures Pandoc with `number-sections: true` + via each document collection's `definition.yaml`. FileAssert validates that the + generated design and verification HTML files each contain at least one + `<span class="header-section-number">` element, which Pandoc inserts into heading + elements only when the number-sections option is enabled. Passing these assertions + proves Pandoc correctly numbers document sections. + tags: [ots] + tests: + - Pandoc_DesignHtml + - Pandoc_VerificationHtml diff --git a/docs/reqstream/ots/reqstream.yaml b/docs/reqstream/ots/reqstream.yaml index 81086ed..6204a46 100644 --- a/docs/reqstream/ots/reqstream.yaml +++ b/docs/reqstream/ots/reqstream.yaml @@ -8,21 +8,62 @@ sections: sections: - title: ReqStream Requirements requirements: - - id: TestResults-OTS-ReqStream + - id: TestResults-OTS-ReqStream-RequirementsProcessing title: >- - ReqStream shall process requirements, generate traceability - artifacts, filter by tags, and enforce test coverage. + ReqStream shall read hierarchical requirements YAML files and produce a structured + requirements document. justification: | - DemaConsulting.ReqStream processes requirements.yaml and the TRX test-result files to - produce a requirements report, justifications document, and traceability matrix. It - supports tag-based filtering for phased pipeline execution. When run with --enforce, - it exits with a non-zero code if any requirement lacks test evidence, making unproven - requirements a build-breaking condition. A successful pipeline run with --enforce proves - all requirements are covered and that ReqStream is functioning. + DemaConsulting.ReqStream parses requirements.yaml and any referenced YAML files to + build a complete, hierarchical requirements model. A successful pipeline run producing + the requirements document proves this ingestion and structuring capability is + functioning. tags: [ots] tests: - ReqStream_RequirementsProcessing + + - id: TestResults-OTS-ReqStream-TraceMatrix + title: >- + ReqStream shall generate a traceability matrix linking requirements to test evidence. + justification: | + DemaConsulting.ReqStream correlates requirements identifiers against TRX test-result + files and emits a traceability matrix. A successful pipeline run producing the trace + matrix proves that requirement-to-test linkage is functioning. + tags: [ots] + tests: - ReqStream_TraceMatrix + + - id: TestResults-OTS-ReqStream-ReportExport + title: >- + ReqStream shall export generated reports to the declared output paths. + justification: | + DemaConsulting.ReqStream writes its requirements document and traceability matrix to + the output paths specified in the pipeline configuration. A successful pipeline run + with the expected files present at their declared paths proves this export capability + is functioning. + tags: [ots] + tests: - ReqStream_ReportExport + + - id: TestResults-OTS-ReqStream-TagsFiltering + title: >- + ReqStream shall filter requirements by tag to support phased pipeline execution. + justification: | + DemaConsulting.ReqStream respects the tags: field on requirements, allowing pipelines + to process only the subset of requirements relevant to a given phase. A successful + pipeline run exercising tag-scoped invocations proves this filtering capability is + functioning. + tags: [ots] + tests: - ReqStream_TagsFiltering + + - id: TestResults-OTS-ReqStream-EnforcementMode + title: >- + ReqStream shall exit non-zero when any requirement lacks test evidence. + justification: | + DemaConsulting.ReqStream is invoked with --enforce in the CI pipeline. When any + requirement has no linked passing test, it exits with a non-zero code, making coverage + gaps a build-breaking condition. A successful pipeline run with --enforce proves both + that all requirements are covered and that this enforcement mechanism is functioning. + tags: [ots] + tests: - ReqStream_EnforcementMode diff --git a/docs/reqstream/ots/reviewmark.yaml b/docs/reqstream/ots/reviewmark.yaml index aea9d43..7e57d63 100644 --- a/docs/reqstream/ots/reviewmark.yaml +++ b/docs/reqstream/ots/reviewmark.yaml @@ -21,18 +21,58 @@ sections: - id: TestResults-OTS-ReviewMark-PlanGeneration title: ReviewMark shall generate a review plan from the review configuration. justification: | - DemaConsulting.ReviewMark reads the .reviewmark.yaml configuration and the review - evidence store index to produce a review plan documenting file review coverage and - currency. It supports working directory override, enforcement, and elaboration. A - successful pipeline run with --enforce proves all files are assigned to a review-set - and that review plan generation is functioning. + DemaConsulting.ReviewMark reads the .reviewmark.yaml configuration to produce a review + plan documenting file review coverage and currency. A successful pipeline run confirms + that review plan generation is functioning. tags: [ots] tests: - ReviewMark_ReviewPlanGeneration + - id: TestResults-OTS-ReviewMark-IndexScan + title: ReviewMark shall scan the evidence store index to determine review currency. + justification: | + DemaConsulting.ReviewMark reads the evidence store index to incorporate review currency + data into the review plan and report. A successful scan confirms the tool can locate + and parse the index at its configured URL. + tags: [ots] + tests: - ReviewMark_IndexScan + - id: TestResults-OTS-ReviewMark-WorkingDirectoryOverride + title: ReviewMark shall accept a working directory override flag. + justification: | + DemaConsulting.ReviewMark supports a --working-dir flag that allows the tool to run + from a directory other than the repository root. This is required for CI matrix builds + that invoke the tool from a different working directory. + tags: [ots] + tests: - ReviewMark_WorkingDirectoryOverride + - id: TestResults-OTS-ReviewMark-Enforcement + title: ReviewMark shall exit non-zero when review coverage gaps are detected. + justification: | + DemaConsulting.ReviewMark supports an --enforce flag that causes it to exit non-zero + if any file matched by the needs-review patterns is not assigned to a review-set. + This makes coverage gaps build-breaking in the CI pipeline. + tags: [ots] + tests: - ReviewMark_Enforce + - id: TestResults-OTS-ReviewMark-Elaboration + title: ReviewMark shall expand a named review-set into a review checklist. + justification: | + DemaConsulting.ReviewMark supports an --elaborate flag that expands a named review-set + into a checklist artifact used during the formal review session. This confirms the + elaboration workflow is functional. + tags: [ots] + tests: - ReviewMark_Elaborate + - id: TestResults-OTS-ReviewMark-Lint + title: ReviewMark shall validate the review configuration file. + justification: | + DemaConsulting.ReviewMark supports a --lint flag that validates the .reviewmark.yaml + configuration against its schema and reports any structural or semantic errors. It is + invoked in lint.ps1 as a CI gate to prevent malformed configuration from reaching the + pipeline. + tags: [ots] + tests: + - ReviewMark_Lint - id: TestResults-OTS-ReviewMark-ReportGeneration title: ReviewMark shall generate a review report from the review configuration and evidence store. justification: | diff --git a/docs/reqstream/ots/sarifmark.yaml b/docs/reqstream/ots/sarifmark.yaml index 6ed4ef0..616eb2b 100644 --- a/docs/reqstream/ots/sarifmark.yaml +++ b/docs/reqstream/ots/sarifmark.yaml @@ -19,4 +19,12 @@ sections: tests: - SarifMark_SarifReading - SarifMark_MarkdownReportGeneration + - id: TestResults-OTS-SarifMark-Enforcement + title: SarifMark shall exit non-zero when SARIF findings exceed the configured threshold. + justification: | + Enforcement makes quality-gate failures build-breaking: when the number of SARIF findings + exceeds the configured threshold SarifMark returns a non-zero exit code, causing the CI + pipeline to fail and preventing a release until the findings are resolved. + tags: [ots] + tests: - SarifMark_Enforcement diff --git a/docs/reqstream/ots/weasyprint.yaml b/docs/reqstream/ots/weasyprint.yaml index c829a44..724e85a 100644 --- a/docs/reqstream/ots/weasyprint.yaml +++ b/docs/reqstream/ots/weasyprint.yaml @@ -24,3 +24,5 @@ sections: - WeasyPrint_ReviewReportPdf - WeasyPrint_DesignPdf - WeasyPrint_UserGuidePdf + - WeasyPrint_RequirementsPdf + - WeasyPrint_TraceMatrixPdf diff --git a/docs/reqstream/test-results-library/io.yaml b/docs/reqstream/test-results-library/io.yaml index 26619b0..cc85a70 100644 --- a/docs/reqstream/test-results-library/io.yaml +++ b/docs/reqstream/test-results-library/io.yaml @@ -38,7 +38,6 @@ sections: children: - TestResults-Serializer-TrxDeserialization - TestResults-Serializer-JUnitDeserialization - - TestResults-Serializer-RoundTrip - TestResults-Trx-Deserialize - TestResults-JUnit-Deserialize tests: @@ -47,7 +46,6 @@ sections: - IO_Deserialize_NullContent_ThrowsArgumentNullException - IO_Deserialize_WhitespaceContent_ThrowsArgumentException - IO_Deserialize_UnknownContent_ThrowsInvalidOperationException - - IO_TrxSerializedResults_RoundTripsViaJUnit_PreservesCoreTestData - id: TestResults-IO-Serialize title: The IO subsystem shall serialize test results to supported file formats. justification: | @@ -63,3 +61,15 @@ sections: tests: - IO_Serialize_TestResults_ProducesTrxContent - IO_Serialize_TestResults_ProducesJUnitContent + - id: TestResults-IO-RoundTrip + title: The IO subsystem shall preserve core test data during round-trip conversion between TRX and JUnit. + justification: | + Round-trip preservation ensures that test results can be serialized and deserialized + without data loss. This is critical for scenarios where test results are stored, + transmitted, or converted between formats, ensuring data integrity throughout the + test result lifecycle. Round-trip fidelity is an emergent behavior of the IO subsystem + as a whole: the Serializer dispatches to TrxSerializer and JUnitSerializer, each of + which independently handles format-specific encoding and decoding. + tests: + - Serializer_RoundTrip_TrxToJUnit_PreservesCoreTestData + - IO_TrxSerializedResults_RoundTripsViaJUnit_PreservesCoreTestData diff --git a/docs/reqstream/test-results-library/io/serializer.yaml b/docs/reqstream/test-results-library/io/serializer.yaml index 6a2f2db..7168312 100644 --- a/docs/reqstream/test-results-library/io/serializer.yaml +++ b/docs/reqstream/test-results-library/io/serializer.yaml @@ -39,13 +39,3 @@ sections: the format, providing a single entry point for JUnit consumption. tests: - Serializer_Deserialize_JUnitContent_ReturnsTestResults - - - id: TestResults-Serializer-RoundTrip - title: The Serializer shall preserve core test data during round-trip conversion between TRX and JUnit. - justification: | - Round-trip preservation ensures that test results can be serialized and deserialized - without data loss. This is critical for scenarios where test results are stored, - transmitted, or converted between formats, ensuring data integrity throughout the - test result lifecycle. - tests: - - Serializer_RoundTrip_TrxToJUnit_PreservesCoreTestData diff --git a/docs/verification/ots/buildmark.md b/docs/verification/ots/buildmark.md index 5dd1af4..1a5a42c 100644 --- a/docs/verification/ots/buildmark.md +++ b/docs/verification/ots/buildmark.md @@ -11,15 +11,15 @@ exercised. ### Test Environment -The GitHub Actions CI pipeline environment with access to the GitHub API, a valid `GITHUB_TOKEN` +The GitHub Actions CI pipeline environment with access to the GitHub API, a valid `GH_TOKEN` environment variable, and an active repository with workflow run history. BuildMark cannot be verified in a purely local offline environment because it requires live GitHub API access. ### Acceptance Criteria - The CI pipeline step invoking BuildMark completes without a non-zero exit code. -- The output file `docs/build_notes/generated/build_notes.md` is produced and contains a - non-trivial amount of content. +- The output file `docs/build_notes/generated/build_notes.md` is produced, is at least 200 bytes + in size, and contains the workflow run ID, branch name, and commit SHA. - The requirement `TestResults-OTS-BuildMark-ReportGeneration` is linked to `BuildMark_MarkdownReportGeneration`, `TestResults-OTS-BuildMark-GitIntegration` to `BuildMark_GitIntegration`, `TestResults-OTS-BuildMark-IssueTracking` to `BuildMark_IssueTracking`, and @@ -28,7 +28,7 @@ verified in a purely local offline environment because it requires live GitHub A ### Test Scenarios **Markdown report generation**: BuildMark shall produce a markdown build-notes document from the -current workflow run metadata, satisfying `TestResults-OTS-BuildMark`. This scenario is +current workflow run metadata, satisfying `TestResults-OTS-BuildMark-ReportGeneration`. This scenario is confirmed by `BuildMark_MarkdownReportGeneration`. **Git integration**: BuildMark shall record the commit SHA and branch associated with the build. diff --git a/docs/verification/ots/pandoc.md b/docs/verification/ots/pandoc.md index ca9238c..5eab35c 100644 --- a/docs/verification/ots/pandoc.md +++ b/docs/verification/ots/pandoc.md @@ -8,6 +8,13 @@ contains a valid `<title>` element, and includes the expected document content. six HTML assertions proves Pandoc executed correctly across all document types produced by the repository. The tests are named in the OTS requirements and tracked by ReqStream. +In addition to basic conversion, the build pipeline depends on three document-structure behaviors +that Pandoc must support: template-driven output, table of contents generation, and numbered +sections. These behaviors are configured via `template: template.html`, +`table-of-contents: true`, and `number-sections: true` in each document collection's +`definition.yaml`. FileAssert validates the design and verification HTML outputs for structural +evidence of each behavior. + ### Test Environment The standard CI pipeline environment with Pandoc installed via the `dotnet-tools.json` manifest @@ -19,8 +26,15 @@ are required. - All six FileAssert HTML assertions pass without error. - Each generated HTML file exists at the expected path under `docs/{collection}/generated/`. - Each HTML file contains a `<title>` element and the expected document title text. +- The design and verification HTML files each contain a `<nav id="TOC">` element confirming + table of contents generation. +- The design and verification HTML files each contain at least one + `<span class="header-section-number">` element confirming section numbering was applied. - The requirement `TestResults-OTS-Pandoc` is linked to all six named test identifiers in the ReqStream trace matrix. +- The requirements `TestResults-OTS-Pandoc-Template`, `TestResults-OTS-Pandoc-Toc`, and + `TestResults-OTS-Pandoc-NumberSections` are each linked to the design and verification HTML + test identifiers in the ReqStream trace matrix. ### Test Scenarios @@ -41,3 +55,21 @@ Markdown inputs. This scenario is confirmed by `Pandoc_DesignHtml`. **User guide HTML generation**: Pandoc shall produce a valid HTML document from the user guide Markdown inputs. This scenario is confirmed by `Pandoc_UserGuideHtml`. + +**Template-driven output**: Pandoc shall apply the custom HTML template (`template.html`) and +produce a standalone document with a `<head>` section including a `<title>` element. Without +the template, Pandoc would emit an HTML fragment with no `<head>` or `<title>`. This scenario +is confirmed by the `//head/title` HTML assertion in `Pandoc_DesignHtml` and +`Pandoc_VerificationHtml`. + +**Table of contents generation**: Pandoc shall insert a `<nav id="TOC">` element into the +output when the `table-of-contents: true` option is active. The design and verification +documents each contain multiple top-level sections, making TOC generation observable. This +scenario is confirmed by the `//nav[@id='TOC']` HTML assertion in `Pandoc_DesignHtml` and +`Pandoc_VerificationHtml`. + +**Numbered section headings**: Pandoc shall prepend sequential numbers to section headings when +the `number-sections: true` option is active, inserting a +`<span class="header-section-number">` element inside each heading. This scenario is confirmed +by the `//span[@class='header-section-number']` HTML assertion in `Pandoc_DesignHtml` and +`Pandoc_VerificationHtml`. diff --git a/docs/verification/ots/reqstream.md b/docs/verification/ots/reqstream.md index f985212..e2ead13 100644 --- a/docs/verification/ots/reqstream.md +++ b/docs/verification/ots/reqstream.md @@ -20,23 +20,30 @@ available as input. - The CI pipeline step invoking ReqStream with `--enforce` exits with code 0. - The generated requirements document and trace matrix are written to their expected output paths under `docs/requirements_doc/generated/` and `docs/requirements_report/generated/`. -- The requirement `TestResults-OTS-ReqStream` is linked to all five named test identifiers in - the ReqStream trace matrix. +- Each requirement below is linked to its corresponding named test identifier in the ReqStream + trace matrix: `TestResults-OTS-ReqStream-RequirementsProcessing`, + `TestResults-OTS-ReqStream-TraceMatrix`, `TestResults-OTS-ReqStream-ReportExport`, + `TestResults-OTS-ReqStream-TagsFiltering`, and `TestResults-OTS-ReqStream-EnforcementMode`. ### Test Scenarios **Requirements processing**: ReqStream shall read hierarchical requirements YAML files and -produce a structured requirements document. This scenario is confirmed by +produce a structured requirements document. This scenario confirms requirement +`TestResults-OTS-ReqStream-RequirementsProcessing` and is evidenced by `ReqStream_RequirementsProcessing`. **Trace matrix generation**: ReqStream shall produce a traceability matrix linking requirements -to test names. This scenario is confirmed by `ReqStream_TraceMatrix`. +to test names. This scenario confirms requirement `TestResults-OTS-ReqStream-TraceMatrix` and is +evidenced by `ReqStream_TraceMatrix`. **Report export**: ReqStream shall write its generated documents to the declared output paths. -This scenario is confirmed by `ReqStream_ReportExport`. +This scenario confirms requirement `TestResults-OTS-ReqStream-ReportExport` and is evidenced by +`ReqStream_ReportExport`. **Tags filtering**: ReqStream shall respect the `tags:` field on requirements to filter scope -during phased pipeline execution. This scenario is confirmed by `ReqStream_TagsFiltering`. +during phased pipeline execution. This scenario confirms requirement +`TestResults-OTS-ReqStream-TagsFiltering` and is evidenced by `ReqStream_TagsFiltering`. **Enforcement mode**: ReqStream shall exit non-zero when any requirement lacks test evidence, -making coverage gaps build-breaking. This scenario is confirmed by `ReqStream_EnforcementMode`. +making coverage gaps build-breaking. This scenario confirms requirement +`TestResults-OTS-ReqStream-EnforcementMode` and is evidenced by `ReqStream_EnforcementMode`. diff --git a/docs/verification/ots/reviewmark.md b/docs/verification/ots/reviewmark.md index 540d2a7..ee3001b 100644 --- a/docs/verification/ots/reviewmark.md +++ b/docs/verification/ots/reviewmark.md @@ -6,15 +6,15 @@ ReviewMark is verified through CI pipeline execution evidence combined with self The tool is invoked during the standard CI pipeline to generate the review plan and review report; a successful run proves it is operational and correctly integrated. Additional self-validation tests (`ReviewMark_VersionDisplay`, `ReviewMark_HelpDisplay`) confirm the tool -binary loads and responds to basic flags. All eight named tests are tracked by ReqStream to +binary loads and responds to basic flags. All ten named tests are tracked by ReqStream to confirm each functional aspect was exercised. ### Test Environment The standard CI pipeline environment with `.reviewmark.yaml` present at the repository root and the evidence store index accessible at its configured URL. The `ReviewMark_IndexScan`, -`ReviewMark_Enforce`, and `ReviewMark_Elaborate` tests require a fully populated `.reviewmark.yaml` -and the CI network access to the evidence store. +`ReviewMark_Enforce`, `ReviewMark_Elaborate`, and `ReviewMark_Lint` tests require a fully +populated `.reviewmark.yaml` and CI network access to the evidence store. ### Acceptance Criteria @@ -23,8 +23,10 @@ and the CI network access to the evidence store. - The review report is written to `docs/code_review_report/generated/report.md`. - `reviewmark --version` and `reviewmark --help` exit with code 0. - The requirements `TestResults-OTS-ReviewMark-Operational`, `TestResults-OTS-ReviewMark-PlanGeneration`, - and `TestResults-OTS-ReviewMark-ReportGeneration` are each linked to their named test identifiers - in the ReqStream trace matrix. + `TestResults-OTS-ReviewMark-IndexScan`, `TestResults-OTS-ReviewMark-WorkingDirectoryOverride`, + `TestResults-OTS-ReviewMark-Enforcement`, `TestResults-OTS-ReviewMark-Elaboration`, + `TestResults-OTS-ReviewMark-Lint`, and `TestResults-OTS-ReviewMark-ReportGeneration` are each + linked to their named test identifiers in the ReqStream trace matrix. ### Test Scenarios @@ -52,3 +54,7 @@ covered by a review-set. This scenario is confirmed by `ReviewMark_Enforce`. **Review elaboration**: ReviewMark shall expand a named review-set into a review checklist when invoked with `--elaborate`. This scenario is confirmed by `ReviewMark_Elaborate`. + +**Configuration validation**: ReviewMark shall validate the `.reviewmark.yaml` configuration +when invoked with `--lint` and exit with code 0 when the configuration is valid. This scenario +is confirmed by `ReviewMark_Lint`. diff --git a/docs/verification/ots/sarifmark.md b/docs/verification/ots/sarifmark.md index 950b44e..658d9f9 100644 --- a/docs/verification/ots/sarifmark.md +++ b/docs/verification/ots/sarifmark.md @@ -19,8 +19,12 @@ without a SARIF file produced by an actual CodeQL scan. - The CI pipeline step invoking SarifMark exits with code 0. - The markdown code quality document is produced at the expected output path under `docs/code_quality/generated/`. -- The requirement `TestResults-OTS-SarifMark` is linked to all three named test identifiers in - the ReqStream trace matrix. +- SarifMark exits non-zero when the configured finding threshold is exceeded, confirmed by + `SarifMark_Enforcement`. +- The requirement `TestResults-OTS-SarifMark` is linked to `SarifMark_SarifReading` and + `SarifMark_MarkdownReportGeneration` in the ReqStream trace matrix. +- The requirement `TestResults-OTS-SarifMark-Enforcement` is linked to `SarifMark_Enforcement` + in the ReqStream trace matrix. ### Test Scenarios diff --git a/docs/verification/ots/weasyprint.md b/docs/verification/ots/weasyprint.md index 27fb3a9..7c1b36c 100644 --- a/docs/verification/ots/weasyprint.md +++ b/docs/verification/ots/weasyprint.md @@ -5,7 +5,7 @@ WeasyPrint is verified through document output validation using FileAssert. After WeasyPrint converts each HTML document to PDF, FileAssert asserts that the PDF file exists, contains at least one page, has the expected metadata fields (Title, Author, Subject), and includes the -expected text content. Passing all six PDF assertions proves WeasyPrint executed correctly +expected text content. Passing all eight PDF assertions proves WeasyPrint executed correctly across all document types. The tests are named in the OTS requirements and tracked by ReqStream. ### Test Environment @@ -16,10 +16,10 @@ are required. ### Acceptance Criteria -- All six FileAssert PDF assertions pass without error. +- All eight FileAssert PDF assertions pass without error. - Each generated PDF file exists at the expected path under `docs/generated/`. - Each PDF contains at least one page and the expected metadata fields. -- The requirement `TestResults-OTS-WeasyPrint` is linked to all six named test identifiers in +- The requirement `TestResults-OTS-WeasyPrint` is linked to all eight named test identifiers in the ReqStream trace matrix. ### Test Scenarios @@ -41,3 +41,9 @@ This scenario is confirmed by `WeasyPrint_DesignPdf`. **User guide PDF generation**: WeasyPrint shall produce a valid PDF from the user guide HTML. This scenario is confirmed by `WeasyPrint_UserGuidePdf`. + +**Requirements PDF generation**: WeasyPrint shall produce a valid PDF from the requirements HTML. +This scenario is confirmed by `WeasyPrint_RequirementsPdf`. + +**Trace matrix PDF generation**: WeasyPrint shall produce a valid PDF from the trace matrix HTML. +This scenario is confirmed by `WeasyPrint_TraceMatrixPdf`. diff --git a/docs/verification/test-results-library.md b/docs/verification/test-results-library.md index 1948ad9..b952ce2 100644 --- a/docs/verification/test-results-library.md +++ b/docs/verification/test-results-library.md @@ -9,7 +9,8 @@ tests exercise the real in-memory model and the real serializer implementations TRX and JUnit XML payloads; no mocks or stubs are introduced at the system boundary because the library has no external service dependencies. The system scenarios provide direct coverage for requirements `TestResultsLibrary-Model-FormatIndependence`, -`TestResultsLibrary-IO-ReadResults`, and `TestResultsLibrary-IO-WriteResults`. +`TestResultsLibrary-IO-ReadResults`, `TestResultsLibrary-IO-WriteResults`, and +`TestResultsLibrary-IO-IdentifyFormat`. ## Test Environment @@ -26,6 +27,7 @@ and embedded example files included in the repository. storage formats. - The library reads supported TRX and JUnit inputs into `TestResults` objects. - The library writes `TestResults` objects to both TRX and JUnit XML outputs. +- The library identifies TRX, JUnit, and unknown content formats without deserializing. ## Test Scenarios @@ -50,3 +52,9 @@ satisfying `TestResultsLibrary-IO-WriteResults`. This scenario is tested by **Write JUnit output from the model**: The system shall serialize an in-memory run to JUnit XML, satisfying `TestResultsLibrary-IO-WriteResults`. This scenario is tested by `TestResultsLibrary_Serialize_InMemoryModel_ProducesJUnitContent`. + +**Identify test result file format**: The system shall determine the format of test result +content without deserializing it, satisfying `TestResultsLibrary-IO-IdentifyFormat`. This +scenario is tested by `TestResultsLibrary_Identify_TrxContent_ReturnsTrx`, +`TestResultsLibrary_Identify_JUnitContent_ReturnsJUnit`, and +`TestResultsLibrary_Identify_UnknownContent_ReturnsUnknown`. diff --git a/docs/verification/test-results-library/test-result.md b/docs/verification/test-results-library/test-result.md index 16a37b1..c05c377 100644 --- a/docs/verification/test-results-library/test-result.md +++ b/docs/verification/test-results-library/test-result.md @@ -6,7 +6,8 @@ The `TestResult` unit is verified with xUnit v3 tests in `test/DemaConsulting.TestResults.Tests/TestResultTests.cs`. Verification uses direct construction and mutation of the real `TestResult` class, with no mocked or injected dependencies because the unit is a data container. The tests provide direct evidence for -requirements `TestResults-Model-TestOutput` and `TestResults-Model-ErrorInfo`, and they also +requirements `TestResults-Model-SystemOutput`, `TestResults-Model-SystemError`, +`TestResults-Model-ErrorMessage`, and `TestResults-Model-ErrorStackTrace`, and they also verify supporting design behaviors such as generated identifiers, default metadata values, and the default `NotExecuted` outcome used by the serializers. @@ -41,13 +42,14 @@ record. This scenario is tested by **Captured output streams**: The unit shall support empty-by-default standard output and standard error properties and preserve assigned content, satisfying -`TestResults-Model-TestOutput`. This scenario is tested by -`TestResult_SystemOutput_Default_IsEmpty`, `TestResult_SystemOutput_Set_RetainsValue`, +`TestResults-Model-SystemOutput` and `TestResults-Model-SystemError`. This scenario is tested +by `TestResult_SystemOutput_Default_IsEmpty`, `TestResult_SystemOutput_Set_RetainsValue`, `TestResult_SystemError_Default_IsEmpty`, and `TestResult_SystemError_Set_RetainsValue`. **Captured error details**: The unit shall support empty-by-default error fields and preserve -assigned error text, satisfying `TestResults-Model-ErrorInfo`. This scenario is tested by +assigned error text, satisfying `TestResults-Model-ErrorMessage` and +`TestResults-Model-ErrorStackTrace`. This scenario is tested by `TestResult_ErrorMessage_Default_IsEmpty`, `TestResult_ErrorMessage_Set_RetainsValue`, `TestResult_ErrorStackTrace_Default_IsEmpty`, and `TestResult_ErrorStackTrace_Set_RetainsValue`. diff --git a/docs/verification/test-results-library/test-results.md b/docs/verification/test-results-library/test-results.md index 0b8f2d5..339e4ae 100644 --- a/docs/verification/test-results-library/test-results.md +++ b/docs/verification/test-results-library/test-results.md @@ -6,8 +6,9 @@ The `TestResults` unit is verified with xUnit v3 tests in `test/DemaConsulting.TestResults.Tests/TestResultsTests.cs`. Verification uses direct construction of the real `TestResults` class because the unit is a simple collection and metadata container with no external dependencies to mock or stub. The tests provide direct -evidence for requirements `TestResults-Model-RunIdentity`, `TestResults-Model-RunNaming`, -and `TestResults-Model-RunCollection` by checking run-level identity, +evidence for requirements `TestResults-Model-RunIdentity`, `TestResults-Model-RunName`, +`TestResults-Model-RunUserName`, `TestResults-Model-RunCollectionNonNull`, and +`TestResults-Model-RunCollectionEmpty` by checking run-level identity, metadata defaults, and collection initialization. ### Test Environment @@ -29,12 +30,21 @@ test run can be serialized without additional caller setup, satisfying `TestResults-Model-RunIdentity`. This scenario is tested by `TestResults_Id_Default_IsNotEmpty` and `TestResults_Id_TwoInstances_AreUnique`. -**Run metadata defaults**: A new `TestResults` instance shall initialize `Name` and -`UserName` to empty strings so the model is safe to use before enrichment, satisfying -`TestResults-Model-RunNaming`. This scenario is tested by -`TestResults_Name_Default_IsEmpty` and `TestResults_UserName_Default_IsEmpty`. +**Run name default**: A new `TestResults` instance shall initialize `Name` to an empty string +so the model is safe to use before enrichment, satisfying `TestResults-Model-RunName`. This +scenario is tested by `TestResults_Name_Default_IsEmpty`. -**Results collection initialization**: A new `TestResults` instance shall expose a non-null, -empty `Results` list ready for callers to populate, satisfying -`TestResults-Model-RunCollection`. This scenario is tested by -`TestResults_Results_Default_IsNotNull` and `TestResults_Results_Default_IsEmpty`. +**Run user name default**: A new `TestResults` instance shall initialize `UserName` to an +empty string so the model is safe to use before enrichment, satisfying +`TestResults-Model-RunUserName`. This scenario is tested by +`TestResults_UserName_Default_IsEmpty`. + +**Results collection non-null initialization**: A new `TestResults` instance shall expose a +non-null `Results` list ready for callers to populate, satisfying +`TestResults-Model-RunCollectionNonNull`. This scenario is tested by +`TestResults_Results_Default_IsNotNull`. + +**Results collection empty initialization**: A new `TestResults` instance shall expose an +empty `Results` list so callers can build a result set from scratch, satisfying +`TestResults-Model-RunCollectionEmpty`. This scenario is tested by +`TestResults_Results_Default_IsEmpty`. diff --git a/test/DemaConsulting.TestResults.Tests/IO/SerializerTests.cs b/test/DemaConsulting.TestResults.Tests/IO/SerializerTests.cs index 7c3d719..a57243e 100644 --- a/test/DemaConsulting.TestResults.Tests/IO/SerializerTests.cs +++ b/test/DemaConsulting.TestResults.Tests/IO/SerializerTests.cs @@ -478,8 +478,8 @@ public void Serializer_Identify_TestRunInWrongNamespace_ReturnsUnknown() /// Proves that the Serializer unit correctly dispatches to TrxSerializer and JUnitSerializer /// so that test count, names, class names, and outcomes survive a serialize-deserialize- /// serialize-deserialize chain through both formats. This exercises - /// <c>TestResults-Serializer-FormatConversion</c> and <c>TestResults-Serializer-RoundTrip</c> - /// at the unit level without relying on the IO subsystem test. + /// <c>TestResults-Serializer-TrxDeserialization</c>, <c>TestResults-Serializer-JUnitDeserialization</c>, + /// and <c>TestResults-IO-RoundTrip</c> at the unit level. /// </remarks> [Fact] public void Serializer_RoundTrip_TrxToJUnit_PreservesCoreTestData() From d8ea586480711fb1efb93f92b262cf6d19aec198 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon <Malcolm.Nixon@hiarc.inc> Date: Fri, 19 Jun 2026 13:22:45 -0400 Subject: [PATCH 07/10] Fix lint issues in docs/design/ots.md - Wrap four bullet-list lines exceeding 120-character limit - Fix British spelling 'organisation' -> 'organization' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/design/ots.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/design/ots.md b/docs/design/ots.md index e622a59..0e625cc 100644 --- a/docs/design/ots.md +++ b/docs/design/ots.md @@ -9,16 +9,16 @@ detailed integration design for each is in the corresponding `docs/design/ots/` OTS items are selected based on the following criteria: -- **License compatibility**: only items published under a permissive open-source license (MIT, Apache 2.0, or equivalent) are - accepted, ensuring compliance with the project's distribution requirements. -- **Ecosystem alignment**: items must be available as a `dotnet` tool or NuGet package so they integrate naturally with the - .NET SDK toolchain and do not require separate runtime environments. +- **License compatibility**: only items published under a permissive open-source license (MIT, + Apache 2.0, or equivalent) are accepted, ensuring compliance with the project's distribution requirements. +- **Ecosystem alignment**: items must be available as a `dotnet` tool or NuGet package so they integrate naturally + with the .NET SDK toolchain and do not require separate runtime environments. - **Maturity and community support**: items should have an active release history, publicly accessible source, and demonstrated use in comparable software projects. -- **Security track record**: no unresolved critical CVEs at the time of adoption; items from the DemaConsulting organisation - are under the same source-control and review processes as the host project. -- **Qualification feasibility**: the item must be verifiable through CI pipeline execution, document output assertions, or - library test exercise without requiring access to vendor internal test suites. +- **Security track record**: no unresolved critical CVEs at the time of adoption; items from the DemaConsulting + organization are under the same source-control and review processes as the host project. +- **Qualification feasibility**: the item must be verifiable through CI pipeline execution, document output + assertions, or library test exercise without requiring access to vendor internal test suites. ## Version Management Policy From 26cbc9723606386062a5cd1a26bce1ba417bd02c Mon Sep 17 00:00:00 2001 From: Malcolm Nixon <Malcolm.Nixon@hiarc.inc> Date: Fri, 19 Jun 2026 13:46:52 -0400 Subject: [PATCH 08/10] Comment out post-ReqStream WeasyPrint tests from requirement WeasyPrint_RequirementsPdf and WeasyPrint_TraceMatrixPdf run after ReqStream in the CI pipeline and cannot be used as test evidence for the requirement at publishing time (temporal paradox). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/reqstream/ots/weasyprint.yaml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/reqstream/ots/weasyprint.yaml b/docs/reqstream/ots/weasyprint.yaml index 724e85a..5dab94b 100644 --- a/docs/reqstream/ots/weasyprint.yaml +++ b/docs/reqstream/ots/weasyprint.yaml @@ -24,5 +24,9 @@ sections: - WeasyPrint_ReviewReportPdf - WeasyPrint_DesignPdf - WeasyPrint_UserGuidePdf - - WeasyPrint_RequirementsPdf - - WeasyPrint_TraceMatrixPdf + # NOTE: WeasyPrint_RequirementsPdf and WeasyPrint_TraceMatrixPdf cannot be listed + # here because they run AFTER ReqStream in the CI pipeline. ReqStream publishes + # the requirements document and trace matrix first, then those PDFs are generated + # from the published HTML — a temporal dependency that prevents their test evidence + # from being available at requirements-publishing time. Adding them here causes the + # requirement to appear unsatisfied (temporal paradox). From d50d35c691ea797992b91fb4f2ef3d3eb0f1921d Mon Sep 17 00:00:00 2001 From: Malcolm Nixon <Malcolm.Nixon@hiarc.inc> Date: Fri, 19 Jun 2026 13:59:47 -0400 Subject: [PATCH 09/10] Fix WeasyPrint verification doc counts and temporal paradox note Update stale 'eight' counts to reflect that only 6 tests are linked in the OTS requirement. RequirementsPdf and TraceMatrixPdf run after ReqStream and cannot be requirement evidence (temporal dependency). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/verification/ots/weasyprint.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/verification/ots/weasyprint.md b/docs/verification/ots/weasyprint.md index 7c1b36c..d27ea4a 100644 --- a/docs/verification/ots/weasyprint.md +++ b/docs/verification/ots/weasyprint.md @@ -6,7 +6,9 @@ WeasyPrint is verified through document output validation using FileAssert. Afte converts each HTML document to PDF, FileAssert asserts that the PDF file exists, contains at least one page, has the expected metadata fields (Title, Author, Subject), and includes the expected text content. Passing all eight PDF assertions proves WeasyPrint executed correctly -across all document types. The tests are named in the OTS requirements and tracked by ReqStream. +across all document types. Six of these tests run before ReqStream and are named in the OTS +requirements and tracked by ReqStream. The remaining two (Requirements and Trace Matrix PDFs) +run after ReqStream due to pipeline ordering and cannot be listed as requirement test evidence. ### Test Environment @@ -19,8 +21,10 @@ are required. - All eight FileAssert PDF assertions pass without error. - Each generated PDF file exists at the expected path under `docs/generated/`. - Each PDF contains at least one page and the expected metadata fields. -- The requirement `TestResults-OTS-WeasyPrint` is linked to all eight named test identifiers in - the ReqStream trace matrix. +- The requirement `TestResults-OTS-WeasyPrint` is linked to the six test identifiers that run + before ReqStream in the CI pipeline. `WeasyPrint_RequirementsPdf` and + `WeasyPrint_TraceMatrixPdf` are excluded from requirement tracing because they run after + ReqStream publishes the requirements document (temporal dependency). ### Test Scenarios From c2cc4ee1ff9c58bc1593c39aa874d000f9e33873 Mon Sep 17 00:00:00 2001 From: Malcolm Nixon <Malcolm.Nixon@hiarc.inc> Date: Fri, 19 Jun 2026 14:15:49 -0400 Subject: [PATCH 10/10] Fix Pandoc/WeasyPrint verification docs and reqstream evidence Corrects the documentation to reflect that FileAssert validates all 9 Pandoc HTML and 9 WeasyPrint PDF outputs serving two distinct purposes: build guards (all 9) and requirements evidence (7 pre-ReqStream only). - pandoc.yaml: add Pandoc_VerificationHtml to TestResults-OTS-Pandoc; comment out RequirementsHtml/TraceMatrixHtml with temporal paradox note - weasyprint.yaml: add WeasyPrint_VerificationPdf (was missing) - verification/ots/pandoc.md: explain dual role, update counts (9/7/2), add scenarios for Verification, Requirements, and TraceMatrix HTML - verification/ots/weasyprint.md: explain dual role, update counts (9/7/2), add Verification PDF scenario, clarify Requirements/TraceMatrix as build guards excluded from requirement tracing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/reqstream/ots/pandoc.yaml | 7 ++++++ docs/reqstream/ots/weasyprint.yaml | 1 + docs/verification/ots/pandoc.md | 34 ++++++++++++++++++++++++----- docs/verification/ots/weasyprint.md | 29 +++++++++++++++++------- 4 files changed, 57 insertions(+), 14 deletions(-) diff --git a/docs/reqstream/ots/pandoc.yaml b/docs/reqstream/ots/pandoc.yaml index 725ca80..53db92a 100644 --- a/docs/reqstream/ots/pandoc.yaml +++ b/docs/reqstream/ots/pandoc.yaml @@ -23,7 +23,14 @@ sections: - Pandoc_ReviewPlanHtml - Pandoc_ReviewReportHtml - Pandoc_DesignHtml + - Pandoc_VerificationHtml - Pandoc_UserGuideHtml + # NOTE: Pandoc_RequirementsHtml and Pandoc_TraceMatrixHtml cannot be listed + # here because they run AFTER ReqStream in the CI pipeline. ReqStream publishes + # the requirements document and trace matrix first, then those HTML files are + # generated from the Markdown output — a temporal dependency that prevents their + # test evidence from being available at requirements-publishing time. Adding them + # here causes the requirement to appear unsatisfied (temporal paradox). - id: TestResults-OTS-Pandoc-Template title: Pandoc shall apply an HTML template to produce a standalone HTML document. diff --git a/docs/reqstream/ots/weasyprint.yaml b/docs/reqstream/ots/weasyprint.yaml index 5dab94b..cf640c1 100644 --- a/docs/reqstream/ots/weasyprint.yaml +++ b/docs/reqstream/ots/weasyprint.yaml @@ -23,6 +23,7 @@ sections: - WeasyPrint_ReviewPlanPdf - WeasyPrint_ReviewReportPdf - WeasyPrint_DesignPdf + - WeasyPrint_VerificationPdf - WeasyPrint_UserGuidePdf # NOTE: WeasyPrint_RequirementsPdf and WeasyPrint_TraceMatrixPdf cannot be listed # here because they run AFTER ReqStream in the CI pipeline. ReqStream publishes diff --git a/docs/verification/ots/pandoc.md b/docs/verification/ots/pandoc.md index 5eab35c..e759e0a 100644 --- a/docs/verification/ots/pandoc.md +++ b/docs/verification/ots/pandoc.md @@ -4,9 +4,16 @@ Pandoc is verified through document output validation using FileAssert. After Pandoc converts each Markdown document collection to HTML, FileAssert asserts that the output HTML file exists, -contains a valid `<title>` element, and includes the expected document content. Passing all -six HTML assertions proves Pandoc executed correctly across all document types produced by the -repository. The tests are named in the OTS requirements and tracked by ReqStream. +contains a valid `<title>` element, and includes the expected document content. + +FileAssert validates all nine Pandoc HTML outputs across the pipeline, serving two distinct +purposes. First, all nine assertions act as build guards: if any HTML file is missing, malformed, +or lacks expected content, the build fails immediately. Second, seven of the nine assertions +(Build Notes, Code Quality, Review Plan, Review Report, Design, Verification, and User Guide) +run before ReqStream and serve as requirements evidence for the OTS requirements. The remaining +two assertions (Requirements and Trace Matrix) run after ReqStream publishes the requirements +document, so they cannot be listed as requirement test evidence due to pipeline ordering +(temporal dependency). In addition to basic conversion, the build pipeline depends on three document-structure behaviors that Pandoc must support: template-driven output, table of contents generation, and numbered @@ -23,15 +30,17 @@ are required. ### Acceptance Criteria -- All six FileAssert HTML assertions pass without error. +- All nine FileAssert HTML assertions pass without error. - Each generated HTML file exists at the expected path under `docs/{collection}/generated/`. - Each HTML file contains a `<title>` element and the expected document title text. - The design and verification HTML files each contain a `<nav id="TOC">` element confirming table of contents generation. - The design and verification HTML files each contain at least one `<span class="header-section-number">` element confirming section numbering was applied. -- The requirement `TestResults-OTS-Pandoc` is linked to all six named test identifiers in the - ReqStream trace matrix. +- The requirement `TestResults-OTS-Pandoc` is linked to the seven test identifiers that run + before ReqStream in the CI pipeline. `Pandoc_RequirementsHtml` and `Pandoc_TraceMatrixHtml` + are excluded from requirement tracing because they run after ReqStream publishes the + requirements document (temporal dependency). - The requirements `TestResults-OTS-Pandoc-Template`, `TestResults-OTS-Pandoc-Toc`, and `TestResults-OTS-Pandoc-NumberSections` are each linked to the design and verification HTML test identifiers in the ReqStream trace matrix. @@ -53,9 +62,22 @@ report Markdown inputs. This scenario is confirmed by `Pandoc_ReviewReportHtml`. **Design HTML generation**: Pandoc shall produce a valid HTML document from the software design Markdown inputs. This scenario is confirmed by `Pandoc_DesignHtml`. +**Verification HTML generation**: Pandoc shall produce a valid HTML document from the +verification Markdown inputs. This scenario is confirmed by `Pandoc_VerificationHtml`. + **User guide HTML generation**: Pandoc shall produce a valid HTML document from the user guide Markdown inputs. This scenario is confirmed by `Pandoc_UserGuideHtml`. +**Requirements HTML generation**: Pandoc shall produce a valid HTML document from the generated +requirements Markdown. This assertion acts as a build guard confirming the requirements document +was produced correctly. It runs after ReqStream and is not included as requirement test evidence. +This scenario is confirmed by `Pandoc_RequirementsHtml`. + +**Trace matrix HTML generation**: Pandoc shall produce a valid HTML document from the generated +trace matrix Markdown. This assertion acts as a build guard confirming the trace matrix was +produced correctly. It runs after ReqStream and is not included as requirement test evidence. +This scenario is confirmed by `Pandoc_TraceMatrixHtml`. + **Template-driven output**: Pandoc shall apply the custom HTML template (`template.html`) and produce a standalone document with a `<head>` section including a `<title>` element. Without the template, Pandoc would emit an HTML fragment with no `<head>` or `<title>`. This scenario diff --git a/docs/verification/ots/weasyprint.md b/docs/verification/ots/weasyprint.md index d27ea4a..e8ef6c1 100644 --- a/docs/verification/ots/weasyprint.md +++ b/docs/verification/ots/weasyprint.md @@ -5,10 +5,16 @@ WeasyPrint is verified through document output validation using FileAssert. After WeasyPrint converts each HTML document to PDF, FileAssert asserts that the PDF file exists, contains at least one page, has the expected metadata fields (Title, Author, Subject), and includes the -expected text content. Passing all eight PDF assertions proves WeasyPrint executed correctly -across all document types. Six of these tests run before ReqStream and are named in the OTS -requirements and tracked by ReqStream. The remaining two (Requirements and Trace Matrix PDFs) -run after ReqStream due to pipeline ordering and cannot be listed as requirement test evidence. +expected text content. + +FileAssert validates all nine WeasyPrint PDF outputs across the pipeline, serving two distinct +purposes. First, all nine assertions act as build guards: if any PDF file is missing, malformed, +or lacks expected content, the build fails immediately. Second, seven of the nine assertions +(Build Notes, Code Quality, Review Plan, Review Report, Design, Verification, and User Guide) +run before ReqStream and serve as requirements evidence for the OTS requirements. The remaining +two assertions (Requirements and Trace Matrix) run after ReqStream publishes the requirements +document, so they cannot be listed as requirement test evidence due to pipeline ordering +(temporal dependency). ### Test Environment @@ -18,10 +24,10 @@ are required. ### Acceptance Criteria -- All eight FileAssert PDF assertions pass without error. +- All nine FileAssert PDF assertions pass without error. - Each generated PDF file exists at the expected path under `docs/generated/`. - Each PDF contains at least one page and the expected metadata fields. -- The requirement `TestResults-OTS-WeasyPrint` is linked to the six test identifiers that run +- The requirement `TestResults-OTS-WeasyPrint` is linked to the seven test identifiers that run before ReqStream in the CI pipeline. `WeasyPrint_RequirementsPdf` and `WeasyPrint_TraceMatrixPdf` are excluded from requirement tracing because they run after ReqStream publishes the requirements document (temporal dependency). @@ -43,11 +49,18 @@ HTML. This scenario is confirmed by `WeasyPrint_ReviewReportPdf`. **Design PDF generation**: WeasyPrint shall produce a valid PDF from the software design HTML. This scenario is confirmed by `WeasyPrint_DesignPdf`. +**Verification PDF generation**: WeasyPrint shall produce a valid PDF from the verification +HTML. This scenario is confirmed by `WeasyPrint_VerificationPdf`. + **User guide PDF generation**: WeasyPrint shall produce a valid PDF from the user guide HTML. This scenario is confirmed by `WeasyPrint_UserGuidePdf`. **Requirements PDF generation**: WeasyPrint shall produce a valid PDF from the requirements HTML. -This scenario is confirmed by `WeasyPrint_RequirementsPdf`. +This assertion acts as a build guard confirming the requirements document was produced correctly. +It runs after ReqStream and is not included as requirement test evidence. This scenario is +confirmed by `WeasyPrint_RequirementsPdf`. **Trace matrix PDF generation**: WeasyPrint shall produce a valid PDF from the trace matrix HTML. -This scenario is confirmed by `WeasyPrint_TraceMatrixPdf`. +This assertion acts as a build guard confirming the trace matrix was produced correctly. It runs +after ReqStream and is not included as requirement test evidence. This scenario is confirmed by +`WeasyPrint_TraceMatrixPdf`.