Add --lint support for requirements YAML files

[Copilot]

ReqStream had no way to validate requirements file structure short of attempting a full processing run that aborts on the first error. This adds --lint to inspect all requirement files and report all issues in one pass.

Description

Core Linting (src/DemaConsulting.ReqStream/Linter.cs)

New Linter class using YamlDotNet's representation model (not the deserializer) to walk YAML trees with full position info:

• Detects: malformed YAML, non-mapping document roots (e.g. top-level sequences), unknown fields at document/section/requirement/mapping level, wrong YAML type for sequence fields (sections, mappings, requirements), missing required fields (title for sections; id+title for requirements; id for mappings), blank field values (id, title, test names, tag names), duplicate requirement IDs across all files
• Follows includes: directives recursively, deduplicating visited files using case-sensitive path comparison
• Reports all issues before exiting (never stops at first error)
• Error format: {file}({line},{col}): error: {description}
• Success: {firstFile}: No issues found

CLI Integration

• Context.cs: Added Lint bool property + --lint argument
• Program.cs: Lint runs as Priority 4 (after --validate, before requirements processing); added to help text

Self-Validation (src/DemaConsulting.ReqStream/Validation.cs)

Added RunLintTest — total self-validation tests: 5 → 6.

Tests

• LinterTests.cs (new): 29 unit tests covering all lint scenarios including blank-value validation (blank req id, req title, mapping id, test name in requirements and mappings, tag name), non-mapping YAML root, and blank mapping test name
• ProgramTests.cs: Updated Total Tests: 5 → Total Tests: 6
• ValidationTests.cs: Added assertion for ReqStream_Lint - Passed

Requirements

• docs/reqstream/unit-linter.yaml (new): 14 linting requirements covering all detection scenarios including blank-value and type-mismatch checks
• docs/reqstream/unit-program.yaml: Added ReqStream-Prog-Lint
• requirements.yaml: Includes unit-linter.yaml
• docs/reqstream/platform-requirements.yaml: Added ReqStream_Lint to all 6 platform requirements (Windows, Linux, macOS, .NET 8/9/10)

Review-Sets (.reviewmark.yaml)

• Added ReqStream-Linter review-set grouping docs/reqstream/unit-linter.yaml, docs/design/linter.md, src/**/Linter.cs, and test/**/LinterTests.cs
• Added docs/design/{unit}.md to each existing software unit review-set (Context, Program, Validation, Requirements, TraceMatrix)
• Added docs/design/**/*.md to needs-review patterns so all design documents require formal review coverage
• All 8 review-sets formally reviewed by the @code-review agent; actionable findings addressed

Design Documents (docs/design/)

New pandoc folder providing detailed design documentation for all software units, enabling complete formal reviews (requirements + design + source + tests):

• title.txt — YAML front matter for the design document PDF
• definition.yaml — Pandoc definition listing all unit design files as input
• introduction.md — Document introduction chapter (purpose, scope, conventions, references)
• program.md — Program.cs: version reflection, priority-dispatch table, error-handling contract, ProcessRequirements and enforcement algorithms
• context.md — Context.cs: CLI switch-based parser, all properties, ExpandGlobPattern, Write/WriteError, IDisposable pattern
• validation.md — Validation.cs: 6 self-validation tests, TemporaryDirectory and DirectorySwitch helpers, results file format dispatch
• requirements.md — Requirements.cs/Section.cs/Requirement.cs: data model, YAML intermediates, ReadFile algorithm, MergeSection, ValidateCycles DFS, export with tag filtering
• tracematrix.md — TraceMatrix.cs: TestMetrics/TestExecution records, source-specific test matching, coverage calculation, export
• linter.md — Linter.cs: known field sets, error format with position, non-mapping root detection, sequence-type-mismatch detection, cross-file deduplication, full linting call chain

Architecture, User Guide & README

• ARCHITECTURE.md: Added Linter to the components table and mermaid flowchart; updated execution flow to include --lint as step 5; added full Linter component description section.
• docs/guide/guide.md: Added --lint to the key features list and CLI options table; added a "Linting Requirements Files" subsection with usage examples, output examples, and a full table of detected issues; updated the self-validation report example to reflect 6 tests.
• README.md: Added 🔍 Linting to the features list; added --lint to the help output block; updated the self-validation example output to show 6 tests including ReqStream_Lint; added ReqStream_Lint description to the self-validation test list.

Usage

# Lint a requirements file and all its includes
reqstream --lint --requirements requirements.yaml

# Exit 0 with success message when clean
# requirements.yaml: No issues found

# Exit 1 with per-issue diagnostics when problems found
# docs/requirements/unit.yaml(42,5): error: Unknown field 'tittle' in requirement
# docs/requirements/unit.yaml(57,13): error: Duplicate requirement ID 'REQ-001' (first seen in docs/requirements/base.yaml)

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Code quality improvement

Related Issues

Pre-Submission Checklist

Before submitting this pull request, ensure you have completed the following:

Build and Test

• Code builds successfully: dotnet build --configuration Release
• All unit tests pass: dotnet test --configuration Release
• Self-validation tests pass:
  dotnet run --project src/DemaConsulting.ReqStream --configuration Release --framework net10.0
--no-build -- --validate
• Code produces zero warnings

Code Quality

• Code formatting is correct: dotnet format --verify-no-changes
• New code has appropriate XML documentation comments
• Static analyzer warnings have been addressed

Quality Checks

Please run the following checks before submitting:

• Spell checker passes: cspell "**/*.{md,cs}"
• Markdown linter passes: markdownlint "**/*.md"
• YAML linter passes: yamllint .

Testing

• Added unit tests for new functionality
• Updated existing tests if behavior changed
• All tests follow the AAA (Arrange, Act, Assert) pattern
• Test coverage is maintained or improved

Documentation

• Updated README.md (if applicable)
• Updated ARCHITECTURE.md (if applicable)
• Added code examples for new features (if applicable)
• Updated requirements.yaml (if applicable)

Additional Notes

ExpandGlobPattern (existing behavior) only handles relative glob patterns — absolute paths passed to --requirements are silently ignored. This is pre-existing and unrelated to this change; all tests use DirectorySwitch + relative paths to work around it, consistent with existing test patterns.

Several pre-existing OTS dependency gaps were identified during code review (YamlDotNet, TestResults, FileSystemGlobbing lack OTS requirement files). These are unrelated to this PR's scope and should be tracked as separate issues.

---

📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

[Malcolmnixon]

@copilot I don't think you updated any documentation, and I don't think you created a new review-set for the new software unit. Please read https://raw.githubusercontent.com/demaconsulting/ContinuousCompliance/refs/heads/main/docs/file-reviews.md and then update as appropriate. You may also find https://raw.githubusercontent.com/demaconsulting/ContinuousCompliance/refs/heads/main/docs/requirements.md useful for making sure the requirements are good.

[Copilot]

@copilot I don't think you updated any documentation, and I don't think you created a new review-set for the new software unit. Please read https://raw.githubusercontent.com/demacons...

Done in commit a727d95:

Review-set: Added ReqStream-Linter to .reviewmark.yaml grouping docs/reqstream/unit-linter.yaml, src/**/Linter.cs, and test/**/LinterTests.cs.

Documentation updates:

• ARCHITECTURE.md: Added Linter to the components table, mermaid flowchart, execution flow glance and priority order, plus a full Linter component description section.
• docs/guide/guide.md: Added --lint to the key features list and CLI options table; added a "Linting Requirements Files" subsection with usage examples, sample output, and a full table of detected issues; updated the validation report example to show 6 tests.

[Malcolmnixon]

@copilot So it does seem we're missing a docs/design pandoc folder containing:

• title.txt (standard document metadata)
• definition.yaml (pandoc definition file)
• introduction.md (design doc introduction)
• [software-items].md (files for each software item (unit and subsystem)

These should be written, and then added to the .reviewmark.yaml files for each review-set.

The goal would be that the design document contains the detailed design of ReqStream, and the reviews contain enough information (requirements, design, code, tests) for formal reviews.

[Copilot]

@copilot So it does seem we're missing a docs/design pandoc folder containing:

• title.txt (standard document metadata)
• definition.yaml (pandoc definition file)
  ...

Done in commit 8d2c56c. Created docs/design/ pandoc folder with:

• title.txt / definition.yaml / introduction.md — standard pandoc boilerplate
• program.md, context.md, validation.md, requirements.md, tracematrix.md, linter.md — one detailed design file per software unit

Each design doc covers the unit's responsibilities, data model, key algorithms, method contracts, and interactions with other units.

Also updated .reviewmark.yaml:

• Added docs/design/**/*.md to needs-review so all design docs require formal review coverage
• Added the unit-specific design doc to each software unit review-set (Context, Program, Validation, Requirements, TraceMatrix, Linter)

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://storage.googleapis.com/chromium-browser-snapshots/Linux_x64/1108766/chrome-linux.zip
  • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node node install.js (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[Malcolmnixon]

@copilot please use the @code-review agent to review all review-sets in the .reviewmark.yaml file. Run the reviews separately so the contexts dont collide.

[Malcolmnixon]

@copilot please continue with the last request

[Copilot]

@copilot please continue with the last request

All 8 review-sets reviewed separately and actionable findings addressed in commit 91047bc:

Reviews completed (all: Pass with Minor Issues, except OTS-Dependencies which flagged missing dependency docs):

• ReqStream-Context, ReqStream-Program, ReqStream-Validation, ReqStream-Requirements, ReqStream-TraceMatrix, ReqStream-Linter, Platform-Support, OTS-Dependencies

Issues fixed:

• platform-requirements.yaml: Added ReqStream_Lint to all 6 platform requirements (Windows, Linux, macOS, .NET 8/9/10)
• program.md: Fixed 3 inaccuracies — version query doesn't call PrintBanner, PrintBanner writes 3 lines not 1, EnforceRequirementsCoverage takes 2 params not 3
• validation.md: Added TemporaryDirectory section; corrected DirectorySwitch to only describe directory switching (not temp dir creation)
• requirements.md: Fixed _allRequirements type from Dictionary<string,string> to Dictionary<string,Requirement>
• tracematrix.md: Renamed LoadTestResultFile → ProcessTestResultFile; fixed GetTestResult signature (source filter is encoded in testName, not a separate parameter)
• ValidationTests.cs: Added missing assertion for ReqStream_Lint - Passed
• LinterTests.cs: Added 5 new tests for blank values (req id, req title, mapping id, test name, tag name)

The OTS-Dependencies review flagged that several NuGet packages (YamlDotNet, TestResults, FileSystemGlobbing) lack OTS requirement files, and that some existing OTS files reference test names from external tools' self-validation runs. These are pre-existing gaps unrelated to this PR's changes — I'd suggest tracking them as separate issues.

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all 6 review findings in commit bb9fa05:

• visitedFiles comparator (line 77): changed OrdinalIgnoreCase → Ordinal to match platform path case-sensitivity.
• Non-mapping root (ParseYaml): changed return type to YamlNode?; LintFile now emits "Document root must be a mapping" when the root is a sequence/scalar instead of silently passing.
• Wrong-type for sequence fields: added GetSequenceChecked helper; LintDocumentRoot and LintSection now emit "Field 'X' must be a sequence" when sections/mappings/requirements keys exist but hold non-sequence values.
• Missing mapping blank-test-name test: added Linter_Lint_WithBlankMappingTestName_ReportsError (and Linter_Lint_WithNonMappingRoot_ReportsError for the new root check).
• ReqStream-Lint-UnknownMappingField split: WithMappingMissingId_ReportsError moved to new ReqStream-Lint-MissingMappingId; blank mapping-id test also linked there.
• Blank-value requirements added: ReqStream-Lint-MissingRequirementFields extended for blank id/title; new ReqStream-Lint-BlankTestName (req + mapping) and ReqStream-Lint-BlankTagName requirements.

All 167 tests pass; all linters clean; CodeQL: 0 alerts.

[Malcolmnixon]

@copilot did you remember to update the README

[Copilot]

@copilot did you remember to update the README

Updated in commit 0ebedf5:

• Added 🔍 Linting to the Features list
• Added --lint to the help output block
• Updated the self-validation example output to show 6 tests (added ReqStream_Lint - Passed)
• Added ReqStream_Lint description to the self-validation test list

All linters pass.