diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 2b88433..4ce3e6d 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -662,6 +662,16 @@ jobs: --metadata date="$(date +'%Y-%m-%d')" --output docs/code_review_report/report.html + - name: Generate Design HTML with Pandoc + shell: bash + run: > + dotnet pandoc + --defaults docs/design/definition.yaml + --filter node_modules/.bin/mermaid-filter.cmd + --metadata version="${{ inputs.version }}" + --metadata date="$(date +'%Y-%m-%d')" + --output docs/design/design.html + # === GENERATE PDF DOCUMENTS WITH WEASYPRINT === # This section converts HTML documents to PDF using Weasyprint. # Downstream projects: Add any additional Weasyprint PDF generation steps here. @@ -715,6 +725,13 @@ jobs: docs/code_review_report/report.html "docs/ReqStream Review Report.pdf" + - name: Generate Design PDF with Weasyprint + run: > + dotnet weasyprint + --pdf-variant pdf/a-3u + docs/design/design.html + "docs/ReqStream Design.pdf" + # === UPLOAD ARTIFACTS === # This section uploads all generated documentation artifacts. # Downstream projects: Add any additional artifact uploads here. diff --git a/.reviewmark.yaml b/.reviewmark.yaml index 79c746b..ffac09b 100644 --- a/.reviewmark.yaml +++ b/.reviewmark.yaml @@ -34,6 +34,14 @@ reviews: - "docs/design/system.md" - "test/**/IntegrationTests.cs" + # Design review — system-level requirements traced against the complete design + - id: ReqStream-Design + title: Review of ReqStream Design Against Requirements + paths: + - "docs/reqstream/reqstream-system.yaml" + - "docs/reqstream/platform-requirements.yaml" + - "docs/design/**/*.md" + # Software unit reviews - one per class - id: ReqStream-Context title: Review of ReqStream Context Unit diff --git a/docs/design/context.md b/docs/design/context.md index 89e3cb8..4356bdf 100644 --- a/docs/design/context.md +++ b/docs/design/context.md @@ -43,44 +43,23 @@ when the enclosing `using` block in `Program.Main` exits. ### `Create(args)` -`Create` is the static factory method that constructs and returns a fully initialized `Context`. -It implements a sequential switch-based parser over the `args` array. - -**Parse loop**: - -1. Iterate `args` with an index variable `i`. -2. Match `args[i]` against known flags using a `switch` statement. -3. For flags that consume the next element (e.g., `--requirements`), check `i + 1 >= args.Length` - before advancing; if the check fails an `ArgumentException` is thrown. -4. An unrecognized argument causes an `ArgumentException` listing the unknown argument. - -**`--filter` handling**: - -The value following `--filter` is split on `','`. Each non-empty token is added to `FilterTags`. -If `FilterTags` is `null` at the point the first `--filter` is encountered, the `HashSet` is -created before adding tokens. Multiple `--filter` arguments are accumulated into the same set. - -**`--requirements` and `--tests` handling**: - -Each value is passed to `ExpandGlobPattern`; the resulting paths are appended to -`RequirementsFiles` or `TestFiles` respectively. - -**Log file**: - -If `--log` was specified, `Create` opens the named file for writing and assigns the resulting -`StreamWriter` to `_logWriter` before returning. +`Create` is the static factory method that constructs and returns a fully initialized `Context`. It +implements a sequential switch-based parser over the `args` array. Each recognized flag sets the +corresponding property; flags that consume the next element (e.g., `--requirements`) advance the +index by one additional step. An unrecognized argument or a missing value for a flag that requires +one causes an `ArgumentException`, which surfaces to the caller as a user-actionable error message +rather than an unhandled exception. + +`--filter` values are split on `','` and accumulated into `FilterTags`; multiple `--filter` +arguments merge into the same set. `--requirements` and `--tests` values are passed to +`ExpandGlobPattern` and appended to the respective file lists. If `--log` is specified, the named +file is opened for writing and assigned to `_logWriter` before the method returns. ### `ExpandGlobPattern(pattern)` `ExpandGlobPattern` resolves a single pattern (which may contain `*` or `**` wildcards) to a list -of absolute file paths. - -**Implementation**: - -1. Construct a `Microsoft.Extensions.FileSystemGlobbing.Matcher`. -2. Add `pattern` as an include pattern. -3. Execute the matcher against `Directory.GetCurrentDirectory()`. -4. Return the matched absolute paths. +of absolute file paths using `Microsoft.Extensions.FileSystemGlobbing.Matcher` against the current +working directory. **Known limitation**: the `Matcher` library silently ignores patterns that are themselves absolute paths. Callers that pass absolute paths directly will receive an empty result set. This is an @@ -88,19 +67,14 @@ accepted limitation of the underlying library; users should use relative paths o ### `WriteLine(message)` -`WriteLine` writes a message to the output channel. - -1. If `Silent` is `false`, write to `Console.WriteLine`. -2. If `_logWriter` is not `null`, write to `_logWriter`. +`WriteLine` writes a message to the console (unless `Silent` is `true`) and to `_logWriter` if a +log file is open. ### `WriteError(message)` -`WriteError` records an error and writes it to the error channel. - -1. Set `_hasErrors = true`. -2. If `Silent` is `false`, set `Console.ForegroundColor` to red, write to `Console.Error`, then - restore the original foreground color. -3. If `_logWriter` is not `null`, write to `_logWriter`. +`WriteError` sets `_hasErrors = true`, writes the message to `Console.Error` in red (unless +`Silent` is `true`), and also writes it to `_logWriter` if a log file is open. Setting +`_hasErrors` ensures that `ExitCode` returns `1` after any error is reported. ### `Dispose()` diff --git a/docs/design/linter.md b/docs/design/linter.md index 401ac47..7b10875 100644 --- a/docs/design/linter.md +++ b/docs/design/linter.md @@ -45,165 +45,95 @@ causes the process to exit with code `1`. ### `Lint(context, files)` -`Lint` is the single public entry point. - -1. If `files` is empty, print `"No requirements files specified."` and return. -2. Initialize a `Dictionary seenIds` (maps requirement ID to source file path) and - a `HashSet visitedFiles` shared across all files in `files`. -3. For each path in `files`, accumulate the return value of `LintFile(context, path, seenIds, visitedFiles)` - into a local `issueCount`. -4. If the total issue count is zero, print `"{files[0]}: No issues found"` via `context.WriteLine`. +`Lint` is the single public entry point. If `files` is empty it prints a notice and returns. For +each file in `files` it calls `LintFile`, accumulating the returned issue count across all files. +After all files are processed, it prints `"{files[0]}: No issues found"` via `context.WriteLine` +only if the total issue count is zero. Accumulating all issues before deciding on the success +message ensures that a clean run produces exactly one affirmative line of output and that a run +with issues lists every problem without any misleading success message. ### `LintFile(context, path, seenIds, visitedFiles)` -`LintFile` lints a single YAML file and follows its `includes:` entries. - -1. Resolve `path` to its full absolute path (`fullPath`). On failure, emit an error and return `1`. -2. If `fullPath` is already in `visitedFiles`, return `0` immediately (deduplication). -3. Add `fullPath` to `visitedFiles`. -4. Verify the file exists; emit an error and return `1` if not. -5. Read the file text. On I/O failure, emit an error and return `1`. -6. Parse the text via `ParseYaml`. If a `YamlException` or `InvalidOperationException` is thrown - (malformed YAML), emit an error at the reported source position and return `1`. -7. If the parsed root is `null` (empty document), return `0` — empty files are valid. -8. If the root node is not a `YamlMappingNode`, emit a type-mismatch error and return `1`. -9. Delegate to `LintDocumentRoot(context, path, root, seenIds)`. -10. Delegate to `LintIncludes(context, fullPath, GetStringList(root, "includes"), seenIds, visitedFiles)` - to follow included files. -11. Return the accumulated issue count from steps 9–10. +`LintFile` lints a single YAML file and follows its `includes:` entries. Three design points +govern its behavior: + +- **Deduplication**: `path` is resolved to a full absolute path and checked against + `visitedFiles`. If already visited, the method returns `0` immediately. This mirrors the + deduplication in `ReadFile` and prevents the same file from being linted twice when it is + included from multiple parents. +- **Error-at-position reporting**: the file text is parsed via `YamlDotNet`'s representation model + rather than the deserializer, so every issue is emitted with the exact line and column of the + offending node. I/O errors, YAML parse exceptions, and structural type mismatches are all caught + and reported as positioned errors rather than unhandled exceptions. +- **Recursive includes**: after linting the document root via `LintDocumentRoot`, the method + delegates to `LintIncludes` to follow and lint any files listed in the `includes:` sequence, + accumulating their issue counts alongside the root document's counts. ### `LintIncludes(context, parentFullPath, includes, seenIds, visitedFiles)` -`LintIncludes` resolves and recursively lints all files listed in an `includes:` sequence. - -1. If `includes` is `null`, return `0`. -2. Derive `baseDirectory` from `parentFullPath`. -3. Filter out any blank entries (using `!string.IsNullOrWhiteSpace`). -4. For each non-blank include path, call `LintFile(context, Path.Combine(baseDirectory, include), - seenIds, visitedFiles)` and accumulate the returned issue count. -5. Return the total issue count. +`LintIncludes` resolves each path in the `includes:` sequence relative to the parent file's +directory and recursively lints it via `LintFile`. Blank entries are skipped. It returns the +accumulated issue count from all included files. ### `LintDocumentRoot(context, path, root, seenIds)` -`LintDocumentRoot` validates the top-level structure of a single YAML document. - -1. For each key in the root mapping, check that it is a member of `KnownDocumentFields`; if not, - emit an unknown-field error at the key's position. -2. Delegate to `LintDocumentSections(context, path, root, seenIds)`. -3. Delegate to `LintDocumentMappings(context, path, root)`. +`LintDocumentRoot` validates the top-level structure of a single YAML document. It checks every +key in the root mapping against `KnownDocumentFields`, emitting an unknown-field error for any +unrecognized key, then delegates to `LintDocumentSections` and `LintDocumentMappings` to validate +the document's content. ### `LintDocumentSections(context, path, root, seenIds)` -`LintDocumentSections` retrieves the `sections:` sequence from the document root and lints each -child. - -1. Call `GetSequenceChecked` for the `"sections"` key on `root`. If the key is absent, return `0`. - If it is present but not a sequence, emit a type-mismatch error and return `1`. -2. Iterate `sections.Children`. For each child: - - - If it is a `YamlMappingNode`, call `LintSection(context, path, child, seenIds)`. - - Otherwise emit a `"Section must be a mapping"` error. +`LintDocumentSections` validates that the `sections:` key, if present, is a sequence, and that +each element of that sequence is a mapping node. It delegates each valid element to `LintSection`. ### `LintDocumentMappings(context, path, root)` -`LintDocumentMappings` retrieves the `mappings:` sequence from the document root and lints each -child. - -1. Call `GetSequenceChecked` for the `"mappings"` key on `root`. If the key is absent, return `0`. - If it is present but not a sequence, emit a type-mismatch error and return `1`. -2. Iterate `mappings.Children`. For each child: - - - If it is a `YamlMappingNode`, call `LintMapping(context, path, child)`. - - Otherwise emit a `"Mapping must be a mapping node"` error. +`LintDocumentMappings` validates that the `mappings:` key, if present, is a sequence, and that +each element of that sequence is a mapping node. It delegates each valid element to `LintMapping`. ### `LintSection(context, path, section, seenIds)` -`LintSection` validates one section mapping node. The caller (`LintDocumentSections` or -`LintSectionChildren`) has already asserted that the node is a `YamlMappingNode`. - -1. For each key in `section`, check against `KnownSectionFields`; emit an unknown-field error for - any unknown key. -2. Check that `title` is present and non-blank via `GetScalar`; emit an error at the section start - if missing, or at the scalar start if blank. -3. Delegate to `LintSectionRequirements(context, path, section, seenIds)`. -4. Delegate to `LintSectionChildren(context, path, section, seenIds)`. +`LintSection` validates one section mapping node. It checks all keys against `KnownSectionFields`, +validates that `title` is present and non-blank, then delegates to `LintSectionRequirements` and +`LintSectionChildren` for the section's contents. ### `LintSectionRequirements(context, path, section, seenIds)` -`LintSectionRequirements` retrieves the `requirements:` sequence from a section and lints each -child. - -1. Call `GetSequenceChecked` for the `"requirements"` key on `section`. If the key is absent, - return `0`. If it is present but not a sequence, emit a type-mismatch error and return `1`. -2. Iterate `requirements.Children`. For each child: - - - If it is a `YamlMappingNode`, call `LintRequirement(context, path, child, seenIds)`. - - Otherwise emit a `"Requirement must be a mapping"` error. +`LintSectionRequirements` validates that the `requirements:` key, if present, is a sequence, and +that each element is a mapping node. It delegates each valid element to `LintRequirement`. ### `LintSectionChildren(context, path, section, seenIds)` -`LintSectionChildren` retrieves the `sections:` sequence from a section and lints each child -section recursively. - -1. Call `GetSequenceChecked` for the `"sections"` key on `section`. If the key is absent, return - `0`. If it is present but not a sequence, emit a type-mismatch error and return `1`. -2. Iterate `sections.Children`. For each child: - - - If it is a `YamlMappingNode`, call `LintSection(context, path, child, seenIds)` recursively. - - Otherwise emit a `"Section must be a mapping"` error. +`LintSectionChildren` validates that the `sections:` key within a section, if present, is a +sequence, and that each element is a mapping node. It delegates each valid element back to +`LintSection` for recursive validation. ### `LintRequirement(context, path, requirement, seenIds)` -`LintRequirement` validates one requirement mapping node. The caller (`LintSectionRequirements`) -has already asserted that the node is a `YamlMappingNode`. - -1. For each key in `requirement`, check against `KnownRequirementFields`; emit an unknown-field - error for any unknown key. -2. Call `LintRequirementId(context, path, requirement, seenIds, ref issueCount)` to validate and - register the `id` field; capture the returned ID string (or `null` on error). -3. Call `LintRequirementTitle(context, path, requirement, reqId)` to validate the `title` field. -4. If `tests:` is present, find blank entries using - `.OfType().Where(s => string.IsNullOrWhiteSpace(s.Value)).Select(s => s.Start)` - and emit a `"Test name cannot be blank"` error for each. -5. If `tags:` is present, apply the same method chain and emit a `"Tag name cannot be blank"` error - for each blank entry. +`LintRequirement` validates one requirement mapping node. It checks all keys against +`KnownRequirementFields`, then delegates ID validation to `LintRequirementId` and title validation +to `LintRequirementTitle`. It also checks `tests:` and `tags:` sequences for blank string entries, +emitting a positioned error for each one found. ### `LintRequirementId(context, path, requirement, seenIds, ref issueCount)` -`LintRequirementId` validates the `id` field of a requirement, checks for duplicates, and -registers the ID. - -1. Look up the `id` scalar via `GetScalar`. If absent, emit a `"Requirement missing required field - 'id'"` error (at the mapping start), increment `issueCount`, and return `null`. -2. If the scalar value is blank, emit a `"Requirement 'id' cannot be blank"` error (at the scalar - start), increment `issueCount`, and return `null`. -3. Check `seenIds` for the ID. If already present, emit a duplicate-ID error referencing the first - file, increment `issueCount`, and return `reqId` (the ID is still returned so downstream - validation can include it in error messages). -4. Register `seenIds[reqId] = path` and return the ID string. +`LintRequirementId` validates the `id` field and registers it in `seenIds` to detect +cross-file duplicates. If the `id` is absent or blank it emits an error and returns `null`. If the +ID is a duplicate, it emits an error referencing the first file but still returns the ID string so +that downstream validators (`LintRequirementTitle`) can include it in their error messages for +better context. ### `LintRequirementTitle(context, path, requirement, reqId)` -`LintRequirementTitle` validates the `title` field of a requirement. - -1. Look up the `title` scalar via `GetScalar`. If absent, emit an error whose description uses - `"requirement '{reqId}'"` when `reqId` is non-null, or `"requirement"` otherwise (at the - mapping start), and return `1`. -2. If the scalar value is blank, emit a `"Requirement 'title' cannot be blank"` error (at the - scalar start) and return `1`. -3. Return `0`. +`LintRequirementTitle` validates that the `title` field is present and non-blank. When `reqId` is +non-null it includes the ID in the error description, making the error actionable even when +the offending requirement is one of many in a large file. ### `LintMapping(context, path, mapping)` -`LintMapping` validates one mapping entry. The caller (`LintDocumentMappings`) has already -asserted that the node is a `YamlMappingNode`. - -1. For each key in `mapping`, check against `KnownMappingFields`; emit an unknown-field error for - any unknown key. -2. Check that `id` is present and non-blank; emit an error at the mapping start if missing, or at - the scalar start if blank. -3. If `tests:` is present, apply the same blank-entry method chain used in `LintRequirement` and - emit a `"Test name cannot be blank"` error for each blank entry. +`LintMapping` validates one mapping entry. It checks all keys against `KnownMappingFields`, +validates that `id` is present and non-blank, and checks any `tests:` sequence for blank entries. ## Issue Accumulation and No-Issues Message diff --git a/docs/design/program.md b/docs/design/program.md index f077877..910a054 100644 --- a/docs/design/program.md +++ b/docs/design/program.md @@ -42,12 +42,12 @@ never throws and never returns `null`. | -------------- | -------- | | `ArgumentException` | Message written to `Console.Error`; returns exit code `1` | | `InvalidOperationException` | Message written to `Console.Error`; returns exit code `1` | -| Any other exception | Message written to `Console.Error`; exception re-thrown for event-log capture | +| Any other exception | Message written to `Console.Error`; exception re-thrown | -`ArgumentException` originates in `Context.Create` during argument parsing. -`InvalidOperationException` originates in the execution layer (e.g., YAML validation failures, -test result parse errors). Unexpected exceptions are intentionally re-thrown so that the operating -system or process supervisor can capture the full stack trace. +`ArgumentException` is thrown by `Context.Create` for invalid arguments and is user-actionable. +`InvalidOperationException` signals a domain error (YAML validation, test-result parse failure); +its message is sufficient for diagnosis. All other exceptions are re-thrown so the operating +system or process supervisor captures the full stack trace for unexpected failures. ### `Run(context)` @@ -77,31 +77,22 @@ argument, grouped logically. It is only called when `--help` is present. ### `ProcessRequirements` `ProcessRequirements` orchestrates the normal (non-version, non-help, non-validate, non-lint) run. -Its internal sequence is: - -1. Call `Requirements.Read(context.RequirementsFiles)` to build the parsed requirement tree. -2. If `context.RequirementsReport` is set, export the requirements report at - `context.ReportDepth`. -3. If `context.JustificationsFile` is set, export the justifications report at - `context.JustificationsDepth`. -4. If `context.TestFiles` is non-empty, construct a `TraceMatrix` from the requirements tree and - the test result files. -5. If `context.Matrix` is set and a trace matrix was constructed, export the matrix report at - `context.MatrixDepth`. -6. If `context.Enforce` is `true`, call `EnforceRequirementsCoverage(context, traceMatrix)`. - -All export methods respect `context.FilterTags` for tag-filtered output. +It begins by calling `Requirements.Read(context.RequirementsFiles)` to build the parsed requirement +tree. It then conditionally generates the requirements report (if `--report` is set) and the +justifications report (if `--justifications` is set). If `--tests` files are provided, a +`TraceMatrix` is constructed from the requirement tree and the test result files to enable coverage +analysis. If `--matrix` is set and a `TraceMatrix` was built, the trace matrix report is exported. +If `--enforce` is active, `EnforceRequirementsCoverage` is called last so that all reports are +generated even when coverage fails. All export methods respect `context.FilterTags` for tag-filtered +output. ### `EnforceRequirementsCoverage` -`EnforceRequirementsCoverage` evaluates whether all requirements are covered by passing tests. - -1. If no `TraceMatrix` was built (i.e., no `--tests` argument was provided), call - `context.WriteError` with a message indicating that enforcement requires test results; return. -2. Call `traceMatrix.CalculateSatisfiedRequirements(context.FilterTags)` to obtain satisfied and - total counts. -3. If `satisfied < total`, iterate all requirements that are unsatisfied and call - `context.WriteError` for each unsatisfied requirement ID. +`EnforceRequirementsCoverage` evaluates whether all requirements are covered by passing tests. If +no `TraceMatrix` was built (i.e., no `--tests` argument was provided), it reports an error +indicating that enforcement requires test results. Otherwise, it calls +`traceMatrix.CalculateSatisfiedRequirements(context.FilterTags)` to obtain satisfied and total +counts, and reports each unsatisfied requirement via `context.WriteError` if any are found. This method never throws; all failure signalling goes through `context.WriteError`, which sets the internal error flag and eventually produces a non-zero exit code. diff --git a/docs/design/requirements.md b/docs/design/requirements.md index 8a2b65e..f5a10d1 100644 --- a/docs/design/requirements.md +++ b/docs/design/requirements.md @@ -63,45 +63,37 @@ These intermediate types are discarded after `ReadFile` completes; the resulting ### `Requirements.Read(paths)` `Read` is the static factory method that constructs and returns a fully loaded `Requirements` -instance. - -1. Create a new `Requirements` with empty collections. -2. For each path in `paths`, call `ReadFile(path)`. -3. Call `ValidateCycles()` to detect circular child-requirement references. -4. Return the populated `Requirements` instance. +instance. It calls `ReadFile` for each supplied path to merge content into the tree, then calls +`ValidateCycles()` to confirm the child-requirement graph is acyclic before returning. ### `ReadFile(path)` -`ReadFile` loads a single YAML file and merges its content into the `Requirements` tree. - -1. Normalize `path` to an absolute path. -2. If the path is already in `_includedFiles`, return immediately (loop prevention). -3. Add the path to `_includedFiles`. -4. Read the file text and deserialize it into a `YamlDocument` using `YamlDotNet`. If the document - is empty or `null`, return silently. -5. Validate each section title (must not be blank) and each requirement ID and title (must not be - blank). Duplicate IDs are detected against `_allRequirements`; a duplicate causes an - `InvalidOperationException` with the file path and conflicting ID. -6. Call `MergeSection` for each top-level section in the document. -7. Apply each entry in the document's `mappings` block: find the matching `Requirement` by ID in - `_allRequirements` (skipping unknown IDs silently) and append the mapping's tests to - `Requirement.Tests`. -8. For each path in the document's `includes` block, resolve it relative to the current file's - directory and call `ReadFile` recursively. +`ReadFile` loads a single YAML file and merges its content into the `Requirements` tree. Four +design points govern its behavior: + +- **Deduplication**: `path` is normalized to an absolute path and checked against `_includedFiles` + before any work is done. If already present, the method returns immediately. This prevents + infinite loops when files include each other directly or transitively. +- **YAML deserialization**: the file text is deserialized into a `YamlDocument` using `YamlDotNet` + with `HyphenatedNamingConvention`. An empty or `null` document is silently accepted. +- **Validation and merging**: each section is validated (title must not be blank) and each + requirement is validated (ID and title must not be blank; ID must not duplicate an entry already + in `_allRequirements`). Validated sections are merged into the tree via `MergeSection`. Mapping + entries append additional test IDs to already-registered requirements. +- **Recursive includes**: each path in the document's `includes` block is resolved relative to the + current file's directory and passed to `ReadFile` recursively, enabling modular file + organization. ### `MergeSection(parent, yamlSection)` -`MergeSection` integrates a newly parsed section into an existing section tree. - -1. Search `parent.Sections` for an existing `Section` whose `Title` equals `yamlSection.Title`. -2. If a match is found: - - Append all requirements from `yamlSection` to the existing section's `Requirements` list. - - Recursively call `MergeSection` for each child section in `yamlSection`. -3. If no match is found: - - Create a new `Section` from `yamlSection` and append it to `parent.Sections`. +`MergeSection` integrates a newly parsed section into an existing section tree. If `parent.Sections` +already contains a section whose `Title` matches `yamlSection.Title`, the incoming requirements are +appended to that existing section and child sections are recursively merged. If no match is found, a +new `Section` is created and appended to `parent.Sections`. -This algorithm ensures that sections with the same title at the same hierarchy level are merged -across multiple files, enabling modular requirements management. +This same-title merge strategy is the key design decision that enables modular requirements +management: multiple YAML files can contribute requirements to the same logical section without +requiring a single monolithic file. ### `ValidateCycles()` diff --git a/docs/design/tracematrix.md b/docs/design/tracematrix.md index 80c1357..e664230 100644 --- a/docs/design/tracematrix.md +++ b/docs/design/tracematrix.md @@ -46,87 +46,56 @@ result file. ### `TraceMatrix(requirements, testResultFiles)` -The constructor builds the internal test-execution index: - -1. Store `requirements` for later iteration. -2. For each path in `testResultFiles`, call `ProcessTestResultFile(path)`. -3. After all files are loaded, `_testExecutions` contains every unique test name seen, each mapped - to a list of `TestExecution` records (one per file that contained that test name). +The constructor stores the `Requirements` tree for later iteration and calls +`ProcessTestResultFile` for each path in `testResultFiles` to populate `_testExecutions`. After +construction, `_testExecutions` contains every unique test name seen, each mapped to one +`TestExecution` record per result file that contained that test name. ### `ProcessTestResultFile(filePath)` -`ProcessTestResultFile` reads and parses one test-result file. - -1. Read the file text. -2. Call `DemaConsulting.TestResults.IO.Serializer.Deserialize(content)` to auto-detect the format - (TRX or JUnit) and parse the results. -3. If parsing fails, wrap the underlying exception in an `InvalidOperationException` that includes - `filePath` so the caller can identify the offending file. -4. For each test case in the deserialized result set, create a `TestExecution` with: - - `FileBaseName` = `Path.GetFileNameWithoutExtension(filePath)` - - `Name` = test case name - - `Metrics` = `TestMetrics(passes, fails)` derived from the test case outcome -5. Append the `TestExecution` to `_testExecutions[name]`, creating the list entry if absent. +`ProcessTestResultFile` reads one test-result file, auto-detects its format (TRX or JUnit) via +`DemaConsulting.TestResults.IO.Serializer.Deserialize`, and adds a `TestExecution` record to +`_testExecutions` for each test case found. If parsing fails, the underlying exception is wrapped +in an `InvalidOperationException` that includes `filePath` — this ensures the caller can identify +the offending file without inspecting nested exception detail. ## Methods ### `GetTestResult(testName)` -`GetTestResult` returns aggregated `TestMetrics` for a named test, with optional source filtering -encoded in the `testName` parameter itself. - -**Source-specific format** (`testName` contains `'@'` not at position 0 or end): - -1. Split `testName` on the first `'@'` to obtain `sourcePart` and `namePart`. -2. Look up `_testExecutions[namePart]`. -3. Filter the list to entries where `FileBaseName.Contains(sourcePart, OrdinalIgnoreCase)`. -4. Sum the `Metrics.Passes` and `Metrics.Fails` of the filtered entries. -5. Return `TestMetrics(totalPasses, totalFails)`. - -**Plain format** (`testName` does not contain a valid `'@'` separator): - -1. Look up `_testExecutions[testName]`. -2. Sum all `Metrics.Passes` and `Metrics.Fails` without source filtering. -3. Return `TestMetrics(totalPasses, totalFails)`. +`GetTestResult` returns aggregated `TestMetrics` for a named test. When `testName` contains a +`'@'` separator (not at position 0 or end), it applies source-specific filtering: the part before +`'@'` is matched case-insensitively against each `TestExecution.FileBaseName`, so only results +from files whose base name contains that prefix are summed. This lets a requirement reference a +test from a specific result file (e.g., `ubuntu@TestFeature_Valid_Passes`) without excluding that +test from plain-name lookups in other requirements. -If the test name is not found in `_testExecutions`, return `TestMetrics(0, 0)`. +When no `'@'` separator is present, all executions for the test name are summed across all result +files. If the test name is not found in `_testExecutions`, the method returns `TestMetrics(0, 0)`, +ensuring callers always receive a valid object. See the [Test Name Format Summary](#test-name-format-summary) +table for a quick reference of both formats. ### `CalculateSatisfiedRequirements(filterTags)` -`CalculateSatisfiedRequirements` iterates every requirement in the tree and returns a two-element -tuple `(satisfied, total)`. - -For each requirement (subject to `filterTags` filtering): - -1. Increment `total`. -2. Call `IsRequirementSatisfied(requirement)`. -3. If satisfied, increment `satisfied`. - -Returns `(satisfied, total)`. +`CalculateSatisfiedRequirements` iterates every requirement in the tree (subject to `filterTags` +filtering) and returns a `(satisfied, total)` tuple. It calls `IsRequirementSatisfied` for each +requirement to determine whether all associated tests have passed. This provides `Program` with the +counts needed to report coverage status and determine whether `--enforce` should fail. ### `CollectAllTests(requirement)` -`CollectAllTests` recursively collects every test name associated with a requirement and its -descendants. - -1. Add all entries from `requirement.Tests` to the result set. -2. For each ID in `requirement.Children`: - - Look up the child `Requirement` by ID. - - If found, recurse into `CollectAllTests(child)` and union the results. -3. Return the union set. - -Because `Requirements.ValidateCycles()` has already confirmed the child graph is acyclic, this -method recurses without a cycle guard. +`CollectAllTests` returns the union of all test names associated with a requirement and its +entire descendant subtree. Child requirements inherit their parent's coverage obligations, so a +requirement is only considered covered when all tests across its whole subtree pass. Because +`Requirements.ValidateCycles()` has already confirmed the child graph is acyclic, this method +recurses without a cycle guard. ### `IsRequirementSatisfied(requirement)` -`IsRequirementSatisfied` returns `true` if and only if the requirement has passing test coverage. - -1. Call `CollectAllTests(requirement)` to obtain the complete set of test names. -2. If the set is empty, return `false` (no tests mapped — requirement is unsatisfied). -3. For each test name, call `GetTestResult(testName)`. -4. If any result has `AllPassed == false`, return `false`. -5. Return `true`. +`IsRequirementSatisfied` returns `true` if and only if the requirement has at least one test +mapped (directly or via descendants) and every one of those tests has `AllPassed == true`. A +requirement with no tests is never satisfied, enforcing the design expectation that every +requirement must be traced to at least one passing test. ### `Export(filePath, depth, filterTags)` diff --git a/docs/design/validation.md b/docs/design/validation.md index e66826c..aa4f026 100644 --- a/docs/design/validation.md +++ b/docs/design/validation.md @@ -13,15 +13,17 @@ All tests run in temporary directories to avoid side effects and are isolated fr ### `Run(context)` -`Run` is the single public entry point. Its sequence is: - -1. Print a header block to `context` containing the tool version, machine name, operating system, - .NET runtime version, and current UTC timestamp. -2. Execute the six validation tests in order, collecting a `TestResult` for each. -3. Print a summary line showing the number of passed and failed tests. -4. If `context.ResultsFile` is set, call `WriteResultsFile(context, testResults)`. - -The six validation tests are listed in the order they are executed: +`Run` is the single public entry point. It prints a header block to `context` containing the tool +version, machine name, operating system, .NET runtime version, and current UTC timestamp. It then +executes the six validation tests in order and prints a multi-line summary block showing the total +number of tests, how many passed, and how many failed (using `WriteError` for the failed count when +any tests have failed). If `context.ResultsFile` is set, it calls `WriteResultsFile(context, testResults)` +to persist the results. +The six validation tests exist to provide structured, machine-readable evidence that ReqStream +correctly processes its own input formats. This evidence can be fed back into ReqStream to verify +the tool's own requirements coverage, enabling a self-hosting compliance workflow. + +The six tests are listed in the order they are executed: | # | Method | What it verifies | | - | ------ | ---------------- | @@ -32,14 +34,9 @@ The six validation tests are listed in the order they are executed: | 5 | `RunEnforcementModeTest` | `--enforce` produces a non-zero exit code when coverage fails | | 6 | `RunLintTest` | The linter detects and reports structural issues in YAML files | -Each test method: - -1. Creates a `TemporaryDirectory` for isolation and uses `DirectorySwitch` (see below) to operate - within it. -2. Writes one or more YAML or test-result fixture files to the temporary directory. -3. Invokes a `Program` method or builds a `Context` and executes the relevant workflow. -4. Asserts the expected outcomes (file content, exit code, error messages). -5. Returns a `TestResult` with outcome `Passed` or `Failed`. +Each test runs in a dedicated `TemporaryDirectory` with `DirectorySwitch` active, writes fixture +files, invokes the relevant workflow, asserts expected outcomes, and returns a `TestResult` with +outcome `Passed` or `Failed`. ### `WriteResultsFile(context, testResults)` @@ -59,32 +56,17 @@ The serializer is invoked with the assembled `TestResults` object and the resolv ### `TemporaryDirectory` (nested helper class) -`TemporaryDirectory` is an `IDisposable` helper that creates and manages the lifetime of a -temporary directory. - -**Construction**: - -1. Compose a unique path under `Path.GetTempPath()` using a GUID suffix. -2. Call `Directory.CreateDirectory` to create the directory. -3. Expose the path via the `DirectoryPath` property. - -**Disposal**: - -1. If the directory still exists, call `Directory.Delete` recursively to remove it and all - contents. +`TemporaryDirectory` is an `IDisposable` helper that creates a uniquely named directory under +`Path.GetTempPath()` on construction and deletes it recursively on disposal. It exists to give +each validation test a clean, isolated file-system workspace that is guaranteed to be removed after +the test completes, regardless of whether the test passes or fails. ### `DirectorySwitch` (nested helper class) -`DirectorySwitch` is an `IDisposable` helper that temporarily changes the process working directory. - -**Construction**: - -1. Capture `Directory.GetCurrentDirectory()` as the original directory. -2. Call `Directory.SetCurrentDirectory` to switch to the supplied `newDirectory`. - -**Disposal**: - -1. Call `Directory.SetCurrentDirectory` to restore the original directory. +`DirectorySwitch` is an `IDisposable` helper that changes the process working directory to a +supplied path on construction and restores the original directory on disposal. It exists because +ReqStream resolves relative paths against the working directory; tests must operate within their +temporary directory for file references to resolve correctly. Each test uses both classes together: `TemporaryDirectory` owns the directory lifetime and `DirectorySwitch` makes it the working directory for the duration of the test. This pattern