Paged documents: source and derived targets

claude.md

@@ -132,6 +132,13 @@ dotnet tool restore --tool-manifest src/.config/dotnet-tools.json
 
 **Builder Pattern**: `SettingsTask` provides fluent API with ~50 configuration methods that are lazily evaluated at verification time.
 
+**Source and derived targets**: A converter can say which of its targets is the document and which were computed from it (`ConversionResult(info, source, derived)`, or the `PagedConversion` builder for documents with pages, both in `src/Verify/Splitters/`):
+- `InnerVerifier.Adopt` (`InnerVerifier_Stream.cs`) is the only place a `ConversionToken` is created. It stamps the targets of a conversion, names them relative to the target that was converted, and marks the source, which is never converted again.
+- `VerifyEngine.HandleResults` decides every file name first (`Plan`), compares sources ahead of the rest (`CompareOrder`, so a differing source makes its derived targets bypass their comparers), and hands results on in planned order, which is the order callbacks and the exception message have always used.
+- `VerifyEngine.Report` runs once, after the delete/new/not-equal callbacks, and is the only caller of DiffEngine for pending files: deletes that stand alone, moves that stand alone (sources among them), moves derived from a pending source (`DiffRunner.LaunchDerived*`), then deletes derived from one. Test seams: `VerifyEngine.LaunchDiff` and `RaisedDeletes.AddDelete`.
+- The settings a paged converter reads (`PageText`, `PagesToInclude`, `ExcludeDerivedTargets`) follow the `ExcludeTargets` pattern: a static on `VerifierSettings`, a `Context` key on `VerifySettings`, a `SettingsTask` wrapper, and a reader on the converter's `context`.
+- User and plugin-author docs: `docs/paged-documents.md` and the "Source and derived targets" section of `docs/converter.md`.
+
 **Counter Pattern**: Deduplicates repeated values in filenames:
 - First occurrence: `Date`, second: `Date_1`, third: `Date_2`, etc.
 - Separate counters for DateTime, DateTimeOffset, Guid, etc.

docs/comparer.md

@@ -205,6 +205,8 @@ public static ConversionResult ConvertDocument(Stream document, IReadOnlyDiction
 
 The flag must be set on the source target, and that target must precede the derived targets in the conversion result.
 
+A converter that says which of its targets is the source, and which were derived from it, needs no flag. Verify compares the source first, and when it differs compares its derived targets exactly. That applies only to the targets derived from that source, where the flag applies to every target after it. See [Source and derived targets](/docs/converter.md#source-and-derived-targets).
+
 
 ## Default Comparison
 

docs/context.md

@@ -90,7 +90,7 @@ Values that are the same for every test do not need Context. A static field is s
 
 ## Reserved keys
 
-Verify uses the same dictionary for some per-verification state, under keys prefixed with `Verify.`. For example `ExcludeTargets` stores its extensions under `Verify.ExcludeTargets`, which is what allows a converter to call `context.IsTargetExcluded("png")`. Keys prefixed with `Verify.` should be treated as reserved.
+Verify uses the same dictionary for some per-verification state, under keys prefixed with `Verify.`. For example `ExcludeTargets` stores its extensions under `Verify.ExcludeTargets`, which is what allows a converter to call `context.IsTargetExcluded("png")`. The settings for [paged documents](/docs/paged-documents.md) are read the same way: `context.PageTextPlacement()`, `context.IsPageIncluded(1)` and `context.IsDerivedTargetExcluded("png")`. Keys prefixed with `Verify.` should be treated as reserved.
 
 
 ## Copy behavior

docs/converter.md

@@ -11,12 +11,14 @@ Converters are used to split a target into its component parts, then verify each
 
 When a target is split the result is:
 
- * An info file (containing the metadata of the target) serialized as json. File name: `{TestType}.{TestMethod}.info.verified.txt`
- * Zero or more documents of a specified extension. File name: `{TestType}.{TestMethod}.{Index}.verified.{Extension}`
+ * An info file (containing the metadata of the target) serialized as json. File name: `{TestType}.{TestMethod}.verified.txt`
+ * Zero or more targets of a specified extension. File name: `{TestType}.{TestMethod}#{Name}.verified.{Extension}` for a target the converter named. Targets with no name that share an extension are told apart by an index: `{TestType}.{TestMethod}#00.verified.{Extension}`.
 
 
 Converters are registered globally. The `context` parameter passed to a conversion carries per-test information. See [Context](/docs/context.md).
 
+A converter for a document with pages is best built on `PagedConversion`. See [Paged documents](/docs/paged-documents.md).
+
 
 ## Usage scenarios
 
@@ -229,6 +231,58 @@ return new(
 <!-- endSnippet -->
 
 
+## Source and derived targets
+
+Many converters return the document they were given, alongside what they computed from it: a csv for each sheet of a workbook, an image of each page of a pdf. A converter can say which is which, by passing the document as the `source` and the rest as `derived`:
+
+<!-- snippet: SourceAndDerivedTargets -->
+<a id='snippet-SourceAndDerivedTargets'></a>
+```cs
+// A converter of a workbook: the workbook is the source, and a csv of each sheet is derived
+static ConversionResult ConvertWorkbook(string? name, Stream stream, IReadOnlyDictionary<string, object> context)
+{
+    var workbook = Workbook.Load(stream);
+
+    var sheets = new List<Target>();
+    foreach (var sheet in workbook.Sheets)
+    {
+        // Named by what it is. The name of the target being converted is added by Verify
+        sheets.Add(new("csv", sheet.ToCsv(), sheet.Name));
+    }
+
+    Target? source = null;
+    if (!context.IsTargetExcluded("xlsx"))
+    {
+        source = new("xlsx", workbook.Save());
+    }
+
+    return new(
+        info: new
+        {
+            workbook.Author
+        },
+        source,
+        derived: sheets);
+}
+```
+<sup><a href='/src/Verify.Tests/Snippets/PagedDocumentSnippets.cs#L53-L82' title='Snippet source file'>snippet source</a> | <a href='#snippet-SourceAndDerivedTargets' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+Verify then does the following, so that no converter has to:
+
+ * **Names.** The targets are named relative to the target that was converted, so the `name` the converter is passed is not used. A sheet named `Sheet1` becomes `{TestType}.{TestMethod}#Sheet1.verified.csv`. Where the workbook is itself a target named `Attachment1`, the sheet becomes `#Attachment1.Sheet1` and the workbook takes `#Attachment1`. So does its info file when the workbook is a target passed to the verification. For a workbook found inside another converted document, the info is gathered into that document's info file.
+ * **No second conversion.** The source is not converted again, whatever its extension. A converter registered for both `xls` and `xlsx` can return an `xls` it was given as an `xlsx`.
+ * **Comparison.** The source is compared first. When it differs, its derived targets skip their registered [comparers](/docs/comparer.md#bypass-comparers-for-derived-targets) and are compared exactly. Only its own: a second document of the same verification is not affected.
+ * **Review.** The diff tool is told the derived files came from the source. [DiffEngineViewer](https://github.com/VerifyTests/DiffEngine/blob/main/docs/viewer.md#files-derived-from-a-document) shows a document it can draw as one row, with what was derived from it beneath, and accepts them together. Other diff tools are given each file as before. Verified files that the conversion no longer produces, such as a page a document has lost, are deleted along with the accept of the document.
+ * **Exclusion.** `ExcludeDerivedTargets` applies to the derived targets and to nothing else. See [Leaving out what was derived](/docs/paged-documents.md#leaving-out-what-was-derived).
+
+The info file counts as derived when it holds only what converters returned. With an `info` argument passed to the verification, or a [JsonAppender](/docs/jsonappender.md) in play, it holds something of the test's as well and stands alone.
+
+`source` is null where the document is not wanted as a target. The derived targets then stand alone as well, and are still named the same way.
+
+A target that is not the source can opt out of further conversion with `performConversion: false`.
+
+
 ## Excluding targets
 
 Some converters emit the source document (for example a `pdf`, `docx`, or `xlsx`) alongside the info file and the derived targets. That source document is then committed as a `.verified.{extension}` file. Where the document is large, or where its bytes cannot be made deterministic, it can be excluded from the snapshot. The info file and the derived targets continue to verify.
@@ -291,6 +345,8 @@ static ConversionResult ConvertExcludeCheck(string? name, Stream stream, IReadOn
 
 `IsTargetExcluded` reflects both the global and the per-verification `ExcludeTargets`, so shipping this check lets a caller opt out of the document build itself, rather than only its snapshot.
 
+`IsDerivedTargetExcluded` is the same check for a [derived target](#source-and-derived-targets). It reflects `ExcludeDerivedTargets` as well as `ExcludeTargets`.
+
 
 ## Shipping
 

docs/inline-snapshots.md

@@ -230,6 +230,8 @@ new Target("md", page1)
 
 The whole verification then falls back to files.
 
+The info file of a converter that [names its source](/docs/converter.md#source-and-derived-targets) is opted out the same way by Verify. It is a file of the document, reviewed and accepted together with the document and the files derived from it.
+
 
 ## Calling Verify through a wrapper
 

docs/mdsource/comparer.source.md

@@ -82,6 +82,8 @@ snippet: BypassComparersForSubsequentOnDifference
 
 The flag must be set on the source target, and that target must precede the derived targets in the conversion result.
 
+A converter that says which of its targets is the source, and which were derived from it, needs no flag. Verify compares the source first, and when it differs compares its derived targets exactly. That applies only to the targets derived from that source, where the flag applies to every target after it. See [Source and derived targets](/docs/converter.md#source-and-derived-targets).
+
 
 ## Default Comparison
 

docs/mdsource/context.source.md

@@ -28,7 +28,7 @@ Values that are the same for every test do not need Context. A static field is s
 
 ## Reserved keys
 
-Verify uses the same dictionary for some per-verification state, under keys prefixed with `Verify.`. For example `ExcludeTargets` stores its extensions under `Verify.ExcludeTargets`, which is what allows a converter to call `context.IsTargetExcluded("png")`. Keys prefixed with `Verify.` should be treated as reserved.
+Verify uses the same dictionary for some per-verification state, under keys prefixed with `Verify.`. For example `ExcludeTargets` stores its extensions under `Verify.ExcludeTargets`, which is what allows a converter to call `context.IsTargetExcluded("png")`. The settings for [paged documents](/docs/paged-documents.md) are read the same way: `context.PageTextPlacement()`, `context.IsPageIncluded(1)` and `context.IsDerivedTargetExcluded("png")`. Keys prefixed with `Verify.` should be treated as reserved.
 
 
 ## Copy behavior

docs/mdsource/converter.source.md

@@ -4,12 +4,14 @@ Converters are used to split a target into its component parts, then verify each
 
 When a target is split the result is:
 
- * An info file (containing the metadata of the target) serialized as json. File name: `{TestType}.{TestMethod}.info.verified.txt`
- * Zero or more documents of a specified extension. File name: `{TestType}.{TestMethod}.{Index}.verified.{Extension}`
+ * An info file (containing the metadata of the target) serialized as json. File name: `{TestType}.{TestMethod}.verified.txt`
+ * Zero or more targets of a specified extension. File name: `{TestType}.{TestMethod}#{Name}.verified.{Extension}` for a target the converter named. Targets with no name that share an extension are told apart by an index: `{TestType}.{TestMethod}#00.verified.{Extension}`.
 
 
 Converters are registered globally. The `context` parameter passed to a conversion carries per-test information. See [Context](/docs/context.md).
 
+A converter for a document with pages is best built on `PagedConversion`. See [Paged documents](/docs/paged-documents.md).
+
 
 ## Usage scenarios
 
@@ -77,6 +79,27 @@ If cleanup needs to occur after verification a callback can be passes to `Conver
 snippet: ConversionResultWithCleanup
 
 
+## Source and derived targets
+
+Many converters return the document they were given, alongside what they computed from it: a csv for each sheet of a workbook, an image of each page of a pdf. A converter can say which is which, by passing the document as the `source` and the rest as `derived`:
+
+snippet: SourceAndDerivedTargets
+
+Verify then does the following, so that no converter has to:
+
+ * **Names.** The targets are named relative to the target that was converted, so the `name` the converter is passed is not used. A sheet named `Sheet1` becomes `{TestType}.{TestMethod}#Sheet1.verified.csv`. Where the workbook is itself a target named `Attachment1`, the sheet becomes `#Attachment1.Sheet1` and the workbook takes `#Attachment1`. So does its info file when the workbook is a target passed to the verification. For a workbook found inside another converted document, the info is gathered into that document's info file.
+ * **No second conversion.** The source is not converted again, whatever its extension. A converter registered for both `xls` and `xlsx` can return an `xls` it was given as an `xlsx`.
+ * **Comparison.** The source is compared first. When it differs, its derived targets skip their registered [comparers](/docs/comparer.md#bypass-comparers-for-derived-targets) and are compared exactly. Only its own: a second document of the same verification is not affected.
+ * **Review.** The diff tool is told the derived files came from the source. [DiffEngineViewer](https://github.com/VerifyTests/DiffEngine/blob/main/docs/viewer.md#files-derived-from-a-document) shows a document it can draw as one row, with what was derived from it beneath, and accepts them together. Other diff tools are given each file as before. Verified files that the conversion no longer produces, such as a page a document has lost, are deleted along with the accept of the document.
+ * **Exclusion.** `ExcludeDerivedTargets` applies to the derived targets and to nothing else. See [Leaving out what was derived](/docs/paged-documents.md#leaving-out-what-was-derived).
+
+The info file counts as derived when it holds only what converters returned. With an `info` argument passed to the verification, or a [JsonAppender](/docs/jsonappender.md) in play, it holds something of the test's as well and stands alone.
+
+`source` is null where the document is not wanted as a target. The derived targets then stand alone as well, and are still named the same way.
+
+A target that is not the source can opt out of further conversion with `performConversion: false`.
+
+
 ## Excluding targets
 
 Some converters emit the source document (for example a `pdf`, `docx`, or `xlsx`) alongside the info file and the derived targets. That source document is then committed as a `.verified.{extension}` file. Where the document is large, or where its bytes cannot be made deterministic, it can be excluded from the snapshot. The info file and the derived targets continue to verify.
@@ -102,6 +125,8 @@ snippet: ConverterExcludeCheck
 
 `IsTargetExcluded` reflects both the global and the per-verification `ExcludeTargets`, so shipping this check lets a caller opt out of the document build itself, rather than only its snapshot.
 
+`IsDerivedTargetExcluded` is the same check for a [derived target](#source-and-derived-targets). It reflects `ExcludeDerivedTargets` as well as `ExcludeTargets`.
+
 
 ## Shipping
 

docs/mdsource/doc-index.include.md

@@ -42,6 +42,7 @@
   * [Kill process locking file](/docs/kill-process-locking-file.md)
   * [Comparers](/docs/comparer.md)
   * [Converters](/docs/converter.md)
+    * [Paged documents](/docs/paged-documents.md)
   * [Context](/docs/context.md)
   * [Recording](/docs/recording.md)
   * [Explicit Targets](/docs/explicit-targets.md)

docs/mdsource/inline-snapshots.source.md

@@ -124,6 +124,8 @@ new Target("md", page1)
 
 The whole verification then falls back to files.
 
+The info file of a converter that [names its source](/docs/converter.md#source-and-derived-targets) is opted out the same way by Verify. It is a file of the document, reviewed and accepted together with the document and the files derived from it.
+
 
 ## Calling Verify through a wrapper
 

docs/mdsource/naming.source.md

@@ -376,6 +376,25 @@ if (maps.TryGetVerified(receivedPath, out var verifiedPath))
 
 `ReceivedMaps.Pairs` enumerates every pair instead, for accepting a whole run at once.
 
+A file that a [converter](converter.md#source-and-derived-targets) derived from a document, such as the image of a page, has a third line while that document is itself pending: the received path of the document.
+
+```
+C:\code\MyProject\Tests\TheTest.TheMethod#page_0001.DotNet11_0.received.png
+C:\code\MyProject\Tests\TheTest.TheMethod#page_0001.verified.png
+C:\code\MyProject\Tests\TheTest.TheMethod.DotNet11_0.received.pdf
+```
+
+`ReceivedMaps.TryGetSource` reads it, so that a tool can present a document and what was derived from it as one change, and accept them together:
+
+```cs
+if (maps.TryGetSource(receivedPath, out var documentReceivedPath))
+{
+    // receivedPath was derived from the document at documentReceivedPath
+}
+```
+
+It answers false for a file that stands alone: one that was not derived from a document, the document itself, and a file whose document has since been accepted.
+
 It scans the directory recursively, so it can be pointed at a project or a repository root. `.git` and `node_modules` are skipped.
 
 Notes: