Release 0.21.0 - #349
Merged
Merged
Conversation
Add ExtractorBase<TSource>, LoaderBase<TDestination>, and TransformerBase<TSource, TDestination> that inherit the two/three-type-parameter bases with TProgress fixed to the built-in Report and a default CreateProgressReport() => new(CurrentItemCount, StartedAt, Elapsed). A component that doesn't need a custom progress type now implements ONLY its worker method — no progress record, no CreateProgressReport override. The default isn't sealed, so a component can still enrich the report. The existing bases are unchanged; purely additive. This is the lighter-weight alternative to the #96 source generator for the common boilerplate. Part B (lifting the timer-injection guard into the base) tracked separately in #344. - 4 ConvenienceBaseTests (extractor/loader/transformer smoke + report content + overridable default). 456 tests; 100% mutation; all 11 TFMs clean. - PublicAPI.Unshipped updated (RS0016 completeness + RS0017 correctness validated). - CHANGELOG [Unreleased] Added entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New second package built from this repo, versioned in lockstep with Wolfgang.Etl.Abstractions (same <Version>, one tag — like TestKit + TestKit.Xunit). Hosts the shared generic ItemErrorPolicy factory for the family's OnError hook so the policies are defined once instead of per format: - Skip / Abort (Func<ItemErrorContext, ItemErrorAction> properties) - SkipAndLog(ILogger) - SkipAndDeadLetter / SkipDeadLetterAndLog, each overloaded for a caller-owned ICollection<ItemErrorContext> or a ChannelWriter<ItemErrorContext> The core Abstractions assembly keeps its minimal deps; only this package takes Microsoft.Extensions.Logging.Abstractions (all TFMs) and System.Threading.Channels (netFx / netstandard2.0 only — in-box on net5.0+). Bumps Abstractions to 0.21.0 for the lockstep release; PackageValidation baseline stays at last-published 0.20.0. No release.yaml change needed — it already packs every src/ project, validates the tag against any src <Version>, and pushes all nupkgs. 14 unit tests pass net462 -> net10.0; full-matrix build clean; pack emits the correct per-TFM dependency groups (Abstractions 0.21.0 lockstep dep on every TFM). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convenience base classes fixing TProgress=Report (#344 Part A)
# Conflicts: # CHANGELOG.md
Makes ErrorPolicies' assign-a-policy design real. ExtractorBase / LoaderBase /
TransformerBase gain:
public Func<ItemErrorContext, ItemErrorAction> ErrorPolicy { get; init; }
non-null, defaulting to a fail-fast AbortPolicy, throw-on-null. The base
OnItemError now delegates to it, so `ErrorPolicy = ItemErrorPolicy.SkipAndLog(...)`
works on any stage with no per-type property or override — while overriding
OnItemError is still available for stage-internal logic. Supersedes the earlier
"no public base property" decision; the rationale comment is rewritten and the
resume caveat (Skip = swallow-and-stop on formats that can't resume) documented.
- ErrorPolicy on the two/three-type-parameter bases (convenience bases inherit it)
- PublicAPI.Unshipped entries; +6 base tests (default fail-fast, assigned policy is
used by the base OnItemError, null throws) across all three stages
- ItemErrorPolicy package docs + csproj description + CHANGELOG updated from the
override-and-invoke wording back to the assignable `ErrorPolicy = ...` form
Full-matrix build clean (0/0); Abstractions PackageValidation passes vs 0.20.0
(additive); Abstractions unit suite 463 green, ErrorPolicies 14 green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The channel dead-letter policies use non-blocking TryWrite because the OnItemError hook is synchronous. On a full bounded channel that drops the failure record. SkipDeadLetterAndLog(ChannelWriter, ILogger) now logs a distinct warning (EventId 2, ItemDeadLetterDropped) when the write is dropped, so the loss is never silent — throwing would convert an error-sink overflow into a full pipeline abort, which is worse. The logger-less SkipAndDeadLetter(ChannelWriter) overload keeps the documented silent drop (no logger to escalate to; caller controls overflow via BoundedChannelFullMode). Adds a bounded-channel-full test and CHANGELOG note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add Wolfgang.Etl.ErrorPolicies package (lockstep 0.21.0)
- CHANGELOG: [Unreleased] -> [0.21.0] - 2026-08-03 (convenience bases #344, assignable ErrorPolicy on base stages, new Wolfgang.Etl.ErrorPolicies package); fresh [Unreleased] scaffold. - PublicAPI: promote Unshipped -> Shipped for Wolfgang.Etl.Abstractions (convenience bases + ErrorPolicy) and Wolfgang.Etl.ErrorPolicies (8 entries). Version stays 0.21.0; baseline stays 0.20.0 for the release build (bumped to 0.21.0 post-release). Full Release build clean (0/0); both packages pack with PackageValidation passing vs 0.20.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
BenchmarkDotNet
Details
| Benchmark suite | Current: 7e34b02 | Previous: 5e5e958 | Ratio |
|---|---|---|---|
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 1000) |
33628.030100504555 ns (± 140.382034768086) |
31583.873189290363 ns (± 115.28689686427228) |
1.06 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 1000) |
35353.487854003906 ns (± 118.80215133501537) |
35160.96792602539 ns (± 104.53835631316898) |
1.01 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 100000) |
3261763.8424479165 ns (± 3195.930693519073) |
3107534.1640625 ns (± 17584.772051305485) |
1.05 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 100000) |
3509169.6341145835 ns (± 36119.37224833264) |
3448047.61328125 ns (± 4145.027564723044) |
1.02 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 1000) |
33587.276875813805 ns (± 126.47704481617426) |
29626.396423339844 ns (± 122.33354576748134) |
1.13 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 1000) |
32454.964721679688 ns (± 121.07504204063846) |
28745.347229003906 ns (± 56.763752733836895) |
1.13 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 1000) |
73267.29903157552 ns (± 309.7379453091387) |
69584.28100585938 ns (± 220.36328762445214) |
1.05 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 100000) |
3223357.76953125 ns (± 39177.578276698216) |
2835179.5182291665 ns (± 5152.038095929195) |
1.14 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 100000) |
3482333.3020833335 ns (± 6590.353358168164) |
2925295.2018229165 ns (± 3396.484964473814) |
1.19 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 100000) |
7023545.411458333 ns (± 19756.057860249108) |
6793926.755208333 ns (± 24027.364375768884) |
1.03 |
This comment was automatically generated by workflow using github-action-benchmark.
Contributor
There was a problem hiding this comment.
Pull request overview
Release PR for 0.21.0, merging additive API/features from vNext into main for the ETL abstractions library and introducing a companion error-policy package.
Changes:
- Added convenience stage base classes (
ExtractorBase<TSource>,LoaderBase<TDestination>,TransformerBase<TSource, TDestination>) that default progress reporting toReport. - Introduced a configurable per-item
ErrorPolicyhook on the three core stage base classes and updated tests accordingly. - Added new package
Wolfgang.Etl.ErrorPolicies(plus unit tests), solution wiring, and release documentation updates (CHANGELOG/PublicAPI/version).
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Wolfgang.Etl.ErrorPolicies.Tests.Unit/Wolfgang.Etl.ErrorPolicies.Tests.Unit.csproj | New unit test project for the ErrorPolicies package across multiple TFMs. |
| tests/Wolfgang.Etl.ErrorPolicies.Tests.Unit/ItemErrorPolicyTests.cs | Unit tests covering ItemErrorPolicy factory behaviors (skip/abort/log/dead-letter). |
| tests/Wolfgang.Etl.Abstractions.Tests.Unit/ItemErrorHandlingTests.cs | Tests for default and assigned ErrorPolicy behavior on base stages. |
| tests/Wolfgang.Etl.Abstractions.Tests.Unit/BaseClassTests/ConvenienceBaseTests.cs | Tests verifying the new convenience base classes’ default Report progress behavior. |
| src/Wolfgang.Etl.ErrorPolicies/Wolfgang.Etl.ErrorPolicies.csproj | New packaged library project for ready-made error policies. |
| src/Wolfgang.Etl.ErrorPolicies/PublicAPI.Unshipped.txt | Initializes API tracking for the new package. |
| src/Wolfgang.Etl.ErrorPolicies/PublicAPI.Shipped.txt | Declares shipped public surface for ItemErrorPolicy. |
| src/Wolfgang.Etl.ErrorPolicies/ItemErrorPolicyLog.cs | Cached LoggerMessage delegates for allocation-free warning logs. |
| src/Wolfgang.Etl.ErrorPolicies/ItemErrorPolicy.cs | Implements ready-made policies: Skip/Abort/SkipAndLog/DeadLetter variants. |
| src/Wolfgang.Etl.Abstractions/Wolfgang.Etl.Abstractions.csproj | Bumps package version to 0.21.0. |
| src/Wolfgang.Etl.Abstractions/TransformerBase{TSource,TDestination}.cs | Adds convenience transformer base fixing progress to Report. |
| src/Wolfgang.Etl.Abstractions/TransformerBase.cs | Adds ErrorPolicy property and delegates base OnItemError to it. |
| src/Wolfgang.Etl.Abstractions/PublicAPI.Shipped.txt | Updates shipped API surface for new bases and ErrorPolicy. |
| src/Wolfgang.Etl.Abstractions/LoaderBase{TDestination}.cs | Adds convenience loader base fixing progress to Report. |
| src/Wolfgang.Etl.Abstractions/LoaderBase.cs | Adds ErrorPolicy property and delegates base OnItemError to it. |
| src/Wolfgang.Etl.Abstractions/ExtractorBase{TSource}.cs | Adds convenience extractor base fixing progress to Report. |
| src/Wolfgang.Etl.Abstractions/ExtractorBase.cs | Adds ErrorPolicy property and delegates base OnItemError to it. |
| ETL-Abstractions.sln | Adds the new package + test project to the solution. |
| CHANGELOG.md | Adds 0.21.0 release notes. |
Per the design decision, ErrorPolicy stays init-only for 0.21.0 (settable PR #350 closed). Add a doc note on all three base stages that the policy is assigned once at construction and cannot change during a run, and correct the CHANGELOG wording from "settable" to "init-only". This satisfies the Copilot review's alternative ("document that immutability is intentional"). The broader move of the whole base config surface (MaximumItemCount / SkipItemCount / ReportingInterval + ErrorPolicy) to init-only is a coordinated pre-1.0 change tracked for a future release, here and downstream. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Copilot's init-only point — decision: This PR now documents the intent (a doc note on all three bases: "assigned once, at construction; cannot change during a run") and the CHANGELOG says init-only. Downstream audit (in #351) confirmed no production consumer sets these post-construction — the init-only flip is safe, with effort concentrated in the TestKit.Xunit contract tests. 0.21.0 content is unchanged in behaviour. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/Wolfgang.Etl.Abstractions/ExtractorBase.cs:498
ErrorPolicyuses aninitaccessor that assigns to_errorPolicy, but_errorPolicyis declaredreadonlyabove (line 482). Assigning to areadonlyfield from a property accessor is illegal and should fail compilation. Also, the PR description and prior review threads indicate this is intended to be assignable after construction (e.g., between runs), sosetis likely the correct shape.
public Func<ItemErrorContext, ItemErrorAction> ErrorPolicy
{
get => _errorPolicy;
init => _errorPolicy = value ?? throw new ArgumentNullException(nameof(value));
}
src/Wolfgang.Etl.Abstractions/LoaderBase.cs:496
ErrorPolicyassigns to_errorPolicy, but_errorPolicyis declaredreadonlyabove (line 480). That assignment from an accessor should not compile. If the intent is a configurable policy (as described in the PR description / earlier threads), switch this to a normalsetand make the backing field non-readonly.
public Func<ItemErrorContext, ItemErrorAction> ErrorPolicy
{
get => _errorPolicy;
init => _errorPolicy = value ?? throw new ArgumentNullException(nameof(value));
}
src/Wolfgang.Etl.Abstractions/TransformerBase.cs:507
ErrorPolicyassigns to_errorPolicyvia aninitaccessor, but_errorPolicyis declaredreadonlyabove (line 491). Assigning to areadonlyfield from a property accessor should fail compilation. If this is meant to be assignable configuration, usesetand a non-readonlybacking field.
public Func<ItemErrorContext, ItemErrorAction> ErrorPolicy
{
get => _errorPolicy;
init => _errorPolicy = value ?? throw new ArgumentNullException(nameof(value));
}
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0.21.0 release
Merges the 0.21.0 content from
vNexttomain. Minor, purely additive — validates against the 0.20.0 baseline, no breaking change.What's in it
ExtractorBase<TSource>/LoaderBase<TDestination>/TransformerBase<TSource, TDestination>fixTProgress = Reportand supply a defaultCreateProgressReport().ErrorPolicyon the base stages (Provide base-class defaults to eliminate component boilerplate (convenience base + timer-injection hook) #344 follow-up): settableFunc<ItemErrorContext, ItemErrorAction>the baseOnItemErrorconsults; default fail-fast.Wolfgang.Etl.ErrorPolicies(lockstep 0.21.0):ItemErrorPolicyfactory —Skip,Abort,SkipAndLog, and dead-letter families (ICollection/ChannelWriter), incl. observable full-channel drop logging.Release-readiness (verified locally)
[0.21.0] - 2026-08-03; PublicAPI Unshipped→Shipped for both packages.release.yamlpacks/publishes glob-over-src/**→ the new ErrorPolicies package publishes automatically.After merge
Tag
v0.21.0(release title e.g. "0.21.0 — Convenience bases, assignable error policy, ErrorPolicies package") to firerelease.yaml. Then the post-release baseline bump + branch cleanup.Closes #344 — Part A (convenience base classes) ships here; Part B (timer-injection hook) was delivered in ETL-Test-Kit#280 (ManualProgressTimerCore) and ships in TestKit 0.14.0.