Release v0.20.0 — retry seam, middleware, error aggregation - #342
Merged
Conversation
…ses (#94) Add a protected virtual WrapWorkerExecution hook to ExtractorBase, LoaderBase, and TransformerBase, wrapped around every worker invocation (both the no-progress and with-progress paths). The default implementation is a no-op, so behaviour is unchanged; an override receives a re-invocable worker factory (call it again to retry a transient failure) and stream-level semantics are documented on the method. Kept dependency-free per the 0.20.0 decision (Option 1): Abstractions stays zero-dep and roots 8 downstream packages, so no Polly dependency here. A ready-made Polly integration will ship as a separate opt-in Wolfgang.Etl.Polly package (#332). - 13 new RetrySeamTests (default no-op, seam-invoked-once on both paths, factory re-invocability, genuine transient-failure recovery, null guard) across all 3 bases. 426 unit tests pass; Stryker 100.00% (0 survivors) on the full project. - PublicAPI.Unshipped updated with the 3 protected virtual entries (RS0017-validated). - CHANGELOG [Unreleased] Added entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add IItemMiddleware<T> + MiddlewareResult<T> (Continue/Drop) and WithMiddleware(...) extensions that attach cross-cutting per-item behaviour (logging, validation, metrics, throttling, dedup) to any IAsyncEnumerable<T> stream — extractor/transformer output or loader input — without changing the component, and compose inside an EtlPipeline via Through(s => s.WithMiddleware(...)). Single and ordered-chain overloads; a dropped item short-circuits the rest of the chain. Dependency-free. - 15 MiddlewareTests (transform, drop, chain order, stop-at-drop, empty chain, token flow, pipeline composition, null guards) + MiddlewareResult value semantics. 441 unit tests pass; Stryker 100.00% (0 survivors). - PublicAPI.Unshipped updated (RS0016 completeness + RS0017 correctness validated). - CHANGELOG [Unreleased] Added entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix the base-class asymmetry: CurrentErrorItemCount lived only on the concrete base
classes, so EtlPipelineProgress.ErrorItemCount could only surface the extractor's errors
(Through takes ITransformAsync, an interface that didn't expose it).
- Add IReportsItemErrors { int CurrentErrorItemCount { get; } }, implemented by
ExtractorBase, LoaderBase, TransformerBase (property already existed — additive).
- EtlRunState now holds a list of error-count readers and sums them in Snapshot().
- Register a reader from every stage that reports errors: the extractor (From), each
transformer (Through, via 'is IReportsItemErrors'), and the loader (sink). ErrorItemCount
now reflects items ANY stage's error policy discarded. Pre-1.0 behaviour change.
- 5 AggregateErrorsTests (sum across all three stages, delegate stage contributes 0, zero
when nothing reports, plain-ITransformAsync overload, interface implemented by all bases).
446 unit tests pass; Stryker 100.00% (0 survivors).
- PublicAPI.Unshipped updated (RS0016 completeness + RS0017 correctness validated).
- CHANGELOG [Unreleased] Added + Changed entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # CHANGELOG.md # src/Wolfgang.Etl.Abstractions/EtlPipeline/EtlRunState.cs # src/Wolfgang.Etl.Abstractions/PublicAPI.Unshipped.txt
Reconcile complete: the 0.20 feature stack (#94 retry seam / #93 middleware / #335 error aggregation) merged in, plus vNext's 0.19.0 prep + #338 clock seam merged down. CHANGELOG [Unreleased] holds the 0.20 features; [0.19.0] section carried in. PublicAPI merged (features in Unshipped, Report ctor promoted to Shipped). 452 tests pass; all 11 TFMs clean; pack + Package Validation green vs 0.18.1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MiddlewareExtensions: validate arguments EAGERLY (public wrapper + private iterator local function) so a null source/middleware/chain-member fails fast at the call site, matching the base classes' TransformAsync pattern instead of throwing only on first enumeration. Tests for the null cases become synchronous (Assert.Throws). - Reinstate ConfigureAwait(false) on the WithMiddleware 'await foreach' loops for net462/netstandard2.0 sync-context safety (public extension is more exposed than the internal worker loops); the equivalent bool mutants are marked Stryker-disable. - README: add Per-item Error Handling / Middleware / Retry Seam feature rows (the #84 error row was also missing). - CHANGELOG: note the internal #338 time-source seam under [Unreleased]. 452 tests pass; 100% mutation (0 survivors); all 11 TFMs clean. No public API change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PackageValidationBaselineVersion 0.18.1 -> 0.19.0; the int->long break is gone vs 0.19.0 (0.19.0 already ships long counters), so 0.20 is purely additive — CompatibilitySuppressions.xml removed and pack validates clean vs 0.19.0. - Promote CHANGELOG [Unreleased] -> [0.20.0] (2026-07-30). - Promote the 0.20 public surface (retry seam / middleware / IReportsItemErrors) PublicAPI.Unshipped -> Shipped (216 entries, ordinal-sorted); reset Unshipped. Version already 0.20.0. 452 tests; all 11 TFMs clean; PublicAPI RS0016+RS0017 validated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…; avoid reverting them on release) # Conflicts: # CHANGELOG.md
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
BenchmarkDotNet
Details
| Benchmark suite | Current: 4850525 | Previous: 5e5eff8 | Ratio |
|---|---|---|---|
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 1000) |
26180.478098551434 ns (± 163.09071239051553) |
34346.27861531576 ns (± 767.7271166066424) |
0.76 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 1000) |
27760.10508219401 ns (± 330.4004072958602) |
35690.69081624349 ns (± 121.99168762232769) |
0.78 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 100000) |
2585459.24609375 ns (± 2664.7329475021124) |
3607541.7135416665 ns (± 4820.553954320889) |
0.72 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 100000) |
2706979.3723958335 ns (± 7911.503269506926) |
3485849.25 ns (± 1446.7723543303268) |
0.78 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 1000) |
25344.441060384113 ns (± 89.81456462245298) |
31901.971710205078 ns (± 107.64253574413279) |
0.79 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 1000) |
24542.37175496419 ns (± 77.88746524664231) |
32595.139221191406 ns (± 132.60242047288244) |
0.75 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 1000) |
58122.23839314779 ns (± 98.74054413246576) |
74138.58180745442 ns (± 339.1681844140048) |
0.78 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 100000) |
2494574.7083333335 ns (± 1691.8449564856976) |
3219619.8020833335 ns (± 3141.7580180829386) |
0.77 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 100000) |
2487880.421875 ns (± 3388.7047087723986) |
3208296.451171875 ns (± 1993.4468905203664) |
0.78 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 100000) |
5410841.114583333 ns (± 2483.5733481380953) |
8852946.604166666 ns (± 19787.514036475975) |
0.61 |
This comment was automatically generated by workflow using github-action-benchmark.
This was referenced Jul 30, 2026
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release: Wolfgang.Etl.Abstractions 0.20.0
Minor release. Bundles the 0.20.0 line from
vNext-plus-one→main. Purely additive — no breaking change (validates against the published 0.19.0 baseline;CompatibilitySuppressions.xmlremoved).What's in it
WrapWorkerExecutionhook on each base stage (default no-op) for plugging in a retry / resilience strategy. Polly glue ships separately (Create Wolfgang.Etl.Polly package — ready-made Polly retry over the WrapWorkerExecution seam #332).IItemMiddleware<T>+MiddlewareResult<T>+WithMiddleware(...)— composable per-item behaviour on any stream; composes inEtlPipelineviaThrough(s => s.WithMiddleware(...)).IReportsItemErrors;EtlPipelineProgress.ErrorItemCountnow sums errors across every stage (source, transformers, loader), not just the extractor.Reporttiming/ETA from a fake clock (unblocks ETL-Test-Kit#262).Release-prep done
CompatibilitySuppressions.xmlremoved (additive vs 0.19.0). Pack + Package Validation green.[Unreleased]→[0.20.0] - 2026-07-30.0.20.0.0.19.0's NuGet date is 2026-07-29, so
v0.20.0must be tagged at/after 2026-07-30T00:00Z to land on a distinct NuGet date. Merging this PR is safe anytime (release.yaml fires only onrelease:published); create thev0.20.0tag tomorrow night.No protected files in the diff → no admin bypass needed.