Add fluent Pipeline API for Extract -> Transform -> Load (#83)#134
Merged
Conversation
Implements Pipeline.Extract(...).Transform(...).Load(...).RunAsync() with compile-time stage-to-stage type safety, optional per-stage WithProgress, one-shot execution, and raw exception propagation. Design decisions are recorded in the issue thread. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TestKit doubles implement the progress-capable interfaces, so overload resolution on Pipeline.Extract/.Transform/.Load always bound to the progress-capable overloads — leaving ExtractStage, TransformStage, and PipelineImpl at 0-50% coverage and failing the 90% module gate. Add 9 tests that cast the test doubles to the no-progress interfaces to force overload resolution onto the previously-untested paths. Module coverage now 94.9% (was 82%). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
352913e to
f2692cc
Compare
The Pipeline API previously required every stage to support cancellation. Expand Pipeline.Extract / IExtractStage.Transform / IExtractStage.Load / ITransformStage.Transform / ITransformStage.Load to 4 overloads each: - Base : IExtractAsync<T> etc. - With cancellation : IExtractWithCancellationAsync etc. - With progress : IExtractWithProgressAsync etc. - With progress + cancel : IExtractWithProgressAnd... etc. Overload resolution picks the most-specific interface a given class implements, so existing code is unaffected. No-cancellation stages receive no CancellationToken; the pipeline's RunAsync token is simply not forwarded to those stages (documented). Rewrite the test suite around 12 real test-double classes (4 capability combos x 3 roles) instead of casting TestKit doubles to bind overloads. Adds BareTests, CancelOnlyTests, ProgressOnlyTests, FullTests, MixedCapabilityTests, and PipelineBehaviorTests. 191/191 tests pass in Release on net10.0; module line coverage 96.4%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add two focused test files to close the last gaps in the Pipeline internals: - NullGuardOnChainedStagesTests: 8 null-argument guards on mid-chain ITransformStage.Transform/Load overloads (the PipelineBehaviorTests guards only hit the IExtractStage overloads, so the ITransformStage throws were previously unreachable in tests). - NoProgressPathTests: 9 tests that exercise the "progress-capable stage appended without WithProgress" code paths in ExtractStageWithProgress, TransformStageWithProgress, and the no-progress lambdas inside ExtractStage/TransformStage. All 9 Pipeline classes now at 100% line coverage (208/208 tests pass in Release on net10.0). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spells out in the XML <remarks> that RunAsync does not catch, wrap, or aggregate — exceptions (including OperationCanceledException) propagate unchanged and the caller should wrap in try/catch. Includes a short code example. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
S3246 (Sonar): TProgress should be declared 'out' (covariant) on IPipelineWithLoadProgress, IExtractStageWithProgress, and ITransformStageWithProgress. TProgress only appears inside IProgress<T> (contravariant) parameter positions, which makes it effectively covariant at the outer interface level. VSTHRD200 (vs-threading): private Source methods returning IAsyncEnumerable<T> must have the Async suffix. Rename Source -> SourceAsync in ExtractStageWithProgress and TransformStageWithProgress. Both errors only fired on net462/net472/net48/net481 (the Stage 2 Windows matrix); Linux Stage 1 (net5-net10) was green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 24, 2026
Addresses findings #6, #7, and #11 from the review. #6 — Use Volatile.Read when reading the Interlocked-incremented CurrentItemCount and CurrentSkippedItemCount fields on ExtractorBase, LoaderBase, and TransformerBase. Writers use Interlocked.Increment; pairing the reads with Volatile.Read closes a theoretical weak-memory-model read hazard (relevant primarily on older ARM NetFx targets) and is free on x86/x64. #7 — Replace the raw InvalidOperationException throws in the CancelOnly* test doubles with a sentinel WrongOverloadCalledException. A future Pipeline refactor that accidentally routes through a bare overload will now produce a unique, named exception in test output instead of a generic "Operation is not valid" message. #11 — Assert Assert.Empty(loader.Loaded) after the pre-cancelled-token cancellation tests on CancelOnlyTests and FullTests. Previously the tests only verified that SOME OperationCanceledException was thrown; a regression where cancellation fires mid-stream after one or two items would have still passed. 208/208 tests pass in Release on net10.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the fluent Pipeline API agreed on in #83 (see design-decisions comment).
What's included
Public API (all in
Wolfgang.Etl.Abstractionsnamespace):Pipeline— static entry with overloadedExtract<T>/Extract<T, TProgress>IExtractStage<T>/IExtractStageWithProgress<T, TProgress>ITransformStage<T>/ITransformStageWithProgress<T, TProgress>IPipeline/IPipelineWithLoadProgress<TProgress>— runnable terminals withWithName,RunAsync(),RunAsync(CancellationToken)Key properties:
.WithProgress(...)only appears on stage types whose underlying extractor/transformer/loader supports progressRunAsynccall throwsInvalidOperationExceptionOut of scope (per design)
PipelineReportcomposite return typeUse(...)(depends on Add middleware / interceptor support for cross-cutting per-item concerns #93)RunAsynccall timeTests
19 new tests covering:
E -> L,E -> T -> L, and multi-transformerE -> T -> T -> T -> Lcomposition (with type-changing transformers)WithProgressforwarding on extractor, transformer, and loaderWithNamedefault and assignmentAll 151 unit tests pass in Release on net10.0.
Test plan
dotnet build -c Release -f net10.0clean (0 warnings, 0 errors) on src + testsdotnet test -c Release -f net10.0— 151/151 passCloses #83.
🤖 Generated with Claude Code