Aggregate per-item error counts across all pipeline stages (#335) - #336
Merged
Conversation
Fix the base-class asymmetry: CurrentErrorItemCount lived only on the concrete base
classes, so EtlPipelineProgress.ErrorItemCount could only surface the extractor's errors
(Through takes ITransformAsync, an interface that didn't expose it).
- Add IReportsItemErrors { int CurrentErrorItemCount { get; } }, implemented by
ExtractorBase, LoaderBase, TransformerBase (property already existed — additive).
- EtlRunState now holds a list of error-count readers and sums them in Snapshot().
- Register a reader from every stage that reports errors: the extractor (From), each
transformer (Through, via 'is IReportsItemErrors'), and the loader (sink). ErrorItemCount
now reflects items ANY stage's error policy discarded. Pre-1.0 behaviour change.
- 5 AggregateErrorsTests (sum across all three stages, delegate stage contributes 0, zero
when nothing reports, plain-ITransformAsync overload, interface implemented by all bases).
446 unit tests pass; Stryker 100.00% (0 survivors).
- PublicAPI.Unshipped updated (RS0016 completeness + RS0017 correctness validated).
- CHANGELOG [Unreleased] Added + Changed entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 30, 2026
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #335. Stacked on #334 (#93) — base is
feat/93-middleware.Summary
Fixes the base-class asymmetry surfaced while reviewing the #93/#94 error interaction: the #84 error mechanism is on all three base classes, but
EtlPipelineProgress.ErrorItemCountonly reflected the extractor's errors. Root cause —CurrentErrorItemCountlived only on the concrete bases, andThrough(ITransformAsync<…>)takes an interface that didn't expose it, so the pipeline structurally couldn't read a transformer's error count.Changes
IReportsItemErrors—int CurrentErrorItemCount { get; }, implemented byExtractorBase,LoaderBase,TransformerBase(the property already existed — purely additive).EtlRunStatenow holds a list of error-count readers and sums them inSnapshot().From), each transformer (Through, viais IReportsItemErrors— so a bareITransformAsyncthat opts in counts too), and the loader (sink).EtlPipelineProgress.ErrorItemCountnow reports the total items any stage's error policy discarded.Behaviour change (pre-1.0)
ErrorItemCountwas extractor-only; it now aggregates source + transformers + loader. 0.19.0 hasn't shipped, so no released consumer depends on the old meaning. Documented in the CHANGELOG.Verification
AggregateErrorsTests: sum across all three stages2+4+1=7, delegate stage contributes 0, zero when nothing reports, plain-ITransformAsyncoverload, and the interface is implemented by all three bases).Notes for release-prep
Targets the 0.20.0 line (
vNext-plus-one). Version bump 0.19.0 → 0.20.0 + baseline → 0.19.0 deferred to 0.20.0 release-prep (gated on 0.19.0 publishing).