release: Wolfgang.Etl.Abstractions 0.18.0 — per-item error handling (#84) [HOLD until 2026-07-25] - #326
Merged
Conversation
Adds the vocabulary + plumbing for skip-or-abort error handling, with NO public
knob on the base (error recovery is a per-format capability):
- ItemErrorAction { Abort, Skip } and ItemErrorContext (record ordinal, exception,
optional lazy raw content).
- protected virtual OnItemError(context) => Abort on ExtractorBase/LoaderBase/
TransformerBase — the policy hook a format overrides.
- protected HandleItemError(context) — the worker calls this from its own
try/catch (a C# async iterator can't resume after throw, so the base can't
catch on the worker's behalf); it invokes OnItemError and, on Skip, increments
the skipped-item count so a skip is never silent.
- EtlPipelineProgress.RecordsSkipped (additive init-only property) surfaced from a
skip-aware ExtractorBase source via EtlRunState; 0 for a raw IAsyncEnumerable.
Default is Abort — existing stages are unaffected. Additive/MINOR; ApiCompat clean
(RecordsSkipped is a property, not a new positional parameter). 382 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adopting #84 in ETL-Json revealed that error-skips must NOT share the CurrentSkippedItemCount counter, which the format already uses for intentional SkipItemCount-budget skips — conflating them would silently change the reference's semantics and hide failures in an intentional-skip count. - Add CurrentErrorItemCount to the 3 base classes; HandleItemError increments it (not the skip counter) on Skip. Reset per run. - EtlPipelineProgress.RecordsSkipped -> RecordsErrored, surfaced from the extractor's CurrentErrorItemCount. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…aths Stryker on the #84 branch surfaced that HandleItemError/OnItemError were only exercised through an ExtractorBase double — the identical code on LoaderBase and TransformerBase was entirely uncovered (10 NoCoverage mutants), plus survivors for the CurrentErrorItemCount reset-between-runs and the EnsureStarted() call. - Add ConfigurableLoader/Transformer + DefaultPolicy doubles and mirror the extractor #84 unit tests onto both (default Abort, Skip-increments, Abort-no-count, null-throws, worker-skips-and-counts). - Add reset-across-runs tests for all three bases (kills the Volatile.Write reset survivors). - Add "skip marks the run started" tests via a StartedAt accessor (kills the EnsureStarted survivors). - Mark the OnItemError default-Abort block as an equivalent mutant (Abort is the enum's 0-default), matching the repo's Stryker-disable convention. Mutation score 93.13% -> 97.23%; all #84-specific mutants now killed or documented-equivalent. Remaining survivors are pre-existing timer code (CreateProgressTimer / SystemProgressTimer), unrelated to #84. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CHANGELOG: add the [0.18.0] section describing the #84 per-item error-handling mechanism (ItemErrorAction / ItemErrorContext / HandleItemError / OnItemError / CurrentErrorItemCount / RecordsErrored). Dated 2026-07-25 (releases tomorrow). - PublicAPI: promote PublicAPI.Unshipped.txt -> PublicAPI.Shipped.txt (19 #84 entries) and reset Unshipped. Shipped now reflects everything through 0.18.0. - <Version> already at 0.18.0 and PackageValidationBaselineVersion at 0.17.0 (MINOR, additive-only). AssemblyVersion stays pinned at 1.0.0.0. src builds clean across all TFMs, 0 warnings; ApiCompat additive vs 0.17.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
BenchmarkDotNet
Details
| Benchmark suite | Current: 2cb4703 | Previous: 33992ff | Ratio |
|---|---|---|---|
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 1000) |
31618.23424275716 ns (± 237.11072389470615) |
32635.177775065105 ns (± 95.51647773189825) |
0.97 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 1000) |
33947.89496866862 ns (± 90.31310002095744) |
35028.11218261719 ns (± 387.8657479408494) |
0.97 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_NoProgress(RecordCount: 100000) |
3088830.7981770835 ns (± 5675.843614808854) |
3220211.7493489585 ns (± 5734.59641892182) |
0.96 |
Wolfgang.Etl.Abstractions.Benchmarks.ExtractorBenchmarks.Extract_WithProgress(RecordCount: 100000) |
3389977.6731770835 ns (± 4011.346760580946) |
3439904.2486979165 ns (± 5769.397030124281) |
0.99 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 1000) |
29464.134033203125 ns (± 68.93775297010534) |
33119.71512858073 ns (± 40.01966686404218) |
0.89 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 1000) |
29725.03340657552 ns (± 110.71074830586022) |
31863.15596516927 ns (± 98.21299228608584) |
0.93 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 1000) |
71162.22298177083 ns (± 259.3403267854645) |
69701.81351725261 ns (± 118.3873780955796) |
1.02 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.FluentPipeline(RecordCount: 100000) |
2972141.91796875 ns (± 11662.746542527511) |
3182391.859375 ns (± 3983.3645830260316) |
0.93 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.ManualComposition(RecordCount: 100000) |
2931481.2330729165 ns (± 42345.05871642324) |
3207955.4934895835 ns (± 7473.333760456796) |
0.91 |
Wolfgang.Etl.Abstractions.Benchmarks.PipelineBenchmarks.BaseClassComposition(RecordCount: 100000) |
6684308.9140625 ns (± 8841.178588433817) |
6936332.84375 ns (± 12973.308060016172) |
0.96 |
This comment was automatically generated by workflow using github-action-benchmark.
Align the three progress counters with the base stages' Current*ItemCount naming: RecordsExtracted -> ExtractedItemCount RecordsLoaded -> LoadedItemCount RecordsErrored -> ErrorItemCount (new in 0.18.0, never shipped) RecordsExtracted/RecordsLoaded shipped in 0.16.0, so removing them is a breaking change — but EtlPipelineProgress is only days old with negligible adoption, so renaming now is far cheaper than later. Pre-1.0. - PublicAPI.Shipped.txt updated + re-sorted. - CompatibilitySuppressions.xml waives the two removed 0.17.0 accessors (CP0002 baseline suppressions) so Package Validation passes. - CHANGELOG [0.18.0]: breaking "### Changed" note. - AssemblyVersion stays pinned at 1.0.0.0. src + pack clean across all TFMs; 398 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the last "record"-named member of the new #84 surface with the rest of the library's Item vocabulary (ItemErrorAction, ItemErrorContext, CurrentErrorItemCount, EtlPipelineProgress.*ItemCount). ItemErrorContext is new in 0.18.0 (absent from the 0.17.0 baseline), so this is NOT a breaking change and needs no compatibility suppression. src + pack clean across all TFMs; 398 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…unters The EtlPipelineProgress rename (RecordsExtracted/RecordsLoaded -> ExtractedItemCount/ LoadedItemCount) missed the two Example8-EtlPipeline programs (Net4.8 + Net8.0), which the full-solution CI build compiles — causing CS1061 and cascading failures in Stage 1 tests, Stryker, CodeQL, and InspectCode. Full solution now builds clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 29, 2026
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release: Wolfgang.Etl.Abstractions 0.18.0 — per-item error handling (#84)
Minor release. Adds an opt-in per-item error-handling mechanism to the three base stages so a worker can skip a bad item and keep going instead of aborting the whole run — plus a small, deliberate breaking rename of the
EtlPipelineProgresscounters (see below).Added
ItemErrorAction(Abort,Skip) andItemErrorContext(item number, exception, optional lazy raw-content accessor).ExtractorBase/LoaderBase/TransformerBase:HandleItemError(ItemErrorContext)— call from a worker'scatch, re-throw onAbort;virtualOnItemError(ItemErrorContext)policy hook (defaultAbort) that a derived stage overrides to surface its own knob;CurrentErrorItemCount, distinct fromCurrentSkippedItemCount(skip-budget), so a failure is never silently absorbed.EtlPipelineProgress.ErrorItemCount— surfaces the extractor's error-item count in the progress snapshot.Changed (breaking)
Renamed the
EtlPipelineProgresscounters for a consistent...ItemCountscheme:RecordsExtracted→ExtractedItemCountRecordsLoaded→LoadedItemCountErrorItemCount, notRecordsErrored)EtlPipelineProgressshipped in 0.16.0, so removing the first two accessors is a breaking change — but the type is only days old with negligible adoption, so aligning the names now is far cheaper than later. Pre-1.0. Package Validation against the 0.17.0 baseline waives the two removed accessors viaCompatibilitySuppressions.xml(44 CP0002 baseline suppressions, one per accessor × TFM);AssemblyVersionstays pinned at1.0.0.0.Safety / quality
Validated downstream (separate branches, not part of this PR)
Proven by adopting it in ETL-Json (greenfield), ETL-FixedWidth (
#29, pre-existing error model), and ETL-Xml (#11, streaming reader repositioning). All three land on the sameAbort/Skipcontract.Release steps (tomorrow)
main.v0.18.0on the merge commit →release.yamlbuilds, packs, and publishes.Closes #84.