Merge initial development#11
Merged
Merged
Conversation
Configured project to work with .net 4.6.2, 4.7.2, 4.8, 4.8.1, 8.0, 9.0, .netstandard 2.0 and 2.1 Created unit tests to verify that classes implementing the three interfaces work with targets frameworks Created Example1-BasicETL in dotnet 4.8 and 8.0
Updated `IExtractAsync`, `ILoadAsync`, and `ITransformAsync` interfaces to include detailed XML documentation comments, clarifying their roles in the ETL process. Modified method signatures for better readability by using descriptive parameter names. Adjusted `ConsoleLoader` implementation to align with the new `ILoadAsync` method signature.
- Adjusted `<LangVersion>` formatting in `Example1-BasicETL.csproj`. - Corrected filename from `FibinocciExtractor.cs` to `FibanocciExtractor.cs`. - Removed unused `using System.Runtime.CompilerServices;` directive in `AssemblyInfo.cs`.
- IExtractWithCancellationAsync - ITransformWithCancellationAsync - ILoadWithCancellationAsync Created - Tests - .Net 4.8 examples of the new interfaces - .Net 8.0 examples of the new interfaces Updated the solution to include `Example2-WithCancellationToken` and `Example3-WithGracefulCancellation` projects targeting .NET 8.0. Enhanced the ETL framework by implementing cancellation tokens in key classes and interfaces, allowing for graceful cancellation of operations. Added necessary configuration files and updated tests to verify functionality. Improved project structure and dependencies for better integration.
Corrected the filename from `FibanocciExtractor.cs` to `FibonacciExtractor.cs` in project references for `Example1-BasicETL.csproj`, `Example2-WithCancellationToken.csproj`, and `Example3-WithGracefulCancellation.csproj`. Adjusted the `LangVersion` property in `Example2-WithCancellationToken.csproj`, though the value remains unchanged. Updated the namespace in `FibonacciExtractor.cs` to align with project naming conventions.
Updated the namespace from `Example3_ExtractorWithGracefulCancellation` to `Example3_WithGracefulCancellation` in `ConsoleLoader.cs`, `IntToStringTransformer.cs`, and `Program.cs`. Adjusted `using` directives accordingly to ensure all files reflect the new namespace structure.
Updated method signatures in `ConsoleLoader`, `IntToStringTransformer`, and `IntToStringWithCancellationTransformer` classes to replace the parameter name `source` with `items`. This change enhances clarity by indicating that the methods process collections of items. The `await foreach` loops have also been modified to iterate over `items` to maintain consistency across the codebase.
Updated ConsoleLoader and IntToStringTransformer to use cancellation tokens for graceful cancellation. Modified Program.cs to pass cancellation tokens to the ETL methods, enhancing control over asynchronous operations.
Updated class names in ConsoleLoaderWithCancellation.cs and IntToStringTransformerWithCancellation.cs to enhance clarity. Renamed `ConsoleWithCancellationLoader` to `ConsoleLoaderWithCancellation` and `IntToStringWithCancellationTransformer` to `IntToStringTransformerWithCancellation`.
- Updated solution file to include Example4a, Example4b, and Example4c projects targeting .NET 8.0. - Modified build configurations for new projects in the solution. - Added tests for progress reporting in CompatibilityTests.cs. - Created new project files and App.config for the new examples. - Implemented progress reporting interfaces and classes for extraction, transformation, and loading. - Introduced the EtlProgress model to encapsulate progress information. - Enhanced the ETL process structure for improved user feedback during operations.
Updated the ETL framework to support asynchronous loading, extraction, and transformation of data with cancellation tokens. - Added async methods in `ConsoleLoader.cs` for loading data, including overloads for cancellation. - Introduced async extraction methods in `FibonacciExtractor.cs` that yield results incrementally with cancellation support. - Modified `IntToStringTransformer.cs` to include async transformation of integers to strings with cancellation. - Mirrored these changes in the `Example3_WithGracefulCancellation` namespace for consistency. - Updated interfaces in `Wolfgang.Etl.Abstractions` to support cancellation tokens and progress reporting. - Added new test classes to validate the functionality of the updated components. - Improved documentation comments for clarity on method parameters and behaviors related to cancellation and progress.
Updated namespaces from Example4c_WithLoaderProgress.ETL to Example4a_WithExtractorProgress.ETL in multiple files. Removed unnecessary using directives in Program.cs and AssemblyInfo.cs to streamline the project structure.
…cellation and ILoadWithProgressAndCancellation This commit introduces a new example project "Example5a-ExtractorWithProgressAndCancellation" targeting .NET 8.0. It includes the implementation of `FibonacciExtractor`, `IntToStringTransformer`, and `ConsoleLoader` classes, all supporting progress reporting and cancellation. Additionally, the project structure has been updated with necessary configuration files, and the solution file has been modified to include the new project. Interfaces for extraction, loading, and transformation with progress and cancellation capabilities have been defined. Unit tests have also been added to validate the new functionality.
Demonstrates how to redunce duplicate code by writing one method that takes IProgress<T> and CancellationToken and the writing the other methods to call that one method
Revised the `items` parameter description in the `LoadAsync` method across multiple interfaces in the `Wolfgang.Etl.Abstractions` namespace. The new description clarifies that the items are intended for loading into a specific destination, enhancing code readability and understanding.
…to support progress reporting and cancellation tokens for asynchronous operations. Added properties for managing extraction, loading, and transformation processes. Added unit tests for new functionality and updated project file with necessary package references. These changes improve the usability and robustness of the ETL framework.
Significantly refactored the codebase to enhance compatibility of extractor, transformer, and loader classes with various .NET versions. Rewrote `CompatibilityTests.cs` to include comprehensive tests for basic interfaces, cancellation tokens, and progress reporting. Updated loader and extractor classes to support cancellation and progress reporting. Added new test classes for interface functionality and organized tests into new namespaces. Updated project file to target additional .NET frameworks. Added test to verify compatability with .net 4.72, 4.8.1 and net 5, 6, 7
Updated `Wolfgang.Etl.Abstractions.Tests.Unit.csproj` to set Removed unnecessary file type references for `ETL\**`, simplifying the project
This commit introduces the `[NotNull]` attribute from JetBrains.Annotations to method parameters across various asynchronous extractor, loader, and transformer interfaces. This enhancement improves nullability checks and overall code safety. Additionally, method signatures have been reformatted for better readability, with parameters now listed on separate lines.
This commit introduces the `[ExcludeFromCodeCoverage]` attribute to several internal classes across multiple test files, improving code coverage accuracy. The visibility of several classes has been changed from `public` to `internal` to enhance encapsulation. Additionally, `using` directives have been updated to include `System.Diagnostics.CodeAnalysis`. The `FibonacciExtractor` and `ConsoleLoader` classes have been modified to implement various extraction and loading interfaces, while the `IntToStringTransformer` class has been updated to support progress and cancellation in asynchronous transformations.
…erBase and LoaderBase which implement the interfaces
Increased code coverage Improved exception messages for null parameters in `ExtractorBase`, `LoaderBase`, and `TransformerBase`. Added `CancellationToken` support to relevant methods. Updated tests to verify handling of null parameters and cancellation scenarios. Changed `EtlProgress` visibility from public to internal for better encapsulation.
Removed netstandard2.0 and netstandard2.1 from the TargetFrameworks property to streamline the supported frameworks, focusing on .NET versions from net462 to net9.0.
- Added binary file `ETL.png` to the project. - Updated `Wolfgang.Etl.Abstractions.sln` to include `README.md` as a solution item. - Modified target frameworks and updated version to `1.0.0-rc1` in `Wolfgang.Etl.Abstractions.csproj`. - Added additional metadata such as title, authors, description, copyright, package URLs, assembly version, license expression, and license acceptance requirement. - Set `PackageIcon` to `ETL.png` and included `ETL.png` and `README.md` in the package contents.
This commit introduces a `<Version>` element set to `1.0.0` in the `<PropertyGroup>` section of multiple `.csproj` files. The projects updated include `Example1-BasicETL`, `Example2-WithCancellationToken`, `Example3-WithGracefulCancellation`, `Example4a-WithExtractorProgress`, `Example4b-WithTransformerProgress`, `Example4c-WithLoaderProgress`, `Example5a-ExtractorWithProgressAndCancellation`, and `Example6-ReducingDuplicateCode`, standardizing their versioning.
Eliminate the use of JetBrains.Annotations across multiple files in the Wolfgang.Etl.Abstractions namespace. Replace `[NotNull]` and `[UsedImplicitly]` attributes with data validation attributes like `[Range(0, int.MaxValue, ErrorMessage = "...")]` for properties such as `CurrentItemCount`, `MaximumItemCount`, and `SkipItemCount`. This change improves property value validation and simplifies project dependencies.
Removed unused references from project files (Example1-6) to streamline dependencies. Eliminated `using System.Runtime.CompilerServices;` in `AssemblyInfo.cs` as it is no longer needed. Updated `Wolfgang.Etl.Abstractions.csproj` by removing README references, indicating a change in documentation handling. Cleaned up `ExtractorBaseTests.cs` by removing unnecessary alias for `OperationCanceledException`.
The `<PackageReference>` for `coverlet.collector` version `6.0.4` has been removed from the project file. This change means the project will no longer utilize this package for code coverage during testing, potentially impacting the collection of code coverage metrics in the test project.
- Updated `Microsoft.NET.Test.Sdk` from `17.10.0` to `17.13.0`. - Added `allowedVersions` attribute to `xunit.runner.visualstudio` to restrict version to `2.8.2`.
Added allowedVersions attribute to specify version 17.13.0.
- Incremented project version from 0.1.0 to 1.0.0. - Updated `Microsoft.NET.Test.Sdk` and `xunit.runner.visualstudio` to include version tags.
Updated the version of `Microsoft.Bcl.AsyncInterfaces` to `9.0.7`, `System.Runtime.CompilerServices.Unsafe` to `6.1.2`, and `System.Threading.Tasks.Extensions` to `4.6.3` across multiple project files and `packages.config`. This ensures the projects utilize the latest compatible versions, incorporating bug fixes, performance enhancements, and new features.
…o long in ExtractorBase, TransformerBase and Loaderbase
…se to allow a minimum value of 0
…nt etc. from long to int
Chris-Wolfgang
added a commit
that referenced
this pull request
Apr 25, 2026
Addresses findings #6, #7, and #11 from the review. #6 — Use Volatile.Read when reading the Interlocked-incremented CurrentItemCount and CurrentSkippedItemCount fields on ExtractorBase, LoaderBase, and TransformerBase. Writers use Interlocked.Increment; pairing the reads with Volatile.Read closes a theoretical weak-memory-model read hazard (relevant primarily on older ARM NetFx targets) and is free on x86/x64. #7 — Replace the raw InvalidOperationException throws in the CancelOnly* test doubles with a sentinel WrongOverloadCalledException. A future Pipeline refactor that accidentally routes through a bare overload will now produce a unique, named exception in test output instead of a generic "Operation is not valid" message. #11 — Assert Assert.Empty(loader.Loaded) after the pre-cancelled-token cancellation tests on CancelOnlyTests and FullTests. Previously the tests only verified that SOME OperationCanceledException was thrown; a regression where cancellation fires mid-stream after one or two items would have still passed. 208/208 tests pass in Release on net10.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chris-Wolfgang
added a commit
that referenced
this pull request
Apr 25, 2026
Addresses findings #6, #7, and #11 from the review. #6 — Use Volatile.Read when reading the Interlocked-incremented CurrentItemCount and CurrentSkippedItemCount fields on ExtractorBase, LoaderBase, and TransformerBase. Writers use Interlocked.Increment; pairing the reads with Volatile.Read closes a theoretical weak-memory-model read hazard (relevant primarily on older ARM NetFx targets) and is free on x86/x64. #7 — Replace the raw InvalidOperationException throws in the CancelOnly* test doubles with a sentinel WrongOverloadCalledException. A future Pipeline refactor that accidentally routes through a bare overload will now produce a unique, named exception in test output instead of a generic "Operation is not valid" message. #11 — Assert Assert.Empty(loader.Loaded) after the pre-cancelled-token cancellation tests on CancelOnlyTests and FullTests. Previously the tests only verified that SOME OperationCanceledException was thrown; a regression where cancellation fires mid-stream after one or two items would have still passed. 208/208 tests pass in Release on net10.0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Initial release 0.4.0.