Merged
Conversation
* WIP: S3 File Sync * Remove FileStorage project. Move interface into environment setup * Remove redundant if statement * Enforce patterns for file searches Previously, both the Offloc Cleaner and Delius Parser would attempt to process any files in the input directories. * Removes redundant docker file processing service Also tightens some file path concatenation * Cleanup + implementation of S3FileSource service * Removes Parallel parsing (and other related redundant code) + hooks in FileSync * Fix for Delius post-parser encoding and line endings * MinIO integration * Bump packages (excludes Aspire) Issues remain with bumping Aspire to .NET 10 with the community toolkit packages * Enables S3 file synchronization and processing Refactors the file synchronization process to support S3 as a file source. Introduces configurable file sources for Delius and Offloc files (FileSystem, MinIO, S3). Adds database interaction to track file processing status and associate Offloc files with archive names. Enhances file processing workflow with new database messaging and processing start events. Updates stored procedures to handle file processing status and archive associations. * Configures dotnet environment * Configures and initialises MinIO bucket * Clarifies Visualiser setup requirement * Fixes S3 file source path resolution * Orders files by date + handles ddMMyyyy and dMMyyyy formats * Ensures `ArchiveFileName` is nullable in queries * Adds AWS Security Token package * Enables pre-kickoff tasks execution * Enable periodic file sync * Adds safe guard for file processing in Delius and Offloc
Adds a README file to the Matching.Core project, providing Also, it disables package generation on build.
* Updates Aspire dependencies to v13 * Configures parameters for various services * Updates CommunityToolkit packages to stable version
Removes the known issue section regarding potential SQL project startup timing issues, as it was fixed in CommunityToolkit's v13.
* Overwrite files when extracting archive * Adds logging for unavailable files * Clears input directories before download * Adds file date guarding Implements logic to prevent processing of files older than the last processed files. * Prevents concurrent file processing Adds a semaphore to prevent concurrent executions of the file processing logic. * Adds additional logging to file sync process
* Removes unused package versions * Bump Rebus * Remove unused Visual Studio packages * Adds Dependabot configuration file
carlsixsmith-moj
approved these changes
Nov 26, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP: S3 File Sync
Remove FileStorage project. Move interface into environment setup
Remove redundant if statement
Enforce patterns for file searches
Previously, both the Offloc Cleaner and Delius Parser would attempt to process any files in the input directories.
Also tightens some file path concatenation
Cleanup + implementation of S3FileSource service
Removes Parallel parsing (and other related redundant code) + hooks in FileSync
Fix for Delius post-parser encoding and line endings
MinIO integration
Bump packages (excludes Aspire)
Issues remain with bumping Aspire to .NET 10 with the community toolkit packages
Refactors the file synchronization process to support S3 as a file source.
Introduces configurable file sources for Delius and Offloc files (FileSystem, MinIO, S3).
Adds database interaction to track file processing status and associate Offloc files with archive names.
Enhances file processing workflow with new database messaging and processing start events.
Updates stored procedures to handle file processing status and archive associations.
Configures dotnet environment
Configures and initialises MinIO bucket
Clarifies Visualiser setup requirement
Fixes S3 file source path resolution
Orders files by date + handles ddMMyyyy and dMMyyyy formats
Ensures
ArchiveFileNameis nullable in queriesAdds AWS Security Token package
Enables pre-kickoff tasks execution
Enable periodic file sync
Adds safe guard for file processing in Delius and Offloc