File Sync service #17
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, both the Offloc Cleaner and Delius Parser would attempt to process any files in the input directories.
Also tightens some file path concatenation
Issues remain with bumping Aspire to .NET 10 with the community toolkit packages
Refactors the file synchronization process to support S3 as a file source. Introduces configurable file sources for Delius and Offloc files (FileSystem, MinIO, S3). Adds database interaction to track file processing status and associate Offloc files with archive names. Enhances file processing workflow with new database messaging and processing start events. Updates stored procedures to handle file processing status and archive associations.
PaulCooperWorkJustice
approved these changes
Nov 20, 2025
PaulCooperWorkJustice
left a comment
PaulCooperWorkJustice
left a comment
There was a problem hiding this comment.
Changes look good. Ran locally ok
carlsixsmith-moj
pushed a commit
that referenced
this pull request
Nov 26, 2025
* File Sync service (#17) * WIP: S3 File Sync * Remove FileStorage project. Move interface into environment setup * Remove redundant if statement * Enforce patterns for file searches Previously, both the Offloc Cleaner and Delius Parser would attempt to process any files in the input directories. * Removes redundant docker file processing service Also tightens some file path concatenation * Cleanup + implementation of S3FileSource service * Removes Parallel parsing (and other related redundant code) + hooks in FileSync * Fix for Delius post-parser encoding and line endings * MinIO integration * Bump packages (excludes Aspire) Issues remain with bumping Aspire to .NET 10 with the community toolkit packages * Enables S3 file synchronization and processing Refactors the file synchronization process to support S3 as a file source. Introduces configurable file sources for Delius and Offloc files (FileSystem, MinIO, S3). Adds database interaction to track file processing status and associate Offloc files with archive names. Enhances file processing workflow with new database messaging and processing start events. Updates stored procedures to handle file processing status and archive associations. * Configures dotnet environment * Configures and initialises MinIO bucket * Clarifies Visualiser setup requirement * Fixes S3 file source path resolution * Orders files by date + handles ddMMyyyy and dMMyyyy formats * Ensures `ArchiveFileName` is nullable in queries * Adds AWS Security Token package * Enables pre-kickoff tasks execution * Enable periodic file sync * Adds safe guard for file processing in Delius and Offloc * Adds Windows x64 runtime identifier for publishing * Removes extracted offloc file from input directory if already processed * Fixes issue with change tracker displaying node id instead of upci/group (#19) * Adds README to Matching.Core project (#21) Adds a README file to the Matching.Core project, providing Also, it disables package generation on build. * Upgrade aspire (#20) * Updates Aspire dependencies to v13 * Configures parameters for various services * Updates CommunityToolkit packages to stable version * Removes known issue from README (#22) Removes the known issue section regarding potential SQL project startup timing issues, as it was fixed in CommunityToolkit's v13. * Guarding for dates (#23) * Overwrite files when extracting archive * Adds logging for unavailable files * Clears input directories before download * Adds file date guarding Implements logic to prevent processing of files older than the last processed files. * Prevents concurrent file processing Adds a semaphore to prevent concurrent executions of the file processing logic. * Adds additional logging to file sync process * Dependabot & package cleanup (#24) * Removes unused package versions * Bump Rebus * Remove unused Visual Studio packages * Adds Dependabot configuration file
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.