#3887 - Limit Parallel Processing #3933
Conversation
andrewsignori-aot
changed the title
| progress?: (currentRecord: number) => Promise<void> | void; | ||
| partialResults?: (results: P[]) => Promise<void> | void; |
There was a problem hiding this comment.
Amazing way to extend the the capability of the method. ❤️
| progress?: (currentRecord: number) => Promise<void> | void; | ||
| partialResults?: (results: P[]) => Promise<void> | void; |
There was a problem hiding this comment.
what about processPartialResults ?
There was a problem hiding this comment.
I like Dheepak's suggestion.
There was a problem hiding this comment.
For progress it could be updateProgress.
There was a problem hiding this comment.
processPartialResults can indicate that a process will be executed with the records, which may be or may not be the case.
Yes, the current method consuming it is executing some processing, but the processInParallel is just communicating that some partial results are done.
Between progress and updateProgress I would stick with progress.
Please let me know if some of those are not suggestions.
| { | ||
| progress: (currentBulk) => { | ||
| this.logger.log( | ||
| `Inserted record bulk ${currentBulk} of ${fedRestrictionsBulks.length}.`, |
| } | ||
| } catch (error: unknown) { | ||
| result.errorsSummary.push( | ||
| "Error while generating notifications. See logs for details", |
There was a problem hiding this comment.
Period at the end of the phrase.
There was a problem hiding this comment.
Added the period.
Added period.
|
Backend Unit Tests Coverage Report
|
E2E Queue Consumers Coverage Report
|
E2E Workflow Workers Coverage Report
|
E2E SIMS API Coverage Report
|
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
temporarily deployed
to
DEV
GitHub Actions
Inactive
andrewsignori-aot
deleted the
feature/#3887-limit-parallel-processing
branch
processInParallelmethod.processInParallelto keep the existing features from the methods refactored.currentRecordallows progress report (added to satisfy SFAS integration and also used for Fed restrictions now).partialResultsallow access to lasted awaited promises batch (added to satisfy offering bulk upload).Federal Restriction Benchmark
The amount of parallel promises allowed was reduced from 4 to 2. Even though the bulk size of 750 was faster in local tests it was kept as 500 to avoid overloading the DB.
Bulk Offering Upload