Parallel Processing Limits #3887
Comments
andrewsignori-aot
added this to the 2.1 Post Part-Time Primera Importante Mucho milestone
andrewsignori-aot
added a commit
that referenced
this issue
Nov 13, 2024
|
@astridSABC Testing note. This has the low potential to impact bulk offering upload. Ensure we test bulk upload to ensure no impact. |
andrewsignori-aot
added a commit
that referenced
this issue
Nov 14, 2024
github-merge-queue bot
pushed a commit
that referenced
this issue
Nov 18, 2024
- Refactoring methods that rely on OS CPUs as a parallel limit to use the existing `processInParallel` method. - New options added to the `processInParallel` to keep the existing features from the methods refactored. - `currentRecord` allows progress report (added to satisfy SFAS integration and also used for Fed restrictions now). - `partialResults` allow access to lasted awaited promises batch (added to satisfy offering bulk upload). - While doing the review, please try to identify existing code that was just moved vs new code. ### Federal Restriction Benchmark The amount of parallel promises allowed was reduced from 4 to 2. Even though the bulk size of 750 was faster in local tests it was kept as 500 to avoid overloading the DB. - 2 parallel promises, 500 bulk inserted records, inserting records time: 2:24.584 (m:ss.mmm) - 4 parallel promises, 500 bulk inserted records, inserting records time: 2:25.388 (m:ss.mmm) - 2 parallel promises, 250 bulk inserted records, inserting records time: 4:24.722 (m:ss.mmm) - 2 parallel promises, 750 bulk inserted records, inserting records: 1:51.329 (m:ss.mmm) ### Bulk Offering Upload - Tested importing 1000 new offerings and it was imported almost instantaneously.
|
@Joshua-Lakusta @astridSABC @CarlyCotton, below are the files impacted by this refactoring.
|
DemoFederal RestrictionsLogs now show the import progress, as shown below. The restrictions are inserted in bulks of 500 records in parallel, usually leading to 1000 records inserted per progress line below.
SFAS Integration
Offerings Bulk UploadInstitution: SIMS_COLLE
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Code is starting 32 parallel processes where each one will create its own transaction. Once the typeorm pool reaches its maximum (configured to 10 right now) it will be waiting forever.
The next PROD release changed the default configuration and should already resolve this issue for the e-Cert.
os.cpus()as a limit when no other limit was provided.The text was updated successfully, but these errors were encountered: