Add multiprocessing to meeting simulation workflow #972
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I'll keep this under WIP while I test it out on my actual simulation tasks. |
desh2608
changed the title
desh2608
changed the title
desh2608
commented
Feb 8, 2023
| for spk_id in this_batch_spk_ids: | ||
| sampler = self.samplers[spk_id] | ||
| try: | ||
| this_batch = next(sampler) |
There was a problem hiding this comment.
@pzelasko An issue I am facing here is that the whole sampler gets exhausted after sampling just 1 batch. I'm not sure why this is happening. Could you take a look?
There was a problem hiding this comment.
Okay very basic mistake --- forgot to sort the cuts by speaker id before groupby!
desh2608
changed the title
|
The simulation should be quite fast now. As an example, creating mixtures from ~3.2M source utterances (each used once) takes ~1h using 4 jobs. |
pzelasko
approved these changes
Feb 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes in this PR
parallel_mapmethod.Recommendations for
num_jobsWhen using the
simulate()method in the workflow. we recommend using 1 job when the number of source utterances is small (up to 50k, for example). For larger inputs, this number can be scaled up slowly, but not more than 4-8 jobs to avoid slow-down due to multiprocessing overhead.