Add multiprocess mechanism for Common Voice

[yfyeung]

• Add multiprocess mechanism to commonvoice recipe.
• Use context manager to disable ffmpeg-torchaudio when process Common Voice dataset.
• Fix bug for tsvreader.

Here is the example using num_jobs=16:

Distributing tasks: 16373it [00:00, 27021.30it/s]                                                                        | 0/1 [00:00<?, ?it/s]
Processing: 100%|███████████████████████████████████████████████████████████████████████████████████████| 16373/16373 [00:39<00:00, 419.75it/s]
Distributing tasks: 16373it [00:00, 18163.42it/s]
Processing: 100%|███████████████████████████████████████████████████████████████████████████████████████| 16373/16373 [00:18<00:00, 878.26it/s]
Distributing tasks: 1013969it [00:35, 28613.47it/s]
Processing: 100%|██████████████████████████████████████████████████████████████████████████████████| 1013969/1013969 [15:41<00:00, 1076.88it/s]
Spliting: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [18:57<00:00, 379.13s/it]
Processing CommonVoice languages: 100%|████████████████████████████████████████████████████████████████████████| 1/1 [18:57<00:00, 1137.40s/it

[pzelasko]

This doc is outdated with your changes because the manifests are no longer created or opened lazily here. That may make the recipe quite memory-intensive; would it be possible to keep the .open_writer() mechanism with your changes? You can open the writers like RecordingSet.open_writer() around line 206-207 and instead of appending to a list, write to disk.

[yfyeung]

Sure

[desh2608]

Even with 16 jobs, preparing CommonVoice en takes 8+ hours? That's quite surprising --- do you know where is the bottleneck?

[yfyeung]

Even with 16 jobs, preparing CommonVoice en takes 8+ hours? That's quite surprising --- do you know where is the bottleneck?

@desh2608 Please check this issue #1026.
It seems there is something wrong with Recording.from_file("*.mp3"). Without multithreading, the cpu% is over 3500%. With multithreading, no matter what num_jobs is, it takes about 8 hours.

@pzelasko @mthrok If we disable

lhotse/lhotse/audio.py

Line 1765 in 7edc9ae

if (is_mp3 or is_fileobj) and torchaudio_supports_ffmpeg():

method	time	CPU%
without ThreadPoolExecutor	4 hours	100%
with ThreadPoolExecutor, num_jobs=1	4 hours	100%
with ThreadPoolExecutor, num_jobs=2	2.3 hours	200%
with ThreadPoolExecutor, num_jobs=4	1.5 hours	350%
with ThreadPoolExecutor, num_jobs=8	50 minutes	600%
with ThreadPoolExecutor, num_jobs=16	40 minutes	800%

[pzelasko]

Thanks for reporting this @yfyeung. AFAIK this mechanism was introduced becausetorchaudio.info would return invalid duration/num samples for some mp3 files and/or file objects. I think maybe the best solution here is to introduce an env var or top-level API (such aslhotse.set_ffmpeg_torchaudio_info_enabled(True/False) that lets the user control the behavior. We could disable it at the beginning of CV recipe and re-enable at the end (context manager would be great for this). Would you or @desh2608 be willing to make that change?

BTW with ProcessPoolExecutor instead of ThreadPoolExecutor you may be able to observe more linear speedup.

[pzelasko]

This shouldn't be in the global scope, because importing this file is going to have a side effect. Please move this function call to the place just before RecordingSet is created inside prepare_commonvoice, and then re-enable it just after the RecordingSet is constructed.

[desh2608]

I am thinking perhaps we can make these functions possible to be used with a context manager? Would be more "pythonic" I feel.

[pzelasko]

+1

[yfyeung]

@desh2608 @pzelasko This is finally ready for review. Could you please re-check the code? Thx.

[pzelasko]

LGTM, thanks!