add multi-proc in to_csv

[bhavitvyamalik]

This PR extends the multi-proc method used in #2747 forto_json to to_csv as well.

Results on my machine post benchmarking on ascent_kb dataset (giving ~45% improvement when compared to num_proc = 1):

Time taken on 1 num_proc, 10000 batch_size  674.2055702209473
Time taken on 4 num_proc, 10000 batch_size  425.6553490161896

Time taken on 1 num_proc, 50000 batch_size  623.5897650718689
Time taken on 4 num_proc, 50000 batch_size  380.0402421951294

Time taken on 4 num_proc, 100000 batch_size  361.7168130874634

This is a WIP as writing tests is pending for this PR.

I'm also exploring this approach for which I'm using pyarrow-5.0.0.

[lhoestq]

Thanks ! This is going in the right direction :)

I think it's better if we stick with the pandas CSV writer rather than Arrow's, for consistency with JSON but also because the pandas one may be more mature

[lhoestq]

I think you can just add a test test_dataset_to_csv_multiproc in tests/io/test_csv.py and we'll be good

[bhavitvyamalik]

Hi @lhoestq,
I've added test_dataset_to_csv apart from test_dataset_to_csv_multiproc as no test was there to check generated CSV file when num_proc=1. Please let me know if anything is also required!

[lhoestq]

Looks all good now, thanks @bhavitvyamalik @mariosasko :)