This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 6.8k
[v1.x] provide a faster PrefetchedDataLoader #19748
Merged
Merged
Commits on Jan 13, 2021
-
provide a faster PrefetchedDataLoader
Now, since my programming skill is very poor, this `PrefetchedDataLoader` only allow generate a single iter at the same time. the benefit of `PrefetchedDataLoader` is that, `PrefetchedDataLoader` provides better performance with a simple replacement in most of the existing codes. test: ```python $ cat iternew.py && python iternew.py import mxnet as mx from mxnet.gluon.data import PrefetchedDataLoader as DataLoader,ArrayDataset from time import sleep,perf_counter_ns train_data=ArrayDataset(mx.nd.array([[i] for i in range(50000)]),mx.nd.array([[99-i] for i in range(50000)])) test_data=ArrayDataset(mx.nd.array([[i] for i in range(10000)]),mx.nd.array([[99-i] for i in range(10000)])) def transform_train(sample): sleep(0.0016) return sample def transform_test(sample): sleep(0.0008) return sample train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10) test_iter =DataLoader(test_data .transform_first(transform_test ),batch_size=500,num_workers=10) if True: tic=perf_counter_ns() for epoch in range(10): print("epoch"+str(epoch)+" start at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") for i in train_iter: sleep(0.1) print(" finished train phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") for i in test_iter: sleep(0.05) print(" finished test phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") print("cost="+str((perf_counter_ns()-tic)*1e-9)+"s") epoch0 start at 0.0s finished train phase at 11.25s finished test phase at 12.31s epoch1 start at 12.31s finished train phase at 22.62s finished test phase at 23.68s epoch2 start at 23.68s finished train phase at 34.03s finished test phase at 35.09s epoch3 start at 35.09s finished train phase at 45.41s finished test phase at 46.48s epoch4 start at 46.48s finished train phase at 56.82s finished test phase at 57.88s epoch5 start at 57.88s finished train phase at 68.24s finished test phase at 69.3s epoch6 start at 69.3s finished train phase at 79.65s finished test phase at 80.71s epoch7 start at 80.71s finished train phase at 91.04s finished test phase at 92.11s epoch8 start at 92.11s finished train phase at 102.46s finished test phase at 103.53s epoch9 start at 103.53s finished train phase at 113.89s finished test phase at 114.95s cost=114.94954171600001s ``` (cost ~`129.67192333600002s` if we are using `Dataloader` rather than `PrefetchedDataLoader`)
Configuration menu - View commit details
-
Copy full SHA for 47eb585 - Browse repository at this point
Copy the full SHA 47eb585View commit details -
provide a faster PrefetchedDataLoader
there already exists some faster dataloader in mxnet 2.0, but in v1.x, the exist dataloader is slower and could be improved by changing its prefetch behavior as what 2.0 have done. ```python $ cat iternew.py && python iternew.py import mxnet as mx from mxnet.gluon.data import PrefetchedDataLoader as DataLoader,ArrayDataset from time import sleep,perf_counter_ns train_data=ArrayDataset(mx.nd.array([[i] for i in range(50000)]),mx.nd.array([[99-i] for i in range(50000)])) test_data=ArrayDataset(mx.nd.array([[i] for i in range(10000)]),mx.nd.array([[99-i] for i in range(10000)])) def transform_train(sample): sleep(0.0016) return sample def transform_test(sample): sleep(0.0008) return sample train_iter=DataLoader(train_data.transform_first(transform_train),batch_size=500,num_workers=10) test_iter =DataLoader(test_data .transform_first(transform_test ),batch_size=500,num_workers=10) if True: tic=perf_counter_ns() for epoch in range(10): print("epoch"+str(epoch)+" start at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") for i in train_iter: sleep(0.1) print(" finished train phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") for i in test_iter: sleep(0.05) print(" finished test phase at "+str(round((perf_counter_ns()-tic)*1e-9,2))+"s") print("cost="+str((perf_counter_ns()-tic)*1e-9)+"s") epoch0 start at 0.0s finished train phase at 11.28s finished test phase at 12.35s epoch1 start at 12.35s finished train phase at 22.73s finished test phase at 23.79s epoch2 start at 23.79s finished train phase at 34.15s finished test phase at 35.21s epoch3 start at 35.22s finished train phase at 45.59s finished test phase at 46.66s epoch4 start at 46.66s finished train phase at 57.01s finished test phase at 58.07s epoch5 start at 58.07s finished train phase at 68.43s finished test phase at 69.5s epoch6 start at 69.5s finished train phase at 79.87s finished test phase at 80.93s epoch7 start at 80.93s finished train phase at 91.3s finished test phase at 92.37s epoch8 start at 92.37s finished train phase at 102.74s finished test phase at 103.8s epoch9 start at 103.8s finished train phase at 114.17s finished test phase at 115.23s cost=115.23376344s ```
Configuration menu - View commit details
-
Copy full SHA for ef9153a - Browse repository at this point
Copy the full SHA ef9153aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7e44bcf - Browse repository at this point
Copy the full SHA 7e44bcfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1c6170c - Browse repository at this point
Copy the full SHA 1c6170cView commit details
Commits on Jan 14, 2021
-
Configuration menu - View commit details
-
Copy full SHA for aee2b3b - Browse repository at this point
Copy the full SHA aee2b3bView commit details -
Configuration menu - View commit details
-
Copy full SHA for f1c40b3 - Browse repository at this point
Copy the full SHA f1c40b3View commit details -
previous test shows that there may be something wrong with the `_MultiWorkerIter` according to the inappropriate __iter__() is called, I tried to fix it by moving the call here.
Configuration menu - View commit details
-
Copy full SHA for 9aa4548 - Browse repository at this point
Copy the full SHA 9aa4548View commit details
Commits on Jan 15, 2021
-
add an auto_reload flag into the dataloader
the added flag is set to `True` rather than the default `False` since in mxnet 2.0, the default `nopython` mode prefetch data and auto reload it.
Configuration menu - View commit details
-
Copy full SHA for 23cbe5a - Browse repository at this point
Copy the full SHA 23cbe5aView commit details -
merge `prefetcheddataloader` into `dataloader`
Configuration menu - View commit details
-
Copy full SHA for 4bfd7ab - Browse repository at this point
Copy the full SHA 4bfd7abView commit details -
Configuration menu - View commit details
-
Copy full SHA for 46bc121 - Browse repository at this point
Copy the full SHA 46bc121View commit details
Commits on Jan 16, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 15f4d6f - Browse repository at this point
Copy the full SHA 15f4d6fView commit details
Commits on Jan 17, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 54c0f38 - Browse repository at this point
Copy the full SHA 54c0f38View commit details -
Configuration menu - View commit details
-
Copy full SHA for ca5315b - Browse repository at this point
Copy the full SHA ca5315bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c8d858 - Browse repository at this point
Copy the full SHA 2c8d858View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0af3b38 - Browse repository at this point
Copy the full SHA 0af3b38View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6ae84bd - Browse repository at this point
Copy the full SHA 6ae84bdView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.