[Windows] Use multiprocessing.dummy on Windows for parallelism (plus adding Azure test pipeline.)#251
Conversation
|
Here is the Azure CI result: https://dev.azure.com/seanyen2018/default/_build/results?buildId=386 |
| # This restriction requires downstream tools to be modified to | ||
| # work. Instead, let's fallback to threading wrapper to get | ||
| # around it and still keep the parallelism on Windows. | ||
| import multiprocessing.dummy as multiprocessing |
There was a problem hiding this comment.
What is the impact of using threading for this on Windows? Is the performance still better than without?
There was a problem hiding this comment.
Well... ran some benchmark tests today and in a test case where it has 1000 packages (and each has dummy 1000 <build_depend> nodes), the result looks not good for threading wrapper in general on Windows...
On CPython 2.7:
- Disable parallel: 26 sec
- Using multiprocessing: 7.5 sec
- Using multiprocessing.dummy: 98.5 sec
On Anaconda Python 3.7:
- Disable parallel: 12.8 sec
- Using multiprocessing: 4.8 sec
- Using multiprocessing.dummy: 14.5 sec
(My environment is using Intel® Xeon® Processor E5-1650 v4 6-cores CPU.)
(My test case is test_find_packages_with_large_amount_packages in https://github.com/seanyen-msft/catkin_pkg/blob/windows_port_multiprocessing_disabled/test/test_packages.py)
There was a problem hiding this comment.
The results don't look like this should be merged as is.
There was a problem hiding this comment.
That make senses. Since now we explored some approaches, any pointers what to investigate next at this moment?
|
Closed it because it is very likely |
2 participants
This is an attempt to address the discussion in #250.
The change is to propose using multiprocessing.dummy (which is a threading wrapper) on Windows to avoid the restriction
Safe importing of main module(which requires the downstream tools to be updated to work). This way can keep parallelism but not requires the downstream tools to be modified.Plus, this change adds an Azure pipeline yaml to exercise the test suite on Windows machines.