Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speedup @everywhere #11687

Merged
merged 1 commit into from
Jun 14, 2015
Merged

speedup @everywhere #11687

merged 1 commit into from
Jun 14, 2015

Conversation

amitmurthy
Copy link
Contributor

Current implementation executes two calls - a remotecall for expression execution and a remotecall_fetch to wait for execution completion on each worker. Now there is a single call of remotecall_fetch.

@andreasnoack , the linear slowdown in the first @everywhere 1+1 was because all code paths related to remotecall_fetch was being compiled for the first time and it was being executed serially.
Actually remotecall_fetch was being called as part of the addprocs. The extra time may just have been due to the serial execution of remotecall_fetch against all workers.

cc @JeffBezanson

@andreasnoack
Copy link
Member

Great. I can confirm that this is much faster. On the Slurm managed cluster I now get

nworkers second
4 0.630646
8 0.309988
16 0.31393
32 0.325648
64 0.334107
128 0.374203
256 0.38756
512 0.443842
1024 💣

but the last row could easily be Slurm related and would have to be investigated further.

@ViralBShah
Copy link
Member

👍

@ViralBShah
Copy link
Member

cc @alanedelman

amitmurthy added a commit that referenced this pull request Jun 19, 2015
(cherry picked from commit bf9b491)
ref PR #11687
@tkelman
Copy link
Contributor

tkelman commented Jun 19, 2015

backported in 426c5be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants