Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added parallel tips #109

Merged
merged 2 commits into from
Oct 1, 2018
Merged

added parallel tips #109

merged 2 commits into from
Oct 1, 2018

Conversation

zsunberg
Copy link
Collaborator

This adds some tips to the readme to help people with parallel computing (e.g. #96 and #97). The new text is as follows:

Tips for parallel programming

When multiple processes or tasks are being used for a computation, the workers should communicate back to a single task for displaying the progress bar. This can be accomplished with a RemoteChannel:

using ProgressMeter
using Distributed

p = Progress(10)
channel = RemoteChannel(()->Channel{Bool}(10), 1)

@sync begin
    # this task prints the progress bar
    @async while take!(channel)
        next!(p)
    end

    # this task does the computation
    @async begin
        @distributed (+) for i in 1:10
            sleep(0.1)
            put!(channel, true)
            i^2
        end
        put!(channel, false) # this tells the printing task to finish
    end
end

@zsunberg zsunberg merged commit 7b8ad52 into timholy:master Oct 1, 2018
@zsunberg zsunberg deleted the parallel_help branch October 1, 2018 16:36
@rjkat
Copy link
Contributor

rjkat commented Sep 12, 2019

Hi. Given that this is in the help, is there any chance we could make this into a macro in the package? I find that I need @distributed more often than pmap, and often I want to reduce the results. Something like

# equivalent of @showprogress for a distributed for loop with a reducer
macro showprogressdistributed(args...)
    if length(args) < 2
        throw(ArgumentError("@showprogressdistributed requires at least 2 arguments"))
    end
    progressargs = args[1:end-2]
    reducer = args[end-1]
    loop = args[end]
    if loop.head !== :for
        throw(ArgumentError("malformed @showprogressdistributed loop"))
    end
    var = loop.args[1].args[1]
    r = loop.args[1].args[2]
    body = loop.args[2]

    quote
        n = length($(esc(r)))
        p = Progress(n, $([esc(arg) for arg in progressargs]...))
        ch = RemoteChannel(() -> Channel{Bool}(n))
        meter = @async while take!(ch) next!(p) end

        job = @async begin
            results = @distributed $(esc(reducer)) for $(esc(var)) in $(esc(r))
                x = $(esc(body)) 
                put!(ch, true)
                x
            end 
            put!(ch, false)
            results
        end

        results = fetch(job)
        wait(meter)
        results
    end
end

Then the example above would become

using Distributed
using ProgressMeter
result = @showprogressdistributed (+) for i in 1:10
    sleep(0.1)
    i^2
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants