make @sync lexically scoped and merge @schedule with @async

[JeffBezanson]

With the upcoming threading runtime we will likely be adding more parallel constructs. Unfortunately, we have a few too many similar things (@async, @schedule, @threads, @spawn, ...). Confusion between @async and @schedule is particularly common and unnecessary; for example some code in the Sockets stdlib uses @async where it should have used @schedule.

This change improves the situation by removing @schedule. The key change to make this possible is to make @sync lexically scoped. That way, if an @async appears on its own it just spawns a task and nothing else happens. But if it's lexically surrounded by a @sync, then that sync waits for all the async stuff inside.

It looks to me like basically all uses of @sync in the stdlib work with lexical scope. If they don't, it's pretty hard to find that out since dynamic scope is so hard to reason about, giving another reason to make this change :)

[vchuravy]

My one comment would be that it is easier to deprecate @async since that is the behaviour this changes. But in general I don't mind much and I think that @async is the better name.

[quinnj]

Probably worth mentioning the deprecation of @schedule, no?

[samoconnor]

I'm a little worried by the removal of @schedule.

Some piece of code might call the new @async with the intention of starting a long-running background task. That will work fine until someone else adds a second short-lived @async operation in the same function and wraps the whole block in @sync.
This is not too hard to debug if the original long-running task never terminates, because the program will just hang and it shouldn't be too hard to spot the cause. However, if the original long-running task usually runs for a few seconds then terminates the result will be a hard to track down performance problem.

Another potential pitfall is a small function that starts out with a @sync block and a couple of @async tasks. Over time the function grows, and eventually someone decides to refactor the guts into seperate functions (or a future automated refactoring tool does this). A seemingly simple refactoring step "move this chunk of code into a function" will result in code that might work fine some of the time but might fail in subtle ways because the @async task is no longer waited for.

I can sympathise with @JeffBezanson's desire to remove unneeded confusion between @async and @schedule, however it might be better to do something like:

• leave @schedule as it was, but maybe only document it under e.g. "Tasks - Advanced",
• make it an error to use @async without an enclosing @sync block.

Another possibility would be to remove @sync and have

tasks = []
push!(tasks, @async begin
    ...
end)
push!(tasks, @async begin
    ...
end)
wait(tasks)

[iamed2]

I think that's an important concern.

Bringing back schedule but keeping the new behaviour for async seems like a good idea to me.