Stackless coroutines

[vadimcn]

Given resurgence of interest in async IO libraries, I would like to re-introduce my old RFC for stackless coroutine support in Rust, updated to keep up with the times.

The proposed syntax is intentionally bare-bones and does not introduce keywords specific to particular applications (e.g. await for async IO). It is my belief that such uses would be better covered with macros and/or syntax extensions.

Rendered

[nagisa]

I’d like to see some elaboration on how dropping happens in case a panic inside a coroutine occurs. Especially interesting to my currently sleep deprived brain is such case:

yield a;
drop(b);
panic!("woaw");
drop(c);
return d;

where all of a, b, c and d are moved.

[erickt]

The drop semantics are identical to enum variant drop semantics. Variables are then just lifted up into a variant, as in:

enum State {
    bb1 { x: usize },
    bb2 { x: usize, y: usize }
    ...
}

loop {
    match state {
        bb1 { x } => { ... state = bb2 { x, y } }
        bb2 { x, y } => { ... }
    }
}

A drop in bb1 naturally drops on the variable in scope. This is what I do in stateful and the semantics all seem to work out quite well.

[plietar]

@nagisa a way of handling it would be an empty "Panic" variant. Just after each yield point, the locals are restored from the state enum, and the enum is set to this "Panic" variant.
If an actual panic happens the locals are dropped as usual, and the state enum is empty, so everything is fine. Otherwise the new state is stored at the next yield. Sort of like the classic option dance.

Invoking a coroutine which has been left in the Panic variant just panics again.

[nagisa]

MIR-speak for this is Assert terminator with false for cond.

[nagisa]

It also seems to me that a somewhat builder-like pattern could be used to avoid this failure case. Namely, if coroutine instead was:

enum CoResult<Y,R,C> {
    Yield<Y, C>,
    Return<R>
}

and the coroutine closure was required to be passed around via value, then you’d evade the issue with cleanups on panic within a coroutine and also would never be able to “call” coroutine again after it has already returned.

[glaebhoerl]

The same thing applies to Iterators, which could've returned Option<(Self, Self::Item)> instead of Option<Self::Item>. I'm not sure if that's why we didn't do it, but two drawbacks are that it's less ergonomic to use manually, and that it precludes object safety.

Here, I think a third drawback would be that, as a consequence, it would preclude a blanket Iterator impl.

(I think it would have been a better idea to have done this for Iterator as well in the first place, but...)

[Ericson2314]

I regret this not being done. I think this would have been the killer usecase for #1736 .

[Ericson2314]

If we had &move, we could avoid the object safety issues..... :/

[tomaka]

Can a regular function be a coroutine as well?

For example if the async I/O example was much larger, I'd like to extract some code from the closure and put it in a function. Maybe this can be solved by making that function return a coroutine itself, although that would be confusing.
But if a coroutine calls a coroutine, can the inner coroutine yield to the outside of the outer coroutine?

Other than that I've seen an alternative proposal on IRC which has some advantages and drawbacks compared to this one.

[ahicks92]

With the caveat that I'm massively multitasking, a few things:

• Regular functions need to be able to be coroutines, or to return coroutines. I prefer the latter because it can probably be done with impl trait, provided that a coroutine implements a coroutine trait of some description. This gives us the benefit of being explicit, one of the motivating factors behind things like async/await.

• I like the coro keyword, for the same reason as above. Explicitness is better.

• Some thought should go into how this will interact with things like @withoutboats's RFC for associated type constructors. if we get the ability to express streaming iterators, we also get the ability to let coroutines return references to locals.

• The RFC should go ahead and say that it's going to make parameterless coroutines iterators, not just say that we could. I think this is pretty uncontroversial, most other languages do it, and implementing your own iterators is currently a major pain point of Rust.

• I like the solution for passing values into coroutines.

• We may wish to consider adding something like Python's yield from to delegate to other coroutines.

• This looks like a place where my work on struct layout optimization could benefit everyone.

• Finally, the initial implementation should absolutely figure out how to not keep extraneous variables around if at all possible, as it is harder to add these optimizations later and we definitely will want it as far as I can see. Reordering structs was incredibly expensive, and it seems to me that leaving this off and doing it later will also be expensive as compared to doing it to begin with.

[ranma42]

I like this proposal, but I would like to explore what happens pushing it even further: I believe we could just merge the function closure and coroutine concepts.
IIUIC if the type of the state field was an enumeration with one value for the initial state and a different value for each yield point, the very same lowering could apply just fine to functions closures and coroutines.

[krdln]

This information is quite crucial and I think it would be easier to read the RFC if somewhere near the beginning there was a sentence like “Coroutine is compiled to a closure which implements FnMut(Args...) -> CoResult<Y, R> trait. For example, the coro1 coroutine is an impl FnMut() -> CoResult<i32, ()>”.

[vadimcn]

@tomaka, @camlorn, @ranma42: Regular functions cannot be merged with coroutines. Coroutines need to keep their state somewhere (the "coroutine environment"), so a regular function pointer cannot point to coroutine.

Coroutines can be merged with closures, and this is exactly what I am proposing. In fact, to the outside observer a coroutine is not distinguishable from a "normal" closure that just happens to return CoResult.

And yes, regular functions can return coroutines: see all the examples at the end.

[vadimcn]

@camlorn

We may wish to consider adding something like Python's yield from to delegate to other coroutines.

Python's yield from returns reference to the inner generator up the call chain, till it reaches the code doing iteration over the outermost generator and so avoids doing multiple iterations over the same data. In Python this works, because the language transparently handles iteration over an iterator returned in place of a regular value.

To do this in Rust, we'd have to add a third variant into CoResult:

enum CoResult<'a, Y: 'a, R> {
    Yield(Y),
    YieldFrom(&'a Iterator<Item=Y>),
    Return(R)
}

and callers would have to deal with this new variant explicitly.
The coroutine signature would become something like for<'a> FnMut() -> CoResult<'a, Y, R>

IMO, this isn't worth it. Inlining should take care of the trivial inner loops.

[ahicks92]

My point about yield from was more along the lines of the compiler just writing the inner loop, though I do understand that this can be done with a macro. The interesting question is whether such a construct opens up optimization opportunities, for example letting us flatten multiple coroutines into one, and if those opportunities are meaningful.

[eddyb]

@vadimcn I'm confused, yield* in ES6 creates an inner loop (since it works like Rust iterators).
Does Python do something else as some sort of optimization?
"multiple iterations over the same data" only makes sense if there's a .collect() in between.

[eddyb]

In fact, to the outside observer a coroutine is not distinguishable from a "normal" closure that just happens to return CoResult.

This is a very important observation and should be the centerpiece of the proposal IMO.
Everything else I've seen introducing a new kind of state machine that implements some trait, whereas this extends closures themselves, like going from struct to enum.

I'm slightly worried the ergonomics of generators might suffer with this model, but I like it.
Not to mention that it actually allows multityped (or even generic!) yield because Fn is generic over the arguments, so I don't know of any generator feature that can't work with it.

I'm also wary about a typing ambiguity with tuples, for example one (T, U) argument and two arguments T and U, they both have a yield typed (T, U).
This is not trivially a problem in practice AFAICT, because of the syntactical argument listing of the closure, but I feel like I'd prefer it to be limited to one argument, at least initially.

[vadimcn]

I'm confused, yield* in ES6 creates an inner loop (since it works like Rust iterators).

Not sure how it works in ES6.

Does Python do something else as some sort of optimization?
"multiple iterations over the same data" only makes sense if there's a .collect() in between.

I was referring to this. I remember reading a longer piece about optimizing nested yields several years ago (not even sure it was about Python), but all my searches come back empty so far. :( Maybe I'll find it later.

Edit: If yield from were just a syntax sugar for a loop, I'd rather implement it as a macro.

[eddyb]

@vadimcn Yeah, that's an optimization for the fact that there is no optimizing compiler.
I wonder if pypy has any gains from it over the "naive" implementation.

[vadimcn]

I'm also wary about a typing ambiguity with tuples, for example one (T, U) argument and two arguments T and U, they both have a yield typed (T, U).

The former one would be typed ((T, U)), wouldn't it? Also, we could do this.

[eddyb]

@vadimcn Huh, have a I seen a draft that used T instead of (T,) for single-type?
Re-binding would be a PITA wrt pattern-matching and destructors, but I can see it work well enough.

[erickt]

@vadimcn: Nice! This is very much along the lines of what I was exploring in stateful, which never was supposed to be a long term solution to this problem. MIR is definitely the way to go. A couple observations:

• You should explicitly mention that temporaries will need to be lifted into the state, in order to work with RAII like patterns, suck as Mutex::Lock.
• I believe the proper semantics for await should be this:

macro_rules! await {
    ($e:expr) => ({
        let mut future = $e;
        loop {
            match future.poll() {
                Ok(Async::Ready(r)) => break Ok(r),
                Ok(Async::NotReady) => yield,
                Err(e) => break Err(e.into()),
            }
        }
    })
}

In order to let the coroutine to handle errors if it so chooses.

• You should either spell out that yield is equivalent to yield (), or replace the yield with yield ().
• We could also build coroutines on top of tailcalls, or if we added state machine support. I personally think this transformation is fine (and is almost exactly what I'm doing in stateful), but I'm not 100% sure if there's a more foundational technology we should be using instead.
• We could have shorter syntax sugar for generators and async functions that are built upon coroutines, like gen fn foo() { yield 1 } or async fn foo() { let x = await(future)?; ... }.
• Is it worthwhile having traits like this?

enum CoResult<Y, R> {
    Yield(Y),
    Return(R),
}

trait Coroutine {
    type Yield;
    type Return;
    fn resume(&mut self) -> CoResult<Self::Yield, Self::Return>;
}

trait CoroutineRef {
    type Yield;
    type Return;
    fn resume<'a>(&'a mut self) -> CoResult<&'a Self::Yield, &'a Self::Return>;
}

trait CoroutineRefMut {
    type Yield;
    type Return;
    fn resume<'a>(&'a mut self) -> CoResult<&'a mut Self::Yield, &'a mut Self::Return>;
}

• If we do get Coroutines, all functions are coroutines. Should functions automatically implement the Coroutine trait
• @nikomatsakis had some thoughts on ways to represent pinned values.
• I used to think yield from wasn't necessary (since it naturally falls out of the state machine), but it could be handy as a mechanism to pass resume values into sub-coroutines. It doesn't need to be represented in the state machine, but it could just be syntax sugar.
• It feels a little odd to me that Coroutine implements FnMut.

[vadimcn]

@erickt:

You should explicitly mention that temporaries will need to be lifted into the state

I thought I did? here

I believe the proper semantics for await should be this: [...]

Good idea!

You should either spell out that yield is equivalent to yield (), or replace the yield with yield ().

Yes, I sorta implicitly assumed we'd do that (unless there are parsing problems).

We could also build coroutines on top of tailcalls, or if we added state machine support [...]

I am afraid you lost me here...

We could have shorter syntax sugar for generators and async functions [...]

I'm thinking this should be doable with syntax extensions:

#[gen]
fn foo() -> impl Iterator<Item=i32> { yield 1; }
#[async]
fn foo() -> impl Future { let x = await(future)?; ...

...But do we really need that? Feels a bit like parroting C# and Python.

Is it worthwhile having traits like this? [...]

What would these buy us over FnMut?

[ranma42]

@vadimcn sorry, when I wrote "function" in the previous message I actually meant "closures".
In several places the RFC explicitly distinguishes between closures and coroutines; for example in the translation the RFC mentions that "Most rustc passes stay the same as for regular closures."

You mention that to an outside observer they will be the same. I am suggesting that we might also try to make them look the same from the point of view of the compiler.

[tomaka]

Regular functions need to be able to be coroutines, or to return coroutines. I prefer the latter because it can probably be done with impl trait, provided that a coroutine implements a coroutine trait of some description. This gives us the benefit of being explicit, one of the motivating factors behind things like async/await.

Something like this I guess?

fn main() {
    some_lib::do_something(|| -> CoResult<u8, String> {
        yield 12;
        foo();
        "hello world".to_owned()
    });
}

fn foo() -> impl FnOnce() -> CoResult<u8, ()> {
    || {
        yield 54;
    }
}

I don't understand how that would work.
As far as I can see, the closure inside main would just build and immediately drop the coroutine of foo without executing it. And you can't yield the second coroutine either, because you can only yield u8s in that example.
Not that yielding the second coroutine would be a good solution either, because that'd add an overhead for something that shouldn't have one.

Also I don't think this syntax would obtain the usability seal of approval.

[eddyb]

@tomaka Instead of foo(); you need the equivalent of ES6 yield* foo(); or Python's yield from.

[vadimcn]

@tomaka: what @eddyb said, or simply for x in foo() { yield x; }. If foo was meant to return an iterator, I don't see anything wrong with using a for loop.

I would also declare foo like so:

fn foo() -> impl Iterator<Item=u8>
    || {
        yield 54;
    }
}

[tomaka]

@vadimcn What I had in mind was async I/O.

A more real-world example would be this:

let database = open_database();

server.set_callback_on_request(move |request| {
    move || {
        if request.url() == "/" {
            yield* home_page(&database)
        } else if request.url() == "/foo" {
            yield* foo_route(&database)
        } else {
            error_404()
        }
    }
});

fn home_page(database: &Database) -> impl Coroutine<(), Response> {
    move || {
        let news = await!(database.query_news_list());
        templates::build_home_page(&news)
    }
}

fn foo_route(database: &Database) -> impl Coroutine<(), Response> {
    move || {
        let foo = await!(database.query_foos());
        let file_content = await!(read_file_async("/foo"));
        templates::build_foo(&foo, &file_content)
    }
}

fn error_404() -> Response {
    templates::build_404()
}

Usually I'm not the first one to complain about an ugly syntax, but here it looks ugly even to me.

I'm also not sure how async I/O would work in practice. In your example there's no way to wake up a coroutine when an async I/O has finished.

If the idea is to yield objects that implement Future, then you need boxing, which isn't great and may not even work if multiple futures return different types of objects.

I saw a proposal on IRC the other day where you were able to pass a "hidden" parameter (in the sense that it doesn't appear in the arguments list) when executing a coroutine, and you could await on a value foo only if the hidden parameter implements Await<the type of foo>. Calling await would then modify the state of the hidden parameter, for example by adding an entry that tells when the coroutine should be waken up. I found that it was a good idea.

[petrochenkov]

@vadimcn @eddyb
There's some kind of controversy currently happening in C++ world regarding "suspend-up" vs "suspend-down" approaches to coroutines and their standardization.
How does this proposal fits into the picture described in the linked paper?
(I'm not prepared to discuss this myself in detail, just posting some possibly relevant link.)

[Amanieu]

@petrochenkov As far as I understand, suspend-up refers to stackless coroutines while suspend-down refers to stackful coroutines.

[plietar]

While I like the overall design of the RFC, I'm worried that the presence of a yield statement changes the behaviour of return statements, and the return type of the closure.

Compare for example a once iterator vs an empty one :

fn once<T>(value: T) -> impl Iterator<Item = T> {
  move || {
    yield value;
    return ();
  }
}

fn empty<T>() -> impl Iterator<Item = T> {
  || {
    return CoResult::Return(());
  }
}

Here the empty iterator needs to return CoResult::Return(()) instead of a simple (), since it is not detected as a coroutine.

Essentially, I think the distinction should be explicit, since a coroutine which happens to never yield should be valid. Something like coro || { ... }.

[ahicks92]

@plietar
Agreed. You also come close to a point which I would like to raise: making functions that take and return coroutines is really ugly right now.

Proposal: Come up with and implement coroutine traits, then make a shorthand for constraining a type parameter to be a coroutine. Introduce the coro keyword, so that coro fn is a coroutine function and coro || is a coroutine closure.

I like yield from the more I think about it, specifically because this would allow embedding the fields of the yielded-from closure into the parent. The more we can do this, the more likely it is that my field reordering PRs can kick in. I don't know how much this would save in practice, but it may be significant. Also, on this issue I am biased.

@erickt
It looks like your CoroutineRef and CoroutineRefMut are trying to be streaming iterators or something to that effect. Will this work now? I thought this needed another inference rule or something, otherwise we'd have streaming iterators already.

[Zoxc]

Does anyone think we should not have immovable generators based immovable types? Immovable types RFC

An alternative to immovable types is to allocate generators on the heap (which is C++'s solution). This turns out to be quite messy however, especially when dealing with allocation failure. It doesn't guarantee 1-allocation per task (in futures-rs terms) like immovable types does, but most allocations can be optimized away.

Immovable generators would allow references to local variables to live across suspend points. This almost recovers normal rules for references. If the resumption of generators can take for<'a> &'a T as an argument, it cannot cross suspend points because the resume function isn't allowed to capture it. If we have arguments to the generators be implicit (like in my generators RFC) we may be able to hide this oddity a bit. I'm not sure how we should access this implicit argument. I'm open to idea on this.

We could also restrict argument to generators to 'static avoiding the issues with references. I would prefer a more flexible solution though. Specifically for the use case of passing down &mut EventLoop in asynchronous code. See my generators RFC for examples of this.

[mark-i-m]

Also, this raises an interesting point: With closures, it seems that the divide is how self is taken. With generators, it seems to be mobility. Maybe there should be two generator traits: Gen (immovable) and GenMov (movable)?

[nikomatsakis]

So, the discussion on this thread is still fairly active, but my feeling is that we're probably not ready to accept this RFC yet. I tend to think it should be postponed, with conversation moving onto internals thread, and perhaps with some nice summaries capturing the highlights of this conversation and others in the related area.

That said, I'm basically way behind on this thread right now, and perhaps I am wrong. Does anyone think I am totally off-base in wanting to postpone?

[Zoxc]

@mark-i-m You can have immovable and movable generators which both implement the same trait Generator: ?Move

An more flexible and complex alternative is to have immovable generators return a movable value which implements a trait GeneratorConstructor. This value can then be used to construct the real immovable generator. Given how flexible the immovable types RFC is, that may not be desired.

[mark-i-m]

@Zoxc

@mark-i-m You can have immovable and movable generators which both implement the same trait Generator: ?Move

This means that all types of generators must have the same method signatures, though... right? In other words, having different traits means that you can have different method signatures, just as with closures. I don't know if this is the right approach for generators, though... I don't really understand how we could get the same effect with trait Generator: ?Move.

An more flexible and complex alternative is to have immovable generators return a movable value which implements a trait GeneratorConstructor. This value can then be used to construct the real immovable generator. Given how flexible the immovable types RFC is, that may not be desired.

👍 This seems very much like an idea presented way earlier in the thread (but I cannot find it) by someone else: have each yield produce both a yield value and the next value of the generator itself. Personally, I find this more elegant than having a generator with mutable state hanging around that may or may not move.

---

@nikomatsakis This thread has a lot of interesting discussion, but I don't think is really conclusive in any direction. It is a big discussion of different design alternatives from syntax, to underlying implementation, to immovability, to typing. I think I have kept up with the thread more or less, and I would also really love a(nother) summary of the different points.

[ayosec]

I tend to think it should be postponed, with conversation moving onto internals thread, and perhaps with some nice summaries capturing the highlights of this conversation and others in the related area.

+1

I guess that a proof of concept of the feature can be implemented as a procedural macro when Macros 1.2 is ready.

[bjorn3]

@ayosec A proof of concept can be created with a nightly compiler plugin already.

[vadimcn]

@Zoxc:

An more flexible and complex alternative is to have immovable generators return a movable value which implements a trait GeneratorConstructor. This value can then be used to construct the real immovable generator.

Can you please explain in more detail, how this would work? That returned value would still need to contain both the borrowed object and the reference, so how would it stay movable?

@nikomatsakis: I'll try to come up with a summary soon. But yeah, there's a lot of people unconvinced about various aspects of this design. I think it will take a proof-of-concept implementation to show that this is viable.

[mark-i-m]

@vadimcn

That returned value would still need to contain both the borrowed object and the reference

Would it? You could conceive of a system in which the generator-generator does not contains self-references but only metadata. When the generator is later constructed, the metadata is consumed to produce an immovable generator (immovable because it now actually contains self-references), which can then be consumed to produce the next generator-generator and a value.

It does sound like a rather elaborate system though...

[vadimcn]

It does sound like a rather elaborate system though...

Well, yes. That's why I'd like to hear more.

[Zoxc]

@mark-i-m There isn't a need for different signatures for movable and immovable generators. A single trait suffices. However having generators returning a new generator when resumed is incompatible with immovable generators.

@vadimcn The generator-generator would only need to contain the upvars (like regular closures) and wouldn't have a storage for values crossing suspend points. So it will remain movable until we construct the proper immovable generator which does have this storage.

An issue I ran into when playing with an implementation of generators is figuring out when an generator implements OIBITs. For example, our generator can't implement Send if some Rc crosses the suspend point. Whether or not an OIBIT is implemented depends on the types of values that lives across suspend points. Figuring out these values is a hard since they in turn can contain references to more values. In the presence of unsafe code, we may have to assume all values that have their address taken may live across suspend points. Movable generators resolve this issue by banning such references.

An related issue is the layout of generators. The same set of values decides the size of the generator. We really want to run MIR optimizations on generators before committing to a layout. If we allow size_of to be used at compile-time, we'd need the ability to compute layout during type checking, but the layout of generators might depend on type checking!

[nikomatsakis]

@rfcbot fcp postpone

Based on the last few comments, I am going to move that we postpone this RFC. It seems clear that there isn't quite enough consensus to adopt anything just yet, and more experimentation is needed. I think it would be very helpful if someone could try to produce a summary on internals of the many points raised in this thread, and we can continue the conversation there.

[rfcbot]

Team member @nikomatsakis has proposed to postpone this. The next step is review by the rest of the tagged teams:

• @aturon
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[pnkfelix]

super nit: this should be |mut which_end: IterEnd|, right?

[vadimcn]

So, let me try to cap off the discussion by enumerating outstanding issues and where I stand on them.

Asynchronous Streams

IMO, the biggest issue with this RFC at the moment, is handling of async streams from futures-rs.
To recap, the problem here is that a coroutine implementing an async stream needs to yield two types of values: the first one to signal that it is currently waiting on another async operation to complete, the second -- to return a value when it is finally available.

Some people (@eddyb was first, I think) had proposed adding a third variant, Wait (or Suspend), into CoResult: enum CoResult<Y,R> { Yield(Y), Return(R), Wait }.
This variant would be returned to signal the "not ready" case, while Yield could be used to return values. The syntax used to create this third value might be yield; i.e. yield without any value (which means that yielding "nothing" would have to be spelled explicitly as yield ();).

I agree that this would work nicely for async streams. However... if we abstract ourselves from the needs of futures-rs, this approach leaves a feeling of incompleteness. One might ask, for instance: Why only one extra variant? Why not make this extensible to an arbitrary number of "extra" cases?

So, after some head-scratching, I think I've found a way to implement both futures and async stream generators using the same await!() macro. It does require an extra macro to "yield" a future, but otherwise works out quite nicely, IMO.

The Self-Borrowing Problem

i.e. the inability to hold references to coroutines's "local" variables across a yield point, is the other Big Unresolved Question of this RFC.

This restriction comes about because borrowing of local variables that are hoisted into the coroutine closure is equivalent to having a regular structure, which stores a reference to another part of self. Obviously, such a structure cannot be moved, because that would invalidate the references!

This problem merits a bigger discussion and perhaps a separate RFC, but briefly, I think there are two possible ways we can go here:

Ignore the problem (for now).

I would say that the "Self-Borrowing Problem" is not as acute as it might appear: in most cases the borrowed value will be external to the coroutine and thus will not cause self-borrowing.
In cases where borrowing of a hoisted local is unavoidable, it can be worked around by moving both the borrowed object and the reference to the heap.
I think that coroutines would be still useful, even if we punted on this problem until a better idea comes along.

Make self-borrowing coroutines immovable

We could allow self-borrowing, but then we must make sure that such coroutines do not ever get moved. Well, "ever" is perhaps too strict: we'd still want to be able to encapsulate creation of a coroutine in a function, and thus we must be able to return them. A reasonable compromise would be that a self-borrowing coroutine becomes pinned after its first invocation. Here's an RFC by @Zoxc, that proposes a possible mechanism for doing that.

Btw, here's another approach to making data immovable - without introducing any new features into the language. The downside is that this will conflict with mutable borrowing needed to invoke a FnMut.

Anyhow, let's continue discussion of self-borrowing on Discourse forums (perhaps in this thread ?)

Traits

Some folks feel that FnMut() -> CoResult<Y,R> does not look pretty enough and would prefer that coroutines auto-implement something like this instead:

trait Coroutine<Args> {
    type Yield;
    type Return;
    fn call(&mut self, args: Args) -> CoResult<Self::Yield, Self::Return>;
}

I am leaning towards 👎 on putting this in the language, for the following reasons:

• It is fully isomorphic to FnMut()->CoResult<...>, and so doesn't add anything new.
• The FnMut()->CoResult<...> will almost never appear in signatures that "normal" people have to deal with. What will show up in method signatures will be the various traits that coroutines implement, such as impl Iterator<Item=XXX>, impl Future<Item=XXX, Error=YYY> and so on.
• I see no reason to disallow regular closures whose signature matches the right pattern from acting as coroutines.
• If desired, the Coroutine trait is trivially implementable as library code.

Coroutine Declaration Syntax

Some people feel that inference of "coroutine-ness" from the presence of yield statements in the body of the closure is too magic.

However, coming up with a pleasant and concise syntax proved to be not so easy, mainly because prefixing any unreserved keyword to a closure is ambiguous with <variable> <logical or> <code block> sequence, which currently is a valid (if unlikely) Rust syntax.

I've come to agree that we need some way of forcing a closure into being a coroutine. If nothing else, this is needed for writing reliable macros, otherwise macro writers would have to perform considerable gymnastics to handle the "degenerate coroutine" (i.e. one that never yields) case.

That said, I also think it is fine to make such declarations optional. The experience of other languages (C# and Python) shows that inferring "coroutine-ness" from presence of yield does not cause any significant confusion among users.

A very basic solution for the "macro problem", proposed earlier in this thread, might be this:

|| { ...; return <something>; yield panic!(); }

Top-Level Generator Functions

Some people would like to have syntax sugar for declaring a top-level function returning a coroutine, e.g.

fn* range_iterator(lo:i32, hi:i32) -> yield i32 {
    for i in lo..hi { yield i; }
}

instead of

fn range_iterator(lo:i32, hi:i32) -> impl Iterator<i32> {
    move || {
        for i in lo..hi { yield i; }
	}
}

Again, I am skeptical that we need this, because:

• The overhead of writing move || { } is not so great.
• The second variant is more explicit about what's going on: we are creating a closure that captures the hi and lo variables.
• If desired, generator syntax may be sweetened with a procedural macro, e.g.

#[generator] 
fn range_iterator(lo:i32, hi:i32) ...

"Coroutines"

Some people took issue with the name "stackless coroutines" (and wanted to call these "generators"?). Sheesh.
Well, here's the evidence that I am right and y'all clearly are not: 😁

• boost.Coroutine documentation explains the difference between stackful and stackless.
• The C++ RFC defining roughly the same feature also uses the term "stackless coroutine" (although, they prefer to call the C++ implementation thereof a "resumable function"). They also seem to agree that "a generator" is a subtype of a coroutine that "provides a sequence of values".
• LLVM uses the term "coroutines". (Also, they do not mind using the word "coro", apparently)
• Python documentation also seems to agree that "generators" != "coroutines", but that they are rather a particular incarnation of coroutines.

I rest my case.

impl Clone for coroutines

With respect to cloning, coroutines are not any different than regular closures. When (if) Rust implements cloning for closures, coroutines will get it for free. So - not in scope for this RFC.

[vadimcn]

And here's IRLO thread I've created in case anyone has left anything to say.

Thanks for participation in this discussion, everyone!

[udoprog]

Small typo (no closing parenthesis)

[nikomatsakis]

@withoutboats still waiting on your FCP comment here: #1823 (comment)

@vadimcn thanks for that great summary!

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period is now complete.

[aturon]

I'm closing as postponed, per previous discussion. Be sure to check out the final summary and continue discussion on internals.

Thanks @vadimcn!

[takanuva]

Support for resumable functions in Clang (as of n4649) and on LLVM 4.0 itself seems fine; it can also do really nice optimizations on chained coroutines (see this presentation by Gor Nishanov). Are you considering using these? These semantics worked really nice on C++, and for sure they would be even better in Rust!

[eddyb]

Resumable functions have to use unsafe pointer tricks to pass values in and out.
The closest Rust equivalent is MoveCell and then it'd still impose way too many restrictions to be safe.