Champion "Async Streams" (including async disposable) (16.3, Core 3)

[gafter]

• Proposal added: https://github.com/dotnet/csharplang/blob/master/proposals/csharp-8.0/async-streams.md
• Discussed in LDM
• Decision in LDM
• Finalized (done, rejected, inactive)
• Spec'ed

See dev spec: https://github.com/dotnet/roslyn/blob/main/docs/features/async-streams.md
See also dotnet/roslyn#261

LDM notes:

• LDM 2017-08-30
• LDM 2017-01-18
• LDM 2015-09-08
• https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-05-21.md
• https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-10-03.md (interface shape)
• https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-10-15.md (syntax)
• https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-11-28.md#async-iterators-and-await-foreach
• https://github.com/dotnet/csharplang/blob/master/meetings/2018/LDM-2018-12-12.md#cancellation-in-getasyncenumerator (CancellationToken, pattern-based binding)
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-01-09.md#async-streams-getasyncenumerator-revisited
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-01-16.md (pattern-based disposal)
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-03-25.md#cancellation-tokens-in-async-streams (cancellationToken variable)
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-04-15.md (cancellation token using attribute)
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-04-22.md#defaultcancellationattribute-and-overridinghidinginterface-implementation (cancellation token using attribute)
• https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-04-29.md (cancellation, local functions)

[gafter]

This would be something like

interface IAsyncEnumerable<T>
{
        IAsyncEnumerator<T> GetEnumerator();
}
interface IAsyncEnumerator<T>: IAsyncDisposable
{
        Task<bool> MoveNextAsync();
        T Current { get; }
}

With the following elements yet to be designed

• IAsyncDisposable
• async foreach loops
• Declaring async iterator methods
• Some kind of query expression support

[clairernovotny]

@gafter I thought we settled on

interface IAsyncEnumerable<T>
{
        IAsyncEnumerator<T> GetEnumerator(CancellationToken cancellation);
}

And potentially

static class AsyncEnumerableExtensions
{
    static IAsyncEnumerator<T> GetEnumerator(this IAsyncEnumerable<T> source) => source.GetEnumerator(CancellationToken.None);
}

[juepiezhongren]

After the appearance of "IAsyncEnumerable" in Azure Service Fabric, we want more.........
We want the entire async implementation of .net collection BCL part!!!!!!
AsyncLinq: SelectAsync... AsyncParalellLinq: AsAsyncParalell... await yield!!!!!

As ecmascript is proposing with generator async, there is to be the fulfilment in .net!
And with what i have mentioned being implemented, .net(C# and F#) is going to be the most developer-friendly language for big-data, and service-fabric is gonna boosting a completely new ecosystem with a firm asynchronous programming foundation.

wish these come true earlier!!!!

[oliverjanik]

Any news on this? Seems this interface has been defined in several places already (IX, EF7)

[benaadams]

As brought up in roslyn thread

interface IAsyncEnumerator<T>: IAsyncDisposable
{
    Task<bool> MoveNextAsync();
    T Current { get; }
}

Has issues with parallelazation/concurrency/atomicity which is a desirable property of an async system (e.g. double gets on Current or missed items on Current)

[IvanKonov]

Maybe then:

interface IAsyncEnumerator<T> : IAsyncDisposable
{
    Task<bool> GetNextAsync(out T next);;
    Task<T> Current { get; }
}

[HaloFour]

An async method can't have out parameters.

[IvanKonov]

Another version:

interface IAsyncEnumerator<T> : IAsyncDisposable
{
    Task<(bool has_value, T next)> GetNextAsync();
    Task<T> Current { get; }
}

[clairernovotny]

How about just adding out to async methods? That would make this far cleaner

[vladd]

@IvanKonov This definition would make IAsyncEnumerator<T> not covariant by T.

[benaadams]

An async method can't have out parameters.

but

Task<bool> GetNextAsync(out T next);

isn't async; its just Task<bool> returning, and you can await with out params so no problems? 😉

async Task DoThing()
{
    var enumerator = new Enumerator<int>();
    while(await enumerator.GetNextAsync(out var next))
    {
        // Do thing with next
    }
}

struct Enumerator<T>
{
    public Task<bool> GetNextAsync(out T next)
    {
        next = default(T);
        return Task.FromResult(false);
    }
}

[yaakov-h]

Doesn't the out param have to be set before the task is returned? At that point, the task will most likely not be completed, so you won't have a "next" to pass as the out param. Then, by the time you have your next, it's too late to fill the out param.

[benaadams]

Doesn't the out param have to be set before the task is returned?

😢 I'll go with @onovotny's suggestion instead then #43 (comment)

[stephentoub]

Has issues with parallelazation/concurrency/atomicity which is a desirable property of an async system

While true, if you have a source that supports multiple readers, you can also simply change the model to one where each reader is given its own enumerator. Then the concurrency issues evaporate / are left up to the coordinator that's giving the items out to the individual enumerators.

I'll go with @onovotny's suggestion instead

I don't understand the suggestion. When I do:

Task<bool> t = enumerator.GetMoveNext(out T result);
Use(result);
await t;

how is that intended to work?

covariant

Note that most of this was already discussed in dotnet/roslyn#261.

[clairernovotny]

I was trolling, but what if using out parameters with Task required use of await? Then the assignment order is guaranteed.

[stephentoub]

I was trolling, but what if using out parameters with Task required use of await? Then the assignment order is guaranteed.

a) that would be a breaking change, b) there are actually scenarios where you want to access the out value after the synchronous return of the method, and c) anything can be made awaitable, so task isn't special in this regard.

[yaakov-h]

@stephentoub is there a Cliffs Notes version of that thread?

[stephentoub]

In the near future I'm planning to take some time to write down where we are, which would include summarizing salient points from that thread.

[scottt732]

Have you flushed out how CancellationTokens would work here? Is the intent to explicitly call GetEnumerator... something like foreach (await var item in ae.GetEnumerator(cancellationToken)) { ... } ?

Task<bool> MoveNextAsync(CancellationToken cancellationToken) in the interface feels more consistent, but I gather that would require additional language changes to accommodate in foreach/while, LINQ query/method syntax.

Would it be possible to get the IEnumerable/LINQ and IObservable/Rx extension methods wired up to IAsyncEnumerable? Do IAsyncQueryable/IAsyncQbservable factor in to the plans at all?

[Igorbek]

discriminated unions would be useful here.

[quinmars]

@alrz I guess that would be very tricky. Who will dispose the reader? I think that will only work with buffering of the complete list or with IAsyncEnumerable<T>. The first MoveNextAsync will take sometime because the reader needs to be awaited, the following MoveNextAsync will be fast. For them, the state machine can simply return Task.FromResult(true) or Task.FromResult(false) or even better a cached or global reference of them.

[alrz]

RE "Who will dispose the reader" The iterator.

async Task<IEnumerable<T>> IteratorAsync() {
  var reader = await ExecuteReader();
  IEnumerable<T> Iterator() {
    using (reader)
      while (reader.Read())
        yield return RowParser(reader);
  }
  return Iterator();
}

It's not always easy to workaround this, would be nice if the compiler can see if the iterator is not awaiting and permit Task<IEnumerable<T>> as the return type.

[quinmars]

@alrz I see. But it's rather unnoticeable, that you will leak resources if you do not use the IEnumerable<T> once.

[bbarry]

If the alternative interfaces are used, I don't think there is any benefit to permitting Task<IEnumerable<T>> (maybe 1 virtual call? If the pattern form was used they could potentially even be inlineable).

[TylerBrinkley]

I was watching the Build demo of this feature and was wondering how we can specify .ConfigureAwait(false) for async disposal and async enumeration?

[TylerBrinkley]

Nevermind, it appears this is being considered in the proposal.

[svick]

From May 21 2018 LDM notes:

foreach await over dynamic

Block it. For synchronous foreach we resort to the nongeneric IEnumerable, but there is no nongeneric IAsyncEnumerable, and there won't be.

Since the IAsyncEnumerable interface is going to be covariant, would it make sense if foreach await over dynamic worked with IAsyncEnumerable<object>? It would work only for async enumerables of reference types, but maybe that's better than nothing?

[Neme12]

How will async disposables interact with #1174? Will there be an async form of that?

[masonwheeler]

• Proposal added: https://github.com/dotnet/csharplang/blob/master/proposals/async-streams.md

This link at the top of the document returns a 404

[stephentoub]

Fixed. It moved into the C# 8 folder.

[IanKemp]

Should this be closed now? I was under the impression it's completed.

[BreyerW]

Should this be closed now? I was under the impression it's completed.

feature itself is done but they close such issues only when "paperwork" is also done (spec must be written if memory serves right)

[333fred]

Correct. We'll close this when a spec is merged.