Improve FileStream WriteAsync buffering and performance

[stephentoub]

Replaces PR #2802
Fixes #1531

This set of changes improves how FileStream WriteAsync handles buffering, with some significant performance benefits. Two primary issues addressed:

• Buffers were handled relatively poorly with asynchronous writes, often allocating an entirely new internal buffer in order to satisfy a request.
• Flushes were performed synchronously, which meant doing the write asynchronously and blocking for it to complete.

The changes in this PR:

• Avoid ever needing to allocate a secondary buffer
• Enable flushing asynchronously

I tested this out with a variety of small benchmarks that involve writing asynchronously, e.g.

using (FileStream fs = new FileStream("test.bin", FileMode.Create, FileAccess.Write, FileShare.None, 0x1000, useAsync:true))
{
    for (int i = 0; i < 100000; i++)
    {
        await fs.WriteAsync(arr, 0, arr.Length).ConfigureAwait(false);
    }
}

with various arr sizes.

As an example, on my laptop:

• On 4.6, with an arr length of 0x100,such a test took ~3.3 seconds and incurred ~27 gen0 GCs and ~2 gen1 GCs.
• With changes we previously made for .NET Core, prior to this PR the same test took ~2.8 seconds with ~14 gen0 GCs and no gen1s.
• After this PR, this same test took ~0.23 seconds with ~1 gen0 GC.

In other words, for this test, this PR improved perf by ~10x. The biggest benefit comes when writing out smaller than the internal buffer size. When writing out >= the internal buffer size, throughput stayed about the same as before the change, but GCs dropped by ~6x.

The change should incur few behavioral differences. The primary observable difference is that if a WriteAsync occurs and we need to flush the buffer, we're now flushing asynchronously rather than synchronously, but because we need to ensure that Position is correct when WriteAsync returns to its synchronous caller, the subsequent asynchronous write may execute concurrently with the flush. That means that if the flush fails asynchronously, whereas before we wouldn't have issued the subsequent write and updated the position, now we will have. I believe this is a worthwhile change for the benefits.

(There are some subsequent improvements we could look at as well. For example, now that we're never allocating a secondary buffer, it becomes easier to consider using a PreAllocatedOverlapped; we would need to ensure that we only use it when there aren't concurrent writes, but that's not the common use case.)

Thanks to @tymlipari for kicking this off.

cc: @ericstj, @FiveTimesTheFun, @ianhays, @KrzysztofCwalina

[ianhays]

nit: typo - extra 't' after period

[stephentoub]

Will fix

[ianhays]

A few minor comments about comments, but otherwise LGTM.

[stephentoub]

@ericstj, could you please review as well?

[factormystic]

awesome 👍