[Proposal] Passing custom buffer to FileStream

[sakno]

Background and Motivation

FileStream has its own buffer for I/O. But it cannot be passed from outside. If the application requires a lot of FileStream objects with short lifetime there is not way to reuse the buffer between them. For instance, logging to files.Rotation of log files leads to dispose of old streams and create new ones. Also, it's reasonable to use memory pool (e.g. ArrayPool<T> or MemoryPool<T>) as a source of such buffers.

Proposed API

public class FileStream
{
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, Memory<byte> buffer, FileOptions options)
}

Usage Examples

var buffer = ArrayPool<byte>.Shared.Rent(4096);
using var fs = new FileStream("dummy.txt", FileMode.OpenOrCreate, FileAccess.Wrtie, FileShare.None, buffer.AsMemory(), FileOptions.None);

Alternative Designs

To mitigate risk of sharing the same buffer between different instances of file stream we could offer passing of memory pool:

public class FileStream
{
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, ArrayPool<byte> buffer, FileOptions options);
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, MemoryPool<byte> buffer, FileOptions options);
}

Risks

• The same buffer can be passed to two or more streams
• The buffer has been returned to the pool while FileStream still in use

[ghost]

Tagging subscribers to this area: @carlossanlop
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and Motivation

FileStream has its own buffer for I/O. But it cannot be passed from outside. If the application requires a lot of FileStream objects with short lifetime there is not way to reuse the buffer between them. For instance, logging to files.Rotation of log files leads to dispose of old streams and create new ones. Also, it's reasonable to use memory pool (e.g. ArrayPool<T> or MemoryPool<T>) as a source of such buffers.

Proposed API

public class FileStream
{
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, Memory<byte> buffer, FileOptions options)
}

Usage Examples

var buffer = ArrayPool<byte>.Shared.Rent(4096);
using var fs = new FileStream("dummy.txt", FileMode.OpenOrCreate, FileAccess.Wrtie, FileShare.None, buffer.AsMemory(), FileOptions.None);

Alternative Designs

To mitigate risk of sharing the same buffer between different instances of file stream we could offer passing of memory pool:

public class FileStream
{
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, ArrayPool<byte> buffer, FileOptions options);
  public FileStream(string path, FileMode mode, FileAccess access, FileShare share, MemoryPool<byte> buffer, FileOptions options);
}

Risks

• The same buffer can be passed to two or more streams
• The buffer has been returned to the pool while FileStream still in use

Author:	sakno
Assignees:	-
Labels:	api-suggestion, area-System.IO, untriaged
Milestone:	-

[davidfowl]

Pass a buffer size of 1 a d you'll disable the FileStream's internal buffer. This shouldn't be needed

[sakno]

Not sure that this works on Linux/MacOS/FreeBSD. In .NET 6, Net5CompatFileStreamStrategy is used for Unix-like platforms. For Windows, there is a condition which analyzes value 1. Moreover, I don't want to disable internal buffer and move the buffer management outside of FileStream.

[davidfowl]

That strategy should work on all platforms (or it's a bug). You pass a buffer to FileStream anyways. Why would you want to have it use the array pool rather than do that yourself? It's much better and you can pre-pin buffers to avoid fragmentation.

I don't think we should add new overloads.

[sakno]

That's why overload with Memory<byte> parameter looks better that overloads with pools. Moving buffer management outside of FileStream increases complexity of my code: I need to track Position and Seek calls to flush the buffer when needed. All this stuff is completely done in Strategy abstraction which requires minimal modifications to support Memory<byte>.

[sakno]

Btw, the current implementation in .NET 6 doesn't use pinned arrays. new byte[_bufferLength] can be replaced with GC.AllocUninitializedArray<byte>(_bufferLength, true)

[scalablecory]

Btw, the current implementation in .NET 6 doesn't use pinned arrays. new byte[_bufferLength] can be replaced with GC.AllocUninitializedArray<byte>(_bufferLength, true)

pre-pinned arrays are placed into Gen2 so we must be careful to only use them when we expect the array to last a while.

[davidfowl]

They're placed in the POH actually which is different than Gen2 and this would be a bad usage of the POH.

[jkotas]

It is different than Gen2 internally in the GC. From the outside, you cannot tell the difference. POH allocations put pressure on Gen2 GCs with the current implementation. #51363 (comment)

[davidfowl]

Yea, I was being pedantic 😄

[adamsitnik]

I've created a new proposal related to this feature request: #52446

[sakno]

OK, we can move any further discussion to one place. Closing issue.