Skip to content

ClientModel: AsyncResultCollection<T> and SSE event collection implementation#43840

Merged
annelo-msft merged 53 commits intoAzure:mainfrom
annelo-msft:clientmodel-sse-nodispose
May 14, 2024
Merged

ClientModel: AsyncResultCollection<T> and SSE event collection implementation#43840
annelo-msft merged 53 commits intoAzure:mainfrom
annelo-msft:clientmodel-sse-nodispose

Conversation

@annelo-msft
Copy link
Copy Markdown
Contributor

@annelo-msft annelo-msft commented May 3, 2024

Overview

System.ClientModel-based clients need to be able to return collections of values from service methods, for example, for endpoints such as OpenAI's streaming chat completions where incremental updates are delivered sequentially via SSE streams. In .NET, we represent such collections as async streams.

What's in this PR

This PR adds sync and async abstractions for collections of results, ResultCollection<T> and AsyncResultCollection<T>. It also adds an internal implementation of an SSE parser and event enumerable that will likely be replaced by dotnet/runtime#98105 when it becomes available. The internal implementation is provided here to enable test coverage and review, as well as illustrate how clients would implement the abstractions to support both convenience method return values (see MockSseClient.cs) and factory-method helpers for use with protocol methods (see MockSseClientExtensions.cs).

Additional notes

Given our timelines, the current implementation uses System.IO.StreamReader, which converts SSE lines to UTF-16. This adds a performance cost in cases where we convert the data payload back to UTF-8, e.g. for use with ModelReaderWriter.Create APIs. It is beyond the scope of this PR to provide UTF-8 stream reading capabilities, but part of our longer term plans for third-party clients.

Also: This builds on a great deal of prior work on the SSE reader by @KrzysztofCwalina and @trrwilson.

Addresses #42284

@azure-sdk
Copy link
Copy Markdown
Collaborator

API change check

APIView has identified API level changes in this PR and created following API reviews.

System.ClientModel

Comment thread sdk/core/System.ClientModel/api/System.ClientModel.net6.0.cs Outdated
Comment thread sdk/core/System.ClientModel/src/Convenience/ClientResult.cs
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/AsyncServerSentEventEnumerator.cs Outdated
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/StreamingClientResult.cs Outdated
Comment thread sdk/core/System.ClientModel/api/System.ClientModel.net6.0.cs Outdated
@annelo-msft annelo-msft changed the title ClientModel: SSE result collection that uses enumerator to dispose ClientModel: AsyncResultCollection<T> and SSE event collection implementation May 10, 2024
@annelo-msft annelo-msft marked this pull request as ready for review May 10, 2024 19:06
@annelo-msft annelo-msft requested review from a team and tg-msft as code owners May 10, 2024 19:06
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/AsyncServerSentEventEnumerable.cs Outdated
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/ServerSentEvent.cs Outdated
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/ServerSentEventReader.cs Outdated
Comment thread sdk/core/System.ClientModel/src/Internal/SSE/ServerSentEventReader.cs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants