[FEATURE REQ] Allow per-request model

[verdie-g]

Describe the feature or improvement you are requesting

The OpenAI protocol allows select a different model for each request, though this is not exposed as ChatCompletionOptions.Model is internal.

Additional context

Fixing #306 would also fix that issue.

[jsquire]

Hi @verdie-g. Thank you for reaching out and for your suggestion. Today, it is possible to specify models at the operation level when dropping down to the protocol overloads, but they are otherwise intentionally governed by the client. We have been discussing whether or not there should be an override on each operation for the model, but no conclusion has been reached yet.

[jsquire]

@KrzysztofCwalina, @christothes: Your thoughts would be appreciated.

[KrzysztofCwalina]

I am not totally opposed to exposing Model, but there are negatives of exposing it, and so I would like to understand better why it's a blocker for M.E.AI.A? M.E.AI.A adapter should probably be calling protocol APIs to avoid converting various small datatypes.

As to the negatives, I worry a bit that users will see the Model property and will unnecessarily worry that they have to set it.

In general, I wonder if it's not the time to expose "protocol models", i.e. types that are similar in shape to options/results but ones that correspond 100% to the request/response content.

[verdie-g]

Invoking MEAI's developer @eiriktsarpalis that could give a better explanation than I.

[verdie-g]

This is how MEAI create the options

https://github.com/dotnet/extensions/blob/0b4c4d53f728a0d4ea12c099bba3f3c1218dc15a/src/Libraries/Microsoft.Extensions.AI.OpenAI/OpenAIChatClient.cs#L534-L614

They can't set the model here because it's internal.

You said there was another way to set it?

[KrzysztofCwalina]

You said there was another way to set it?

Yeah, the adapter should not be using the options class. It should call so called "protocol" overloads that take RequestContext, e.g.
public virtual ClientResult CompleteChat(BinaryContent content, RequestOptions options = null);

This way, the adapter would not be translating from MEAI types to OpenAI Options to JSON, and instead go directly from MEAI types to JSON (the BinaryContent parameter is the JSON)

[stephentoub]

From my perspective, the main benefit of the OpenAI library is in using the non-protocol overloads. If M.E.AI.OpenAI is just doing its own writing of request payloads and parsing of response payloads, the OpenAI library provides relatively little benefit over just using HTTP directly. Further, we use the types exposed by the OpenAI library for better breaking-glass support, e.g. a developer can give us the ChatCompletionOptions they want to use as part of the request, or for responses we can expose the OpenAI types for cases where the developer wants to access some niche information that's not exposed via the abstraction. It'd be very helpful if Model could be set. Concerns around developers not knowing if they need to set it can be addressed with documentation.

[KrzysztofCwalina]

FYI: here is a simple usage sample:

BinaryContent content = BinaryContent.Create(BinaryData.FromObjectAsJson(
    new {
        messages = new[] {
            new {
                role = "user",
                content = "Write a one-sentence bedtime story about a unicorn."
            }
        },
        model = "o4-mini"
    }
));
ClientResult result = chat.CompleteChat(content);
ChatCompletion completion = ModelReaderWriter.Read<ChatCompletion>(result.GetRawResponse().Content)!;

[KrzysztofCwalina]

From my perspective, the main benefit of the OpenAI library is in using the non-protocol overloads. If M.E.AI.OpenAI is just doing its own writing of request payloads and parsing of response payloads, the OpenAI library provides relatively little benefit over just using HTTP directly. Further, we use the types exposed by the OpenAI library for better breaking-glass support, e.g. a developer can give us the ChatCompletionOptions they want to use as part of the request, or for responses we can expose the OpenAI types for cases where the developer wants to access some niche information that's not exposed via the abstraction. It'd be very helpful if Model could be set. Concerns around developers not knowing if they need to set it can be addressed with documentation.

Sure, for end users, the convenience overloads are the main value prop as they have superior UX/productivity. But for reusable libraries (like the adapter), UX is (should be?) less important, isn't it? ... than being lightweight and having direct access to the protocol (for example, to be able to implement semantics that are important to it (change model on per request basis) and not to the convenience library.