Account for OpenAI's search models

[dkotter]

Issue

While testing something out that utilizes this client to make requests, things were working great yesterday but started to fail today with the following error: Invalid arguments while trying to generate text result: Bad Request (400) - Model incompatible request argument supplied: temperature.

In debugging further, found that if I only have OpenAI configured and I don't specify a model, seems today it started picking gpt-4o-mini-search-preview as the model. Not sure if that's a new model that OpenAI just released recently or if they changed the model list their API returns but somehow that now ends up on the top of the list when we try and determine which model works best.

But as the error shows, that model doesn't support standard parameters like temperature or n and so the request doesn't work.

This does highlight a downside of the approach taken in this SDK, where we try and intelligently choose the provider and model. Anytime a new model is released or the model APIs we use change the order of their results, we could end up breaking integrations. I was just testing things locally so while this issue was annoying it wasn't critical but for any production integrations, would be ideal if there was a bit more certainty things won't just break one day.

There's potentially a conversation we could have around that (for instance, should we be setting preferred models within the client to avoid breaking changes like this?) but for now, seems like we need to update the OpenAiModelMetadataDirectory class to account for this search model.

Steps to Reproduce

Things may change but as of right now, I can easily reproduce this by calling our cli command:

OPENAI_API_KEY=sk-proj-XXXX php cli.php 'Hello' --temperature=0.9

If the model the SDK chooses changes in the future, can reproduce by running this:

OPENAI_API_KEY=sk-proj-XXXX php cli.php 'Hello' --modelId=gpt-4o-mini-search-preview --temperature=0.9

[felixarntz]

There are a few problems that all come together in this issue 🙃

1. Our logic to assign model capabilities and model capabilities to each model is definitely incomplete / inaccurate. We knew this from the beginning, and realistically we'll never get this 100% right, unless we have a reliable way to automatically track new models and model changes, plus have enough people on top of those changes to ensure everything is always accurate and up to date. There are some high-level ideas for improving this (see Abstracting away the need to manually maintain tons of provider and model metadata #4), but it'll remain a problem for the foreseeable future. If there is a concrete problem here where we annotate one or more new OpenAI models incorrectly, we should simply fix it and move on. Given the model was returned as eligible but then caused an error, I assume the model does in fact not support temperature but we annotate it as if it did.
2. The much more straightforward problem (which we have talked about already many times) is that currently, if you don't specify a model preference list, the alphabetically first model from the list of eligible models will be returned. This is problematic for many reasons:
   • As you're saying @dkotter, it can easily lead to an unexpected change in model, and maybe that model will have different capabilities or different options.
   • More holistically, the model returned is often not a useful model to have as default. Simply look at this current example: Why would we return a -search-preview model by default - that seems super specific. To fix that, we should introduce a concept of centrally controlled model preference list per provider: If the caller doesn't give one, we should use the provider's model preference list. This way we can ensure one of the generally applicable flagship models is preferred rather than some random experimental model or a model that may be from an outdated generation.

The latter is holistically more important, but it's also totally fair to review our current model capability and option lists for OpenAI to try to bring them up to date.

[dkotter]

To fix that, we should introduce a concept of centrally controlled model preference list per provider: If the caller doesn't give one, we should use the provider's model preference list. This way we can ensure one of the generally applicable flagship models is preferred rather than some random experimental model or a model that may be from an outdated generation

This makes a lot of sense to me (and is more elegantly phrased then my comment 😄). While it would be great to maintain an up-to-date list of model capabilities, the reality of doing that is difficult, so having something like this in place makes things more stable/reliable