Summary
Microsoft.Extensions.AI.Evaluation.Quality 10.7.0 evaluators (RelevanceEvaluator, CoherenceEvaluator, FluencyEvaluator, CompletenessEvaluator, EquivalenceEvaluator, etc.) require an IChatClient for the LLM-as-judge call. The current docs at learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries describe what each evaluator measures but never state which judge model families are supported. In particular:
- Reasoning models (
gpt-5, gpt-5-mini, o1, o3, o-series more broadly) are not supported as judges via the default AzureAIInferenceChatClient because the SDK still sends max_tokens and reasoning models require max_completion_tokens. The 400 response is then swallowed by the evaluator (see sibling issue on silent error handling).
- The evaluators' built-in judge prompts are tuned for chat-completion-style responses. There's no published guidance on whether responses-API models, embeddings-only deployments, or fine-tuned variants are supported.
Requested
Add a "Judge model requirements" subsection to learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries (and to each Quality evaluator's API doc) covering:
- Supported model families for the judge
IChatClient. At minimum, name the ones we know work:
gpt-4o, gpt-4o-mini, gpt-4-turbo ✅
- Known incompatibilities and why. At minimum:
- Reasoning models (
gpt-5*, o-series) — Azure.AI.Inference sends legacy max_tokens that these models reject with HTTP 400.
- Recommended pinning strategy for reproducible judge scores. (Judge scores drift across model versions; pin the judge model to one version, document the change, re-baseline when bumping.)
- What to do when the agent's production model is a reasoning model. The recommended pattern of splitting agent vs judge deployments (e.g. via a
JUDGE_DEPLOYMENT_NAME indirection in the user's IChatClient factory).
Example failure mode that drove this request
Pointed the judge IChatClient at a Foundry gpt-5-mini deployment — dotnet test exits 0, 4/4 passed, but every Quality column in report.html is Status: 400 (Bad Request) ErrorCode: unsupported_parameter — 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead. Took ~2 hours to track down because there's nothing in the docs that says "don't point the judge at a reasoning model."
After switching to a gpt-4o-mini deployment, the same harness produced clean scores for all five Quality metrics in 44 seconds. The only thing that changed was the judge deployment.
Environment
Microsoft.Extensions.AI.Evaluation 10.7.0 (GA)Microsoft.Extensions.AI.Evaluation.Quality 10.7.0 (GA)Aspire.Azure.AI.Inference 13.2.1-preview
Related
Filed from a dogfood session of the setup-maf-evals skill against an Aspire+Foundry MAF agent. Skill now hard-warns about this in its common-pitfalls doc; happy to upstream the warning text if useful.
Summary
Microsoft.Extensions.AI.Evaluation.Quality10.7.0 evaluators (RelevanceEvaluator,CoherenceEvaluator,FluencyEvaluator,CompletenessEvaluator,EquivalenceEvaluator, etc.) require anIChatClientfor the LLM-as-judge call. The current docs at learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries describe what each evaluator measures but never state which judge model families are supported. In particular:gpt-5,gpt-5-mini,o1,o3,o-seriesmore broadly) are not supported as judges via the defaultAzureAIInferenceChatClientbecause the SDK still sendsmax_tokensand reasoning models requiremax_completion_tokens. The 400 response is then swallowed by the evaluator (see sibling issue on silent error handling).Requested
Add a "Judge model requirements" subsection to learn.microsoft.com/en-us/dotnet/ai/evaluation/libraries (and to each Quality evaluator's API doc) covering:
IChatClient. At minimum, name the ones we know work:gpt-4o,gpt-4o-mini,gpt-4-turbo✅gpt-5*,o-series) — Azure.AI.Inference sends legacymax_tokensthat these models reject with HTTP 400.JUDGE_DEPLOYMENT_NAMEindirection in the user'sIChatClientfactory).Example failure mode that drove this request
Pointed the judge
IChatClientat a Foundrygpt-5-minideployment —dotnet testexits 0, 4/4 passed, but every Quality column inreport.htmlisStatus: 400 (Bad Request) ErrorCode: unsupported_parameter — 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.Took ~2 hours to track down because there's nothing in the docs that says "don't point the judge at a reasoning model."After switching to a
gpt-4o-minideployment, the same harness produced clean scores for all five Quality metrics in 44 seconds. The only thing that changed was the judge deployment.Environment
Microsoft.Extensions.AI.Evaluation10.7.0 (GA)Microsoft.Extensions.AI.Evaluation.Quality10.7.0 (GA)Aspire.Azure.AI.Inference13.2.1-previewRelated
max_tokens→max_completion_tokenstranslation inAzureAIInferenceChatClient).Filed from a dogfood session of the setup-maf-evals skill against an Aspire+Foundry MAF agent. Skill now hard-warns about this in its common-pitfalls doc; happy to upstream the warning text if useful.