fix(copilot): clamp reasoning effort to nearest supported level instead of capping xhigh - #51480
Conversation
…not xhigh->high The Copilot provider profile unconditionally mapped ``xhigh`` to ``high`` before checking the model's catalog, so models that DO support ``xhigh`` (e.g. the gpt-5.x family per the live /models catalog) were silently capped one level down. Honor the requested effort when the catalog lists it as supported, and only downgrade when it does not, choosing the nearest weaker supported level (xhigh->high, minimal->low, else medium, else the first supported level). This matches the nearest-down clamp behavior used elsewhere for the ``max`` effort. Adds tests/plugins/model_providers/test_copilot_profile.py covering forward, downgrade, and fallback paths (catalog lookup stubbed).
Related: this fixes the xhigh-clamp in the Copilot provider profile ( |
|
Local validation on draft PR head python -m pytest tests/plugins/model_providers/test_copilot_profile.py -q -o 'addopts='
# 6 passed, 1 warning
python -m py_compile plugins/model-providers/copilot/__init__.py tests/plugins/model_providers/test_copilot_profile.py
# passedThe clamp behavior is correct when One scope caveat: the profile still gets |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the supported-effort negotiation and adding focused tests.
Problems
- This changes only the chat-completions provider profile. GPT-5 Copilot models route to
codex_responses(hermes_cli/models.py:3293-3307), whererun_agent.py:5454-5455still unconditionally changesxhightohigh. - The profile invokes
github_model_reasoning_efforts(model)without catalog/API-key input (plugins/model-providers/copilot/__init__.py:35); the helper consults live capabilities only when supplied one (hermes_cli/models.py:3511-3517). The new stubbed-xhigh case is therefore not reachable through the current production wiring.
Suggested changes
- Prefer the existing #62028 salvage carrier: it preserves this commit's authorship, covers both request paths, and adds current-main regression coverage. This branch is also currently conflicted with
mainafter7550c594cchanged the same normalization.
Automated hermes-sweeper review.
| # the higher level.) | ||
| if effort not in supported_efforts: | ||
| if effort == "xhigh" and "high" in supported_efforts: | ||
| effort = "high" |
There was a problem hiding this comment.
This fixes only the chat-completions profile. GPT-5 Copilot models use codex_responses (hermes_cli/models.py:3293-3307), whose payload still comes from run_agent.py:_github_models_reasoning_extra_body() and unconditionally clamps xhigh at run_agent.py:5454-5455; update both paths (as carried by #62028).
|
This landed (in spirit) a while back — the nearest-weaker catalog clamp shipped for the copilot profile, and as of PR #90350 it routes through the canonical |
Problem
The Copilot provider profile (
plugins/model-providers/copilot/__init__.py) normalized the reasoning effort by unconditionally mappingxhightohighbefore consulting the model's supported-effort set:But the live Copilot
/modelscatalog reports supported efforts per model, and several models (the gpt-5.x family) do supportxhigh. Cappingxhightohighunconditionally silently downgraded those models one level below what the user asked for and what the model supports.Fix
Honor the requested effort when the catalog lists it as supported, and only downgrade when it does not, choosing the nearest weaker supported level:
This is the same nearest-down clamp behavior used elsewhere for the
maxeffort: a requested level the model genuinely supports is forwarded verbatim; an unsupported level steps down to the nearest weaker supported one instead of being dropped or hard-mapped.Scope
The Copilot profile's
build_api_kwargs_extraseffort handling only. No change to the catalog lookup itself or to any other provider.Tests
Adds
tests/plugins/model_providers/test_copilot_profile.py(6 tests, catalog lookup stubbed): a supportedxhighis forwarded verbatim; an unsupportedxhighsteps tohigh;minimalsteps tolow; an unrecognized effort falls tomedium; and when neither the specific rule normediumapplies, the first supported level is chosen.ruff checkand the Windows-footgun check pass.