feat: add automatic reasoning effort resolution - #61410
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for implementing a local, cache-safe resolver. The feature is still needed on current main, but there are a few correctness gaps to address.
Problems
- Gateway
/reasoning autois still rejected by the hard-coded level list ingateway/slash_commands.py:2730-2738; changing the loader docstring ingateway/run.pydoes not reach the command handler. - The GitHub Responses branch still constructs
github_reasoning_extrafrom rawagent.reasoning_configatagent/chat_completion_helpers.py:770;run_agent.py:5433-5449therefore maps rawautoto its fallback instead of the resolved effort. - The LM Studio summary effort is calculated from raw config before
summary_reasoning_configis resolved atagent/chat_completion_helpers.py:1661-1671, soautobecomes LM Studio's default medium (agent/lmstudio_reasoning.py:35-47).
Suggested changes
- Route gateway parsing through the shared parser, pass resolved config to GitHub/LM Studio special paths, and add transport plus summary regression tests.
- Update the public config docs at
website/docs/user-guide/configuration.md:1279-1301andcli-config.yaml.example:739-742.
Automated hermes-sweeper review.
| """Load reasoning effort from config.yaml. | ||
|
|
||
| Reads agent.reasoning_effort from config.yaml. Valid: "none", | ||
| Reads agent.reasoning_effort from config.yaml. Valid: "none", "auto", |
There was a problem hiding this comment.
This documents auto, but gateway /reasoning auto still fails because gateway/slash_commands.py:2730-2738 hard-codes its accepted levels. Please update that handler and add a gateway command test.
| from hermes_constants import resolve_auto_reasoning_config | ||
|
|
||
| tools_for_api = agent.tools | ||
| reasoning_config = resolve_auto_reasoning_config(agent.reasoning_config, api_messages) |
There was a problem hiding this comment.
The resolved value is bypassed for GitHub Responses: the payload at line 770 is still built by _github_models_reasoning_extra_body() from raw agent.reasoning_config, which maps auto to its fallback medium. Pass this resolved config through that special path.
| agent._resolve_lmstudio_summary_reasoning_effort() | ||
| if _is_lmstudio_summary else None | ||
| ) | ||
| try: |
There was a problem hiding this comment.
For LM Studio, _lm_reasoning_effort was already computed from raw agent.reasoning_config at lines 1661-1663. Resolve first, then derive the LM Studio effort from the resolved config; otherwise iteration-limit summaries use medium for auto.
|
Hi @0xSynthex — I’m sorry: I missed that this PR was already open and linked from #40306 when preparing #82578. I should have reviewed and credited your work first; thank you for implementing the deterministic local approach ahead of mine. After comparing them, I think #82578 may still be useful as a separate candidate because it was built around some additional lifecycle requirements—per-session state, full tool-loop restoration, explicit override/Desktop behavior, and delegated-child handling. That is not meant to diminish this PR. I’m happy to follow whatever consolidation path you and the maintainers prefer, including reworking or closing mine. I’ve updated #82578 to acknowledge this PR properly. |
Summary
reasoning_effort: autoto parse as a first-class markerautoper request with deterministic local heuristics (low/medium/high)/reasoninghelp text and gateway config docsWhy
Users can keep a stable config while avoiding one fixed reasoning level for every task. The resolver is local-only and does not mutate prompts or tools, so prompt caching remains stable.
Tests
python -m pytest tests/test_auto_reasoning_effort.py -qruff check agent/chat_completion_helpers.py gateway/run.py hermes_cli/cli_commands_mixin.py hermes_constants.py tests/test_auto_reasoning_effort.py