feat(agent): add OpenAI Ultra multi-agent mode - #62156
Conversation
d133ff4 to
508cf77
Compare
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “Ultra” reasoning mode (/reasoning ultra) that enables OpenAI’s hosted Multi-agent Responses beta (or Codex app-server Ultra turns) without ever sending the unsupported literal reasoning.effort: "ultra" on the wire. The change updates transport request-shaping, app-server capability validation, CLI/gateway surfaces, docs, locales, and adds regression tests.
Changes:
- Implement Ultra request shaping for direct
api.openai.comResponses calls (Multi-agent beta header/body, strongest supported effort projection, hosted tool budget defaults, and fail-closed behavior). - Add Codex app-server Ultra capability validation via live
model/list, and forwardmodel+effort: ultraonly when advertised. - Expose
ultrain CLI/gateway command surfaces, completions, docs/locales, and add test coverage for both routes and response normalization.
Reviewed changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| agent/chat_completion_helpers.py | Adds fail-closed guard when Ultra is selected on unsupported API modes. |
| agent/codex_responses_adapter.py | Preserves hosted multi-agent output items and ensures they are not dispatched as Hermes tools. |
| agent/codex_runtime.py | Forwards model + reasoning effort into Codex app-server turns. |
| agent/transports/codex.py | Implements Ultra → OpenAI Multi-agent beta request shaping (headers/body/effort projection). |
| agent/transports/codex_app_server_session.py | Adds live model/list capability checks and forwards model/effort for Ultra turns. |
| gateway/run.py | Updates gateway config reasoning-effort docstring to include ultra. |
| gateway/slash_commands.py | Accepts /reasoning ultra in gateway command parsing. |
| hermes_cli/cli_commands_mixin.py | Updates CLI /reasoning help/validation strings to include ultra. |
| hermes_cli/commands.py | Adds ultra to /reasoning subcommand completion list. |
| hermes_constants.py | Extends VALID_REASONING_EFFORTS / parsing to accept ultra. |
| locales/af.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/de.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/en.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/es.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/fr.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/ga.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/hu.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/it.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/ja.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/ko.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/pt.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/ru.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/tr.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/uk.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/zh-hant.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| locales/zh.yaml | Updates localized /reasoning unknown-arg help to include ultra. |
| tests/agent/test_codex_app_server_persist.py | Ensures app-server persistence path forwards model + ultra reasoning effort. |
| tests/agent/test_codex_responses_adapter.py | Adds coverage for preserving hosted multi-agent telemetry items and tool continuation. |
| tests/agent/transports/test_codex_app_server_session.py | Verifies Ultra is forwarded only when live catalog advertises it; otherwise fail-closed. |
| tests/agent/transports/test_codex_transport.py | Adds Ultra request-shaping tests (beta headers, effort projection, overrides, fail-closed). |
| tests/cli/test_reasoning_command.py | Updates CLI parsing tests to accept ultra. |
| tests/gateway/test_reasoning_command.py | Adds gateway /reasoning ultra acceptance test (session-scoped). |
| tests/hermes_cli/test_commands.py | Adds CLI completion test ensuring ultra is suggested. |
| tests/run_agent/test_run_agent_codex_responses.py | Adds agent-level _build_api_kwargs tests for Ultra request shaping and fail-closed behavior. |
| tests/test_hermes_constants.py | Updates constants documentation guard test to include ultra. |
| website/docs/user-guide/configuration.md | Documents /reasoning ultra behavior and supported routes in user configuration docs. |
| website/docs/user-guide/features/codex-app-server-runtime.md | Documents Ultra behavior on the Codex app-server runtime. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the thorough transport, replay, and capability-gating coverage. One state-reset path remains blocking.
Problems
agent/codex_runtime.py:390-401maps{enabled: false}toreasoning_effort=None.agent/transports/codex_app_server_session.py:516-528then omitsturn/start.effort. Since/reasoning noneis represented as{enabled: false}(gateway/slash_commands.py:2734-2735), an Ultra → none transition on the same app-server thread leaves no explicit replacement for the sticky Ultra override. The new regression only covers Ultra → high (tests/agent/transports/test_codex_app_server_session.py:302-348).
Suggested changes
- Send an explicit supported disabled/default replacement (or reset the relevant app-server state) for disabled reasoning, and add an Ultra → none regression asserting the second
turn/startcannot retain Ultra.
Automated hermes-sweeper review.
| reasoning_effort = None | ||
| if ( | ||
| isinstance(reasoning_config, dict) | ||
| and reasoning_config.get("enabled") is not False |
There was a problem hiding this comment.
/reasoning none produces {enabled: false}, so this guard leaves reasoning_effort unset. CodexAppServerSession.run_turn() then omits turn/start.effort, allowing a reused app-server thread to retain the prior sticky Ultra override. Please explicitly clear or replace the effort for the disabled case and add an Ultra → none regression.
What changed
Adds an explicit OpenAI Ultra mode behind
/reasoning ultrawithout sending the unsupported literalreasoning.effort: ultravalue.api.openai.comResponses calls enable the Multi-agent beta forgpt-5.6,gpt-5.6-sol,gpt-5.6-terra, andgpt-5.6-luna.?beta=true, theresponses_multi_agent=v1header,multi_agent.enabled: true, a default concurrency of three, andreasoning.effort: max. Positive caller concurrency is preserved; no undocumented upper bound is imposed.ultra,reasoning.summary, andmax_tool_callsnever reach the wire, and unknownmulti_agentoverride keys fail closed.multi_agent_call,multi_agent_call_output,agent_message, reasoning, messages, and agent-attributed function calls. Hosted actions never become Hermes client tools, and only/roottext is rendered as the user-facing answer.model/listcatalog once per subprocess, caches all advertised reasoning capabilities, and stops repeated cursors. Ultra still fails closed unless the selected model advertises it.Why
OpenAI documents Responses Multi-agent as similar to Ultra mode in Codex, but
ultrais not a valid Responses reasoning effort. The public API requires its beta query/header/body contract, while Codex app-server exposes Ultra as a turn-level capability advertised bymodel/list.This intentionally does not implement the generic
maxcommand-surface work in #61638 or provider projection in #61834. The only overlapping command/localization edits add the separateultravalue.Usage
Direct OpenAI Responses API:
ChatGPT/Codex subscription route:
The app-server path fails closed when the per-model capability catalog does not advertise
ultra.Verification
Tested on Windows 10 with Python 3.11.15, OpenAI Python SDK 2.24.0, and Codex CLI 0.144.0.
git diff --check: passedruff checkon all changed Python files: passed/responses?beta=true, the beta header/body, strongest-effort projection, absence of unsupported fields, and serialization of all hosted replay item typesgpt-5.6-soladvertisedultra, accepted the Ultra turn, returnedULTRA_OK, and the subprocess exited cleanlySafety and compatibility
Ultra is opt-in. Existing reasoning levels remain unchanged. App-server turns now carry the selected model and non-empty effort explicitly to prevent sticky cross-turn overrides. The public beta route is limited to the four documented public model identifiers on
api.openai.com; custom endpoints and unsupported providers/models remain fail closed.Closes #61634