Skip to content

fix(reasoning): expose max and ultra for GPT-5.6 - #6018

Open
ruizanthony wants to merge 9 commits into
nesquena:masterfrom
ruizanthony:fix/gpt56-max-ultra-reasoning-webui
Open

fix(reasoning): expose max and ultra for GPT-5.6#6018
ruizanthony wants to merge 9 commits into
nesquena:masterfrom
ruizanthony:fix/gpt56-max-ultra-reasoning-webui

Conversation

@ruizanthony

Copy link
Copy Markdown
Contributor

Summary

  • align the standalone WebUI reasoning-effort mirror with Hermes Agent's current max / ultra contract
  • expose max and ultra for GPT-5.6 on OpenAI-family routes, including gpt-5.6-sol through openai-codex
  • keep older GPT-5 models capped at xhigh and o-series models capped at high
  • add ultra to the composer picker, /reasoning command, autocomplete, and labels
  • preserve safe downgrade behavior for unknown providers, Gemini, and pre-adaptive Claude models

Hermes Agent PR #62650 added the generic max and ultra levels and maps the Codex product tier ultra to the Responses API wire value max for GPT-5.6. The standalone WebUI still removed max from every gpt-5* Codex model and did not recognize ultra, so gpt-5.6-sol stopped at xhigh in the UI.

Validation

  • ./scripts/test.sh tests/*reasoning*.py tests/test_issue1103_reasoning_chip_visibility.py -q — 226 passed
  • python3 -m py_compile api/config.py tests/test_reasoning_effort_model_capabilities.py tests/test_reasoning_show_hide.py tests/test_issue1103_reasoning_chip_visibility.py
  • node --check static/commands.js
  • node --check static/ui.js
  • git diff --check origin/master..HEAD
  • direct capability/coercion probe:
    • GPT-5.5 remains xhigh-capped
    • GPT-5.6 Sol exposes and preserves both max and ultra

Related

@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds GPT-5.6 reasoning levels across the API and WebUI. The main changes are:

  • Adds max and ultra support for GPT-5.6 routes.
  • Limits unsupported reasoning levels by model and provider.
  • Adds provider aliases and explicit custom-provider allowlists.
  • Updates the reasoning command, picker, labels, and mobile dropdown behavior.
  • Expands capability, coercion, routing, and viewport tests.

Confidence Score: 5/5

This looks safe to merge.

  • The updated fallback logic blocks unsupported top-tier values on unknown providers.
  • Routed legacy Claude models are limited by model family across aggregator paths.
  • GPT-5.6 retains max and ultra on the intended routes.
  • No blocking issues were found in the changed code.

Important Files Changed

Filename Overview
api/config.py Adds reasoning-level resolution, provider normalization, model limits, allowlists, and conservative fallback coercion.
static/commands.js Adds ultra to the reasoning command and sends the active model context when saving.
static/index.html Adds Ultra to the composer reasoning picker.
static/style.css Makes the expanded reasoning picker scrollable on short viewports.
static/ui.js Adds the display label for Ultra.
tests/test_reasoning_effort_model_capabilities.py Expands coverage for GPT-5.6, older models, routed providers, allowlists, and coercion.
tests/test_reasoning_dropdown_viewport.py Checks that all reasoning options remain reachable across desktop and mobile viewports.

Reviews (9): Last reviewed commit: "fix(reasoning): close 2026-08-13 gate bl..." | Re-trigger Greptile

Comment thread api/config.py
@nesquena-hermes nesquena-hermes added the size:M Medium PR (≤10 files, ≤250 LOC) label Jul 13, 2026
@cutter-sh

cutter-sh Bot commented Jul 13, 2026

Copy link
Copy Markdown

🎬 Cutter preview — PR #6018

Select Ultra reasoning effort from composer dropdown
Select Ultra reasoning effort from composer dropdown — The dropdown menu is open in image A but closed in image B, showing only the settled Ultra selection result.

@nesquena-hermes

Copy link
Copy Markdown
Collaborator

Read the api/config.py diff at HEAD (8e1ab80) against origin/master, plus the agent-side contract this mirrors. The core change is correct and the agent alignment checks out — one narrow fallback gap is worth closing before merge, which is essentially the same thing Greptile flagged.

The core change is right

_filter_reasoning_efforts_for_provider now gates the OpenAI-family lanes precisely (api/config.py:3402-3408 at HEAD):

if bare.startswith("gpt-5"):
    if "gpt-5.6" in bare:
        return normalized
    return [eff for eff in normalized if eff not in {"max", "ultra"}]

That matches the agent's Codex transport exactly — agent/transports/codex.py:166-169 only maps ultra -> max (the wire value) when the model is GPT-5.6:

_effort_clamp = {"minimal": "low"}
if "gpt-5.6" in (model or "").lower():
    # Ultra is the Codex product tier; the Responses API wire value is max.
    _effort_clamp["ultra"] = "max"

The default-deny extension in coerce_reasoning_effort_for_model (ultra degrades to xhigh for unrecognized providers, same as max) is also consistent with agent/anthropic_adapter.py:67-69 (ADAPTIVE_EFFORT_MAP: "ultra" -> "max") — known adaptive lanes accept it, unknowns don't. Good.

The fallback gap (Greptile's flag, made specific)

The leak is in the Copilot heuristic branch, api/config.py:3497-3501:

if provider in {"copilot", "github-copilot"}:
    if bare.startswith(("gpt-5", "o1", "o3", "o4")):
        if bare.startswith(("o1", "o3", "o4")):
            return ["low", "medium", "high"]
        return list(VALID_REASONING_EFFORTS)   # now includes max + ultra

_heuristic_reasoning_efforts is reached for Copilot only when the from hermes_cli.models import github_model_reasoning_efforts import fails (api/config.py:3828-3829) — the degraded/hermes_cli-unavailable path. In that state, a Copilot GPT-5 model returns the full VALID_REASONING_EFFORTS, and the top-level filter in resolve_model_reasoning_efforts (3757) has no Copilot rule — the openai-family branch keys on {"openai-codex","openai","openai-api","azure*"}, not copilot — so max/ultra pass through unstripped.

That contradicts the agent's own ceiling: hermes_cli/models.py:30 caps Copilot GPT-5 at

COPILOT_REASONING_EFFORTS_GPT5 = ["minimal", "low", "medium", "high"]

So Copilot GPT-5 would advertise (and, since Copilot is in _KNOWN_REASONING_PROVIDERS, not degrade) ultra, which the adapter tops out below. Note this is a pre-existing max leak — the PR just extends the same branch to ultra — so it's arguably out of scope, but it's the exact "fallback exposes a top tier the model can't take" case Greptile is pointing at, and it's a one-liner.

Suggested fix

Either make the heuristic Copilot GPT-5 branch return the agent's capped list instead of the global one:

return ["minimal", "low", "medium", "high"]   # mirror COPILOT_REASONING_EFFORTS_GPT5

or add a Copilot ceiling to _filter_reasoning_efforts_for_provider (strip max/ultra) so the top-level filter closes it regardless of which source produced the list. The second is more robust — it also covers the _models_dev_reasoning_efforts path if metadata ever over-reports for Copilot.

Test note

tests/test_reasoning_effort_model_capabilities.py adds GPT-5.6 Codex + unknown-provider coverage, which is the right shape. A case that stubs the github_model_reasoning_efforts import to raise (forcing the heuristic) and asserts a Copilot GPT-5 model does not surface ultra/max would lock the fallback behavior. Everything else in the diff looks sound.

@nesquena-hermes nesquena-hermes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔬 Gate review (Codex) — SHIP ONLY WITH FIXES (picker vocab correct; backend coercion lets an unsupported effort reach some providers)

Thanks @ruizanthony — the parity intent is right and much of it verifies clean: GPT-5.6/Codex correctly exposes max/ultra (Agent maps ultra→Responses max), older GPT-5 downgrades both top tiers to xhigh, o1/o3/o4 → high, Gemini + pre-adaptive Claude → xhigh, existing minimalxhigh unregressed, and the picker/command/autocomplete vocab is consistent (none, minimal, low, medium, high, xhigh, max, ultra everywhere, labels complete). 79 WebUI + 4 Agent-transport tests pass. But the gate found the runtime coercion lets an unsupported effort reach a provider that will reject it in several cases:

  1. [CORE — custom/unknown provider not default-denied] api/config.py:3932 — custom heuristic models and unknown providers can be sent max/ultra. Fix: default-deny max/ultra for unknown/custom providers unless an explicit provider reasoning_efforts allowlist authorizes them; add recognized-custom-model tests.
  2. [CORE — aggregator ceilings not model-scoped] api/config.py:3402 — older GPT-5 + o-series routed via OpenRouter/Nous (and other aggregators) don't get their ceiling applied, so ultra reaches them. Fix: make the GPT-5/o-series ceilings model-scoped across aggregator routes; add ultra-downgrade tests for those routes.
  3. [CORE — Copilot fallback] api/config.py:3501 — cap the Copilot fallback to its actual high ceiling; add a forced hermes_cli.models-import-failure test.
  4. [SILENT — slash command not coerced] static/commands.js:1856 — the /reasoning effort POST doesn't include _reasoningEffortContext(), so a slash-command-set effort isn't coerced against the active model/provider. Fix: merge _reasoningEffortContext() into the command POST payload + test that slash-command responses are coerced.

Net: a user (or the /reasoning command) selecting max/ultra on a custom/aggregator/older model can send an effort the provider 400s on. Once unknown/custom providers default-deny, the aggregator + Copilot ceilings are model-scoped, and the slash command coerces against the active model, this is a clean parity win. The vocabulary + label work is already right — it's just the coercion that needs to fail-closed. Rebased clean on current master.

@ruizanthony

Copy link
Copy Markdown
Contributor Author

All four max/ultra coercion leaks are now closed on head 1e4aa84:

  1. Custom/unknown providers — DEFAULT-DENY. _filter_reasoning_efforts_for_provider now strips max/ultra for any non-empty provider not recognized as reasoning-capable, unless an explicit provider reasoning_efforts allowlist (providers. or named custom_providers[]) authorizes them. Regression tests cover recognized custom models (kimi, deepseek, GLM, minimax behind custom:*) plus the allowlist exception path.
  2. Aggregator routes — model-scoped ceilings. The o-series cap (low/medium/high) and the older-GPT-5 cap (no max/ultra) now follow the MODEL across every lane instead of being gated on openai-family providers, so openai/gpt-5.5, openai/gpt-5.1, openai/o3, openai/o4-mini via OpenRouter and Nous can never receive ultra (ultra→xhigh for GPT-5, ultra→high for o-series). GPT-5.6 via the same aggregators keeps the top tiers — the ceiling is model-scoped, not aggregator-scoped. Ultra-downgrade tests cover both aggregator routes.
  3. Copilot fallback capped. The Copilot heuristic fallback no longer hands out the full global list for any gpt-5 id — it routes through the provider filter, so only GPT-5.6 keeps max/ultra; older GPT-5 via Copilot coerces ultra→xhigh.
  4. Metadata-fallback gap closed. Every heuristic fallback branch that returns the expanded global effort list when capability metadata is unavailable (prefix heuristics, nested gateway routes, candidate-family detection, Copilot) now routes through _filter_reasoning_efforts_for_provider, so the GPT-5.6 model check and the unknown-provider default-deny apply even with no models.dev metadata.

Also: first-class catalog providers (xai-oauth, zai, kimi-coding, minimax, minimax-cn, opencode-zen, opencode-go, mistralai, alibaba, nvidia, xiaomi) are now explicitly recognized so the default-deny only hits genuinely custom/unknown lanes (caught by test_models_dev_reasoning.py during development).

Tests: 34/34 passed in tests/test_reasoning_effort_model_capabilities.py (29 existing + 5 new regression tests, one per finding + aggregator routes); 121/121 passed across all six reasoning-related test files. git diff --check clean.

Please re-review on exact head 1e4aa84.

@nesquena-hermes nesquena-hermes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watch-tier re-gate — changes still required

Thanks for the July 29 follow-up. The new commit fixes the tested bare-model custom, OpenRouter/Nous, and Copilot coercion paths, and the targeted sandbox gate is green (75 passed). Two objective gaps remain at head 1e4aa8445fc7.

1. Named custom-provider hints bypass the new hard model ceiling

api/config.py::_filter_reasoning_efforts_for_provider() resolves provider at line 3396, but line 3397 calls _strip_provider_hint_for_reasoning(model_id) without passing it. For a production-shaped ID such as @custom:frontier-gw:gpt-5.5, the generic split removes only @custom: and leaves frontier-gw:gpt-5.5. That no longer starts with gpt-5, so the older-GPT ceiling is skipped. An explicit custom-provider allowlist can then preserve max and ultra, despite the new invariant that only GPT-5.6 may expose them.

I reproduced this through the enforced no-network sandbox. A custom provider with reasoning_efforts: [high, xhigh, max, ultra] returned ['high', 'xhigh', 'max', 'ultra'] for @custom:frontier-gw:gpt-5.5; the regression assertion that max must be absent failed.

Fix: pass the already resolved provider into the helper:

bare = _strip_provider_hint_for_reasoning(model_id, provider).lower().rsplit("/", 1)[-1]

Add production-shaped regressions for:

  • @custom:frontier-gw:gpt-5.5 → no max/ultra; both coerce to xhigh
  • @custom:frontier-gw:o3 → capped at high
  • @custom:frontier-gw:gpt-5.6-sol → still preserves max/ultra
  • the existing unknown-model explicit-authorization control → still preserves the allowlisted tiers

2. The prior slash-command coercion item is still unchanged

The July 14 gate explicitly requested _reasoningEffortContext() on the /reasoning effort POST. At static/commands.js:1856, the live head still sends only:

JSON.stringify({effort:arg})

The July 29 commit does not touch static/commands.js, so /reasoning max or /reasoning ultra still omits the active model/provider context needed by the backend coercion path.

Fix: merge _reasoningEffortContext() into that POST payload, matching the picker/settings path, and add a regression that drives the slash-command branch and proves an unsupported effort is downgraded for the active model/provider.

No other assertion weakening was found. Once these two residuals and their regressions land, the model-support addition can return to the normal ship gate.

@nesquena-hermes nesquena-hermes added changes-requested Maintainer left detailed feedback requesting changes; PR is waiting on author to address size:L Large PR (>10 files or >250 LOC) and removed size:M Medium PR (≤10 files, ≤250 LOC) labels Jul 30, 2026
@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from 1e4aa84 to b1d816c Compare August 3, 2026 10:00
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Les deux résiduels de la watch-tier review sont corrigés sur le head exact b1d816cfdae7b16e4860a6d0568c3fa6ae7f1556.

  • Les IDs de fournisseurs custom nommés passent le provider résolu à _strip_provider_hint_for_reasoning, donc les plafonds suivent correctement gpt-5.5, o3 et gpt-5.6-sol.
  • La commande /reasoning transmet _reasoningEffortContext() au backend afin que la coercition utilise le modèle/provider actifs.
  • Les fonctionnalités amont concurrentes ont été conservées pendant la résolution des conflits.
  • Validation post-rebase : 80 tests ciblés/adjacents réussis, compilation Python, syntaxe JavaScript et git diff --check propres.

Merci de re-review ce SHA exact.

@nesquena-hermes nesquena-hermes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-gate on b1d816cfdae7: two reasoning-capability blockers remain

Thanks for the response. The /reasoning command now posts model/provider context, and the prior explicit-provider custom-gateway case is fixed. The exact-head contributor target is green (89 passed), but two deterministic reviewer probes still fail inside the mandatory sandbox.

  1. Model-qualified provider context is discarded when the redundant provider_id argument is omitted. _resolve_model_reasoning_efforts_impl() correctly derives the provider from a qualified model, but resolve_model_reasoning_efforts() then re-applies the final ceiling with the caller's original blank provider_id; coerce_reasoning_effort_for_model() does the same. At this head, resolve_model_reasoning_efforts('@custom:frontier-gw:gpt-5.5') returns ['high', 'xhigh', 'max', 'ultra'] instead of capping at ['high', 'xhigh'].
  2. A negative/stale models.dev answer erases the explicit GPT-5.6 contract. _models_dev_reasoning_efforts() returns before the GPT-5.6 heuristic. With a concrete negative answer ([]), resolve_model_reasoning_efforts('gpt-5.6-sol', provider_id='openai-codex') returns [], hiding max/ultra. The submitted test covers metadata unavailable (None), not negative ([]).

Please canonicalize effective model/provider/base-URL context once at the public boundary and propagate that tuple through resolve, final ceiling filtering, and coercion. Put the recognized first-party GPT-5.6 capability ahead of a negative metadata answer while preserving hard-deny and explicit config precedence. Add regressions for qualified custom models with omitted provider_id through resolve/coerce and GET/POST /api/reasoning, plus GPT-5.6 with a concrete negative metadata result.

The prior /reasoning POST blocker is closed; no change is requested there.

@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from b1d816c to df3144a Compare August 7, 2026 22:06
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Les deux bloqueurs du re-gate sont corrigés sur df3144a494348e54b620aece79c82ebc592aeea5. Le contexte modèle/fournisseur/base URL est désormais canonicalisé une seule fois aux frontières publiques, y compris lorsque provider_id est omis, et le contrat first-party GPT-5.6 prévaut sur une réponse négative/stale de models.dev après la configuration explicite. Régressions ajoutées; 116 tests ciblés/adjacents passent (1 skip), compilation Python et git diff --check propres. Les 7 échecs Z.AI adjacents ont été reproduits à l’identique sur l’ancien head et ne sont pas causés par cette correction. Merci de re-review ce SHA exact.

@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from df3144a to 4877b4f Compare August 7, 2026 22:12
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Complément sur le même re-gate : les shards CI ont exposé un résiduel du diff antérieur, désormais corrigé sur 4877b4f77b19be014bbef8b027cab20ac4e9f842. Le ladder Z.AI GLM-5.2 reste plafonné à max (sans ultra) et l’attente GPT-5/Z.AI reflète le plafond model-scoped. Validation élargie : 197 tests passent, 1 skip; nouvelle CI en cours sur ce SHA exact.

@nesquena-hermes nesquena-hermes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Watch-tier re-gate on 4877b4f77b19: canonicalization still has deterministic gaps

Thanks for the fast follow-up. The new commit fixes the two blockers from the prior review, and the changed contributor slices are green in the mandatory sandbox. The exact-head re-gate found objective residuals in the new canonicalization path:

  1. Canonical values are computed, then bypassed in the empty-capability branch. coerce_reasoning_effort_for_model() resolves (model, provider, base_url) at api/config.py:4181-4183, but lines 4249 and 4251 call _zai_glm_reasoning_efforts_supported() and _provider_known_reasoning_capable() with the original model_id / provider_id. Reproduced in-sandbox: qualified native-ZAI @zai:glm-5.1 preserves high instead of omitting the unsupported field, and a qualified recognized provider degrades ultra to xhigh when capability metadata is empty.

  2. Status classification repeats the same stale-input split. get_reasoning_status() resolves supported efforts, but lines 4316-4318 classify the ZAI toggle with the uncanonicalized inputs. A qualified @zai:glm-5.1 therefore reports supports_thinking_toggle=False although the resolved native-ZAI model is toggle-only.

  3. The first-party fallback is incomplete and over-broad. The provider set at line 4157 omits the existing azure-foundry lane. The substring test at line 4159 treats unrelated not-gpt-5.6 and gpt-5.60 as GPT-5.6. Reviewer probes reproduce both outcomes.

  4. The resolver's exception fallback is not compatible with the supported legacy string config shape. Line 3968 unconditionally calls .get("provider") on cfg["model"]; a resolver error with model: "..." raises AttributeError instead of returning a conservative tuple.

Requested fix

  • Reuse the canonical model and provider for every downstream empty-capability and ZAI decision.
  • Canonicalize once in get_reasoning_status() and pass the same tuple to effort resolution, toggle classification, and coercion.
  • Use one exact GPT-5.6-family predicate in both capability paths, include azure-foundry, and reject lookalikes such as not-gpt-5.6 / gpt-5.60.
  • Guard the fallback model config with isinstance(model_cfg, dict) before reading provider.
  • Add durable regressions for those four shapes, including a composed provider-omitted /api/reasoning case.

Mandatory gate evidence at this exact head: threat scan CLEAN; contributor reasoning slices 162 passed; unchanged neighbor 9 passed; six reviewer discriminator cases failed deterministically inside Layer 3. All temporary probes were removed and the final worktree was clean. No screenshot is needed for this backend capability contract.

@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from 4877b4f to 0095fb3 Compare August 8, 2026 22:06
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Les quatre résiduels de la re-gate sont corrigés sur le head exact 0095fb3089f2b91886103264bad63c8fadcc79ef. Les décisions de coercition et de statut réutilisent désormais le contexte modèle/fournisseur canonicalisé ; le fallback GPT-5.6 est exact, inclut Azure Foundry et rejette les lookalikes ; le fallback accepte la configuration legacy où model est une chaîne. Après rebase sur l’amont courant : 200 tests ciblés/adjacents passent (1 skip), compilation Python, lint diff-scoped et git diff --check sont propres. CI et Greptile sont en cours sur ce SHA.

@nesquena-hermes nesquena-hermes left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-gate: canonical context fixes most prior gaps, but two capability contracts remain red

Thanks for the follow-up. I re-gated exact head 0095fb3089f2b91886103264bad63c8fadcc79ef against the 4877b4f77b19 review.

The latest commit fixes the stale pre-canonical Z.AI/empty-capability inputs and the legacy string-config crash. Slash-command context wiring, explicit-provider ceilings, qualified-model inference, and the canonical GPT-5.6 stale-metadata override also survive.

Two deterministic blockers remain.

1. Canonicalize the complete Azure Foundry alias family

The installed Agent declares azure-foundry aliases azure, azure-ai-foundry, and azure-ai. WebUI canonicalizes azure, but _resolve_provider_alias does not map azure-ai-foundry or azure-ai.

Those two documented aliases therefore miss:

  • the first-party GPT-5.6 override,
  • _KNOWN_REASONING_PROVIDERS,
  • and the Azure/Claude family ceiling set.

For provider=azure-ai-foundry|azure-ai, model=gpt-5.6-*, the top-tier default-deny treats the route as unknown and suppresses max/ultra, contrary to the Agent provider contract.

Please map both aliases centrally to azure-foundry, then test resolver, coercer, and status behavior for canonical plus both aliases, explicit-provider and @provider:model forms. GPT-5.6 must retain max/ultra; GPT-5.5 and o-series must retain their lower ceilings.

2. Keep rejected GPT-5.6 lookalikes rejected through coercion

_is_gpt_5_6_family() now has the right anchored boundary, so capability resolution rejects not-gpt-5.6. But the effective coercion contract can re-admit it:

  • the older-GPT ceiling checks only bare.startswith("gpt-5"), so the prefixed lookalike bypasses it;
  • openai is a recognized reasoning provider;
  • an authoritative empty capability result reaches if not supported;
  • _provider_known_reasoning_capable(openai) is true, so ultra is returned unchanged.

The UI advertises no supported effort, while coercion can still send ultra (mapped downstream to wire max). Ensure a first-party candidate that fails the exact GPT-5.6 boundary cannot regain max/ultra through the recognized-provider empty-list fallback. Preserve the intentional unknown/custom future-model behavior.

Add a matrix for not-gpt-5.6 and gpt-5.60 across every first-party lane: resolver exposes no top tier, coercer never returns max/ultra, and status cannot report a hidden unsupported top tier. Keep positive controls for gpt-5.6, gpt-5.6-sol, and qualified forms.

Test mirrors

Please also update the JS reasoning-option fixture that claims to mirror all dropdown options but still has eight and omits ultra, and add ultra to the test that claims to cover all valid persistence levels.

Execution note

This re-gate is static-only. The trusted threat scanner could not obtain the GitHub diff because the API returned HTTP 403 rate-limit, so policy required mandatory NO-RUN. No PR code or tests were executed. Both findings above follow directly from the current source and installed Agent alias contract.

@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from 0095fb3 to 83feb8a Compare August 11, 2026 22:09
@ruizanthony

Copy link
Copy Markdown
Contributor Author

Addressed both blockers at exact head 83feb8ab994ff3d385f974d9ce065c4c43e47912. Azure Foundry now canonicalizes azure, azure-ai-foundry, and azure-ai centrally even without the Agent import; resolver, coercer, and status coverage includes explicit and qualified forms plus lower GPT/o-series ceilings. First-party GPT-5.6 lookalikes can no longer regain max/ultra through the recognized-provider empty-capability fallback, while unknown/custom future-model behavior is unchanged. The JS dropdown mirror and valid persistence-level matrix now include ultra. Post-rebase focused/adjacent validation: 184 passed, 1 skipped; compileall, diff-scoped Ruff, and diff check pass. CI and Greptile are running on this SHA.

@nesquena-hermes

Copy link
Copy Markdown
Collaborator

🔬 Gate certification — RED ⛔

Certified contributor head: d0a171469265f773c24d5cdec05b0d73c41b28a2
Frozen master: 8fb95b43a72dc2961b65bd538603a3b32907980d
Tested integration: 166ead9dd042c3550a1b4937b550d760e0545d42 (clean conflict-free rebase; cumulative stable patch ID preserved as 799d24f58cf8f1658cd905f7a9574f3ea52aad1f)
Installed Agent authority: c4d195b2c415968c2513eed9f8819952037a60f3

Verdict: SHIP ONLY WITH FIXES. Exact-head review independently reproduced two patch regressions, plus one degraded Copilot fallback contract mismatch.

What I ran

  • Threat scan: CLEAN, score 0.
  • Codex regression gate: SHIP ONLY WITH FIXES after inspecting all nine changed files. It identified the model-level allowlist and Actual-provider downgrades below.
  • Opus 4.8 cross-layer review: SHIP ONLY WITH THIS FIX because the standalone Copilot fallback advertises levels above the installed Agent ceiling. Fable reported SHIP-UX; its only UX notes were the pre-existing short-viewport dropdown scrolling weakness and xhigh label wording.
  • Full serial sandbox suite: candidate 14,253 passed, 92 skipped, 1 xfailed, 2 xpassed, with 9 failed + 2 errors; same-box frozen-master control 14,238 passed with the exact same 11 residual node IDs. Those shared environment/topology failures are not the RED basis.
  • Focused PR suite: 184 passed, 1 skipped. Node syntax checks, Python compilation, changed-file scope, git diff --check, and diff-scoped Ruff all passed.
  • Installed-Agent contract: the focused GPT-5.6 wire-normalization tests passed 3/3, confirming the Agent maps product ultra to wire max where supported.
  • Visible proof: inspected the 600×375 Cutter animation across sampled frames. The dropdown opens, lists and selects Ultra, and settles to an unclipped Ultra chip. The final visual/command diff is byte-identical to the previously reviewed 83feb8ab state.

Independently reproduced blockers

  1. Model-level top-tier allowlists are no longer authoritative (api/config.py:4024, api/config.py:4148).

    _resolve_model_reasoning_efforts_impl() correctly reads custom_providers[].models.<model>.reasoning_efforts, but the public resolver applies _filter_reasoning_efforts_for_provider() again. That filter consults only provider-level configuration, so a model-only ['high', 'max'] allowlist on an unknown/custom gateway is reduced to ['high']; coercing max no longer returns max.

    Independent sandbox reproduction: frozen master passed the same model-level max contract; this head failed it (['high'] != ['high', 'max']).

  2. The registered actual provider silently loses max (api/config.py:3597).

    actual is absent from _KNOWN_REASONING_PROVIDERS. Even when capability metadata reports the complete supported ladder, resolve_model_reasoning_efforts('glm-5.2-nvfp4', provider_id='actual') strips max, and coercion downgrades it. The installed Actual transport then maps xhigh to effective high, reducing requested reasoning depth.

    Independent sandbox reproduction: frozen master preserved max; this head returned ['minimal', 'low', 'medium', 'high', 'xhigh'] and failed the asserted max contract.

  3. The PR introduces ultra into the standalone Copilot fallback above the Agent's static ceiling (api/config.py:3710).

    With hermes_cli.models unavailable, gpt-5.6-sol on copilot receives the generic ladder including xhigh, max, and ultra. The installed Agent's no-catalog authority for every Copilot GPT-5 model is ['minimal', 'low', 'medium', 'high']. This is a real PR delta: the same assert 'ultra' not in fallback probe passes on frozen master and fails on this head. Production installs with hermes_cli present take the correct high-ceiling path, so this is degraded-path-only, but it violates the fallback's purpose and the prior review requirement.

Required repair

  • Make final top-tier authorization model-aware. Preserve model-level reasoning_efforts before provider-level fallback for both regular and named custom providers, then add resolve + coercion regressions for model-only max/ultra.
  • Add actual to the recognized reasoning-provider contract and prove a metadata-supported Actual model retains max through resolution and coercion.
  • Cap standalone Copilot GPT-5 fallback levels to the Agent's static minimal/low/medium/high ceiling. Add a forced-import-failure regression proving GPT-5.6 cannot gain xhigh, max, or ultra through that fallback.

Recommendation

Do not merge this head. @ruizanthony, the main GPT-5.6, Azure alias, lookalike, Z.AI, slash-command, and visible Ultra paths are otherwise in good shape. Repair the three contract gaps above and request a fresh exact-head gate.

No merge, tag, deploy, close, or contributor-branch push was performed by this lane.

@nesquena-hermes nesquena-hermes added the gate-fail Gate found blocking issue(s); fix-spec in comment; awaiting fix/re-push label Aug 12, 2026
Comment thread api/config.py Outdated
@nesquena-hermes nesquena-hermes removed the changes-requested Maintainer left detailed feedback requesting changes; PR is waiting on author to address label Aug 13, 2026
@nesquena-hermes

Copy link
Copy Markdown
Collaborator

🔬 Gate certification — RED ⛔

Certified contributor head: f420825cf7ee5ea3581c4ee1beadf057ed92b220
Frozen master: f3e0e2981556ff1c93281703e8ebf68f0b0f8a69
Tested integration: 0cdac70d2d634d3ca98a567f2c06a21b6f6c5f2e (clean conflict-free rebase; stable patch ID b8363111020144909794e3631ab3925b7eab0d00)
Installed Agent authority: 6be1ce46849df3afeb22cad866f18ac99c8f0d13

Verdict: SHIP ONLY WITH FIXES. This head closes all three blockers from the prior exact-head RED, but the fresh full gate independently reproduced three new provider/model contract gaps and one short-viewport UI regression.

What passed

  • Threat scan: CLEAN, score 0.
  • Prior RED closure: reviewer-owned production-function probes passed 3/3 for model-only regular/qualified provider allowlists, Actual aliases plus wire ultra → max, and standalone Copilot aliases capped at high.
  • Focused PR suite: 189 passed. Python compilation, Node syntax for both changed JS files, git diff --check, exact changed-file scope, and diff-scoped Ruff all passed.
  • Full serial sandbox suite: candidate 14,264 passed, 92 skipped, 1 xfailed, 2 xpassed, with 9 failed + 2 errors; same-box frozen-master control 14,246 passed with the exact same eleven residual node IDs. Those shared sandbox/topology residuals are not the RED basis.
  • Opus cross-layer review: APPROVE/SHIP, with one latent dotted GPT-5.6 family fast-follow. This does not override reproduced failures outside its accepted contract.
  • Visual evidence: the submitted 600×375 desktop animation shows the menu opening, Ultra being selected, and an Ultra chip. It contains no mobile/short-viewport proof.

Independently reproduced blockers

  1. Unknown/custom ultra still degrades to xhigh, above the stated safe ceiling (api/config.py:4303-4308).

    The final empty-capability fallback handles both max and ultra by returning xhigh. The new code itself calls high the universal fallback for lower-cap providers, while an unknown OpenAI-compatible endpoint has no authority proving xhigh support. A sandboxed production-function probe expected the default-denied ultra request to land on high; this head returned xhigh.

    Required repair: for unknown/custom providers without an explicit provider/model allowlist, degrade both max and ultra to high (or another explicitly proven universal ceiling), while preserving model/provider allowlists as authoritative. Add resolver + coercion + wire regressions.

  2. Pre-adaptive Claude ceilings remain provider-gated and leak through aggregators (api/config.py:3564-3572).

    The comment says the ceiling follows the model across every lane, but _anthropic_lanes omits openrouter and nous. Both are recognized aggregators, so anthropic/claude-sonnet-4.5 resolves with max and ultra; their Agent profiles pass enabled reasoning config through. A sandboxed production-function probe reproduced the full seven-level ladder instead of the expected minimal..xhigh ceiling on both routes.

    Required repair: apply the pre-adaptive Claude ceiling by model across OpenRouter, Nous, and every routed/aggregator lane, then add resolver/coercer tests for prefixed and bare legacy Claude IDs on those providers.

  3. ai-gateway is a registered production provider but is omitted from the recognized reasoning contract (api/config.py:3599-3615).

    The installed Agent registers canonical ai-gateway plus vercel, vercel-ai-gateway, ai_gateway, and aigateway aliases and forwards reasoning config. WebUI resolves all aliases to ai-gateway, then treats it as unknown: a Claude 4.6 model loses max, max degrades to xhigh, and ultra cannot map to max. The sandboxed alias cross-product reproduced this at the real resolver/coercer boundary.

    Required repair: recognize ai-gateway, preserve Claude 4.6 max, map its ultra ceiling to max, and test every installed alias through resolution and coercion.

  4. The new ninth menu row makes Default unreachable on short viewports (static/style.css:2641).

    The reasoning dropdown opens upward with overflow:hidden and no height cap or scroll. A sandboxed Playwright characterization at 390×300 measured menuTop=-50, defaultTop=-45, overflowY=hidden; the override-clearing Default row is above the viewport with no scroll-to-reach path. The submitted animation is desktop-only and does not cover this case.

    Required repair: match sibling dropdowns with a viewport-bounded max-height and overflow-y:auto, then prove desktop, mobile, and short-landscape access to every row, including Default and Ultra.

Recommendation

Do not merge this head. @ruizanthony, the three earlier RED findings are genuinely closed and the broad suite is otherwise clean relative to the frozen-master control. Please repair the four reproduced gaps above, add focused regressions, and request a fresh exact-head gate.

No merge, tag, deploy, close, or contributor-branch push was performed by this lane.

ruizanthony added a commit to ruizanthony/hermes-webui that referenced this pull request Aug 16, 2026
Address the four independently reproduced blockers from the RED gate
certification on f420825 (nesquena#6018):

1. Unknown/custom providers without an explicit provider/model allowlist
   now degrade both max AND ultra to the universally proven 'high'
   ceiling instead of xhigh — nothing proves an unknown OpenAI-compatible
   endpoint accepts xhigh. Explicit provider- and model-level allowlists
   remain authoritative, and an unnamed (empty) provider keeps the
   historical conservative xhigh landing. Resolver + coercion + wire
   (/api/reasoning set/get) regressions added.

2. The pre-adaptive Claude ceiling is now applied BY MODEL on every
   serving lane: the _anthropic_lanes provider gate is removed, so
   OpenRouter, Nous, AI Gateway, and custom gateway routes cap legacy
   Claude (3.x, 4.0-4.5) below max/ultra, for prefixed and bare IDs.
   This also closes the unresolved Greptile P1 thread on api/config.py.

3. ai-gateway is recognized as the registered production provider it is:
   the installed Agent alias family (vercel, vercel-ai-gateway,
   ai_gateway, aigateway) canonicalizes to ai-gateway, adaptive Claude
   4.6 keeps max, and the Codex product-only ultra tier maps down to the
   wire max instead of degrading to xhigh. Every alias is tested through
   resolution, coercion, and status.

4. The 9-row reasoning dropdown gets a viewport-bounded max-height with
   overflow-y:auto (dvh-aware), matching the sibling model/session
   dropdowns, so the Default row stays reachable on short viewports.
   A Playwright characterization proves every row reachable at
   1280x800, 390x844, and the reviewer's 390x300 short-landscape repro
   (fails at head, passes with the fix).
… and copilot lanes

- Default-deny max/ultra for custom/unknown providers unless an explicit
  provider reasoning_efforts allowlist authorizes them (recognized custom
  models covered by regression tests).
- Model-scoped ceilings now follow the model across every lane: older GPT-5
  and o-series via OpenRouter/Nous aggregators are capped so ultra never
  reaches them; GPT-5.6 keeps the top tiers.
- Copilot heuristic fallback routes through the provider filter so only
  GPT-5.6 keeps max/ultra.
- Heuristic/metadata-unavailable fallback branches returning the expanded
  global effort list now apply the GPT-5.6 model check and the
  unknown-provider default-deny.
- Recognize first-class catalog providers (xai-oauth, zai, kimi-coding,
  minimax, opencode-zen/go, mistralai, alibaba, nvidia, xiaomi, nous) so the
  default-deny only hits truly custom/unknown lanes.
Address the four independently reproduced blockers from the RED gate
certification on f420825 (nesquena#6018):

1. Unknown/custom providers without an explicit provider/model allowlist
   now degrade both max AND ultra to the universally proven 'high'
   ceiling instead of xhigh — nothing proves an unknown OpenAI-compatible
   endpoint accepts xhigh. Explicit provider- and model-level allowlists
   remain authoritative, and an unnamed (empty) provider keeps the
   historical conservative xhigh landing. Resolver + coercion + wire
   (/api/reasoning set/get) regressions added.

2. The pre-adaptive Claude ceiling is now applied BY MODEL on every
   serving lane: the _anthropic_lanes provider gate is removed, so
   OpenRouter, Nous, AI Gateway, and custom gateway routes cap legacy
   Claude (3.x, 4.0-4.5) below max/ultra, for prefixed and bare IDs.
   This also closes the unresolved Greptile P1 thread on api/config.py.

3. ai-gateway is recognized as the registered production provider it is:
   the installed Agent alias family (vercel, vercel-ai-gateway,
   ai_gateway, aigateway) canonicalizes to ai-gateway, adaptive Claude
   4.6 keeps max, and the Codex product-only ultra tier maps down to the
   wire max instead of degrading to xhigh. Every alias is tested through
   resolution, coercion, and status.

4. The 9-row reasoning dropdown gets a viewport-bounded max-height with
   overflow-y:auto (dvh-aware), matching the sibling model/session
   dropdowns, so the Default row stays reachable on short viewports.
   A Playwright characterization proves every row reachable at
   1280x800, 390x844, and the reviewer's 390x300 short-landscape repro
   (fails at head, passes with the fix).
@ruizanthony
ruizanthony force-pushed the fix/gpt56-max-ultra-reasoning-webui branch from 13c7308 to 4528baf Compare August 17, 2026 23:04
@nesquena-hermes

Copy link
Copy Markdown
Collaborator

🔬 Gate certification — RED ⛔ (prior capability gaps closed; runtime fallback and selected-row visibility remain)

Certified contributor head: 4528bafea7dc1ddf71e392293ce08b5944d09833
Pinned WebUI base: 63a562f6ed4e20e63377674e4d7b85b75b7fc4ec
Clean rebased integration: 799dc5cfcc27a89fe2e443cbc49cb987c5d1f24b
Installed Hermes Agent authority: dd859ee9725c4c4dabde4d5a999c78ec045bef24
Rebase proof: clean nine-commit rebase; aggregate contributor patch remained equivalent.

Verdict: gate-fail. The four blockers from the prior RED are genuinely closed in the WebUI layer: unknown/custom top tiers fail safely, legacy Claude ceilings follow routed models, ai-gateway aliases preserve their intended ladder, and the short menu is now bounded/scrollable with Default reachable. A current installed-Agent fallback path still forwards persisted ultra unchanged after the destination model becomes GPT-5.5, and the short-viewport picker opens with a selected Ultra row outside the visible scrollport.

What I ran

Gate Result
Exact-head threat scan CLEAN, score 0
Rebase-first / intent equivalence Clean nine-commit rebase; aggregate patch equivalent
Full sandboxed pytest suite, serial, to completion 14,717 passed, 94 skipped, 1 xfailed, 2 xpassed, 34 subtests; 9 failed + 2 errors
Frozen pinned-base attribution Exact inherited 9-fail/2-error baseline already reproduced on this pinned master; zero PR-owned suite residuals
Focused exact-head warm-up suite 198 passed
Codex regression gate SHIP ONLY WITH FIXES on the installed-Agent fallback mismatch
Opus 4.8 cross-layer review GREEN dissent; reverified WebUI/provider closures but did not catch the post-switch wire mismatch below
Fable UX gate SHIP-WITH-UX-FIXES on selected-row visibility
Reviewer installed-runtime probe Reproduced: Agent resolves global Ultra for GPT-5.5 and the real Codex transport emits {"effort":"ultra"}
Reviewer real-browser gate Reproduced selected Ultra outside the 390×300 menu on open; manual scroll reveals it
Static gates Ruff-forward CLEAN, ESLint runtime CLEAN, scope-undef CLEAN, git diff --check clean

The 9 failures + 2 errors are the established same-box/topology baseline on this exact pinned master.

Prior RED blockers — CLOSED ✅

  • Unknown/custom providers without explicit authority now clamp max/ultra to a safe ceiling; explicit model/provider allowlists remain authoritative.
  • Pre-adaptive Claude ceilings are model-scoped across OpenRouter/Nous/routed lanes; adaptive Claude keeps its stronger ladder.
  • ai-gateway/Vercel aliases are centrally recognized and preserve/map max/ultra as intended.
  • The reasoning menu now has a viewport-bounded max-height, overflow-y:auto, and contained overscroll. At 390×300 it measures 180px high with scrollHeight 339; Default is visible and every row is manually reachable.
  • Earlier-round Azure, Actual, Copilot fallback, Z.AI, model-only custom allowlist, lookalike, stale-negative-metadata, canonical-context, status, command, and picker contracts remain covered by the exact-head matrix.

MUST-FIX 1 — Ultra survives an installed-Agent fallback to GPT-5.5 and reaches the wire (CORE)

WebUI exposure: api/config.py adds/persists ultra as a valid reasoning preference.
Installed authority:

  • agent/chat_completion_helpers.py:2795 fallback activation calls resolve_reasoning_config(load_config(), agent.model).
  • agent/agent_runtime_helpers.py:2941 model switching does the same.
  • hermes_constants.resolve_reasoning_config() parses the global agent.reasoning_effort but does not clamp it by the destination model/provider.
  • agent/transports/codex.py:435-458 maps ultra→max only when the target contains gpt-5.6; for GPT-5.5, ultra remains unchanged.

Independent production-composed reproduction against installed Agent dd859ee: resolve config agent.reasoning_effort='ultra' for model gpt-5.5, then pass that real result into ResponsesApiTransport.build_kwargs() for the Codex endpoint. Observed:

{"resolved":{"enabled":true,"effort":"ultra"},"wire_reasoning":{"effort":"ultra","summary":"auto"}}

This conflicts with the same PR's enforced pre-5.6 GPT-5 ceiling (xhigh). A GPT-5.6 Ultra session that activates a GPT-5.5 fallback or /model switch can therefore fail the recovery request instead of degrading.

Required repair: ship a compatible Agent-side destination-aware clamp at both fallback/model-switch reasoning re-resolution chokepoints (or one shared runtime authority called by both). Map GPT-5.6 Ultra to wire Max, pre-5.6 GPT-5 max/ultra→xhigh, o-series max/ultra→high, and apply the corresponding provider/model ceilings. Add a production-composed GPT-5.6 Ultra → GPT-5.5 fallback regression that asserts the final wire request emits xhigh, never ultra.

A release-order note is insufficient: WebUI and Agent are independently versioned, and the current installed authority is the contract this gate must certify.

MUST-FIX 2 — current selection is hidden when the short menu opens (UX)

Sites: static/ui.js:5306-5322, static/style.css:2642-2647.

The new scrollbar closes the old “Default unreachable” blocker, but toggleReasoningDropdown() highlights the current row and opens the menu without bringing that row into view.

Real 390×300 reproduction with Ultra selected: menu rect top=45, bottom=225, clientHeight=178, scrollHeight=339, scrollTop=0. Default is visible at 50..86.8, but selected Ultra is at 344.4..381.2, outside both menu and viewport. After manually setting scrollTop=161, Ultra appears at 183.4..220.2, while Default moves out of view. Desktop 1280×800 and portrait 390×844 fit all rows and are clean.

Required repair: after opening and highlighting, call selected.scrollIntoView({block:'nearest'}) scoped to the reasoning dropdown (or set the equivalent bounded scroll position). Add a real/behavioral 390×300 test proving selected Default, middle, and Ultra options are visible immediately on open while every row remains reachable. Final desktop/mobile/short-landscape screenshots must include Default-at-top, Ultra selected on open, and the selected chip without composer-footer overflow.

Recommendation

Do not merge this head. Preserve @ruizanthony's extensive WebUI capability, alias, coercion, command, and scrollability repairs. Close the installed-runtime transition mismatch and the one-line selected-row visibility issue, then run a fresh exact-head full gate and final visual proof.


Gate-certifier layer only. No merge, tag, deploy, close, or contributor-branch write was performed. This certificate is valid only for contributor head 4528bafea7dc1ddf71e392293ce08b5944d09833.

@nesquena-hermes nesquena-hermes added the changes-requested Maintainer left detailed feedback requesting changes; PR is waiting on author to address label Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changes-requested Maintainer left detailed feedback requesting changes; PR is waiting on author to address gate-fail Gate found blocking issue(s); fix-spec in comment; awaiting fix/re-push size:L Large PR (>10 files or >250 LOC)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants