feat: always show reasoning effort selector, default off for unrecognized models (#3377)

[b3nw]

feat: always show reasoning effort selector, default off for unrecognized models (#3377)

Improvement on the #3379 which fixed #3377.

Thinking Path

• Hermes WebUI allows configuring reasoning-effort levels for models that support thinking.
• Previously, the thinking/reasoning chip was shown or hidden based on a heuristic: recognized reasoning models got the chip, unrecognized models had it completely hidden.
• This caused false negatives where custom providers, aggregator-rewritten model IDs (e.g., claude-sonnet-4-6:free), and new model releases would silently hide the selector even though the user might want to enable reasoning.
• Rather than continuing to chase an ever-growing heuristic checklist, this PR inverts the model: always show the chip, but set the default based on whether the model is positively identified as reasoning-capable.
• Recognized models (GPT-5+, Claude 4/3.7+, Qwen-3+, DeepSeek, Kimi, etc.) default to "Default" (active reasoning). Unrecognized models default to "None" (off), letting users opt-in on any model.
• The benefit is that no model is ever locked out of the reasoning feature — the user always has the choice.

What Changed

1. Backend: Always return full effort list with reasoning_default_on flag

In api/config.py (get_reasoning_status):

• When resolve_model_reasoning_efforts returns an empty list (unrecognized model), fall back to the full VALID_REASONING_EFFORTS list instead of [].
• supports_reasoning_effort is now always True.
• Added reasoning_default_on: True when the model was positively identified as reasoning-capable, False otherwise.

2. Frontend: Always show chip, default to "None" for unrecognized models

In static/ui.js:

• _applyReasoningChip() no longer hides the chip when supported_efforts is empty. The chip is always displayed.
• When reasoning_default_on is False and no effort has been explicitly set by the user, the chip defaults to "None" (inactive state).
• _applyReasoningOptions() now shows all effort levels in the dropdown when the supported set is empty (previously hid them all).
• fetchReasoningChip() error handler defaults to reasoning_default_on: false so the chip remains functional even on API errors.

3. Expanded Test Coverage

In tests/test_reasoning_effort_model_capabilities.py:

• Updated test_get_reasoning_status_includes_supported_efforts to assert reasoning_default_on is True.
• Added test_get_reasoning_status_unrecognized_model_still_offers_efforts: verifies that unrecognized models get the full effort list with reasoning_default_on=False.
• Added test_get_reasoning_status_recognized_model_default_on: verifies that recognized models get reasoning_default_on=True.

4. Changelog Documentation

• Documented the change in CHANGELOG.md under ## [Unreleased].

Why It Matters

Previously, the heuristic-based hide/show logic was a constant source of false negatives: every new model release, custom provider configuration, or aggregator-rewritten model ID risked hiding the reasoning selector. This PR eliminates that class of bugs entirely by never hiding the chip. Users can always opt into reasoning, and the default is informed but not restrictive.

Verification

Automated tests

Ran the pytest suite targeting reasoning effort model capabilities:

uv run --with pytest --with pyyaml --with cryptography pytest \
  tests/test_reasoning_effort_model_capabilities.py \
  tests/test_custom_provider_bare_model_reasoning.py -v

Result: 28 passed successfully.

Risks / Follow-ups

• Current effort persistence: If a user sets an explicit effort (e.g., "high") on an unrecognized model and later switches to another unrecognized model, the persisted effort carries over. The frontend only defaults to "None" when no effort is persisted — an existing persisted value is respected. This is intentional; it matches the pre-PR behavior where the CLI's agent.reasoning_effort config is profile-scoped, not model-scoped.
• Backward compatibility: supports_reasoning_effort is now always True, which changes the API contract. No known consumers rely on this boolean for chip visibility (the frontend uses reasoning_default_on), but third-party integrations should be aware.

AI Usage Disclosure

• Provider: cursor / anthropic
• Model: claude opus 4.6 via cursor
• Tool Use: Explored codebase reasoning-effort flow end-to-end, implemented backend and frontend changes, wrote tests, hotpatched production container for validation, and drafted this PR description.

[nesquena-hermes]

Pulled the branch and read get_reasoning_status (api/config.py:2322), the JS chip logic (static/ui.js:1854-1912), and both test files against origin/master. The inversion is clean and the contract is internally consistent — one nuance about the empty-list semantics is worth a look before merge.

The backend change reads correctly

model_recognized = bool(supported_efforts)
if not supported_efforts:
    supported_efforts = list(VALID_REASONING_EFFORTS)
return {
    ...
    "supported_efforts": supported_efforts,
    "supports_reasoning_effort": True,
    "reasoning_default_on": model_recognized,
}

supports_reasoning_effort is now hard-coded True, and reasoning_default_on carries the old recognition signal. I grepped all consumers of supports_reasoning_effort — nothing in static/ reads it anymore (only tests/test_models_dev_reasoning.py:154 and the capability tests assert is True), so pinning it to True is safe and no test asserts the False branch. Good.

Frontend matches

_applyReasoningChip (ui.js:1866) now unconditionally sets wrap.style.display='' and only flips the default value to 'none' when reasoning_default_on is false and no effort is set:

var defaultOn=(meta&&meta.reasoning_default_on!==undefined)?meta.reasoning_default_on:true;
if(!defaultOn&&(!effort||effort==='')){ effort='none'; }

The error/catch path in fetchReasoningChip (ui.js:1898) was updated to {supported_efforts:null,reasoning_default_on:false}, which keeps the chip visible with options intact (since _applyReasoningOptions now shows all when !supported.size) rather than the old hidden state. The modified assertion in test_reasoning_chip_btw_fixes.py ("wrap.style.display='none'" not in fn) correctly locks in "never hide."

One nuance: [] has two meanings upstream

resolve_model_reasoning_efforts returns [] in two semantically distinct cases:

1. Unrecognized model — genuinely unknown, the case this PR targets. "Show selector, default off, let the user opt in" is exactly right here.
2. Positively known NOT to support reasoning — the ACP subprocess providers return [] deliberately at api/config.py:2285:

if provider in {"cursor-acp", "copilot-acp"}:
    return []

(and the capability layer returns [] when supports_reasoning is False, config.py:~2197).

After this change, a cursor-acp / copilot-acp session shows a reasoning-effort selector even though that provider can't honor it. The practical harm is low: reasoning_default_on=False means it defaults to "none" and won't send anything unless the user explicitly opts in, and the downstream path is defensive — streaming.py:4943 runs the selected value through parse_reasoning_effort and only attaches reasoning_config when non-None and the agent accepts the param (streaming.py:4974). So a stray opt-in on an ACP model degrades to a no-op, not an error. But it is a control that looks actionable and isn't. If you want to preserve the "positively unsupported" signal, the cheap fix is to keep returning reasoning_default_on=False and a supports_reasoning_effort=False for the ACP set, and have the JS hide only when explicitly false — but that partly re-introduces the heuristic the PR is trying to retire, so it's a judgment call. Flagging it rather than blocking on it.

Tests

The two new cases in test_reasoning_effort_model_capabilities.py cover both the recognized (reasoning_default_on True) and unrecognized (> 0 efforts, reasoning_default_on False) paths by monkeypatching resolve_model_reasoning_efforts. Per cron policy I didn't execute them, but the assertions match the backend logic above. Consider adding one case pinning the ACP-provider expectation either way, so the chosen behavior for case (2) is intentional and regression-guarded.

Overall a sensible inversion — replacing an ever-growing recognition checklist with "always available, smart default" is the right direction for #3377.

[b3nw]

updated the implementation, summary of changes below, preformed manual testing to validate. @nesquena-hermes

Changes Made

1. Upstream Merged: Merged origin/master into feat/3377-thinking-level-missing to bring in get_config_for_profile_home and other recent framework fixes, resolving the WebUI runtime crash.
2. Subprocess/ACP Prefix Checks: Modified get_reasoning_status in api/config.py to identify explicitly unsupported models whose ID contains a slash with an ACP subprocess provider namespace (e.g. cursor-acp/* or copilot-acp/*), hiding the controls.
3. Fallback Upstream Model Lookup:
   • Refactored _models_dev_reasoning_efforts to fallback and search upstream providers (openai, anthropic, gemini, google, deepseek, etc.) when capabilities return None under custom proxy providers (such as llm-proxy).
   • Normalizes the lookup model name by stripping any namespace prefixes (e.g., copilot/gpt-4o -> gpt-4o), so it correctly resolves against standard capabilities (e.g. gpt-4o under openai is flagged as supports_reasoning=False).

Automated Tests

Verified the changes locally using:

uv run --with pytest --with pyyaml --with cryptography pytest tests/test_reasoning_effort_model_capabilities.py tests/test_reasoning_chip_btw_fixes.py

• Result: All 28 tests passed successfully.

Manual Verification

• Explicitly Unsupported Model (copilot/gpt-4o): Verified that the reasoning chip is completely hidden from the WebUI.
• Unrecognized/Custom Model (google/gemini-flash-lite-latest): Verified that the reasoning chip is visible and defaults to "None" (opt-in).
• Supported Model (copilot/claude-sonnet-4.6): Verified that the reasoning chip is visible and defaults to "Default".

[nesquena-hermes]

Pulled the updated branch (HEAD 26319406) and read the full get_reasoning_status (api/config.py:2386-2475), the reworked _models_dev_reasoning_efforts fallback (api/config.py:2282-2326), the JS in static/ui.js:1877-1930, and the four new test cases. This cleanly addresses the ACP nuance I raised last round — the cursor-acp/copilot-acp set and the models.dev supports_reasoning=False set now both yield supports_reasoning_effort=False and a hidden chip, and the JS reads that flag directly instead of inferring from list length. The manual matrix you posted (copilot/gpt-4o hidden, google/gemini-flash-lite-latest visible+off, copilot/claude-sonnet-4.6 visible+default) lines up with the code. One real edge case before merge.

The two _models_dev_reasoning_efforts calls can disagree for copilot/lmstudio

get_reasoning_status derives supported_efforts from the primary resolver, then separately re-derives the "positively unsupported" signal by calling _models_dev_reasoning_efforts directly:

supported_efforts = resolve_model_reasoning_efforts(resolve_model, ...)
...
elif resolve_model:
    hinted_model = _strip_provider_hint_for_reasoning(resolve_model)
    metadata_efforts = _models_dev_reasoning_efforts(hinted_model, provider)
    if metadata_efforts == []:
        explicitly_unsupported = True
...
if explicitly_unsupported:
    supported_efforts = []          # <-- overwrites the primary result

The problem: resolve_model_reasoning_efforts (api/config.py:2329-2380) does not route copilot/lmstudio through _models_dev_reasoning_efforts. For copilot/github-copilot it returns github_model_reasoning_efforts(...) and for lmstudio it probes the live endpoint — those are the authoritative sources for those providers and it returns before ever consulting models.dev. But the new elif branch calls _models_dev_reasoning_efforts unconditionally, and your new cross-provider fallback (api/config.py:2299-2316) now resolves the bare model name against standard catalogs:

bare_model = model.rsplit("/", 1)[-1]
standard_providers = ["openai","anthropic","gemini","google","deepseek","xai","mistral","copilot","openrouter"]
for p in standard_providers:
    ...
    caps = get_model_capabilities(provider=p, model=lookup_model)
    if caps is not None:
        capabilities = caps; break

So a copilot model whose GitHub API answer is "reasoning supported" (step-1 returns a non-empty list) but whose bare name matches a supports_reasoning=False entry in some standard catalog (e.g. gpt-4o resolved under openai) gets metadata_efforts == [] from step-2, flips explicitly_unsupported=True, and the non-empty step-1 result is overwritten with [] — hiding a chip the authoritative resolver had enabled. copilot is in PROVIDER_TO_MODELS_DEV (agent/models_dev.py:160 → "github-copilot"), so this path is live, not theoretical.

Recommendation

A non-empty step-1 result is authoritative — the model demonstrably supports reasoning, so it should never be re-marked unsupported. Gate the metadata recovery on the primary resolver having come back empty:

elif resolve_model and not supported_efforts:
    hinted_model = _strip_provider_hint_for_reasoning(resolve_model)
    metadata_efforts = _models_dev_reasoning_efforts(hinted_model, provider)
    if metadata_efforts == []:
        explicitly_unsupported = True

This keeps every passing case you tested (ACP and copilot/gpt-4o both still produce [] at step-1, so the recovery still fires) while preventing the cross-provider bare-name fallback from clobbering an authoritative copilot/lmstudio "supported" answer. The []-collapse-loses-the-distinction problem you're working around only exists when step-1 already returned [], so this guard is exactly the right scope.

Minor: the bare-name fallback's first-match-wins loop is order-sensitive (openai before anthropic before openrouter); for a name that exists under several catalogs the iteration order silently decides. Low risk, but a one-line case in the test file pinning a known collision would lock the chosen precedence. Also a couple of the new blank lines carry trailing whitespace (e.g. the line after supported_efforts = resolve_model_reasoning_efforts(...)).

Solid iteration overall — the contract is now explicit on both sides and the test coverage for the recognized / unrecognized / ACP / metadata-unsupported quadrants is good.

[b3nw]

Changes Made - @nesquena-hermes

1. Gated Metadata Recovery in get_reasoning_status:

   • Gated the _models_dev_reasoning_efforts metadata check in get_reasoning_status (api/config.py) to only run if supported_efforts is empty (not supported_efforts).
   • This ensures that authoritative provider-specific resolver results (such as Copilot or LMStudio) are never clobbered or overridden by fallback checks.
   • Cleaned up trailing whitespaces in the modified sections of api/config.py.

2. Added Verification Tests:

   • test_get_reasoning_status_copilot_disagreement_authoritative: Asserts that when Copilot resolves reasoning capabilities authoritatively, fallback metadata recovery is bypassed and doesn't override the result.
   • test_models_dev_reasoning_efforts_precedence_loop: Pins the deterministic search order of standard providers (openai, anthropic, gemini, etc.) during fallback lookup to prevent order-sensitivity regressions.

[greptile-apps]

Greptile Summary

This PR changes the reasoning-effort chip from a hide-when-unrecognized model to always-show-with-smart-default. Recognized models keep their "Default" (active) starting state; unrecognized/custom-provider models default to "None" (off) while still surfacing the full effort selector. The backend adds reasoning_default_on to the API response and a _models_dev_reasoning_efforts fallback loop that tries standard providers when the primary provider returns nothing.

• Backend (api/config.py): get_reasoning_status now determines explicitly_unsupported (ACP subprocess providers and models with an empty metadata catalog entry) and returns reasoning_default_on + always-True supports_reasoning_effort for every other model; the new standard-provider fallback loop in _models_dev_reasoning_efforts retries up to 9 providers when the primary lookup returns None.
• Frontend (static/ui.js): _applyReasoningChip now reads reasoning_default_on to force effort='none' when the model is unrecognized and no effort was previously persisted; the error-path fallback changes from hiding the chip to showing it with "None" default.
• Tests: Three new test cases cover unrecognized-model exposure, recognized-model positive flag, and provider-loop ordering.

Confidence Score: 5/5

Safe to merge; the always-show chip behavior is well-guarded and the ACP/explicitly-unsupported path still hides the chip correctly.

The backend and frontend changes are logically consistent: the new reasoning_default_on flag flows cleanly from get_reasoning_status through all three _applyReasoningChip call sites, all of which supply a meta object. The explicitly_unsupported guard correctly preserves the hide-chip path for ACP providers and metadata-confirmed non-reasoning models.

The _models_dev_reasoning_efforts exception-handling change in api/config.py is worth a second look before merging.

Important Files Changed

Filename	Overview
api/config.py	Adds reasoning_default_on flag, explicitly_unsupported detection, and a standard-provider fallback loop in _models_dev_reasoning_efforts; the exception-to-fallback path change and the second _models_dev_reasoning_efforts call (flagged in a prior thread) are the areas worth reviewing.
static/ui.js	All three _applyReasoningChip call sites pass a meta object, so the defaultOn fallback-to-true path is never reached in practice; the error-handler and option-visibility changes are correct.
tests/test_reasoning_effort_model_capabilities.py	New tests are well-structured; test_get_reasoning_status_cursor_acp_not_supported is non-hermetic but safe because the ACP early-return fires before any I/O; provider-loop ordering test correctly expects all 9 standard providers after the primary call.
tests/test_reasoning_chip_btw_fixes.py	Only cosmetic change — error message suffix (#3377) added to assertion string; no logic change.
CHANGELOG.md	Conflict marker from previous review thread is resolved; new Unreleased entry accurately describes the behavior change.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[get_reasoning_status called] --> B[resolve_model_reasoning_efforts]
    B --> C{supported_efforts non-empty?}
    C -- Yes --> D[model_recognized = True]
    C -- No --> E{ACP provider?}
    E -- Yes --> F[explicitly_unsupported = True]
    E -- No --> G[_models_dev_reasoning_efforts metadata check]
    G --> H{metadata_efforts empty list?}
    H -- Yes --> F
    H -- No or None --> I[model_recognized = False]
    D --> J[reasoning_default_on=True, supports=True, efforts unchanged]
    I --> K[reasoning_default_on=False, supports=True, efforts=VALID list]
    F --> L[reasoning_default_on=False, supports=False, efforts empty]
    J --> M[API response to frontend]
    K --> M
    L --> M
    M --> N[_applyReasoningChip in ui.js]
    N --> O{supports_reasoning_effort?}
    O -- False --> P[Hide chip entirely]
    O -- True --> Q{reasoning_default_on and no persisted effort?}
    Q -- False --> R[Default effort = none]
    Q -- True --> S[Default effort = active reasoning]
    R --> T[Show chip inactive]
    S --> U[Show chip active]

Loading

_{Reviews (3): Last reviewed commit: "chore(changelog): resolve residual upstr..." | Re-trigger Greptile}

[greptile-apps]

_models_dev_reasoning_efforts not mocked — test can flip if catalog is reachable

The test patches resolve_model_reasoning_efforts to return [], but get_reasoning_status then calls _models_dev_reasoning_efforts directly (the new explicitly_unsupported check). If agent.models_dev is importable in the test environment and get_model_capabilities returns a capabilities object with supports_reasoning=False for "some-unknown-model", the function returns [], explicitly_unsupported becomes True, and both supports_reasoning_effort and len(supported_efforts) assertions fail. Adding monkeypatch.setattr(cfg, "_models_dev_reasoning_efforts", lambda *a, **k: None) makes the intent explicit and removes the environmental dependency.

[nesquena-hermes]

Pulled the updated branch (HEAD 69185804) and read the reworked get_reasoning_status (api/config.py:2416-2472) plus the two new test cases. The gate I suggested landed exactly as intended, and the disagreement case is now regression-guarded — this resolves the copilot/lmstudio clobber I flagged last round.

The gate is correct

if (provider in {"cursor-acp", "copilot-acp"} or model_prefix in {"cursor-acp", "copilot-acp"}):
    explicitly_unsupported = True
elif resolve_model and not supported_efforts:        # <-- the new guard
    hinted_model = _strip_provider_hint_for_reasoning(resolve_model)
    metadata_efforts = _models_dev_reasoning_efforts(hinted_model, provider)
    if metadata_efforts == []:
        explicitly_unsupported = True

The not supported_efforts condition means the metadata recovery only fires when the authoritative resolver already came back empty, so a non-empty copilot/lmstudio result can no longer be overwritten by the bare-name catalog fallback. That's the exact scope — the []-collapse-loses-the-distinction workaround only ever mattered when step-1 returned [].

The new test actually proves it

test_get_reasoning_status_copilot_disagreement_authoritative (tests/test_reasoning_effort_model_capabilities.py:131) is the right shape — it asserts both the result and that the second lookup never runs:

assert status["supported_efforts"] == ["medium", "high"]
assert not called_metadata_check, "Should not query models.dev metadata since resolver returned success"

The not called_metadata_check assertion is the key one: it locks in that the gate short-circuits before the redundant call, which also addresses the double-lookup latency concern greptile raised — for any model the resolver recognizes, there's now exactly one lookup, not two. test_models_dev_reasoning_efforts_precedence_loop (line 159) pins the 9-provider iteration order, so the order-sensitivity nit is covered too.

One small test-isolation note

test_get_reasoning_status_unrecognized_model_still_offers_efforts (line 71) monkeypatches resolve_model_reasoning_efforts → [] but leaves _models_dev_reasoning_efforts un-mocked. With the gate, supported_efforts is [], so the elif branch now does fire and calls the real _models_dev_reasoning_efforts("some-unknown-model", ...). The test passes only because an unknown model returns None (not []) from that path, so explicitly_unsupported stays false. That's correct today, but it's an implicit dependency on the live catalog answering None for an unknown name. A one-line monkeypatch.setattr(cfg, "_models_dev_reasoning_efforts", lambda *a, **k: None) would make it hermetic and immune to CI catalog state — worth adding since the other three cases already mock it.

Contract is now explicit on both sides and the quadrant coverage (recognized / unrecognized / ACP / metadata-unsupported / authoritative-disagreement) is solid. Reads merge-ready to me modulo that one test mock.

[claw-io]

@nesquena-hermes Merged master into feat/3377-thinking-level-missing to bring in recent framework updates and cleanly resolved the merge conflict in CHANGELOG.md.

Also addressed the test isolation concern by adding mock coverage for _models_dev_reasoning_efforts in test_get_reasoning_status_unrecognized_model_still_offers_efforts so the test is fully hermetic. All 59 tests in the reasoning suite pass successfully. This is ready to go!

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[claw-io]

Cleaned up a residual conflict marker in CHANGELOG.md left over from the merge against master. The branch is completely pristine now!

[nesquena-hermes]

Thanks for the thoughtful rework here, @b3nw — and you've correctly identified a real problem: the heuristic produces false negatives, hiding the reasoning selector for custom providers and aggregator-rewritten model IDs (e.g. claude-sonnet-4-6:free) that genuinely do support reasoning.

After a maintainer review, we've decided not to take the "always show the chip" approach. The product direction is that the reasoning-effort selector should appear only when the model actually supports reasoning options — surfacing it on every model (even with a sensible default-off) puts a control in front of users that does nothing on most models, which is the opposite of the intent.

So this is a detection problem, not a visibility-model problem: the fix we want is to make resolve_model_reasoning_efforts (and its callers) correctly recognize the reasoning-capable cases it's currently missing — custom providers, :free/:thinking-suffixed aggregator IDs, and newly-released reasoning models — so the chip shows for those because they're supported, rather than showing for everything.

I'm going to close this PR in favor of that approach. If you'd like to take a run at improving the capability detection itself (the suffix-stripping + custom-provider path in particular), that would be a very welcome contribution and we'd happily review it. Really appreciate the effort and the clear write-up — the false-negative cases you documented here are exactly the ones a detection fix should target.