Skip to content

fix(context): thread custom_providers to all context-length resolution call sites - #75738

Open
TurgutKural wants to merge 2 commits into
NousResearch:mainfrom
TurgutKural:fix/custom-provider-context-length-threading
Open

fix(context): thread custom_providers to all context-length resolution call sites#75738
TurgutKural wants to merge 2 commits into
NousResearch:mainfrom
TurgutKural:fix/custom-provider-context-length-threading

Conversation

@TurgutKural

Copy link
Copy Markdown
Contributor

Problem

custom_providers[].models.<id>.context_length overrides were only honored at agent startup (agent_init) and /model switch (model_switch). Several sibling call paths called get_model_context_length() without passing custom_providers, causing them to fall through to endpoint probing and the 256K/131K hardcoded defaults — even when the user had an explicit per-model override configured.

Observed symptom (log spam on every session start with a custom provider model):

INFO agent.model_metadata: Could not detect context length for model 'qwen3.8-max-preview' at https://… — defaulting to 256,000 tokens (probe-down).
INFO agent.model_metadata: Using hardcoded context length 131,072 for model 'qwen3.8-max-preview' (custom endpoint, catalog match on 'qwen')

This fires twice per startup because both the built-in compressor and the context engine plugin resolve context length independently.

Affected call sites

Call site File Impact
ContextCompressor._resolve_context_length agent/context_compressor.py Deferred first-access probe ignores per-model override
_candidate_context_window agent/auxiliary_client.py Fallback chain screening uses wrong context window
_trim_messages_for_reference agent/moa_loop.py MoA reference trimming uses wrong window
get_model_info hermes_cli/web_server.py WebUI shows wrong auto-detected context

Fix

Thread custom_providers through each path:

  • ContextCompressor: new custom_providers constructor parameter, passed from agent_init (where _custom_providers is already resolved). Used in _resolve_context_length().
  • auxiliary_client: _candidate_context_window lazily loads custom_providers from config via load_config_readonly() + get_compatible_custom_providers() (matching the existing lazy-config pattern in the file).
  • moa_loop: new _load_custom_providers() helper with the same lazy pattern, called from _trim_messages_for_reference.
  • web_server: get_model_info extracts custom_providers from the config it already loads.

All changes are backward-compatible: custom_providers defaults to None and every config load is wrapped in try/except so failures fall through to the existing probe chain.

Tests

20 new tests in tests/agent/test_custom_provider_context_threading.py:

  • ContextCompressor (5): override honored, fallthrough without override, config_context_length precedence, instance storage, default None
  • auxiliary_client (3): override honored, graceful config failure, empty model
  • moa_loop (3): helper returns list, helper returns None on failure, trim respects 1M context
  • web_server (1): custom_providers kwarg passed to resolver
  • get_custom_provider_context_length (8): extended edge-case coverage (list-format models, zero/negative/string values, first-match-wins, non-dict model cfg)

All 90 existing related tests pass without modification:

  • tests/hermes_cli/test_custom_provider_context_length.py (6)
  • tests/agent/test_model_metadata.py (42)
  • tests/agent/test_moa_context_max_tokens.py (2)
  • tests/run_agent/test_switch_model_context.py (6)
  • tests/run_agent/test_invalid_context_length_warning.py (3)
  • tests/hermes_cli/test_model_switch_context_display.py (3)

Closes the sibling-path gap left by #15779.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 1, 2026
@teknium1

teknium1 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Thanks for tracing the missing propagation paths. The core premise is verified on current main: agent/model_metadata.py:2343-2350 only applies a per-model custom-provider override when both the route and custom_providers are supplied; the current auxiliary, MoA, and WebUI callers omit that list.

Problems

  • The claimed all-call-sites coverage is incomplete. model_tools.py:621-641 resolves an active custom endpoint, then calls get_model_context_length() without custom_providers; the tool-search context gate therefore retains the same fallback/default behavior.
  • The inactive-agent /context fallback at gateway/slash_commands.py:785-791 passes only model_name, with no route identity or compatible custom-provider list, so it cannot satisfy the resolver's override preconditions.

Suggested changes

  • Extend the shared propagation to model_tools._resolve_active_context_length() and add a behavioral test using a temporary configured custom provider.
  • Either resolve the configured route/providers for the no-resident-agent /context fallback or reuse a helper that already does so, then test that branch.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Aug 1, 2026
@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch 4 times, most recently from e5e1cc3 to 0298230 Compare August 5, 2026 03:47
@TurgutKural

Copy link
Copy Markdown
Contributor Author

Thanks — both suggested changes are now implemented in e043661470 (on top of today's rebase onto aec331899e):

  1. model_tools._resolve_active_context_length() — now loads the compatible custom_providers list from config and passes it to get_model_context_length(), so the tool-search context gate honors per-model context_length overrides instead of falling back to generic metadata for custom endpoints.

  2. Inactive-agent /context fallback (gateway/slash_commands.py) — the fallback now resolves the configured model.provider route identity and the compatible custom_providers list before calling the resolver, instead of passing only model_name. This satisfies the resolver's override preconditions on the no-resident-agent path.

Regression coverage added in tests/agent/test_custom_provider_context_threading.py:

  • behavioral capture test asserting get_model_context_length() receives custom_providers + route provider from the tool-search gate (and that the gate degrades to 0 on config-load failure),
  • behavioral capture test for the /context fallback (custom_providers + provider + model kwargs), plus a fail-open test when config loading raises.

Local validation on the rebased worktree: tests/agent/test_custom_provider_context_threading.py (24 passed) and tests/test_model_tools.py (24 passed); py_compile clean on all three touched files.

@andrexibiza

Copy link
Copy Markdown
Contributor

Coordination from the web_server.py god-file decomposition (epic #78647): the custom-endpoints family has been extracted into web_routers/custom_endpoints.py in #79127 (slice R3-C1). Verified LOW overlap with this PR — adjacent regions, disjoint code. Both compose; no conflict expected. Interlocked: #79127 and the full slice wave (#79123-#79129).

@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch 4 times, most recently from ebc7caa to 5a8d5d5 Compare August 12, 2026 03:44
@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch 3 times, most recently from f6be6c6 to 984166d Compare August 14, 2026 03:49
@alt-glitch alt-glitch added needs-decision Awaiting maintainer decision before any implementation comp/tools Tool registry, model_tools, toolsets comp/gateway Gateway runner, session dispatch, delivery and removed needs-decision Awaiting maintainer decision before any implementation labels Aug 14, 2026
@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch from 984166d to ff77f14 Compare August 15, 2026 04:15
@alt-glitch alt-glitch added comp/dashboard Web dashboard / control panel UI (dashboard/, landing) and removed comp/cli CLI entry point, hermes_cli/, setup wizard labels Aug 15, 2026
@alt-glitch alt-glitch added the comp/cli CLI entry point, hermes_cli/, setup wizard label Aug 15, 2026
@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch 4 times, most recently from 7e4e674 to c6b00a8 Compare August 18, 2026 04:35
…n call sites

custom_providers[].models.<id>.context_length overrides were only honored
at agent startup (agent_init) and /model switch (model_switch). Several
sibling call paths called get_model_context_length() without passing
custom_providers, causing them to fall through to endpoint probing and
the 256K/131K hardcoded defaults — even when the user had an explicit
per-model override configured.

Affected call sites:
- ContextCompressor._resolve_context_length (deferred first-access probe)
- auxiliary_client._candidate_context_window (fallback chain screening)
- moa_loop._trim_messages_for_reference (MoA reference model trimming)
- web_server.get_model_info (WebUI model info endpoint)

Upstream added the custom_providers parameter to get_model_context_length
(f981d47, NousResearch#15844); this PR threads it through the remaining call
sites, which upstream has not covered yet (verified against current
upstream/main: ContextCompressor construction, _candidate_context_window,
moa_loop, and web_server.get_model_info all still omit it).

Tests: 20 tests covering all four call sites, precedence rules, graceful
degradation, and extended helper coverage.
@TurgutKural
TurgutKural force-pushed the fix/custom-provider-context-length-threading branch from c6b00a8 to a06affd Compare August 19, 2026 04:19
@TurgutKural

Copy link
Copy Markdown
Contributor Author

Rebased onto current upstream/main (395c70d). The three red slices on the previous head (2/12, 6/12: test_session_api.py; 12/12: test_goal_continuation_drain.py + test_goal_max_turns_config.py + test_transcription_tools.py) were the Aug-18 gateway timing races and a transcription idle-timeout flake, de-flaked upstream in a75d1b5 / c70e152 / 5633764 — none reproduce on the rebased head; all 12 test slices and all required checks are green.

The two sweeper review items remain implemented from the previous cycle: model_tools._resolve_active_context_length() passes the configured custom_providers list to get_model_context_length(), and the inactive-agent /context fallback in gateway/slash_commands.py resolves route identity + compatible providers before calling the resolver. Regression coverage in tests/agent/test_custom_provider_context_threading.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/dashboard Web dashboard / control panel UI (dashboard/, landing) comp/gateway Gateway runner, session dispatch, delivery comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants