Skip to content

fix(agent): let context length resolution fall through when failed to get context_length from provider - #6495

Open
nocoo wants to merge 1 commit into
NousResearch:mainfrom
nocoo:fix/context-length-fallthrough-for-custom-endpoints
Open

fix(agent): let context length resolution fall through when failed to get context_length from provider#6495
nocoo wants to merge 1 commit into
NousResearch:mainfrom
nocoo:fix/context-length-fallthrough-for-custom-endpoints

Conversation

@nocoo

@nocoo nocoo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes an early return in get_model_context_length() that causes custom endpoints without context_length in their /v1/models response to default to 128K, even when later resolution steps (hardcoded defaults, models.dev, OpenRouter) could correctly resolve the context length.

Example: A proxy serving claude-opus-4.6-1m that omits context_length from its model metadata would return 128K instead of 1M, because the hardcoded DEFAULT_CONTEXT_LENGTHS entry "claude-opus-4.6" (a valid substring match at step 8) was never reached.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Changes Made

  • agent/model_metadata.py: Remove the early return DEFAULT_FALLBACK_CONTEXT at the end of the custom endpoint probe block (step 2/3). The function now falls through to remaining resolution steps (4–8) when the endpoint probe does not yield a context_length. The logger.info message is preserved for observability.
  • tests/agent/test_model_metadata.py: Update test_custom_endpoint_without_metadata_falls_through_to_hardcoded_default (formerly ..._skips_name_based_default) to assert the new fall-through behavior. Add two new regression tests:
    • test_custom_endpoint_no_context_length_falls_through_to_defaults — endpoint returns model metadata without context_length
    • test_custom_endpoint_no_match_falls_through_to_defaults — endpoint returns unrelated models, requested model falls through to hardcoded defaults

How to Test

from agent.model_metadata import get_model_context_length

# Before fix: returns 128000
# After fix: returns 1000000
result = get_model_context_length(
    "claude-opus-4.6-1m",
    base_url="http://localhost:7024/v1",  # proxy without context_length in /models
    api_key="test-key",
)
assert result == 1000000
python -m pytest tests/agent/test_model_metadata.py tests/agent/test_model_metadata_local_ctx.py -q

Checklist

Code

…dpoint lacks context_length

When a custom endpoint (e.g. an API proxy) recognizes a model name in
its /v1/models response but does not include a context_length field, the
early return on line 912 caused get_model_context_length() to return
DEFAULT_FALLBACK_CONTEXT (128K) immediately. This prevented the function
from reaching later resolution steps (Anthropic API, models.dev,
OpenRouter, hardcoded defaults) that could correctly resolve the context
length.

For example, a proxy serving 'claude-opus-4.6-1m' without context_length
metadata would return 128K instead of 1M, because the hardcoded
DEFAULT_CONTEXT_LENGTHS entry 'claude-opus-4.6' (a substring match)
was never reached.

Remove the early return and let the resolution chain continue. Keep the
logger.info message for observability.

Update the existing test that asserted the old 128K behavior and add two
new regression tests covering the exact scenarios fixed.
@nocoo
nocoo marked this pull request as ready for review April 9, 2026 09:11
@nocoo nocoo changed the title fix(agent): let context length resolution fall through when custom endpoint lacks context_length fix(agent): let context length resolution fall through when failed to get context_length from provider Apr 9, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Apr 30, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #15563, #12059 — same early-return bug in get_model_context_length() step 2 short-circuiting to 128K.

1 similar comment
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #15563, #12059 — same early-return bug in get_model_context_length() step 2 short-circuiting to 128K.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for isolating the custom-endpoint early-return issue.

Problems

  • Current main already fixes the catalog-match case through the targeted fallback at agent/model_metadata.py:2238-2258, with regression coverage at tests/agent/test_model_metadata.py:999-1053 (commit 2e61de063).
  • This PR's broader fall-through changes precedence beyond that fix, but its added tests only exercise hardcoded catalog matches. They do not demonstrate a result obtained specifically from a later provider, models.dev, or OpenRouter lookup.

Suggested changes

  • Add a regression whose expected result cannot be produced by DEFAULT_CONTEXT_LENGTHS, and document why the later metadata source is authoritative for an incomplete custom endpoint.
  • Clarify the intended source precedence before extending the current bounded catalog fallback.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants