Skip to content

fix(agent): default context lookup for empty model IDs (salvage #65515) — un-reds main CI - #85498

Merged
kshitijk4poor merged 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/65515-empty-model-context
Aug 13, 2026
Merged

fix(agent): default context lookup for empty model IDs (salvage #65515) — un-reds main CI#85498
kshitijk4poor merged 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/65515-empty-model-context

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

Every open PR's CI is currently red on one shard with:

FAILED tests/run_agent/test_primary_runtime_restore.py::TestTryRecoverPrimaryTransport::test_allowed_for_nous_anthropic_messages
ValueError: Model  has a context window of 32,000 tokens, which is below the minimum 64,000 required by Hermes Agent.

(Seen on #85444 slice 7/12, #85452 slice 2/12; reproduces locally on plain upstream/main files.)

Root cause: the test constructs an agent with model="" against the live Nous portal URL. get_model_context_length("") reaches _resolve_endpoint_context_length, whose fuzzy matcher

if model in key or key in model:

is vacuously true for an empty model"" in key matches every entry — so it picks an arbitrary model from the live /v1/models response and returns that entry's context length. The portal catalog changed recently; the arbitrary match now lands on a 32K entry, init_agent raises the 64K-floor ValueError, and the junk value is persisted under a @https://inference-api.nousresearch.com/v1 cache key (visible in the failing log: Cached context length @https://... -> 32,000 tokens).

This is a salvage of #65515 by @whirmill (open since July 16), rebased onto current main — the same guard, now positioned after the malformed-base_url normalization that landed since, plus an explanatory comment documenting the fuzzy-match footgun. Contributor authorship preserved on the commit.

Changes

  • agent/model_metadata.py: a blank/empty model id falls back to DEFAULT_FALLBACK_CONTEXT immediately — before any cache write or network probe.
  • tests/agent/test_model_metadata.py: test_empty_model_uses_fallback_context (covers "" and None).

Validation

  • The two previously failing/new tests pass: test_allowed_for_nous_anthropic_messages + test_empty_model_uses_fallback_context.
  • Full tests/agent/test_model_metadata.py + tests/run_agent/test_primary_runtime_restore.py: 92 passed.
  • Mutation check: removing the guard makes test_empty_model_uses_fallback_context fail (arbitrary live-catalog match returns non-default), restoring goes green.
  • ruff check clean.

Closes #65515.

An empty/blank model id reaching get_model_context_length() can't be
meaningfully resolved — and it's worse than a miss: the endpoint
metadata fuzzy matcher ('model in key or key in model') is vacuously
true for "", so it matches an ARBITRARY catalog entry from the live
/v1/models response and returns whatever context length that entry
happens to have, persisting it under a junk '@<base_url>' cache key.

This started failing CI on main when the Nous portal catalog changed:
tests/run_agent/test_primary_runtime_restore.py constructs agents with
model='' against the live portal URL, the arbitrary match now lands on
a 32K entry, and init_agent raises the 64K-floor ValueError
(test_allowed_for_nous_anthropic_messages, red on every PR's slice).

Guard early: a blank model id falls back to DEFAULT_FALLBACK_CONTEXT
immediately, before any cache write or network probe.

Salvaged from NousResearch#65515 by @whirmill (rebased onto current main; the
guard now sits after the malformed-base_url normalization added since,
and carries an explanatory comment for the fuzzy-match footgun).

Fixes the red slice on NousResearch#85444, NousResearch#85452 and every other open PR.

Co-authored-by: whirmill <5079591+whirmill@users.noreply.github.com>
@kshitijk4poor
kshitijk4poor merged commit 4a6d364 into NousResearch:main Aug 13, 2026
45 checks passed
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Aug 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants