Skip to content

fix(gateway): read context_length from custom_providers in session info header - #16579

Closed
JanCong wants to merge 1 commit into
NousResearch:mainfrom
JanCong:fix/gateway-session-info-custom-provider-context-length
Closed

fix(gateway): read context_length from custom_providers in session info header#16579
JanCong wants to merge 1 commit into
NousResearch:mainfrom
JanCong:fix/gateway-session-info-custom-provider-context-length

Conversation

@JanCong

@JanCong JanCong commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Problem

_format_session_info() in gateway/run.py only reads context_length from the top-level model.context_length in config.yaml. If a user configures context_length under custom_providers → specific provider → models.<model>.context_length, this is completely ignored, and the session header always shows "128K tokens (default — set model.context_length in config to override)".

Fix

Added a scan of custom_providers entries when the top-level model.context_length is not set. The resolution order is now:

  1. Top-level model.context_length (existing, highest priority)
  2. custom_providers → matching provider → models.<model>.context_length
  3. Fall through to the existing probe/default behavior

The fix is a ~35 line addition that iterates through custom_providers looking for context_length matching the current model. Also checks both cp[context_length] (provider-level) and cp[models][model][context_length] (per-model).

Testing

  • All 19 gateway-related tests pass
  • 250 existing run_agent tests pass (3 pre-existing unrelated failures)

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery labels Apr 27, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Likely duplicate of #15668, #5096, #8240 — all fix _format_session_info() to read context_length from custom_providers. Note that #15844 (merged) already fixed the runtime resolution path; this addresses the remaining display-only gap in the session info header.

@teknium1

teknium1 commented May 4, 2026

Copy link
Copy Markdown
Contributor

Salvaged via #19708 onto current main. A custom_provs hoist added on main since your PR opened was preserved alongside your data init, so both your session-info fix and the downstream custom-provider usage sites continue to work. Thanks @JanCong!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants