Skip to content

feat(api-server): surface context-window usage and Markdown-aware platform hint - #52600

Open
ladyada-piclaw wants to merge 1 commit into
NousResearch:mainfrom
ladyada-piclaw:feat/api-server-ctx-usage-markdown-hint
Open

feat(api-server): surface context-window usage and Markdown-aware platform hint#52600
ladyada-piclaw wants to merge 1 commit into
NousResearch:mainfrom
ladyada-piclaw:feat/api-server-ctx-usage-markdown-hint

Conversation

@ladyada-piclaw

Copy link
Copy Markdown

Summary

Two related improvements for OpenAI-compatible chat frontends (Open WebUI, LibreChat) served by the Hermes API server.

1. Context-window usage in chat-completions

Adds current-turn context occupancy to the usage block of /v1/chat/completions responses — both the streaming finish chunk and the non-streaming body. Open WebUI's per-message generation-info (ⓘ) popup can then show ctx: 29k / 200k (15%).

  • New helpers: _fmt_token_count, _build_ctx_footer, _ctx_footer_enabled, _ctx_usage_fields.
  • Sourced from agent.context_compressor.last_prompt_tokens / .context_length (the prompt actually sent this turn — the right number for "how full is the window").
  • Gated by display.runtime_footer.show_ctx_in_api (default on). Emits nothing when token data is unavailable, so the usage payload never breaks.

2. Markdown-aware api_server platform hint

The old hint told the model to assume plain text and emit no Markdown. Real consumers of this endpoint (Open WebUI, LibreChat) render Markdown, so the hint now:

  • allows Markdown (headings, bold, lists, tables, code fences) when it aids readability;
  • instructs the model not to emit MEDIA:/path attachment syntax (unsupported by these frontends);
  • points it at real Markdown image URLs (![alt](https://...)) instead.

Real behavior

Verified against a live API-server profile (Open WebUI on a Raspberry Pi 5): the ctx fields appear in the generation-info popup, and replies render Markdown instead of raw asterisks.

Scope / safety

  • No new model tools; no change to the agent loop or prompt caching.
  • Feature flag defaults on but degrades to empty when data is missing.
  • .env untouched; setting lives in config.yaml per the config-not-env convention.

…tform hint

Two related improvements for OpenAI-compatible frontends (Open WebUI,
LibreChat) served by the API server:

- Add context-window occupancy to the usage block of chat-completions
  responses (both streaming finish chunk and non-streaming), so the
  per-message generation-info popup can show "ctx: 29k / 200k (15%)".
  Gated by display.runtime_footer.show_ctx_in_api (default on); emits
  nothing when token data is unavailable so the payload never breaks.

- Update the api_server platform hint: these frontends render Markdown,
  so allow it (was: "assume plain text, no markdown"). Also instruct the
  model not to emit MEDIA:/path attachment syntax (unsupported) and to
  use real Markdown image URLs instead.

Co-authored-by: ladyada <limor@ladyada.net>
@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have labels Jun 25, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for addressing a real API-server gap: current main still prohibits Markdown for api_server in agent/prompt_builder.py:848-852, and its chat-completions usage payloads contain only the standard token fields (gateway/platforms/api_server.py:2424-2428, 2583-2587).

Problems

  • The new instruction to never emit MEDIA:/path would disable an existing API-server delivery path. gateway/platforms/api_server.py:590-636 deliberately validates local image paths and converts them to Markdown data URLs for remote OpenAI-compatible frontends. Please preserve that behavior in the hint.
  • The PR adds no tests. Existing coverage at tests/gateway/test_api_server.py:3041-3062 checks only standard usage fields, leaving the new non-streaming/SSE fields and disabled/unavailable-data behavior unverified.
  • _build_ctx_footer is added but never called; response construction uses _ctx_usage_fields instead.

Suggested changes

  • Keep supported local-image MEDIA: delivery available while allowing normal Markdown.
  • Add API-server tests for both completion modes, missing/sentinel token counts, and the config gate.
  • Remove or use the dead helper.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform area/usage-cost Token accounting, usage reporting, billing, cost tracking labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/usage-cost Token accounting, usage reporting, billing, cost tracking comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants