feat(api-server): surface context-window usage and Markdown-aware platform hint - #52600
Open
ladyada-piclaw wants to merge 1 commit into
Open
Conversation
…tform hint Two related improvements for OpenAI-compatible frontends (Open WebUI, LibreChat) served by the API server: - Add context-window occupancy to the usage block of chat-completions responses (both streaming finish chunk and non-streaming), so the per-message generation-info popup can show "ctx: 29k / 200k (15%)". Gated by display.runtime_footer.show_ctx_in_api (default on); emits nothing when token data is unavailable so the payload never breaks. - Update the api_server platform hint: these frontends render Markdown, so allow it (was: "assume plain text, no markdown"). Also instruct the model not to emit MEDIA:/path attachment syntax (unsupported) and to use real Markdown image URLs instead. Co-authored-by: ladyada <limor@ladyada.net>
Contributor
|
Thanks for addressing a real API-server gap: current main still prohibits Markdown for Problems
Suggested changes
Automated hermes-sweeper review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related improvements for OpenAI-compatible chat frontends (Open WebUI, LibreChat) served by the Hermes API server.
1. Context-window usage in chat-completions
Adds current-turn context occupancy to the
usageblock of/v1/chat/completionsresponses — both the streaming finish chunk and the non-streaming body. Open WebUI's per-message generation-info (ⓘ) popup can then showctx: 29k / 200k (15%)._fmt_token_count,_build_ctx_footer,_ctx_footer_enabled,_ctx_usage_fields.agent.context_compressor.last_prompt_tokens/.context_length(the prompt actually sent this turn — the right number for "how full is the window").display.runtime_footer.show_ctx_in_api(default on). Emits nothing when token data is unavailable, so the usage payload never breaks.2. Markdown-aware api_server platform hint
The old hint told the model to assume plain text and emit no Markdown. Real consumers of this endpoint (Open WebUI, LibreChat) render Markdown, so the hint now:
MEDIA:/pathattachment syntax (unsupported by these frontends);) instead.Real behavior
Verified against a live API-server profile (Open WebUI on a Raspberry Pi 5): the ctx fields appear in the generation-info popup, and replies render Markdown instead of raw asterisks.
Scope / safety
.envuntouched; setting lives inconfig.yamlper the config-not-env convention.