Skip to content

Fix prompt cache routing for SSE Responses proxies - #6461

Closed
nicelnicel wants to merge 2 commits into
NousResearch:mainfrom
nicelnicel:codex/sse-responses-cache-routing
Closed

Fix prompt cache routing for SSE Responses proxies#6461
nicelnicel wants to merge 2 commits into
NousResearch:mainfrom
nicelnicel:codex/sse-responses-cache-routing

Conversation

@nicelnicel

Copy link
Copy Markdown

Summary

  • use a stable prompt cache identity for OpenAI-compatible Responses proxies that stream via SSE
  • add stable routing headers for those proxies: conversation_id, session_id, x-session-id, and x-session-affinity
  • keep prompt_cache_key aligned with the same stable identity instead of a random per-session value

Why

Some local proxy services expose the Responses API over SSE and only reuse prompt cache reliably when consecutive requests are routed with a stable identity. Hermes currently relies on a random session-derived cache key and does not attach the routing headers these proxies expect, which causes avoidable cache misses across turns.

Validation

  • python -m py_compile run_agent.py
  • A/B probing against an SSE streaming Responses proxy showed that prompt_cache_key alone had little effect, while stable routing headers restored high second-turn cache reuse
  • a two-turn Hermes smoke test on the same proxy restored >95% second-turn cache reads

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant