Skip to content

fix(chatgpt): forward prompt cache key - #37630

Draft
steveonjava wants to merge 3 commits into
BerriAI:litellm_internal_stagingfrom
steveonjava:litellm_prompt_cache_key
Draft

fix(chatgpt): forward prompt cache key#37630
steveonjava wants to merge 3 commits into
BerriAI:litellm_internal_stagingfrom
steveonjava:litellm_prompt_cache_key

Conversation

@steveonjava

Copy link
Copy Markdown

TLDR

Problem this solves:

  • ChatGPT Responses requests silently drop prompt_cache_key
  • Session-scoped prompt caching cannot reach ChatGPT Codex

How it solves it:

  • Forward prompt_cache_key through the ChatGPT provider allowlist
  • Retain filtering of unsupported context_management

User Flow

Before: a developer sends a ChatGPT Responses request with a cache key, but the backend never receives it

  1. They send POST https://litellm-domain/v1/responses with model chatgpt/gpt-5.6-sol and "prompt_cache_key": "session_123"
  2. LiteLLM forwards a ChatGPT Codex request without prompt_cache_key
  3. The request cannot use the caller's intended prompt-cache scope

After: the same request forwards the caller's cache key

  1. They send the same POST https://litellm-domain/v1/responses with "prompt_cache_key": "session_123"
  2. LiteLLM forwards the ChatGPT Codex request with prompt_cache_key unchanged
  3. The backend receives the caller's intended prompt-cache scope

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Before (6d47468)

  1. Run uv run pytest tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py::TestChatGPTResponsesAPITransformation::test_chatgpt_drops_unsupported_responses_params -q -o 'addopts=' after asserting prompt_cache_key is preserved.
  2. Result: 2 failures, both KeyError: 'prompt_cache_key'.

After (025438c)

  1. Run uv run pytest tests/test_litellm/llms/chatgpt/responses/test_chatgpt_responses_transformation.py -q -o 'addopts='.
  2. Result: 19 passed, 1 existing dependency warning.
  3. Run uv run pytest tests/test_litellm/llms/openai/responses/test_openai_responses_transformation.py -q -o 'addopts='.
  4. Result: 63 passed, 2 existing warnings.

Type

🐛 Bug Fix

Caveats (if any)

  • GPT-5.6 Sol/Luna/Terra context-management probes received identical 401 responses
  • The stored ChatGPT subscription tokens were expired and refresh was unauthorized
  • Keep context_management filtered pending authenticated backend acceptance proof
  • Codex's documented /responses/compact path remains the likely compaction integration

Final Attestation

  • The test covers allowlist preservation while retaining existing unsupported-field filtering

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing steveonjava:litellm_prompt_cache_key (682ffbf) with litellm_internal_staging (e07a712)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (cde1344) during the generation of this report, so e07a712 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

I removed context_management from the ChatGPT Responses allowlist after\nauthenticated GPT-5.6 probes did not establish normal endpoint support.\nprompt_cache_key remains forwarded, and the regression keeps unsupported\nChatGPT request fields filtered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant