fix(code): inject OpenAI `prompt_cache_key` for any OpenAI-provider endpoint by mdrxy · Pull Request #4995 · langchain-ai/deepagents

Mason Daugherty (mdrxy) · 2026-07-23T13:22:12Z

dcode now attaches the per-thread prompt_cache_key to every model whose provider resolves to openai, regardless of base URL, so reliable prompt-cache prefix routing also applies through the LangSmith gateway and other OpenAI-compatible endpoints — not just api.openai.com. A default-on models.openai_prompt_cache_key opt-out is added for endpoints that reject unknown request fields.

Previously ConfigurableModelMiddleware only injected prompt_cache_key when the resolved ChatOpenAI base URL was api.openai.com (or an official regional subdomain). Any OpenAI-compatible proxy — including the managed LangSmith gateway (gateway.smith.langchain.com/openai/v1) that dcode is provisioned to use — was classified as a custom endpoint and silently dropped the key, so consecutive turns over a stable ~58-59K-token prefix missed prompt caching and were billed at full input price (observed ~7x per-turn cost spikes).

prompt_cache_key is an optional, additive OpenAI request field: it sharpens prefix-cache routing on model families that support it and is otherwise inert. Rather than maintaining a hostname allowlist, _is_openai_model now treats any model reporting ls_provider == "openai" as eligible and attempts the key unconditionally. Endpoints that reject unknown fields can opt out via the new models.openai_prompt_cache_key config option ([models].openai_prompt_cache_key in config.toml, or DEEPAGENTS_CODE_OPENAI_PROMPT_CACHE_KEY, default on). A user-supplied key (on the model or per-invocation) is still preserved, and the Fireworks session-affinity path is unchanged.

Verified end to end that a model created with the gateway base URL now serializes prompt_cache_key in the Responses payload, and that the opt-out suppresses it.

Co-authored-by: open-swe[bot] open-swe@users.noreply.github.com

Made by Open SWE

…endpoint Attach the per-thread `prompt_cache_key` to every model whose provider resolves to `openai`, regardless of base URL, so the cache-key optimization also applies through the LangSmith gateway and other OpenAI-compatible endpoints — not just `api.openai.com`. Adds a default-on `models.openai_prompt_cache_key` opt-out (env `DEEPAGENTS_CODE_OPENAI_PROMPT_CACHE_KEY`) for endpoints that reject unknown request fields. A user-supplied key is still preserved. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

Alexander Olsen (aolsenjazz) · 2026-07-23T13:24:20Z

 flow. Parsed by `is_env_truthy`: accepts `1`, `true`, `yes`, `on` as enabled.
 """

+OPENAI_PROMPT_CACHE_KEY = "DEEPAGENTS_CODE_OPENAI_PROMPT_CACHE_KEY"


I wonder if a better long-term solution is to just use a generic PROMPT_CACHE_KEY env variable for multi-provider coverage

I don't think decomposing this into provider-specific variables is useful to the end user

Also - I could see orgs wanting to set org- or team-specific cache keys to maximal static prefix sharing. imo better to support str | falsy

Keeping this OpenAI-scoped and as a bool + per-thread default for this PR.

Naming: Injection is not one shared behavior across providers. Fireworks already has its own path (prompt_cache_key + x-session-affinity), and Anthropic uses a different caching model entirely. A generic PROMPT_CACHE_KEY name would overpromise common semantics and get ambiguous when those providers disagree. Happy to revisit a shared name later if multiple providers genuinely share one setting and one meaning.

Value shape: Org/team keys for shared static prefixes are a real product use case, but I'm not expanding this into str | falsy here. The bool matches the bug fix (restore key injection behind non-api.openai.com endpoints) without changing the config mental model. Power users can already supply a custom prompt_cache_key on the model / invocation today. If we want first-class org/team keys in config, I'd rather add a dedicated value setting later than overload one knob that means both enable and value.

hmmmmm I think first-class org/team keys routed to cache keys is a logical disconnect and canonicalizing differences in nomenclature under one key (PROMPT_CACHE_KEY or whatever) is better UX. at some point dcode needs to be opinionated rather than saying "here are 5 different levels you can pull

but won't let this get in the way of shipping a fix

curious what your thoughts are vs this LLMs thoughts

The job of this PR is restoring prompt_cache_key injection.

On the longer-term shape: I’m aligned that we shouldn’t end up with a pile of provider-specific “almost the same” knobs. I’d rather unify cache-key semantics when we actually introduce generalizable prompt-caching middleware — i.e. one opinionated setting/path that different providers can plug into, instead of inventing a fake-generic PROMPT_CACHE_KEY ahead of shared behavior existing.

Alexander Olsen (aolsenjazz) · 2026-07-23T13:26:26Z

+        resolve_scalar,
+    )
+
+    option = get_option("models.openai_prompt_cache_key")


This will change if we support str | falsy

Yep — only if we take the str | falsy shape. Staying with the bool toggle here, so this resolver remains enabled -> bool.

Alexander Olsen (aolsenjazz) · 2026-07-23T13:29:30Z

@@ -104,45 +103,20 @@ def _is_fireworks_model(model: object) -> bool:


 def _is_openai_model(model: object) -> bool:


This function can just go away if this is how we're rewriting it

I see this as a good thing - inspecting the base url feels like overloading this function

Keeping the thin helper.

It now only checks ls_provider == "openai" (base-URL allowlisting is gone, which I agree was overload), but that mirrors _is_anthropic_model / _is_fireworks_model and keeps call sites readable. The docstring here is also the natural home for the eligibility policy (any OpenAI-provider endpoint, not just api.openai.com).

I'd only inline if we collapsed all of the provider helpers the same way — out of scope for this PR.

Alexander Olsen (aolsenjazz) · 2026-07-23T13:31:10Z

-    same key is sent to every OpenAI model without a version gate. This mirrors
-    the Fireworks path (`_with_fireworks_session_settings`), which sets
-    `prompt_cache_key` the same way and additionally sends an
+    same key is sent to every OpenAI-provider model — including custom base URLs


these docs feel out of scope of this variable

Good call — trimmed this docstring to injection-only behavior (add key if missing; preserve user-supplied keys). Provider eligibility / endpoint policy / opt-out stay on the caller helpers instead.

Alexander Olsen (aolsenjazz) · 2026-07-23T13:32:21Z

            updated_settings = _with_fireworks_session_settings(settings, ctx.thread_id)
            injected = "Fireworks session settings"
-        elif _is_openai_model(effective_model):
+        elif _is_openai_model(effective_model) and _openai_prompt_cache_key_enabled():


If we do str | falsy we do not need this additional check

True under a key-or-off config shape. Keeping the separate enabled gate for the bool opt-out: resolve once at construction, then if openai_prompt_cache_key: before inject. That only collapses if we redesign the setting as in the str | falsy suggestion.

open-swe

✅ Open SWE Review: No issues found

Open SWE reviewed this PR and found no potential bugs to report.

Open in Web • View Open SWE trace

Read models.openai_prompt_cache_key once when ConfigurableModelMiddleware is built so config.toml never blocks a model call, and cover empty-env opt-out plus Fireworks isolation.

Leave provider eligibility and opt-out policy on the caller helpers.

Keep both openai_prompt_cache_key and goal auto-accept criteria config helpers.

Re-raise BlockingError from the openai_prompt_cache_key config read so event-loop I/O violations are not masked, and cover async opt-out plus that path.

> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.46](deepagents-code==0.1.45...deepagents-code==0.1.46) (2026-07-24) ### Highlights - Auto mode is now generally available. [#4957](#4957) - Added configurable Auto goal-criteria acceptance. [#4940](#4940) - Improved Auto behavior by authorizing actions from active goal/rubric directives, avoiding redundant approval prompts, showing the enable notice only on first global enable, deduplicating classifier-unavailable transcript spam, logging underlying classifier failures, and reporting classifier timeout budgets. [#5017](#5017) [#4993](#4993) [#5012](#5012) [#5013](#5013) [#5011](#5011) [#5025](#5025) - Added Hooks v2 capability snapshots and session transcripts, and hardened Hooks v2 command execution. [#4916](#4916) [#4918](#4918) [#4917](#4917) - Raised the agent recursion limit to 2000 and made it configurable. [#4994](#4994) ### Improvements and fixes - Let the rubric grader inspect working-directory files, show rubric grader defaults, and improved `/rubric` help and empty-state messaging. [#4835](#4835) [#4966](#4966) [#5015](#5015) - Unified goal activation signaling. [#4980](#4980) - Made Version, Model, and CWD copyable in the Debug Console. [#4975](#4975) - Improved `config get` output when a key is missing. [#4976](#4976) - Aborted YOLO launch on `Ctrl+C`/`Ctrl+D` and made the YOLO warning friendlier for new users. [#4953](#4953) [#4950](#4950) - Updated LangSmith handling: secret redaction is disabled by default, `/trace` now flags empty env overrides that shadow the LangSmith key, and the default US endpoint is no longer treated as a custom target. [#4970](#4970) [#4996](#4996) [#5022](#5022) - Injected OpenAI `prompt_cache_key` for any OpenAI-provider endpoint. [#4995](#4995) - Improved tool and schema presentation: finished calls stay on the live tool-group line, first-party tool schemas now include field descriptions, and `web_search`/`fetch_url` tool descriptions were trimmed. [#4927](#4927) [#5019](#5019) [#5016](#5016) - Omitted `plugins/` and `conversation_history/` from the `/agent` picker. [#4991](#4991) - Made selector modal backdrop dimming consistent. [#4990](#4990) - Restored the `"Server log preserved at:"` notice on exit. [#4999](#4999) - Used the SDK pin as the effective editable version. [#4949](#4949) _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>

…ndpoint (langchain-ai#4995) `dcode` now attaches the per-thread `prompt_cache_key` to every model whose provider resolves to `openai`, regardless of base URL, so reliable prompt-cache prefix routing also applies through the LangSmith gateway and other OpenAI-compatible endpoints — not just `api.openai.com`. A default-on `models.openai_prompt_cache_key` opt-out is added for endpoints that reject unknown request fields. --- Previously `ConfigurableModelMiddleware` only injected `prompt_cache_key` when the resolved `ChatOpenAI` base URL was `api.openai.com` (or an official regional subdomain). Any OpenAI-compatible proxy — including the managed LangSmith gateway (`gateway.smith.langchain.com/openai/v1`) that dcode is provisioned to use — was classified as a custom endpoint and silently dropped the key, so consecutive turns over a stable ~58-59K-token prefix missed prompt caching and were billed at full input price (observed ~7x per-turn cost spikes). `prompt_cache_key` is an optional, additive OpenAI request field: it sharpens prefix-cache routing on model families that support it and is otherwise inert. Rather than maintaining a hostname allowlist, `_is_openai_model` now treats any model reporting `ls_provider == "openai"` as eligible and attempts the key unconditionally. Endpoints that reject unknown fields can opt out via the new `models.openai_prompt_cache_key` config option (`[models].openai_prompt_cache_key` in `config.toml`, or `DEEPAGENTS_CODE_OPENAI_PROMPT_CACHE_KEY`, default on). A user-supplied key (on the model or per-invocation) is still preserved, and the Fireworks session-affinity path is unchanged. Verified end to end that a model created with the gateway base URL now serializes `prompt_cache_key` in the Responses payload, and that the opt-out suppresses it. Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com> Made by [Open SWE](https://openswe.vercel.app/agents/09eb0978-379d-0b80-9beb-73aa2ef6ab79) --------- Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>

> [!CAUTION] > Merging this PR will automatically publish to **PyPI** and create a **GitHub release**. For the full release process, see [`.github/RELEASING.md`](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md). --- _Release notes preview: keep this section in sync with the package `CHANGELOG.md`. The published GitHub release body is extracted from the merged `CHANGELOG.md` by `release.yml`, not from this PR description._ --- ## [0.1.46](langchain-ai/deepagents@deepagents-code==0.1.45...deepagents-code==0.1.46) (2026-07-24) ### Highlights - Auto mode is now generally available. [langchain-ai#4957](langchain-ai#4957) - Added configurable Auto goal-criteria acceptance. [langchain-ai#4940](langchain-ai#4940) - Improved Auto behavior by authorizing actions from active goal/rubric directives, avoiding redundant approval prompts, showing the enable notice only on first global enable, deduplicating classifier-unavailable transcript spam, logging underlying classifier failures, and reporting classifier timeout budgets. [langchain-ai#5017](langchain-ai#5017) [langchain-ai#4993](langchain-ai#4993) [langchain-ai#5012](langchain-ai#5012) [langchain-ai#5013](langchain-ai#5013) [langchain-ai#5011](langchain-ai#5011) [langchain-ai#5025](langchain-ai#5025) - Added Hooks v2 capability snapshots and session transcripts, and hardened Hooks v2 command execution. [langchain-ai#4916](langchain-ai#4916) [langchain-ai#4918](langchain-ai#4918) [langchain-ai#4917](langchain-ai#4917) - Raised the agent recursion limit to 2000 and made it configurable. [langchain-ai#4994](langchain-ai#4994) ### Improvements and fixes - Let the rubric grader inspect working-directory files, show rubric grader defaults, and improved `/rubric` help and empty-state messaging. [langchain-ai#4835](langchain-ai#4835) [langchain-ai#4966](langchain-ai#4966) [langchain-ai#5015](langchain-ai#5015) - Unified goal activation signaling. [langchain-ai#4980](langchain-ai#4980) - Made Version, Model, and CWD copyable in the Debug Console. [langchain-ai#4975](langchain-ai#4975) - Improved `config get` output when a key is missing. [langchain-ai#4976](langchain-ai#4976) - Aborted YOLO launch on `Ctrl+C`/`Ctrl+D` and made the YOLO warning friendlier for new users. [langchain-ai#4953](langchain-ai#4953) [langchain-ai#4950](langchain-ai#4950) - Updated LangSmith handling: secret redaction is disabled by default, `/trace` now flags empty env overrides that shadow the LangSmith key, and the default US endpoint is no longer treated as a custom target. [langchain-ai#4970](langchain-ai#4970) [langchain-ai#4996](langchain-ai#4996) [langchain-ai#5022](langchain-ai#5022) - Injected OpenAI `prompt_cache_key` for any OpenAI-provider endpoint. [langchain-ai#4995](langchain-ai#4995) - Improved tool and schema presentation: finished calls stay on the live tool-group line, first-party tool schemas now include field descriptions, and `web_search`/`fetch_url` tool descriptions were trimmed. [langchain-ai#4927](langchain-ai#4927) [langchain-ai#5019](langchain-ai#5019) [langchain-ai#5016](langchain-ai#5016) - Omitted `plugins/` and `conversation_history/` from the `/agent` picker. [langchain-ai#4991](langchain-ai#4991) - Made selector modal backdrop dimming consistent. [langchain-ai#4990](langchain-ai#4990) - Restored the `"Server log preserved at:"` notice on exit. [langchain-ai#4999](langchain-ai#4999) - Used the SDK pin as the effective editable version. [langchain-ai#4949](langchain-ai#4949) _End release notes preview._ --- > [!NOTE] > A **New Contributors** section is appended to the GitHub release notes automatically at publish time (see [Release Pipeline](https://github.com/langchain-ai/deepagents/blob/main/.github/RELEASING.md#release-pipeline), step 2). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: langchain-oss-automated-triage[bot] <248757908+langchain-oss-automated-triage[bot]@users.noreply.github.com>

github-actions Bot added dcode Related to `deepagents-code` feature New feature/enhancement or request for one internal User is a member of the `langchain-ai` GitHub organization size: M 200-499 LOC labels Jul 23, 2026

Alexander Olsen (aolsenjazz) reviewed Jul 23, 2026

View reviewed changes

Mason Daugherty (mdrxy) marked this pull request as ready for review July 23, 2026 13:41

open-swe Bot reviewed Jul 23, 2026

View reviewed changes

Mason Daugherty (mdrxy) and others added 5 commits July 23, 2026 11:16

Merge branch 'main' into mdrxy/code/openai-prompt-cache-key-any-endpoint

5df3e1c

fix(code): resolve OpenAI prompt_cache_key opt-out at construction

a75d791

Read models.openai_prompt_cache_key once when ConfigurableModelMiddleware is built so config.toml never blocks a model call, and cover empty-env opt-out plus Fireworks isolation.

style(code): trim _with_openai_prompt_cache_key docs to injection-only

9a80429

Leave provider eligibility and opt-out policy on the caller helpers.

Merge branch 'main' into mdrxy/code/openai-prompt-cache-key-any-endpoint

ded1560

Merge branch 'main' into mdrxy/code/openai-prompt-cache-key-any-endpoint

3ae59be

Keep both openai_prompt_cache_key and goal auto-accept criteria config helpers.

Mason Daugherty (mdrxy) changed the title ~~feat(code): inject OpenAI prompt_cache_key for any OpenAI-provider endpoint~~ fix(code): inject OpenAI prompt_cache_key for any OpenAI-provider endpoint Jul 24, 2026

github-actions Bot added fix A bug fix (PATCH) and removed feature New feature/enhancement or request for one labels Jul 24, 2026

fix(code): do not fail open on BlockingError for prompt cache opt-out

9468e7a

Re-raise BlockingError from the openai_prompt_cache_key config read so event-loop I/O violations are not masked, and cover async opt-out plus that path.

github-actions Bot added size: L 500-999 LOC and removed size: M 200-499 LOC labels Jul 24, 2026

Mason Daugherty (mdrxy) merged commit 6d6632f into main Jul 24, 2026
54 checks passed

Mason Daugherty (mdrxy) deleted the mdrxy/code/openai-prompt-cache-key-any-endpoint branch July 24, 2026 01:39

github-actions Bot mentioned this pull request Jul 24, 2026

release(deepagents-code): 0.1.46 #4965

Merged

		@@ -104,45 +103,20 @@ def _is_fireworks_model(model: object) -> bool:


		def _is_openai_model(model: object) -> bool:

Uh oh!

Conversation

Mason Daugherty (mdrxy) commented Jul 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mason Daugherty (mdrxy) Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alexander Olsen (aolsenjazz) Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

open-swe Bot left a comment

Choose a reason for hiding this comment

✅ Open SWE Review: No issues found

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Mason Daugherty (mdrxy) Jul 23, 2026 •

edited

Loading

Alexander Olsen (aolsenjazz) Jul 23, 2026 •

edited

Loading