docs(dcode): document cold prompt-cache warning and trusted endpoints - #5524
Merged
Conversation
Draft docs for deepagents PRs: - langchain-ai/deepagents#5439 (cold prompt-cache warning) - langchain-ai/deepagents#5462 (trusted cache endpoints) Adds a 'Cold prompt-cache warning' section to the config.toml reference covering the warning modal, provider retention windows, skip conditions, and warnings.cold_cache_min_delta_usd; a 'Trust a gateway endpoint' subsection for warnings.trusted_cache_endpoints; and the DEEPAGENTS_CODE_DEBUG_COLD_CACHE env var entry.
github-actions
Bot
added
deepagents
Trim implementation details (keybindings, fail-closed behavior, minimum/maximum window theory, pricing internals) and keep the user-facing configuration surface.
This was referenced Aug 18, 2026
Mason Daugherty (mdrxy)
added a commit
to langchain-ai/deepagents
that referenced
this pull request
Aug 18, 2026
[Docs](langchain-ai/docs#5524) When an interactive chat message reaches the front of the queue, dcode now warns before sending it to the model if its prompt cache may need re-warming, showing the estimated cold-input cost and warm-cache delta. ```mermaid flowchart TD A["Queued message reaches front of queue"] --> B{"Qualifying message?"} B -- "No (e.g. using a slash command, ACP mode)" --> S["Send the message"] B -- "Yes - sent inside the interactive TUI and not a slash command" --> C["Load checkpointed last request time + model identity"] C -- "Missing or malformed:\nlog a warning and treat as cold (see Fig. 1 for specifics)" --> D C -- "Loaded" --> D{"Known cache policy for current model + endpoint?"} D -- "No (e.g. custom endpoint or other provider)" --> S D -- Yes --> E{"Idle past window, age unknown, or cache-affecting params changed?"} E -- "No: within window" --> S E -- Yes --> E2{"Context ≥ minimum cacheable tokens?"} E2 -- "No: tiny context" --> S E2 -- Yes --> F{"Re-warm price delta ≥ threshold?"} F -- "No: below threshold or unpriceable" --> S F -- Yes --> M["Show blocking warning modal"] M -- "Keep draft (Esc)" --> R["Restore draft to chat input"] M -- "Cannot be shown (render failure, or another prompt owns the modal slot)" --> R M -- "Send anyway" --> S M -- "Send + don't warn again this session" --> SS["Mute until restart (in memory)"] --> S M -- "Send + don't warn again ever" --> SP["Persist to warnings.suppress in config.toml"] --> S ``` <details><summary><b>Fig. 1</b> — the "treat as cold" fallback</summary> The main flow compares the *checkpointed* identity (what last warmed the cache) against the *current* model (what's about to run). Policy and cost always come from the current model, which is known from live config — so a missing checkpointed identity doesn't need its own policy; it just forces the comparison to "changed": ```mermaid flowchart TD subgraph P["Checkpointed identity (written by middleware after each completed model call)"] P1 -- "Request time bad, absent, or in the future" --> P2["Log, discard time:\nthe age is unknown, not zero"] P1 -- "Model spec bad" --> P3["Log warning, discard spec"] end subgraph Q["Current identity (from live config)"] Q1["Effective model spec\n(always resolvable)"] --> Q2["Resolve policy + price against this model"] end P2 -- "No age to compare against a window" --> OUT2["Continue down the cold-cache path:\nreason = age_unknown"] P3 -- "Discarded spec reads as the model having changed in last vs. current" --> OUT["Continue down the cold-cache path:\nidentity_changed = true"] Q2 --> OUT Q2 --> OUT2 ``` The three causes are kept distinct because each gets its own modal copy, and reporting one as another states something untrue about the user's configuration. In particular an unknown age is never reported as a model change, which would claim a change that never happened. </details> The retention windows come from documented provider policies: | Provider | Condition | Retention window | Warning wording | | --- | --- | --- | --- | | Anthropic | any | 5m | "expired" | | OpenAI | `prompt_cache_retention: in_memory` | 1h | "expired" | | OpenAI | `prompt_cache_retention: 24h` | 24h | "expired" | | OpenAI | GPT-5.6 or newer, no explicit retention | 30m minimum | "may be cold" (provider may retain longer) | The window is either a documented **maximum** or a documented **minimum**, and the wording follows from which. Anthropic's TTL and OpenAI's `prompt_cache_retention` ceilings are maximums, so once the window passes the entry is gone ("expired"). GPT-5.6+'s 30 minutes is a guaranteed minimum that the provider may exceed, so past it the cache can only be called "may be cold". The two OpenAI knobs are independent, which is why the table is ordered as it is: `prompt_cache_retention` states a maximum lifetime while the GPT-5.6+ guarantee states a minimum one, so an explicitly configured retention is the later and firmer bound and takes precedence on those models too. Warning a user who asked for `24h` at the 30-minute mark would contradict their own configuration. Anthropic is always 5m regardless of a configured `cache_control.ttl`. `AnthropicPromptCachingMiddleware` runs inside `ConfigurableModelMiddleware` and rewrites `model_settings["cache_control"]` with its own TTL (5m, which this stack never overrides), so a user-supplied `1h` never reaches the API. Treating it as an hour would suppress the warning for 55 minutes of a cache that died at five. The warning combines three sources of information: 1. **Provider retention policies** (from the table above) — how long each provider keeps cache entries 2. **Checkpointed timing** — when you last sent a request to this model 3. **Pricing overrides** — your configured cost per token If the estimated re-warm cost reaches your threshold, a modal appears before sending with four keyboard-driven choices. Cost is estimated by pricing two synthetic usage payloads — one billed as a full cache read, one as a cold request — through the ordinary cost-tracking path, so the pricing catalog stays the single source of truth. Both figures are upper bounds and are rounded upward so the displayed estimate never reads lower than the modelled spend. How a miss is priced follows the model: Anthropic carries a write premium over plain input, GPT-5.6+ bills a miss as a cache write, and OpenAI before 5.6 charges no write surcharge at all, so those misses price at the plain input rate. Tagging a pre-5.6 miss as a write would apply a premium the provider never charges and overstate both the displayed cost and the threshold comparison. <img width="614" height="357" alt="Screenshot" src="https://github.com/user-attachments/assets/4bdb5ef4-8ee4-47ed-b624-c87ac4a71722" /> The model identity check looks at the model name plus the invocation parameters that actually select or invalidate a cache entry — `prompt_cache_retention`, `prompt_cache_key`, `prompt_cache_options`, and `cache_control`. A change to any of them invalidates the prefix, so it warns regardless of how recent the last turn was. Unrelated knobs are deliberately excluded: `/effort` rewrites `reasoning_effort` wholesale, and `temperature` or `max_tokens` are just as inert for caching, so comparing whole parameter maps would open the modal asserting that the cached prefix "cannot be reused" when nothing about the prefix moved. Anything that prevents the confirmation from being shown keeps the draft rather than sending: if the modal fails to render, or another prompt already owns the modal slot, the message is not dispatched and the text is restored to the chat input. A warning that cannot be presented must not become an implicit authorization to spend. Where a draft genuinely cannot be put back — the input already holds different text, or the active thread changed while the modal was open — the notification says the message was not sent and points at history recall rather than implying it was merely deferred. Only interactive chat submissions are checked — prompts sent programmatically (e.g. via ACP from an editor client), slash commands, requests to custom endpoints, and requests without pricing information all skip this warning entirely. The default threshold is $0.50; adjust it with `warnings.cold_cache_min_delta_usd`. The request time is kept fresh from two directions. A completed turn reads it back from the checkpoint; a turn that reached the model but ended without a readable checkpoint — an interrupt, say — stamps it locally instead. Without that second path the field would never advance past its initial value, and every send after the first would open the modal reporting no record of a turn that plainly happened. If the checkpointed request time or model identity cannot be loaded (for example, a checkpoint written by a newer version, or a malformed value), dcode logs a warning naming the affected channel and proceeds down the cold-cache path rather than silently assuming the cache is warm: an unusable request time makes the age unknown, and a missing model identity is treated as a model change. A request time in the future — a corrected clock, or a thread synced from a skewed machine — is treated as unknown age too, since clamping it to zero would place it inside every retention window and suppress the warning outright. Skipping the warning means sending at full cold-cache price, so the paths that skip it on failure are made visible rather than left to the log alone: if the evaluation itself fails, the session says once that cost checks are unavailable; an unparseable configured `base_url` warns where it is parsed, since it silently disables warnings for that provider; and a pricing catalog that produces a non-finite figure is reported as a data defect rather than reading as "no price published". A failed persistent suppression names its cause — a malformed `[warnings]` table needs one line of TOML, not the `chmod` that a generic "check file permissions" would send the user to. Made by [Open SWE](https://openswe.vercel.app/agents/f30f3674-c02d-e6cd-6bf2-89df728709cb) ## References - Plan: https://openswe.vercel.app/agents/f30f3674-c02d-e6cd-6bf2-89df728709cb/plan --------- Signed-off-by: Mason Daugherty <github@mdrxy.com> Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Mason Daugherty (mdrxy)
added a commit
to langchain-ai/deepagents
that referenced
this pull request
Aug 18, 2026
) Depends on #5439 [Docs](langchain-ai/docs#5524) Some users don't talk to model providers directly — they go through a gateway or corporate proxy (for example, LangSmith's gateway, or a company-internal relay). The cold-cache warning from #5439 stays silent for all of them, because it only trusts the provider's official API address. This PR adds a setting to say "my endpoint plays by the provider's rules," turning the warnings back on: ```toml [warnings] trusted_cache_endpoints = ["smith.langchain.com"] ``` One entry covers every provider routed through that endpoint — no need to repeat it per provider. --- **Why trust is per-endpoint, not global.** The warning's dollar estimates are only accurate if the thing sitting between you and the provider passes your cache settings through untouched and keeps cached data for as long as the provider documents. A proxy that quietly drops those settings would make the warning's numbers fiction. So instead of one blanket "trust everything" switch, you name the specific endpoints you've verified, and everything else stays silent. **One case stays silent even on a trusted endpoint.** LangSmith's gateway can translate between API formats — for example, you can send an OpenAI-shaped request and have it answered by an Anthropic model. During that translation, your cache settings get rewritten to a generic 5-minute cache (or dropped entirely), so the provider's documented cache behavior no longer applies and any estimate would be a guess. When dcode can see a request will take one of these translated routes (the model name carries a cross-provider prefix like `openai:anthropic/claude-...`), it skips the warning rather than show numbers built on wrong assumptions. Requests that stay in their own format — the normal case — are forwarded by the gateway byte-for-byte (verified against the gateway source), so their warnings remain exactly as accurate as going direct. <details><summary>Test plan</summary> - Unit tests cover trusted-endpoint policy resolution, gateway same-format vs cross-format routes, lookalike-host rejection (`notsmith.langchain.com`, `smith.langchain.com.evil.example`), and malformed config tolerance. - App-level tests confirm a trusted gateway endpoint allows the warning through, and a cross-format gateway route suppresses it even when trusted. - Lint, type check, and the cold-cache/manifest suites pass. </details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft documentation for two
deepagents-codefeatures:Generated with AI assistance (Deep Agents Code).