Skip to content

feat(caching): add provider cache writes toggle #2407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
steebchen merged 12 commits into from
May 28, 2026
Merged

feat(caching): add provider cache writes toggle #2407

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
e3921bd
feat(caching): add provider cache writes toggle
RATCHAW May 25, 2026
efa836c
feat(caching): strip caller-supplied cache_control when disabled
RATCHAW May 27, 2026
aa1e7b6
fix(caching): gate Bedrock turn-boundary and retry path
RATCHAW May 27, 2026
1312b8f
Merge upstream/main into RATCHAW/api-timeout-cache-toggle
RATCHAW May 27, 2026
8e490e5
test(caching): narrow cache_control checks without as any
RATCHAW May 27, 2026
5304003
chore(db): reset migrations to main before merge
steebchen May 28, 2026
169bc62
Merge remote-tracking branch 'origin/main' into RATCHAW/api-timeout-c…
steebchen May 28, 2026
6d85e13
chore(db): regenerate migrations after merge
steebchen May 28, 2026
45711ca
chore(db): reset migrations to main before merge
steebchen May 28, 2026
edfe186
Merge remote-tracking branch 'origin/main' into RATCHAW/api-timeout-c…
steebchen May 28, 2026
27ffeb9
chore(db): regenerate migrations after merge
steebchen May 28, 2026
eb5a2c0
test(caching): align prepareRequestBody call with updated signature
steebchen May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions apps/api/src/routes/organization.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ const projectSchema = z.object({
organizationId: z.string(),
cachingEnabled: z.boolean(),
cacheDurationSeconds: z.number(),
providerCacheControlEnabled: z.boolean(),
mode: z.enum(["api-keys", "credits", "hybrid"]),
status: z.enum(["active", "inactive", "deleted"]).nullable(),
});
Expand Down
28 changes: 26 additions & 2 deletions apps/api/src/routes/projects.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ const projectSchema = z.object({
organizationId: z.string(),
cachingEnabled: z.boolean(),
cacheDurationSeconds: z.number(),
providerCacheControlEnabled: z.boolean(),
mode: z.enum(["api-keys", "credits", "hybrid"]),
status: z.enum(["active", "inactive", "deleted"]).nullable(),
});
Expand All @@ -29,13 +30,15 @@ const createProjectSchema = z.object({
organizationId: z.string().min(1),
cachingEnabled: z.boolean().optional(),
cacheDurationSeconds: z.number().min(10).max(31536000).optional(),
providerCacheControlEnabled: z.boolean().optional(),
mode: z.enum(["api-keys", "credits", "hybrid"]).optional(),
});

const updateProjectSchema = z.object({
name: z.string().min(1).max(255).optional(),
cachingEnabled: z.boolean().optional(),
cacheDurationSeconds: z.number().min(10).max(31536000).optional(), // Min 10 seconds, max 1 year
providerCacheControlEnabled: z.boolean().optional(),
mode: z.enum(["api-keys", "credits", "hybrid"]).optional(),
});

Expand Down Expand Up @@ -154,8 +157,13 @@ projects.openapi(updateProject, async (c) => {
}

const { id } = c.req.param();
const { name, cachingEnabled, cacheDurationSeconds, mode } =
c.req.valid("json");
const {
name,
cachingEnabled,
cacheDurationSeconds,
providerCacheControlEnabled,
mode,
} = c.req.valid("json");

const userOrgs = await db.query.userOrganization.findMany({
where: {
Expand Down Expand Up @@ -201,6 +209,10 @@ projects.openapi(updateProject, async (c) => {
updateData.cacheDurationSeconds = cacheDurationSeconds;
}

if (providerCacheControlEnabled !== undefined) {
updateData.providerCacheControlEnabled = providerCacheControlEnabled;
}

if (mode !== undefined) {
updateData.mode = mode;
}
Expand Down Expand Up @@ -234,6 +246,15 @@ projects.openapi(updateProject, async (c) => {
new: cacheDurationSeconds,
};
}
if (
providerCacheControlEnabled !== undefined &&
providerCacheControlEnabled !== project.providerCacheControlEnabled
) {
changes.providerCacheControlEnabled = {
old: project.providerCacheControlEnabled,
new: providerCacheControlEnabled,
};
}
if (mode !== undefined && mode !== project.mode) {
changes.mode = { old: project.mode, new: mode };
}
Expand Down Expand Up @@ -305,6 +326,7 @@ export interface CreateProjectInput {
name: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
mode?: "api-keys" | "credits" | "hybrid";
}

Expand All @@ -318,6 +340,7 @@ export async function createProjectForOrg(
name,
cachingEnabled = false,
cacheDurationSeconds = 60,
providerCacheControlEnabled = true,
mode = "hybrid",
} = input;

Expand Down Expand Up @@ -372,6 +395,7 @@ export async function createProjectForOrg(
organizationId,
cachingEnabled,
cacheDurationSeconds,
providerCacheControlEnabled,
mode,
})
.returning();
Expand Down
3 changes: 3 additions & 0 deletions apps/api/src/routes/v1-master.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ const projectSchema = z.object({
organizationId: z.string(),
cachingEnabled: z.boolean(),
cacheDurationSeconds: z.number(),
providerCacheControlEnabled: z.boolean(),
mode: projectModeEnum,
status: z.enum(["active", "inactive", "deleted"]).nullable(),
});
Expand All @@ -130,6 +131,7 @@ const createProjectBody = z.object({
name: z.string().min(1).max(255),
cachingEnabled: z.boolean().optional(),
cacheDurationSeconds: z.number().min(10).max(31536000).optional(),
providerCacheControlEnabled: z.boolean().optional(),
mode: projectModeEnum.optional(),
});

Expand Down Expand Up @@ -313,6 +315,7 @@ const updateProjectBody = z
name: z.string().min(1).max(255).optional(),
cachingEnabled: z.boolean().optional(),
cacheDurationSeconds: z.number().min(10).max(31536000).optional(),
providerCacheControlEnabled: z.boolean().optional(),
mode: projectModeEnum.optional(),
status: z.enum(["active", "inactive"]).optional(),
})
Expand Down
11 changes: 11 additions & 0 deletions apps/code/src/lib/api/v1.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand All @@ -707,6 +708,7 @@ export interface paths {
name: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -728,6 +730,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -854,6 +857,7 @@ export interface paths {
name?: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
/** @enum {string} */
Expand All @@ -877,6 +881,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7135,6 +7140,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7210,6 +7216,7 @@ export interface paths {
name?: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -7232,6 +7239,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7289,6 +7297,7 @@ export interface paths {
organizationId: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -7310,6 +7319,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -8014,6 +8024,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down
4 changes: 4 additions & 0 deletions apps/docs/content/features/caching/provider-cache-control.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ For most users, prompt caching just works — you do not need to change your req

Providers including OpenAI, Anthropic (when prompts cross the provider's minimum size), Google, DeepSeek, xAI, and Alibaba inspect incoming requests for shared prefixes and cache them automatically. LLM Gateway forwards the provider's cache metadata back to you in the response, and bills the cached portion at the model's `cached_input` rate.

For **Anthropic** and **AWS Bedrock Claude**, prompt caching is strictly opt-in via `cache_control` / `cachePoint` markers on the request body. To get automatic cache benefits without rewriting your requests, LLM Gateway injects those markers for you on long system and user messages by default. If you send long prompts sporadically — with gaps wider than the 5-minute TTL — you may want to disable this entirely, since you would otherwise pay the cache-write premium (1.25× input for 5m, 2× for 1h) without ever benefiting from a cache read.

To disable, open **Project Settings → Caching → Provider Cache Writes** and turn off "Allow provider cache writes". When disabled, the gateway strips **all** `cache_control` markers from outgoing requests for the project — both the ones it adds automatically and any markers your client sends. This covers callers that always emit markers regardless of the user's request cadence (e.g. Claude Code, Cursor, Cline). The change takes up to 5 minutes to take effect due to the project-settings cache.

To take advantage of automatic caching:

- Put stable content (system prompt, instructions, tool definitions, long documents) at the **start** of your messages
Expand Down
8 changes: 6 additions & 2 deletions apps/gateway/src/chat/chat.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3695,8 +3695,11 @@ chat.openapi(completions, async (c) => {
}

// Check if caching is enabled for this project
const { enabled: cachingEnabled, duration: cacheDuration } =
await isCachingEnabled(project.id);
const {
enabled: cachingEnabled,
duration: cacheDuration,
providerCacheControlEnabled,
} = await isCachingEnabled(project.id);
Comment on lines +3793 to +3797

@chatgpt-codex-connector chatgpt-codex-connector Bot May 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip cache lookup for dev-plan chats

For dev-plan organizations this new unconditional isCachingEnabled(project.id) call adds a second project DB dependency even though cachingEnabled is forced to false on the next line. If Postgres is unavailable and the existing swr:project:<id> mirror used by findProjectById is populated but the newly required swr:project:cachingEnabled:<id> mirror is not, dev-plan chat requests that previously proceeded will now fail before reaching the provider because isCachingEnabled rethrows on a fallback miss. The provider cache flag is already available on the fetched project, so the dev-plan branch can avoid this lookup or default from that row.

Useful? React with 👍 / 👎.


let cacheKey: string | null = null;
let streamingCacheKey: string | null = null;
Expand Down Expand Up @@ -4319,6 +4322,7 @@ chat.openapi(completions, async (c) => {
useResponsesApi,
prompt_cache_key,
prompt_cache_retention,
providerCacheControlEnabled,
);

if (forceImageStreamUpstream) {
Expand Down
11 changes: 11 additions & 0 deletions apps/playground/src/lib/api/v1.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -683,6 +683,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand All @@ -707,6 +708,7 @@ export interface paths {
name: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -728,6 +730,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -854,6 +857,7 @@ export interface paths {
name?: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
/** @enum {string} */
Expand All @@ -877,6 +881,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7135,6 +7140,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7210,6 +7216,7 @@ export interface paths {
name?: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -7232,6 +7239,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -7289,6 +7297,7 @@ export interface paths {
organizationId: string;
cachingEnabled?: boolean;
cacheDurationSeconds?: number;
providerCacheControlEnabled?: boolean;
/** @enum {string} */
mode?: "api-keys" | "credits" | "hybrid";
};
Expand All @@ -7310,6 +7319,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down Expand Up @@ -8014,6 +8024,7 @@ export interface paths {
organizationId: string;
cachingEnabled: boolean;
cacheDurationSeconds: number;
providerCacheControlEnabled: boolean;
/** @enum {string} */
mode: "api-keys" | "credits" | "hybrid";
/** @enum {string|null} */
Expand Down
1 change: 1 addition & 0 deletions ...p/dashboard/[orgId]/[projectId]/settings/preferences/_components/caching-settings-rsc.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ export const CachingSettingsRsc = async ({
preferences: {
cachingEnabled: project.cachingEnabled,
cacheDurationSeconds: project.cacheDurationSeconds,
providerCacheControlEnabled: project.providerCacheControlEnabled,
},
},
};
Expand Down
Loading
Loading