From 2b54bcea19b724bde723cbf89e5f8c1d1bc54ee6 Mon Sep 17 00:00:00 2001 From: janiussyafiq Date: Wed, 20 May 2026 04:37:48 +0000 Subject: [PATCH 1/4] =?UTF-8?q?docs:=20correct=20cache-fingerprint=20descr?= =?UTF-8?q?iption=20on=20enable-response-caching=20tutorial=20=E2=80=94=20?= =?UTF-8?q?caller=20key=20is=20excluded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #349. The tutorial previously claimed the proxy keys each request on a fingerprint built from the resolved model alias, **the caller key**, and the request body. The code does the opposite — the caller's ApiKey is intentionally excluded from the cache fingerprint. The incorrect claim recurred at two places in the file (line 19 in the conceptual explanation, line 119 in the verification recap); both fixed. Operationally this means two different callers issuing the same prompt to the same model alias hit the same cache entry — the second caller will see the first caller's cached response. Operators reading the prior text would assume per-caller cache isolation that does not exist, with security-relevant implications when deciding what to cache. Code anchors: - `crates/aisix-cache/src/key.rs:4-6` — `CacheKey` module-level comment explicitly states the caller's ApiKey is excluded so two callers asking the same question hit the same entry. - `crates/aisix-cache/src/key.rs:27-41` — `CacheKey` struct fields: `model`, `messages`, `temperature_milli`, `top_p_milli`, `max_tokens`, `extras`. No caller-key field. - `crates/aisix-cache/src/key.rs:45-57` — `CacheKey::from_request` reads only those six fields; the caller's API key is never read into the fingerprint. Fix shape: - Line 19 (How It Works section): replace the incorrect sentence with an accurate fingerprint description that enumerates the fields actually hashed (model alias + normalized messages + sampling params + tool-calling extras), explicitly notes the caller-key exclusion, and adds an operational callout warning operators that prompts cross caller boundaries. - Line 119 (What Just Happened section): remove the "caller key" reference from the verification recap so it matches the conceptual explanation. --- docs/tutorials/enable-response-caching.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tutorials/enable-response-caching.md b/docs/tutorials/enable-response-caching.md index fa22d8e0..2494a0c0 100644 --- a/docs/tutorials/enable-response-caching.md +++ b/docs/tutorials/enable-response-caching.md @@ -16,7 +16,7 @@ You end with one enabled `CachePolicy` and a reproducible header-level proof tha ## How It Works -The proxy keys each request on a fingerprint built from the resolved model alias, the caller key, and the request body. When an enabled `CachePolicy` matches the request, the proxy: +The proxy keys each request on a fingerprint built from the resolved model alias and the normalized request body (messages, `temperature`, `top_p`, `max_tokens`, plus any tool-calling extras such as `tools`, `tool_choice`, `response_format`, `seed`, `stop`, `presence_penalty`, `frequency_penalty`). **The caller key is intentionally excluded from the cache fingerprint** — two different callers asking the same prompt to the same model alias hit the same cache entry, so the second caller may see the first caller's cached response. When designing what to cache, treat prompts as crossing caller boundaries: prompts and responses that should not be visible to other callers in your deployment should not be cached. When an enabled `CachePolicy` matches the request, the proxy: 1. computes the fingerprint 2. looks up the cache @@ -116,7 +116,7 @@ curl -sSi -X POST http://127.0.0.1:3000/v1/chat/completions \ ## What Just Happened -The proxy hashed the resolved model alias, caller key, and request body for each request. The first call missed the cache and went to the upstream; the second call had the same fingerprint and hit the cache; the third call had a different prompt and therefore a different fingerprint, so it missed again. The upstream saw only two calls instead of three. +The proxy hashed the resolved model alias and the request body for each request. The first call missed the cache and went to the upstream; the second call had the same fingerprint and hit the cache; the third call had a different prompt and therefore a different fingerprint, so it missed again. The upstream saw only two calls instead of three. This contract is exercised by `tests/e2e/src/cases/cache-policy-e2e.test.ts` (identical-request hit) and `tests/e2e/src/cases/cache-scenarios-e2e.test.ts` (different-prompt miss). Both tests assert the `x-aisix-cache` header values and the upstream `receivedRequests.length` count — the cache header is a published contract that cp-api and the dashboard's `/logs` view depend on. From dfc80c70c6c5598cc736842f9a9e79b8d1c8e4ef Mon Sep 17 00:00:00 2001 From: janiussyafiq Date: Wed, 20 May 2026 05:07:39 +0000 Subject: [PATCH 2/4] docs: clarify cache fingerprint extras list as examples, not exhaustive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses Copilot inline `3271255231` on PR #350 round 1. The round-1 text presented the extras list (`tools`, `tool_choice`, `response_format`, `seed`, `stop`, `presence_penalty`, `frequency_penalty`) as if those were the complete set of extra fields hashed into the cache fingerprint. In fact `CacheKey::from_request` at `crates/aisix-cache/src/key.rs:53` reads `extras: canonical_extras(&req.extra)` — the entire `ChatFormat::extra` map, not a filtered tool-calling subset. Any other top-level request field the proxy doesn't pre-process will be included in the fingerprint, including future OpenAI-shape fields that ship through `extra`. Reword to make the 7 tool-calling fields explicitly examples, and add "and any future OpenAI-shape fields that arrive through `ChatFormat::extra`" so operators know the extras list is open-ended. Same shape as PR #344 round-1's playground inline mini-def correction: a positive claim about runtime that turned out to be a subset/superset mismatch. The runtime hashes more than the doc enumerated. --- docs/tutorials/enable-response-caching.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/enable-response-caching.md b/docs/tutorials/enable-response-caching.md index 2494a0c0..e977de65 100644 --- a/docs/tutorials/enable-response-caching.md +++ b/docs/tutorials/enable-response-caching.md @@ -16,7 +16,7 @@ You end with one enabled `CachePolicy` and a reproducible header-level proof tha ## How It Works -The proxy keys each request on a fingerprint built from the resolved model alias and the normalized request body (messages, `temperature`, `top_p`, `max_tokens`, plus any tool-calling extras such as `tools`, `tool_choice`, `response_format`, `seed`, `stop`, `presence_penalty`, `frequency_penalty`). **The caller key is intentionally excluded from the cache fingerprint** — two different callers asking the same prompt to the same model alias hit the same cache entry, so the second caller may see the first caller's cached response. When designing what to cache, treat prompts as crossing caller boundaries: prompts and responses that should not be visible to other callers in your deployment should not be cached. When an enabled `CachePolicy` matches the request, the proxy: +The proxy keys each request on a fingerprint built from the resolved model alias and the normalized request body (messages, `temperature`, `top_p`, `max_tokens`, plus any other top-level request fields the proxy doesn't pre-process — including tool-calling fields such as `tools`, `tool_choice`, `response_format`, `seed`, `stop`, `presence_penalty`, `frequency_penalty`, and any future OpenAI-shape fields that arrive through `ChatFormat::extra`). **The caller key is intentionally excluded from the cache fingerprint** — two different callers asking the same prompt to the same model alias hit the same cache entry, so the second caller may see the first caller's cached response. When designing what to cache, treat prompts as crossing caller boundaries: prompts and responses that should not be visible to other callers in your deployment should not be cached. When an enabled `CachePolicy` matches the request, the proxy: 1. computes the fingerprint 2. looks up the cache From da64831166a6dacca70cc7cf20b5a1a0c239f6cd Mon Sep 17 00:00:00 2001 From: janiussyafiq Date: Wed, 20 May 2026 14:50:19 +0000 Subject: [PATCH 3/4] docs: trim cache-fingerprint paragraph to match tutorial host-style density MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Operator directive (DM `29bc02aa`): trim verbosity, match host page style. The round-2 cache-fingerprint paragraph was ~7 sentences crammed into a single block. The host tutorial (`enable-response-caching.md`) uses short paragraphs (1-2 sentences each), zero callouts, and inline em-dash asides for caveats — the round-2 form stood out as stylistically dense. Restructure the same substance into 3 shorter paragraphs: 1. fingerprint composition (model alias + normalized request body + extras-as-examples-not-exhaustive) 2. caller-key-excluded claim + cross-caller-visibility consequence (the failure-mode-naming sentence preserved as standalone bolded line) 3. operational guidance ("treat prompts as crossing caller boundaries") Drops the explicit `ChatFormat::extra` code-anchor reference (internal naming bleed not appropriate for user-facing tutorial prose). Drops the explicit 7-field tool-calling enumeration (already captured by "tool-calling fields, future OpenAI-shape fields, anything the proxy doesn't pre-process" framing). Substantive content preserved verbatim: caller-key-excluded claim, cross-caller cache-sharing consequence, the security warning's failure-mode-name, the operational don't-cache-content guidance. --- docs/tutorials/enable-response-caching.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/tutorials/enable-response-caching.md b/docs/tutorials/enable-response-caching.md index e977de65..a9383915 100644 --- a/docs/tutorials/enable-response-caching.md +++ b/docs/tutorials/enable-response-caching.md @@ -16,7 +16,13 @@ You end with one enabled `CachePolicy` and a reproducible header-level proof tha ## How It Works -The proxy keys each request on a fingerprint built from the resolved model alias and the normalized request body (messages, `temperature`, `top_p`, `max_tokens`, plus any other top-level request fields the proxy doesn't pre-process — including tool-calling fields such as `tools`, `tool_choice`, `response_format`, `seed`, `stop`, `presence_penalty`, `frequency_penalty`, and any future OpenAI-shape fields that arrive through `ChatFormat::extra`). **The caller key is intentionally excluded from the cache fingerprint** — two different callers asking the same prompt to the same model alias hit the same cache entry, so the second caller may see the first caller's cached response. When designing what to cache, treat prompts as crossing caller boundaries: prompts and responses that should not be visible to other callers in your deployment should not be cached. When an enabled `CachePolicy` matches the request, the proxy: +The proxy keys each request on a fingerprint built from the resolved model alias and the normalized request body — `messages`, `temperature`, `top_p`, `max_tokens`, and any other top-level request fields (tool-calling fields, future OpenAI-shape fields, anything the proxy doesn't pre-process). + +**The caller key is intentionally excluded from the fingerprint.** Two callers asking the same prompt to the same model alias hit the same cache entry, so the second caller sees the first caller's cached response. + +Treat prompts as crossing caller boundaries: don't cache content that shouldn't be visible to other callers in your deployment. + +When an enabled `CachePolicy` matches the request, the proxy: 1. computes the fingerprint 2. looks up the cache From c874971d471a83f7b154ecd40c7dbe315c8de25a Mon Sep 17 00:00:00 2001 From: janiussyafiq Date: Wed, 20 May 2026 15:12:22 +0000 Subject: [PATCH 4/4] docs: drop bearer-sameness implication from Step 3 cache-hit instructions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses Copilot inline `3274925344` on PR #350 round-3. Step 3 ("Send The Same Request Again — Cache Hit") previously instructed readers to repeat the "exact same request body, model alias, and bearer". After the round-2 caller-key-excluded fix on the fingerprint description in Step 1, the bearer-sameness implication re-introduces the wrong mental model that the cache key includes the caller's identity. Since the tutorial uses the same bearer throughout its example, the wrong assumption never gets disproven by observation — a stealth-cost case for tutorial-shaped artifacts. Resolution: drop "and bearer" from the sameness-required claim. Rewrite as "Repeat the request with the same body and model alias:" — the curl example below stays unchanged (still uses `sk-demo-caller`), so the reader's actual auth bearer is still populated; only the prose implication is removed. Consistent with the round-2 caller-key-excluded discipline applied prospectively to Step 3. Single-line edit; no prose added; matches operator's `29bc02aa` trim directive. The companion Copilot inline `3274925286` (fingerprint extras "unknown top-level fields" precision suggestion) is SKIP-with- reasoning per @Umar's ack: (a) no stealth-cost — `stream` field that motivated the precision request bypasses the cache entirely, so wrong-mental-model is fail-fast not stealth, (b) restoring `ChatFormat::extra` internal-struct reference would re-introduce the dim-4 internal-naming leak Muslim flagged on the round-3 trim verdict, (c) operator's trim directive outranks Copilot's specific suggested phrasing per the locked discipline. --- docs/tutorials/enable-response-caching.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/enable-response-caching.md b/docs/tutorials/enable-response-caching.md index a9383915..6a630d91 100644 --- a/docs/tutorials/enable-response-caching.md +++ b/docs/tutorials/enable-response-caching.md @@ -84,7 +84,7 @@ x-aisix-cache: miss ## Step 3: Send The Same Request Again — Cache Hit -Repeat the **exact same** request body, model alias, and bearer: +Repeat the request with the same body and model alias: ```bash title="Second call — should report cache hit" curl -sSi -X POST http://127.0.0.1:3000/v1/chat/completions \