perf(prompt-caching): copy only mutable cache-plan rows

[Xipong]

Closes #106061

Redundant full-history copy

build_prompt_cache_plan() began by deep-copying the complete provider message history, even though the planner mutates only a small set of rows: stale-marker carriers, list-valued content that needs canonical cleanup, the system row selected for decoration, and completed transaction endpoints selected for cache markers.

On long tool-heavy turns, that unconditional copy duplicated nested history structures before the request could be sent.

Copy-on-write contract

The planner now shallow-copies the outer request list and detaches only rows whose provider-visible representation may change:

• rows containing stale cache_control metadata;
• rows with list-valued content that must pass through the existing canonicalizer;
• the direct-native system carrier before system-marker decoration;
• completed transaction endpoints before adding cache markers.

Unmodified rows remain shared only with the caller's request-local structural copy. Canonical history is still isolated at the actual request-assembly boundary. Modified rows are detached before mutation, and the tools array keeps its existing independent cleanup path.

Exact behavior preservation

Unmarked list-valued content still requires cleanup: the legacy helper canonicalizes supported plain-text part shapes even when no stale marker is present. An intermediate candidate skipped that case and was rejected before publication.

The final regression compares plain strings with equivalent one-part user and two-part system content across native/non-native and direct-tool-cache layouts. Provider-visible messages, marker counts, tools, and caller input remain equal to the previous planner.

This PR does not change cache placement policy, TTLs, transcript retention, provider routing, or the four-marker budget.

Measured impact

Real planner, 32 synthetic tool schemas, median of seven calls on Linux/WSL2 with CPython 3.11.15:

Message rows	Before	After	Peak traced allocation before → after
301	1.059 ms	0.283 ms	0.131 → 0.039 MiB
1,501	4.987 ms	0.592 ms	0.746 → 0.091 MiB
6,001	20.106 ms	2.210 ms	3.045 → 0.318 MiB
12,001 stress case	40.182 ms	4.191 ms	6.238 → 0.622 MiB

The direct-native-tool layout showed the same scaling improvement: 41.481 ms → 3.948 ms at 12,001 rows.

Verification

• Allocation and identity invariant: RED on the previous implementation, GREEN on copy-on-write.
• Plain-text canonicalization invariant: RED on the rejected intermediate candidate, GREEN on the final patch.
• Final canonical matrix across seven planner, policy, auxiliary, and request files: 377 passed.
• Generated differential probes matched the baseline across 86,400 combinations, including unmarked lists, stale markers, multimodal/nested content, routing options, and real MoA mutation isolation.
• Independent final review found no security or logic issue.
• git diff --check passed. Local Ruff was unavailable in the shared virtual environment; repository CI remains responsible for that unchanged lint lane.

Boundaries and related work

Unmodified result rows may alias the caller's request-local rows; callers continue to rely on the existing structural-copy boundary before planning. No canonical-history write-through, cache, invalidation owner, schema, or dependency is introduced. The separate tool-pairing optimization in #105660 is not duplicated.

[Enough1122]

AI code review — automated review for reference; please use your judgment.

PR #106074 — perf(prompt-caching): copy only mutable cache-plan rows

Copy-on-write planning: list-copy the transcript, then deepcopy only rows that will carry markers (agent/prompt_caching.py:14-33). The identity-pinning test (plan.messages[1] is messages[1]) plus input-immutability assertions make the aliasing contract explicit. Good perf hygiene for large transcripts.

• Non-blocking — agent/prompt_caching.py:15-19: [dict(msg)] is a shallow copy, so strip_anthropic_cache_control still shares the nested content list with the caller's message. Correctness therefore relies on that helper not mutating nested inputs in place. The tests (messages == original, part_form == original) evidently cover it, but a one-line comment stating the non-mutating assumption (or a deepcopy of the content list when it's a list) would protect against a future edit to the helper silently corrupting caller state.
• agent/prompt_caching.py:27,32: system row and endpoint rows are deep-copied right before marker application — the two mutation sites are both covered. Complete.
• tests/agent/test_prompt_caching.py:75-96: the part-form/plain-text equivalence test guards canonicalization across the new sharing behavior. Nice.

No blocking issues.

[kshitijk4poor]

Thanks for this. It is salvaged into #120406 as part of the perf sweep, with your authorship kept on the commit (cherry-picked with a cherry picked from line, or --author where it had to be re-applied by hand). Kept as written.

Closing this in favour of #120406, which is armed to rebase-merge. If that PR doesn't land, I'll reopen this one.