perf(caching): selective copy instead of deepcopy entire history by kohoj · Pull Request #57046 · NousResearch/hermes-agent

kohoj · 2026-07-02T12:09:48Z

What

Replaces full deepcopy with selective shallow copy in apply_anthropic_cache_control.

Why

Every agent turn calls this function to inject cache markers. The current implementation deep-copies the entire conversation history — but only 4 messages actually get modified (system + last 3 non-system). Everything else is copied for no reason.

In a 100-message conversation:

Before: ~15ms per call
After: ~2ms per call
7.5x faster, scales with conversation length

Long sessions (200+ messages, tool-heavy workflows) see proportionally larger wins. Memory footprint drops too — no duplicated unchanged messages.

How

list(api_messages) creates a shallow copy of the message list. When we need to modify a message to add a cache marker, we deepcopy just that one message and replace it in the list. Unmodified messages stay as references.

The contract is preserved: callers get an independent list they can mutate. Only the 4 messages we touch are isolated copies.

Testing

Existing tests pass — behavior is unchanged. The optimization is invisible to callers: same input, same output, just faster.

Noticed this while reading ee8cbfd — the web_extract optimization removed an auxiliary LLM call, and it made me wonder where else we're doing redundant work on every turn. This was the next obvious one.

Replaces full deepcopy with selective shallow copy in apply_anthropic_cache_control. Only the 4 messages that receive cache_control markers are deep-copied; the rest stay as references. Measured on a 100-message conversation (typical long session): - Before: ~15ms per call - After: ~2ms per call - 7.5x faster, scales with conversation length Memory impact is also significant — no need to duplicate dozens of unchanged messages on every turn. The contract is unchanged: callers still get an independent message list they can mutate. Only messages we modify (by injecting cache markers) are copied. The rest are shared references to immutable history entries, which is safe since the agent never mutates past turns.

teknium1

Thanks for targeting a real per-turn allocation hotspot: current main still performs the full history deepcopy in agent/prompt_caching.py:97.

Problems

The PR's candidate selection at agent/prompt_caching.py:76 predates current main's _can_carry_marker() filter at agent/prompt_caching.py:110-115. Applying it unchanged would select non-carrier OpenRouter-layout turns that main intentionally skips; tests/agent/test_prompt_caching.py:198-224 covers this breakpoint-preservation behavior.
The existing deep-copy test at tests/agent/test_prompt_caching.py:126-132 exercises only a marked message. The selective-copy boundary needs regression coverage for unmarked history and current carrier filtering.

Suggested changes

Salvage the copy-on-write optimization while preserving _can_carry_marker() in the candidate list.
Add a test combining unmarked history with empty assistant/tool messages and assert input immutability plus unchanged usable-marker placement.

This is an automated hermes-sweeper review.

teknium1 · 2026-07-15T13:38:14Z

        breakpoints_used += 1

    remaining = 4 - breakpoints_used
    non_sys = [i for i in range(len(messages)) if messages[i].get("role") != "system"]


Current main filters candidates through _can_carry_marker(..., native_anthropic=...) so empty/non-carrier OpenRouter turns do not spend breakpoint capacity. Preserve that filter when salvaging this selective-copy change; see current agent/prompt_caching.py:110-115 and tests/agent/test_prompt_caching.py:198-224.

…-equivalence Self-review found the deepcopy->shallow-copy change in apply_anthropic_cache_control (NousResearch#57046) had no test pinning the "prompt caching is sacred / never mutate the caller's list" invariant. The old test_returns_deep_copy only exercised the single marked message, never an un-marked shared reference, so a regression that deep-copied too little would pass the whole suite. Add two tests: (1) caller list + every element left byte-identical after the call, un-marked middle messages returned as shared references, marked messages fresh copies, and mutating a returned marked message does not leak upstream; (2) structural byte-equivalence vs a reference full-deepcopy implementation across both native_anthropic modes and two TTLs. Mutation-verified: neutering the per-message deepcopy makes test (1) fail.

…images never reach stored history With the selective prompt-cache copy (NousResearch#57046), un-marked messages on the decorated api_messages list share their nested content parts with the persistent conversation history — the per-message copy in conversation_loop is shallow and decoration now deep-copies only the marked messages. try_shrink_image_parts_in_messages previously wrote the re-encoded image INTO the aliased part/source dicts, so an image-too-large retry on an Anthropic route would silently replace the original image bytes in agent.messages (and persist the degraded copy). Replace the in-place writes with copy-on-write: rebuild the content list with fresh part/source/image_url dicts and reassign msg['content'] — a top-level write on the per-call copy that never reaches history. Adds two regression tests simulating the aliasing; both fail against the old in-place implementation (mutation-verified).

…-equivalence Self-review found the deepcopy->shallow-copy change in apply_anthropic_cache_control (#57046) had no test pinning the "prompt caching is sacred / never mutate the caller's list" invariant. The old test_returns_deep_copy only exercised the single marked message, never an un-marked shared reference, so a regression that deep-copied too little would pass the whole suite. Add two tests: (1) caller list + every element left byte-identical after the call, un-marked middle messages returned as shared references, marked messages fresh copies, and mutating a returned marked message does not leak upstream; (2) structural byte-equivalence vs a reference full-deepcopy implementation across both native_anthropic modes and two TTLs. Mutation-verified: neutering the per-message deepcopy makes test (1) fail.

…images never reach stored history With the selective prompt-cache copy (#57046), un-marked messages on the decorated api_messages list share their nested content parts with the persistent conversation history — the per-message copy in conversation_loop is shallow and decoration now deep-copies only the marked messages. try_shrink_image_parts_in_messages previously wrote the re-encoded image INTO the aliased part/source dicts, so an image-too-large retry on an Anthropic route would silently replace the original image bytes in agent.messages (and persist the degraded copy). Replace the in-place writes with copy-on-write: rebuild the content list with fresh part/source/image_url dicts and reassign msg['content'] — a top-level write on the per-call copy that never reaches history. Adds two regression tests simulating the aliasing; both fail against the old in-place implementation (mutation-verified).

kshitijk4poor · 2026-07-28T14:38:48Z

Merged via #57229 — your commit was cherry-picked onto current main with authorship preserved (now on main as 6a37b5a-era rebase, perf(caching): selective copy instead of deepcopy entire history). Verified byte-identical output vs the old deepcopy path across native/TTL/static-prefix combos, ~39x on a 3 MB history. Thanks for the excellent find!

…-equivalence Self-review found the deepcopy->shallow-copy change in apply_anthropic_cache_control (NousResearch#57046) had no test pinning the "prompt caching is sacred / never mutate the caller's list" invariant. The old test_returns_deep_copy only exercised the single marked message, never an un-marked shared reference, so a regression that deep-copied too little would pass the whole suite. Add two tests: (1) caller list + every element left byte-identical after the call, un-marked middle messages returned as shared references, marked messages fresh copies, and mutating a returned marked message does not leak upstream; (2) structural byte-equivalence vs a reference full-deepcopy implementation across both native_anthropic modes and two TTLs. Mutation-verified: neutering the per-message deepcopy makes test (1) fail.

…images never reach stored history With the selective prompt-cache copy (NousResearch#57046), un-marked messages on the decorated api_messages list share their nested content parts with the persistent conversation history — the per-message copy in conversation_loop is shallow and decoration now deep-copies only the marked messages. try_shrink_image_parts_in_messages previously wrote the re-encoded image INTO the aliased part/source dicts, so an image-too-large retry on an Anthropic route would silently replace the original image bytes in agent.messages (and persist the degraded copy). Replace the in-place writes with copy-on-write: rebuild the content list with fresh part/source/image_url dicts and reassign msg['content'] — a top-level write on the per-call copy that never reaches history. Adds two regression tests simulating the aliasing; both fail against the old in-place implementation (mutation-verified).

alt-glitch added type/perf Performance improvement or optimization P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jul 2, 2026

hashbender mentioned this pull request Jul 2, 2026

perf(caching): selective copy instead of deepcopy entire history hashbender/hermes-agent#846

Open

kshitijk4poor mentioned this pull request Jul 2, 2026

perf(agent): AIAgent hot-path salvage — prompt-cache copy, reasoning-timeout precompute, lazy compressor init #57229

Merged

teknium1 reviewed Jul 15, 2026

View reviewed changes

teknium1 added sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026

kshitijk4poor closed this Jul 28, 2026

Navigation Menu

Uh oh!

perf(caching): selective copy instead of deepcopy entire history - #57046

perf(caching): selective copy instead of deepcopy entire history#57046
kohoj wants to merge 1 commit into
NousResearch:mainfrom
kohoj:perf/cache-selective-copy

kohoj commented Jul 2, 2026

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 15, 2026

Uh oh!

kshitijk4poor commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

kohoj commented Jul 2, 2026

What

Why

How

Testing

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

kshitijk4poor commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants