Skip to content

perf: avoid broad message prep deepcopies - #40632

Closed
pedrommaiaa wants to merge 1 commit into
NousResearch:mainfrom
pedrommaiaa:perf/message-prep-copy-on-write
Closed

perf: avoid broad message prep deepcopies#40632
pedrommaiaa wants to merge 1 commit into
NousResearch:mainfrom
pedrommaiaa:perf/message-prep-copy-on-write

Conversation

@pedrommaiaa

Copy link
Copy Markdown
Contributor

What does this PR do?

This reduces request-prep overhead by replacing broad message-history deepcopy() calls with targeted copy-on-write behavior.

The changed paths only mutate small parts of the outgoing request: provider-only tool call fields, prompt-cache markers, Qwen content normalization, and nested image payloads used during retry recovery. Copying the full message history in those cases is unnecessary, especially in long sessions with tool calls, reasoning metadata, and images.

Local request-prep benchmarks using Hermes-shaped payloads:

  • chat cleanup, 1000-message history with dirty tool-call metadata: 9.699ms -> 0.963ms (10.1x faster)
  • Anthropic prompt-cache prep, 201-message history with large content parts: 0.517ms -> 0.072ms (7.1x faster)
  • Qwen prep, 502-message history with list content and image payloads: 1.185ms -> 0.126ms (9.4x faster)

This also tightens provider-switch safety:

  • strict providers can strip Gemini-only metadata without mutating canonical history
  • Gemini replay metadata survives strict-provider cleanup
  • Gemma no longer receives stale Gemini extra_content
  • Qwen cache_control does not leak into shared history
  • nested image URLs are isolated before retry paths can mutate request payloads

Type of Change

  • Bug fix
  • Tests
  • Refactor

Changes Made

  • Use copy-on-write sanitization in agent/transports/chat_completions.py.
  • Replace prompt-cache deepcopy() with targeted request-container copies in agent/prompt_caching.py.
  • Replace Qwen OAuth deepcopy() with targeted normalization copies in plugins/model-providers/qwen-oauth/__init__.py.
  • Add regression tests for provider switching, prompt caching, Qwen prep, Gemma/Gemini metadata handling, and nested image retry mutation.

How to Test

scripts/run_tests.sh tests/agent/test_prompt_caching.py tests/agent/transports/test_chat_completions.py tests/providers/test_provider_profiles.py tests/providers/test_profile_wiring.py tests/run_agent/test_provider_parity.py tests/run_agent/test_image_shrink_recovery.py -- -q

Result: 270 passed

scripts/run_tests.sh tests/run_agent/test_run_agent.py -- -q

Result: 367 passed

.venv/bin/ruff check agent/prompt_caching.py agent/transports/chat_completions.py plugins/model-providers/qwen-oauth/__init__.py tests/agent/test_prompt_caching.py tests/agent/transports/test_chat_completions.py tests/providers/test_provider_profiles.py tests/run_agent/test_provider_parity.py

Result: passed

@alt-glitch alt-glitch added type/perf Performance improvement or optimization P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/qwen Qwen / Alibaba Cloud (OAuth) labels Jun 6, 2026
kshitijk4poor added a commit to kshitijk4poor/hermes-agent that referenced this pull request Jul 9, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate e8c3ac2; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
kshitijk4poor added a commit to kshitijk4poor/hermes-agent that referenced this pull request Jul 9, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
kshitijk4poor added a commit that referenced this pull request Jul 9, 2026
Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by #57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate e8c3ac2; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(#47868) inside the copy-on-write path.
kshitijk4poor added a commit that referenced this pull request Jul 9, 2026
Review finding: PR #40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via #61133 — your commit was cherry-picked onto current main with your authorship preserved (now 724ab90 on main). Thanks for the contribution! The copy-on-write conversions for ChatCompletionsTransport.convert_messages and QwenProfile.prepare_messages landed together with your mutation-safety tests.

Two pieces of this branch were intentionally scoped out (see #61133 for details): the agent/prompt_caching.py selective-copy (superseded by #57229, which rewrites that same function on current main) and the Gemma extra_content narrowing (reverts deliberate behavior from e8c3ac2 — worth its own PR with its own justification if you want to pursue it). Closing in favor of the merged salvage.

santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate c1b7532; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
justemu pushed a commit to justemu/hermes-agent that referenced this pull request Jul 18, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate e8c3ac2; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
justemu pushed a commit to justemu/hermes-agent that referenced this pull request Jul 18, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate e8c3ac2; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate e8c3ac2; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…rite sites

Trim the salvaged commit to its two still-valid conversions:
- ChatCompletionsTransport.convert_messages (copy-on-write sanitize)
- QwenProfile.prepare_messages (copy-on-write normalize + cache_control)

Dropped from the original PR:
- agent/prompt_caching.py selective-copy: superseded by NousResearch#57229 which
  already rewrites apply_anthropic_cache_control on current main.
- Gemma extra_content narrowing (_model_consumes_thought_signature
  'gemini or gemma' -> 'gemini' only) + its two tests: unrelated
  behavior change reverting deliberate 6d43d01; belongs in its own
  PR with its own justification if pursued.

Conflict resolution: preserved main's newer timestamp-stripping
(NousResearch#47868) inside the copy-on-write path.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Review finding: PR NousResearch#40632's branch had silently dropped gemma-3-27b
from the keep-extra_content test loop (part of its Gemma narrowing,
which this salvage reverts). Restore main's original coverage so a
future narrowing back to Gemini-only fails loudly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have provider/qwen Qwen / Alibaba Cloud (OAuth) type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants