Skip to content

fix(moa): apply prompt-caching decoration to the aggregator's one-shot synthesis call - #58284

Closed
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/moa-aggregate-context-prompt-cache
Closed

fix(moa): apply prompt-caching decoration to the aggregator's one-shot synthesis call#58284
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/moa-aggregate-context-prompt-cache

Conversation

@srojk34

@srojk34 srojk34 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

22c5048d9 ("fix(moa): restore prompt caching for the aggregator and advisors") restored Anthropic-style cache_control for two of MoA's three call paths: the acting aggregator (MoAChatCompletions.create, the persistent provider: moa virtual model) and the advisor fan-out (_run_reference). aggregate_moa_context() — the /moa <prompt> one-shot command's synthesis call — is a third, independent call path and was never covered: its call_llm(task="moa_aggregator", ...) sends a single undecorated user message containing the full joined reference output, re-billing the entire input on every invocation even when the resolved aggregator slot is a cache-honoring route (Claude on OpenRouter/native Anthropic, MiniMax, Qwen/DashScope) — the same cost-impact bug class 22c5048d9 measured at up to 85% cache share lost.

No test file references aggregate_moa_context at all currently, so this gap had zero regression coverage.

Changes

  • Generalized _maybe_apply_advisor_cache_control to _maybe_apply_moa_cache_control — it never had advisor-specific logic (same policy function anthropic_prompt_cache_policy, same breakpoint layout apply_anthropic_cache_control, judged purely on the passed-in runtime dict) — and reused it in aggregate_moa_context the same way _run_reference already does.
  • Computed _slot_runtime(aggregator) once in aggregate_moa_context and reused it for both the decoration call and the call_llm kwargs, instead of calling it twice.

Test plan

  • test_aggregator_synthesis_gets_cache_control_on_native_anthropic_route — a cache-honoring aggregator slot (native Anthropic) gets cache_control content-block decoration on its synthesis message
  • test_aggregator_synthesis_untouched_on_non_caching_route — a plain OpenAI aggregator slot stays undecorated (plain string content), proving the guard doesn't over-fire
  • Mutation-verified: reverting agent/moa_loop.py makes the first test fail (asserts plain string content where the fix produces native cache_control blocks)
  • scripts/run_tests.sh tests/hermes_cli/test_moa_config.py tests/agent/test_moa_trace_streamed_capture.py tests/agent/test_moa_aggregator_cost_slot.py tests/agent/test_moa_aggregator_cache_control.py tests/agent/test_moa_switch_api_mode.py tests/agent/test_moa_slot_api_mode.py tests/cli/test_moa_command.py tests/tui_gateway/test_moa_reference_emit.py tests/run_agent/test_moa_streaming.py tests/run_agent/test_moa_loop_mode.py tests/gateway/test_moa_one_shot_restore.py — 84 passed
  • ruff check clean on all changed files
  • ty check before/after diff on changed files — only line-number shifts, no new diagnostics

…t synthesis call

22c5048 restored Anthropic-style cache_control for two of MoA's three
call paths: the acting aggregator (MoAChatCompletions.create, the
persistent `provider: moa` model) and the advisor fan-out (_run_reference).
aggregate_moa_context() -- the /moa <prompt> one-shot command's synthesis
call -- is the third, independent call path and was never covered: its
call_llm(task="moa_aggregator", ...) sent a single undecorated user message
containing the full joined reference output, re-billing the entire input on
every invocation even when the resolved aggregator slot is a cache-honoring
route (Claude on OpenRouter/native Anthropic, MiniMax, Qwen/DashScope).

- Generalize _maybe_apply_advisor_cache_control to
  _maybe_apply_moa_cache_control (it never had advisor-specific logic --
  same policy function, same breakpoint layout as the main loop, judged
  purely on the passed-in runtime) and reuse it in aggregate_moa_context
  the same way _run_reference already does.
- Compute _slot_runtime(aggregator) once and reuse it for both the
  decoration call and the call_llm kwargs, instead of calling it twice.

Mutation-verified: reverting the moa_loop.py change makes the new
regression test fail by asserting a plain string aggregator-message
content where the cache-honoring case expects native cache_control
content blocks.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/anthropic Anthropic native Messages API sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) P0 Critical — data loss, security, crash loop labels Jul 4, 2026
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via #58334 (commit 2d3eac5 on main) — cherry-picked to preserve your authorship. Thanks @srojk34! This closes the last of the three MoA prompt-caching call paths (after #57675 covered the acting aggregator and advisor fan-out). Verified with an E2E run against the real aggregate_moa_context: a cache-honoring aggregator slot now gets cache_control decoration on its synthesis message while a non-caching slot stays undecorated.

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Comment

Applies the prompt-caching decoration to the MoA aggregator's one-shot synthesis call so that cached prompt prefixes are reused for the aggregation step.

Observations

  • max_tokens=max_tokens at line 85 — this passes the parameter through. Verify the aggregation call actually needs max_tokens (the synthesis output is typically short).

Looks Good

  • No hardcoded secrets
  • No debug artifacts
  • No test changes (behavioral fix)

Reviewed by Hermes Agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P0 Critical — data loss, security, crash loop provider/anthropic Anthropic native Messages API sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants