fix(estimator): stop quadruple-counting Anthropic interleaved-thinking blocks

[adurham]

What does this PR do?

Fixes a token-estimation bug in agent/model_metadata.py's
_estimate_message_tokens_without_images() — the live preflight/compaction-
trigger estimator (reached via estimate_messages_tokens_rough(), called
from context_compressor.py, conversation_loop.py, turn_context.py, and
context_breakdown.py).

When Anthropic's interleaved-thinking feature is active, the same thinking
text ends up duplicated across up to 5 places on a stored assistant message:
content, reasoning, reasoning_content, reasoning_details, and
anthropic_content_blocks (the raw provider content array stashed onto the
message so _convert_assistant_message in agent/anthropic_adapter.py can
replay it verbatim on the next API call — that replay path reads
anthropic_content_blocks alone and never touches the other four).

The estimator had a stale exclusion check for a field named
_anthropic_content_blocks (leading underscore) that doesn't match the real
field name actually written (chat_completion_helpers.py) and read
(anthropic_adapter.py's replay path) — anthropic_content_blocks, no
underscore. So the exclusion never fired, and content/reasoning/
reasoning_content/reasoning_details/anthropic_content_blocks were all
walked into the estimate in full.

Reproduced live (not just theoretically): a message with a single
thinking block replicated across all 5 fields estimated at ~4x the token
cost it should have (~4053 vs ~1000 for a 4000-char thinking block). This is
the dominant driver of a preflight/compaction estimate running far ahead of
the real provider-reported prompt_tokens on any session using interleaved
thinking + reasoning effort — triggering compaction well before the actual
context window is anywhere near full, with no indication to the user that
it was imminent.

Related Issue

No existing issue found for this (searched via gh search issues/gh search prs for "estimate_message_chars", "anthropic_content_blocks double count",
"preflight estimator quadruple", "premature compaction interleaved
thinking" — no hits). Filing directly with a live-reproduced repro instead.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• agent/model_metadata.py: _estimate_message_tokens_without_images() —
  corrected the field-name check from _anthropic_content_blocks to
  anthropic_content_blocks, and extended the same skip to reasoning/
  reasoning_content/reasoning_details when anthropic_content_blocks is
  present. Non-Anthropic providers (no anthropic_content_blocks on the
  message) are unaffected — those fields are their only copy and are still
  counted in full.
• tests/agent/test_model_metadata.py: added TestAnthropicInterleavedThinkingDedup
  (4 tests) — blocks-present dedup counts the thinking text once, the
  non-Anthropic fallback path still counts reasoning fields normally, the
  content-vs-blocks-only path (no reasoning fields) still dedupes correctly,
  and an end-to-end regression guard comparing the fixed estimate against
  what the old unstripped behavior would have produced.

How to Test

1. Construct an assistant message with the same thinking text repeated in
   content, anthropic_content_blocks (as a thinking block),
   reasoning, reasoning_content, and reasoning_details.
2. Before this fix: estimate_messages_tokens_rough([msg]) returns
   roughly 4x the token count of the actual unique text.
3. After this fix: the estimate reflects the deduplicated text once.
4. scripts/run_tests.sh tests/agent/test_model_metadata.py -q — 136
   passed (132 pre-existing + 4 new).
5. Also ran every real caller of the estimator to check for regressions:
   tests/agent/test_context_compressor*.py, test_context_breakdown.py,
   test_turn_context*.py — 302 passed, 0 failed.

Checklist

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs/issues (none found — see Related Issue)
• My PR contains only changes related to this fix
• I've run pytest tests/ -q (targeted: tests/agent/test_model_metadata.py
  + all real callers of the estimator) and all tests pass
• I've added tests for my changes
• Tested on macOS

[teknium1]

Thanks for the focused reproduction and narrow estimator fix. The current-main premise is verified: agent/model_metadata.py:3007 excludes only _anthropic_content_blocks, while agent/chat_completion_helpers.py:1549-1551 stores anthropic_content_blocks; agent/anthropic_adapter.py:2044-2132 replays that ordered-block channel directly.

Problems

• The new blocks-present tests in tests/agent/test_model_metadata.py assert only an upper bound. They would still pass if the estimator accidentally dropped anthropic_content_blocks itself, rather than retaining it as the single counted copy.

Suggested changes

• Add a lower-bound or payload-size-proportional assertion so the tests demonstrate both halves of the contract: duplicate content/reasoning fields are excluded and the ordered replay blocks remain included.

Automated hermes-sweeper review.

[israellot]

Independently hit this on a fork carrying the same estimator, and I can confirm both your premise and the sweeper's review ask — plus add one datum that I think constrains the final shape here and in #73306.

The underscore is load-bearing. As the review notes, the exclusion names _anthropic_content_blocks while the wire path writes anthropic_content_blocks. The consequence is sharper than "one stale exclusion": on current main the ordered replay channel is counted only by accident, because the mismatched name means the exclusion never fires. So any fix that reshapes this function into a field allowlist — the natural way to kill the reasoning/reasoning_content/reasoning_details triple-count — silently drops the channel unless it is added back explicitly.

I did exactly that and measured the result. A ~11K-char interleaved-thinking payload went from 9,026 estimated tokens to 9:

anthropic_content_blocks (11K payload)   before: 9,026   after allowlist: 9
codex_reasoning_items                    before: 9,020   after allowlist: 9
codex_message_items                      before: 9,019   after allowlist: 9

That trades a bounded overcount for a ~1000x undercount, which is the more dangerous direction: compaction fires too late and the turn dies on a hard context error instead of compacting early. Worth flagging that codex_reasoning_items / codex_message_items are the same class of channel (agent_runtime_helpers.py treats a turn carrying them as payload precisely because they are "never wire-empty on any api_mode"), and that counting them appears to have been added deliberately in #55572 / #55756 — so an allowlist regresses that fix too, not just this one.

On the sweeper's lower-bound ask: it catches this exactly, and I'd suggest making it payload-proportional rather than a simple > baseline, since a placeholder-stub bug still clears a plain inequality. What worked for me:

assert result >= baseline + (len(LARGE_PAYLOAD) // 4) * 0.9

Parametrised over all three channel keys, this fails if any one is dropped or stubbed, while still allowing the duplicate-reasoning exclusion to be asserted separately. Emptying my replay-key set fails 5 of 6 new tests; the upper-bound-only version passed all of them.

One trap if you add base64 stripping alongside this. Image source.data inside a replayed block will otherwise be counted raw (~100K tokens for one screenshot), so it needs stripping — but a charset-only pattern such as ^[A-Za-z0-9+/\s]+={0,2}$ also matches ordinary letters-and-spaces prose and silently eats real thinking text. That regressed for me and needed mixed-case+digit density with whitespace forbidden. Also note _count_image_tokens doesn't walk into these channels, so an image carried only inside a replay block is charged 0 rather than _IMAGE_TOKEN_COST — a bounded per-image undercount that predates all of this and is probably out of scope, but it's there.

Happy to push my parametrised replay-channel tests as a PR against this branch if that's useful, or leave them here if you'd rather keep the diff yours. Not filing a competing PR — the reasoning-dedup design belongs in this thread and #73298.

[adurham]

Fixed per the sweeper's review and @israellot's detailed report -- thank you both, the exact ~9-token measurement was invaluable for confirming the fix has real teeth.

Added _assert_payload_proportional() (following the suggested >= floor + (len(payload) // 4) * 0.9 shape) and applied it to every existing test that carries a replay channel, plus 3 new tests covering codex_reasoning_items/codex_message_items explicitly and a meta-test proving the assertion helper itself would catch a stripped channel.

Verified empirically before pushing: simulated the exact allowlist regression locally (reshaped the function to a bare field allowlist) and confirmed 5 of 7 tests in the class correctly fail against it -- reproduced your ~9-token estimate for the stripped payload almost exactly. Restored the real fix and all 7 pass.

@israellot -- appreciate the offer to push parametrized tests directly; I went ahead and added them myself since I had the local repro running, but flagging your name in the commit message for the credit either way. Let me know if you want to open a follow-up for the base64-stripping trap you found in the replay blocks (separate concern from this PR's scope, but a real one) -- happy to look at that too if useful.

[adurham]

Important update: while rebasing this onto current main, found that upstream's own #73298 refactor extracted _estimate_message_tokens_without_images()'s logic into a new shared _wire_message_shadow() helper -- and that refactor introduced the exact bug this PR describes, independently. _wire_message_shadow() excludes the legacy back-compat key _anthropic_content_blocks (underscore) but the live wire field is anthropic_content_blocks (no underscore) -- so on current main right now, the exclusion silently never fires.

Verified empirically before fixing: a message with duplicated thinking text across content/reasoning/reasoning_content/anthropic_content_blocks is estimated at 4.04x the correct token count on unpatched main. Since _wire_message_shadow() now feeds both the char-count and token-estimate paths, this is a bigger-blast-radius version of the original bug, not a smaller one.

Retargeted the fix into _wire_message_shadow() directly (previous version patched the old, now-removed inline logic). All the payload-proportional test hardening from the earlier review round (thanks again @israellot) carried over and still passes -- 7/7, plus the full test_model_metadata.py suite (66/66) and adjacent context-compression/turn-context suites (236/236, since the shared helper needed a wider regression check).

Verified the fix has teeth: reverted just the anthropic_content_blocks dedup and re-ran -- 5/7 tests correctly failed, reproducing the same ~4x pattern. Restored and all 7 pass. Rebased onto current main, now shows mergeable.

[GottZ]

This was generated by AI during triage.

Summary

Seven PRs address or reference three compression defects: #72087, #73306, and #75884 change thinking-token estimation; #74251, #74417, and #75884 persist hygiene failure cooldowns; #75604, #75635, and #75884 bound the short-tool-suffix compression path. #75884 contains the landed cross-issue implementation, while #72087 retains a distinct Anthropic replay-channel deduplication beyond #75884's reasoning_details exclusion.

Related pull requests

• fix(estimator): stop quadruple-counting Anthropic interleaved-thinking blocks #72087 partial — (+223/-1) — n/a: The diff updates _wire_message_shadow() to count anthropic_content_blocks once while suppressing duplicated content, reasoning, and reasoning_content, with lower- and upper-bound tests that preserve Anthropic and Codex replay payload accounting. Consistent with the automated keep-open verdict, this remains a high-salvageability complement to fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 rather than an exact duplicate.
• fix: exclude reasoning_details from preflight token estimate (#73298) #73306 [closed] best fix — (+2/-2) — n/a: The diff narrowly excludes reasoning_details from both preflight estimator helpers while retaining ordinary reasoning content; it remains relevant because this production change was salvaged into fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 with regression coverage and the separate tail-budget correction. Despite the COMMENTED keep_open review, fix: exclude reasoning_details from preflight token estimate (#73298) #73306 is already closed after fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 incorporated its fix with preserved authorship.
• fix: persist session hygiene compression cooldown to state DB #74251 [closed] best fix — (+52/-20) — n/a: The diff replaces the process-local hygiene cooldown check and timeout/abort writes with the existing DB-backed cooldown methods. Despite the COMMENTED keep_open review requesting corrected tests and restart coverage, fix: persist session hygiene compression cooldown to state DB #74251 is already closed because its best-fix implementation was salvaged into fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884, which added a real AsyncSessionDB restart regression.
• fix(gateway): persist session hygiene compression-failure cooldown to state DB (#74136) #74417 [closed] best fix — (+52/-23) — n/a: The diff implements the same DB-backed hygiene cooldown change as fix: persist session hygiene compression cooldown to state DB #74251 and attempts test updates, but those fixtures expose raw MagicMock methods rather than the awaited production facade. Despite the automated keep-open verdict, it is already closed as a duplicate of fix: persist session hygiene compression cooldown to state DB #74251; fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 retained the fix and supplied the missing real-DB restart test.
• fix(agent): clamp tail-cut boundary and summary-scan indices to prevent IndexError #75604 [closed] best fix — (+122/-1) — n/a: The diff clamps the tail-cut result and summary-scan bounds, directly addressing the len(messages)+1 IndexError, with helper-level regression tests. Despite the COMMENTED keep_open review requesting production-path coverage, it is already closed because its best-fix commits were salvaged into fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 with the requested compress()-level no-summary, unchanged-transcript regression.
• fix(context-compressor): clamp out-of-range tail-cut and summary-scan bounds (#75588) #75635 [closed] best fix — (+257/-5) — n/a: The diff applies equivalent tail-cut and scan guards, additionally tolerates non-dict scan rows, and adds broader helper and compress() tests. It remains relevant as the recorded best fix for [Bug] Short tool-only suffix can make context compressor scan past messages #75588 but is already closed as a duplicate of fix(agent): clamp tail-cut boundary and summary-scan indices to prevent IndexError #75604; its keep-open review specifically requested has_content_to_compress() coverage, while fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 landed the competing fix(agent): clamp tail-cut boundary and summary-scan indices to prevent IndexError #75604 implementation with an end-to-end regression.
• fix(compression): index clamps, persisted failure cooldown, user-query preservation, thinking-token estimates #75884 [merged] best fix — (+770/-33) — n/a: This landed consolidation implements the recorded best fixes for all three issues: reasoning_details envelope exclusion in preflight and tail budgeting, DB-persisted hygiene cooldowns with restart coverage, and bounded tail-cut/summary scans with a compress()-level regression. It is the reference implementation that supersedes the narrower closed PRs, while its additional non-empty-user-query guard is outside these three reported causes.

Duplicates

#74417 is a duplicate of #74251; #75635 substantially duplicates #75604. #73306's estimator fix and the best-fix portions of #74251 and #75604 were incorporated into #75884; #72087 overlaps the estimator area but is not an exact duplicate because it deduplicates the live anthropic_content_blocks replay channel while preserving that channel's payload accounting.

Suggested consolidation

Keep #72087 open with a salvage path: retain its payload-proportional replay-channel tests and rebase its Anthropic content/reasoning deduplication on top of #75884's landed reasoning_details handling, explicitly verifying that anthropic_content_blocks and both Codex replay channels still count once. The automated keep-open verdict on #72087 supports this path; #73306, #74251, #74417, #75604, and #75635 should remain closed under the explicit salvage/duplicate chains to #75884, with no additional duplicate closure needed.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I73298(["issue #73298 (closed)"])
    P72087["PR #72087 (open)"]
    P72087 -.->|partial| I73298
    class I73298 closed
    class P72087 open
    class P72087 target
    click I73298 "https://github.com/NousResearch/hermes-agent/issues/73298"
    click P72087 "https://github.com/NousResearch/hermes-agent/pull/72087"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 7 pull requests and 3 issues in this complex. Each diff was read against this issue; Assessment working set: 94 kB of PR diffs, 33 kB of issue/PR text, 27 kB of discussion (27 comments), 20 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.