fix(compression): CJK-aware token estimates and stop re-copying stale head summaries

[teknium1]

Summary

Context compression now budgets CJK/Hangul/Kana text at its real token density and stops re-copying a stale handoff summary into the protected head on every re-compaction. Root causes: estimate_tokens_rough applied the English-centric (len+3)//4 rule to CJK text (undercounting ~4x, inflating the protected tail), and the Phase-4 head-copy loop carried an old protected-head handoff forward verbatim while also inserting its updated replacement, so summaries accumulated across compactions and the live prompt never actually shrank.

Changes

• agent/model_metadata.py: estimate_tokens_rough counts CJK-dense codepoints (Hangul Jamo/Syllables, CJK ideographs, fullwidth forms/halfwidth kana) as ~1 token each; estimate_messages_tokens_rough / estimate_request_tokens_rough route through it via a new image-stripping token shadow.
• agent/model_metadata.py (salvage addition): perf gate — str.isascii() O(1) fast path keeps pure-ASCII text bit-identical to the classic (len+3)//4 rule, and the CJK count runs as a single compiled-regex C pass instead of the PR's per-character Python loop.
• agent/context_compressor.py: Phase-4 head assembly skips the old handoff at summary_idx (_is_context_summary_content guarded) so re-compaction replaces it instead of duplicating it; last_head_role now reads from the assembled head. _estimate_msg_budget_tokens uses the CJK-aware estimator for string content and tool-call envelopes while preserving the Tail-protection token estimate ignores codex_reasoning_items, so Codex/Responses sessions compact too late #55572 _REPLAY_BUDGET_KEYS replay-field accounting.
• Tests: new tests/agent/test_cjk_token_estimation.py (+ salvage parity/fast-path tests against the per-char reference), continuity regression test for the stale-head-summary fix, and two stale expectations updated to the new intended behavior.

Dropped from the original PR: the compression.hygiene_threshold gateway knob (scope creep — split out).

Perf evidence (timeit, 1MB strings)

Case	(len+3)//4 baseline	PR per-char loop	This PR (gated)
1MB ASCII	0.17 µs	~3.0 s (~2.8×10⁷x)	0.23 µs (1.3x)
1MB Hangul	—	~2.1 s	~352 ms
1MB mixed EN/KO	—	—	~127 ms

Validation

	Before	After
estimate_tokens_rough("가"*400)	100	400
Re-compaction with protected-head handoff	old + new summaries both kept, prompt grows	exactly one (updated) summary survives
1MB ASCII tool-output estimate	0.17 µs	0.23 µs (no hot-path regression)

Targeted tests: bash scripts/run_tests.sh tests/agent/ -q -k 'estimate or cjk or continuity or head' → 163 passed, 0 failed; full runs of the 4 touched test files → 144 passed, 0 failed.

Credit

Salvaged from #52517 by @plainOldCode (gateway hygiene knob split out; estimator perf-gated during salvage).

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{running on 42534d6}

CI timings

CI timings · View job

Wall time 7m23s vs 7m46s (-4.9%). 7 job(s) slower, 10 faster, 3 unchanged.

• Python tests / Run tests slice 5/8: -50.0s
• Python tests / Run tests slice 1/8: +46.0s
• Python tests / Run tests slice 3/8: -26.0s
• Python tests / Run tests slice 4/8: -12.0s
• Python tests / Run tests slice 8/8: +11.0s