fix(context): calibrate the chars/4 guard estimate with provider-reported prompt_tokens

[skygunner]

TL;DR (plain language)

The guard estimates input size as "count characters, divide by 4". That is right for prose, but tool-schema JSON's true density depends on the provider's tokenizer: o200k packs it at ~4.7 chars/token while Qwen-family tokenizers reach ~25 — the js-tiktoken cross-check in #14931 showed no single static ratio can cover both (a discount tuned for Qwen would under-estimate for OpenAI-family providers and let genuinely oversized requests through).

So this PR doesn't guess a ratio — it learns one. Every completed request already carries the provider's own usage.prompt_tokens. An exponential moving average of actual / estimated is kept per (provider, model), split into a tools-present and a tools-absent bucket (agent traffic is ~90% tool schema, chat traffic ~0%). The next estimate for the same shape is multiplied by that factor. Cold start is factor 1.0 — byte-identical to today's behavior — and the guard gets more accurate with every request.

This is Part 2 of a 3-part series on #14931 — Part 1: sibling-side limit resolution · Part 2: estimator calibration (this PR, the load-bearing fix per the issue discussion) · Part 3: proactive-compression log basis labels. Sibling PRs: #15053 and #15055. Please keep the issue open until all three land.

Safety properties

• Cold start = status quo. Fewer than 3 trusted observations for a shape → factor 1.0, estimate unchanged.
• Clamped to [0.15, 2.0] — the guard never loses its teeth, and the fix(backend): Context guard rejects Codex CLI at ~50k real tokens: chars/4 estimator over-counts ~1MB of tool schemas 5-6x, checked against a generic 128k fallback limit (model catalog miss → combo-min) #14931 incident ratio (~0.19) sits above the floor.
• Dirty observations rejected: ratios outside [0.05, 20], estimates under 1,000 tokens, and usage objects without a token count never enter the EMA.
• Multi-leg tool-loop sums skipped: their prompt_tokens re-counts resent history and would poison the factor.
• In-memory only. A restart re-learns from scratch; the worst case is exactly today's behavior, never worse.
• Escape hatch: OMNIROUTE_ESTIMATOR_CALIBRATION=off|0|false disables recording and application entirely.

Wiring

Everything lives in the new open-sse/services/estimatorCalibration.ts (164 lines). The chatCore integration sits in handlers/chatCore/contextEstimation.ts — next to the raw estimate it scales, and outside the frozen chatCore.ts size budget (handleChatCoreInner grows by 8 physical lines):

• the calibrated estimate feeds both the last-resort compaction and enforceOutputTokenBudget (all re-computations go through the same factor, so the basis stays consistent);
• the streaming and non-streaming completion paths both record (guard-acted estimate, provider prompt_tokens) pairs.

Claude-native input_tokens is accepted alongside prompt_tokens.

Evidence

• tests/unit/estimator-calibration.test.ts (10 cases): cold-start identity, bucket isolation, clamping, dirty-ratio rejection, EMA outlier absorption, tool-loop skip, input_tokens acceptance, env off-switch.
• tests/unit/chatcore-estimator-calibration-14931.test.ts, end-to-end through handleChatCore: three requests whose stub upstream reports prompt_tokens at 1/4 of the chars/4 estimate teach the bucket; a fourth request whose raw estimate (25,000) exceeds the context limit (20,000) then passes the guard — and the same request with calibration disabled is still rejected at limit 20000 (counterfactual control).
• Container E2E against a mock OpenAI-compatible upstream reporting a 0.2× density (reproducing the fix(backend): Context guard rejects Codex CLI at ~50k real tokens: chars/4 estimator over-counts ~1MB of tool schemas 5-6x, checked against a generic 128k fallback limit (model catalog miss → combo-min) #14931 incident shape): the previously-fatal request shape returns 200; a request 33% above the inherited limit is still rejected — at member A with the calibrated number, at fallback member B (cold bucket) with the raw number — confirming per-member bucket isolation and combo failover behavior.

Relationship to #14931

Combined with Part 1, the incident is closed end-to-end: the limit stops being a baseless 128,000, and the estimate stops pretending 1 MB of Qwen-tokenized tool schemas is prose. qwen3.8-27b deployments can then retire the DISABLE_CONTEXT_WINDOW_CHECKS workaround.

[diegosouzapw]

Re-homed to release/v3.8.52: v3.8.51 entered its release freeze, so the branch now belongs to the release captain and development continues on the next cycle. Nothing is wrong with this PR — it just needed a live base. No action needed from you; CI will re-run against the new base.