feat(compaction): calibrate token-estimate skew per content class

[Kyzcreig]

The problem

The token-estimate skew was a single global ratio, but the rough estimator's error is a rate error and that rate is not uniform across content. One blended ratio is wrong for text-heavy turns and wrong for tool-output-heavy turns.

Measurement first (this is not speculative)

I replayed real data before building anything. 30 real production sessions from the local session store, bucketed by content class, chunked to ~80 KB, and compared the production estimator (estimate_messages_tokens_rough) against provider ground truth (a real POST /v1/messages with max_tokens=1, reading usage.input_tokens, net of a measured empty-request baseline).

15.4 MB of real transcript, 176 measured chunks:

class	n	median	mean	sd	min	max
text	89	1.0048	1.0128	0.0536	0.9335	1.1456
tool	87	1.1629	1.1757	0.0871	1.0008	1.4499
global	176	1.0842	—	—	—	—

Separation is decisive:

• Welch t = 14.89 (df ≈ 142)
• Mann-Whitney z = −10.48
• Cohen's d = 2.26 (d ≥ 0.8 is conventionally "large")
• text p90 = 1.0877 vs tool p10 = 1.0842 — the distributions barely touch

Cost of the single global ratio, applied to each class:

• text-dominated content: over-corrected by 7.9%
• tool-dominated content: under-corrected by 6.8%

That is the "wrong for both" failure, quantified. Structured tool output (JSON, escaped strings, long identifiers, paths) tokenizes denser than prose, so the character-rate estimator reads LOW on it — which makes threshold compaction fire LATE, toward provider overflow.

Full method + reproduction steps: docs/per-class-skew-measurement.md.

What this adds

agent/content_class.py — a cheap, total, never-raising classifier over three classes (text / tool / media):

• Weight is attributed at the part level, not the message level. An assistant message carrying both prose and a tool_calls payload contributes to both classes proportionally, instead of mislabelling the most common message shape in an agent transcript.
• Media is weighed at its flat provider-pricing cost (media_part_token_cost), never at base64 length — so one screenshot cannot out-vote an entire conversation. (This is the bug agent/media_tokens.py exists to prevent; there's a test asserting the classifier doesn't reintroduce it.)
• "Dominant" = strict MAJORITY. With three classes, plurality-without-majority is genuinely ambiguous — a 40/35/25 turn has no single correct per-class ratio, and the honest correction for it is the blended global one. That's a definition, not a tuning value, so it's deliberately not a knob.

Per-class skew histories alongside the existing global one. The class arm is strictly additive: every pair still lands in the global history, which remains the fallback.

Required behaviors

✅ Clean fallback below the sample floor. New knob compression.skew_class_min_samples, default 3, with an explicit reader (_per_class_min_samples) — a config.yaml knob, declared in config_defaults.py (so hermes config set validates it), not a hardcoded constant and not an env var. There's a test asserting the reader is actually consulted and a test asserting the key is declared (an undeclared knob is an inert knob).

Why 3: the class ratio is a median of at most _SKEW_HISTORY (5) readings. A median only rejects an outlier from 3 samples up — at 1 it is the outlier, at 2 it's a mean of two that an outlier still drags half the distance. 3 is the smallest size at which the smoothing the calibration already relies on actually functions. Deliberately not higher: the calibration resets per conversation, so a floor of ~10 would leave the feature inert in exactly the sessions that matter. Set it to 0 to disable the class arm entirely and restore the single-global-ratio behavior exactly.

✅ Scale-up work preserved, not regressed. The per-class median rides the same clamp band as the global one, so an UNDER-count is still correctable upward, bounded by _SKEW_SCALE_UP_MAX. There's a dedicated test class (TestUndercountStillCorrectableUpward) asserting: class ratio may exceed 1.0, is bounded by _SKEW_SCALE_UP_MAX, respects the skew floor, the global ratio still exceeds 1.0, and seed_skew_calibration's band is unchanged — i.e. the exact bands PR #506 lifted from record_skew_from_real + _current_skew and PR #529 fixed in seed_skew_calibration.

✅ Measured constant NOT hardcoded. The ~2.91 body divisor is nowhere in this diff. Each class accumulates its own last-k readings and takes its own median, so the adaptive loop converges per class on its own rather than freezing one workload's number into a fleet-wide constant. Convergence is also now measurable: COMPACTION_SKEW telemetry gained a class= label, so the skew distribution in ~/.hermes/state/skew-samples.log is decomposable by class.

Wiring (the anti-dead-code assertions)

The preflight paths in agent/turn_context.py and agent/conversation_loop.py — the sites that decide compaction for every CLI/gateway turn — now thread the outgoing messages into the calibration via call_with_messages(), which degrades to the old single-argument call for plugin engines predating the parameter (they keep global-only behavior, the documented fallback).

TestProductionPathWiring includes an AST assertion over the real agent/turn_context.py: every calibration entry point (note_rough_sent, calibrated_tokens, should_compress_calibrated, _trigger_calibrated_tokens) must be called with the message list. If someone drops the argument, no request is ever classified and the whole arm is dead — that test fails loudly instead.

Tests

tests/agent/test_per_class_skew_calibration.py — 36 tests, asserting the behavior contract, never the measured constants:

• a tool-heavy turn receives the tool-class ratio, not a blended one (incl. an end-to-end decision test: same raw rough estimate fires for tool-heavy content and defers for text-heavy)
• a class below the sample floor falls back to global; at the floor it engages; floor of 0 disables it
• an under-count is still correctable upward, bounded by _SKEW_SCALE_UP_MAX
• wiring: the classifier is actually consulted on the production estimate path
• recording hygiene: the class label is consumed with the rough (no stale label tagging the next turn), reset clears per-class state, class history is bounded, global history is never starved

RED proof: against the un-implemented baseline (class arm neutered, wiring reverted, knob undeclared — module kept importable so the suite collects): 10 failed / 26 passed. With the implementation: 36/36 pass.

Regression: 589 passed / 2 failed across test_context_compressor, test_skew_persistence, test_compaction_preemption_gate, tests/context_engine/, test_api_content_sidecar. Both failures are pre-existing on the base branch (a DaemonThreadPoolExecutor._initializer environment issue), verified by stashing the change and re-running. ruff check clean on all touched files; scripts/check_subprocess_stdin.py passes.