feat(compression): optimize context compressor via autoresearch

[SSGREAPER]

Summary

Applied Karpathy's autoresearch pattern to autonomously optimize the context compressor. An agent loop ran 50 experiments, evaluating each change against a composite metric (info retention, compression ratio, structural integrity, test regression). 8 improvements were kept.

Changes discovered by the autoresearch loop:

• Summary budget: _SUMMARY_RATIO 0.20 → 0.30 (50% more budget for summaries)
• Min summary tokens: 2000 → 500 (prevents summary inflation on short conversations)
• Max summary tokens: 8000 → 4000 (tighter cap prevents bloated summaries)
• Tail token budget: 20,000 → 8,000 (more aggressive compression)
• Truncation limits: 3000 → 4500 chars per message (retains more tool output)
• File path pre-extraction: New regex-based pass that detects all file paths and appends them with "MUST appear in summary" — forces the summarizer to preserve them
• Template restructure: Moved "Relevant Files" and "Critical Context" above "Progress" sections; merged "Key Decisions" + "Constraints & Preferences"
• Mandatory preservation rules: Explicit instructions in both prompts that file paths, error messages, commands, and specific values must be preserved verbatim

Issues addressed

• fix: detect, warn, and block file re-read/search loops after context compression #705: file re-read loops after compression (file paths now explicitly preserved)
• feat: improve context compaction handoff summaries #1273: handoff summaries losing actionable state (template restructured, critical context prioritized)
• Context drift from lossy summarization (mandatory preservation rules)

Metrics (composite score 0–1, higher is better)

Metric	Before	After	Change
Composite	0.6346	0.6572	+3.6%
Info retention	0.828	0.867	+4.7%
Compression ratio	0.011	0.035	3x
Structural integrity	1.0	1.0	✓
Tests	pass	pass	✓

Test plan

• All existing compression tests pass (test_compression_boundary.py, test_context_compressor.py)
• Structural integrity validated (tool pairs, role alternation)
• 50 automated experiments with keep/revert evaluation
• Manual testing on long gateway sessions (Discord/Telegram)

🤖 Generated with Claude Code + autoresearch

[teknium1]

Closing this PR:

1. PR is unmergeable. 1,733 files changed, 280K+ additions — the branch includes 13 unrelated commits (MCP errors, Cerebras/Groq providers, gateway fixes, lock files) on top of the compressor change. GitHub can't even render the diff. The actual compressor change is 1 commit in 1 file, buried under everything else.

2. Compression tuning changes are risky without real-world validation. The autoresearch loop optimized against its own synthetic evaluation metric — that's circular. A 3.6% composite score improvement on a synthetic benchmark doesn't justify changing defaults that affect every user's compression behavior. Specifically:

• Halving _MAX_SUMMARY_TOKENS (8000 → 4000) conflicts with our Gemini auxiliary model's 12K summary cap
• Halving _DEFAULT_TAIL_TOKEN_BUDGET (20000 → 8000) means substantially more information loss on long sessions
• These parameters were tuned through real-world usage over months

3. Some ideas have merit. The file path pre-extraction regex and mandatory preservation rules for error messages/commands are worth exploring — but as a separate, clean PR with real-world testing on long gateway sessions, not as autoresearch-tuned constants.

If you want to resubmit the compressor changes, please open a PR with only the context_compressor.py changes on a fresh branch from current main.