feat(default): auto-compaction via summarize_at_tokens

[faresobeid]

Auto-compaction for the default harness, validated in multi-day production RL runs (GLM-4.5-Air on scaleswe-v1 / swebench-verified-v1 at 131k context, ~100k rollouts).

Auto-compaction (summarize_at_tokens)

rlm-parity context compaction, matching rlm's engine semantics exactly:

• Trigger: usage.prompt_tokens >= threshold after a turn's tool results land.
• Summary request: rlm's CHECKPOINT_COMPACTION_PROMPT, with tools kept in the request under tool_choice="none" so the prompt renders like a regular turn; tool calls in the reply are ignored.
• Rebuild: system message + the summary wrapped in rlm's POST_COMPACTION_FRAMING — the original task prompt is dropped and the summary carries the goal, exactly as rlm does.
• Config: mirrors rlm's summarize_at_tokens — fixed int, or a (lo, hi) per-task draw seeded by task index so a group's rollouts share one threshold.

Long multi-turn episodes survive past the context window; observed in production that agents resume coherently across the boundary (including completing tasks post-compaction).

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes the default agent chat loop and issues extra model calls when compaction fires; behavior is gated behind config but affects long-running eval/RL rollouts and trace continuity.

Overview
Adds optional context auto-compaction to the default harness so long agent episodes can continue past the context window without manual truncation.

Harness config: New summarize_at_tokens on DefaultHarnessConfig — fixed positive int, (lo, hi) range (per-task threshold seeded by task_idx so rollouts on the same task share one draw), or None to disable. Pydantic validation rejects invalid bounds. DefaultHarness.summarize_threshold() resolves the value and passes --summarize-at-tokens into the uv program when enabled.

Program loop: After each tool-using turn, if usage.prompt_tokens meets the threshold, the agent runs an extra completion with rlm-style checkpoint prompts (CHECKPOINT_COMPACTION_PROMPT, POST_COMPACTION_FRAMING), using tool_choice="none" for the summary call. The message list is then rebuilt to system messages only plus a single user message with the framed summary — the original task user prompt is dropped, matching rlm semantics. chat() now returns prompt token usage to drive the trigger.

^{Reviewed by Cursor Bugbot for commit 85f56fb. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add auto-compaction via --summarize-at-tokens to the default harness

• Adds a summarize_at_tokens field to DefaultHarnessConfig that accepts a fixed int or a (lo, hi) range; invalid values raise a validation error at config instantiation.
• When non-zero, the harness passes --summarize-at-tokens={threshold} to the launched program, with per-task thresholds drawn deterministically from the range using random.Random(task_idx).
• In program.py, the interactive loop now tracks prompt token counts per turn; once the count meets the threshold, it requests a summary with tool_choice='none', then rebuilds the conversation to system messages plus a framed summary, discarding prior history.
• Behavioral Change: after compaction, the model conditions only on system messages and the summary rather than the full conversation history.

^{Macroscope summarized 85f56fb.}

[macroscopeapp]

🟡 Medium default/program.py:359

The compaction check uses context_tokens from the completion returned before the current turn's tool results were appended to messages. A bash, search, or MCP call can return a large payload that pushes messages past the context limit, but the code won't detect this until the next chat call. When it finally does detect it, it appends SUMMARIZE_PROMPT to the already-oversized messages list and sends that as the summary request — which fails because the list is already over the limit, so the episode still terminates at the context boundary instead of compacting. The threshold check needs to account for the size of the newly appended tool-result content before deciding whether to compact, rather than relying on stale token counts from before the tool results were added.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/harnesses/default/program.py around line 359:

The compaction check uses `context_tokens` from the completion returned *before* the current turn's tool results were appended to `messages`. A `bash`, search, or MCP call can return a large payload that pushes `messages` past the context limit, but the code won't detect this until the next `chat` call. When it finally does detect it, it appends `SUMMARIZE_PROMPT` to the already-oversized `messages` list and sends that as the summary request — which fails because the list is already over the limit, so the episode still terminates at the context boundary instead of compacting. The threshold check needs to account for the size of the newly appended tool-result content before deciding whether to compact, rather than relying on stale token counts from before the tool results were added.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 85f56fb. Configure here.}

[cursor]

Wrong task index attribute path

High Severity

summarize_threshold is called with trace.task.idx, but TraceTask only exposes data (and type). The index lives on trace.task.data.idx, as the rlm harness already uses. Because the argument is evaluated eagerly, every DefaultHarness.launch raises AttributeError, including when auto-compaction is disabled.

 

^{Reviewed by Cursor Bugbot for commit 85f56fb. Configure here.}

[cursor]

Skills docs omit new config

Medium Severity

This PR adds user-facing summarize_at_tokens to DefaultHarnessConfig for eval/RL workflows, but skills/evaluate-environments/references/REFERENCE.md still lists only edit and search under that config. The skills-update rule requires matching skill updates when evaluation/training knobs change; RLMHarnessConfig already documents the same field.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 85f56fb. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

1 blocking correctness issue found. New auto-compaction feature with significant runtime behavior changes. Unresolved review comments identify a high-severity bug (wrong attribute path trace.task.idx causing AttributeError) and medium-severity logic issue with stale token counts during compaction checks.

^{You can customize Macroscope's approvability policy. Learn more.}