fix: end the null harness cleanly on context overflow - #2456
Merged
Conversation
The null harness has no compaction, so a relayed overlong prompt error killed its program and failed the rollout. Catch the overflow and return instead, ending the run with what the conversation holds. For the trace to stay ok the rollout must trust that exit: stop re-raising a stashed model-call error after a clean harness completion - a program that dies on it still surfaces the original error through the harness exception path, and the failed call stays recorded on the trace either way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
marked this pull request as ready for review
August 27, 2026 20:04
Contributor
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — The PR fixes context-overflow termination in the null harness, but also changes shared rollout semantics so clean harness exits after model-call failures can produce successful, scored traces. That broader production behavior change warrants focused human review. You can add or adjust custom eligibility rules. Learn more. |
hallerite
reviewed
Aug 27, 2026
Each marker now names the API whose error wording it matches, and the unattributable generics are gone - "too many tokens" also matches Bedrock throttling, and bare "context length"/"context window" substrings matched more than they targeted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 426b36e. Configure here.
The 413 markers could never fire: a 413 arrives as a plain APIStatusError, not BadRequestError. Catch APIStatusError and gate overflow detection on a deterministic status (400 or 413) so marker-shaped text in a transient failure never ends the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
hallerite
approved these changes
Aug 27, 2026
mikasenghaas
pushed a commit
that referenced
this pull request
Aug 28, 2026
## Summary - clear a handled model-call error when a harness segment exits cleanly - prevent that stale provider error from replacing a later segment's real failure ## Why PR #2456 lets a harness complete cleanly after handling a relayed model-call failure, but the resolved error remained in `RolloutSession.error`. An interactive segment can then fail before its next model request resets that slot, causing `Rollout.step()` to record the stale provider error instead of the new request-interceptor or harness failure. The reset happens only after `HarnessSession.turn()` succeeds, so a harness crash in the current segment still records the intercepted provider error. ## Verification - exact two-segment regression probe: handled overflow followed by request-interceptor failure - current-segment probe: harness failure still records the provider error - `uv run pytest tests/ -q` - `uv run ruff check --fix .` - `uv run ty check verifiers` - `uv run pre-commit run --files verifiers/v1/rollout.py` The all-files pre-commit run retains the existing MD033 failures in the unchanged legacy SWE README; the changed-file hooks pass. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Single-line lifecycle fix in rollout stepping with targeted regression tests; no auth or data-path changes. > > **Overview** > After a harness segment finishes **without** raising, **`Rollout.step()`** now sets **`self._session.error = None`**, so a provider failure that was handled in-segment (e.g. context overflow) does not linger on the session. > > That stale slot previously could win in the **`step()`** exception path (**`real = self._session.error`**) when a later segment failed before the next model call cleared it—masking the new interceptor or harness error. Failures in the **current** segment are unchanged: the clear runs only after **`turn()`** succeeds, so a harness crash still surfaces the stashed provider error. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b61b045. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Clear `self._session.error` after segment completion in `Rollout` > Resets the session error state to `None` after a segment finishes and after agent time remaining and deadline are updated. No other logic or return conditions were changed. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized b61b045.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2453: the null harness has no compaction, so the relayed overlong prompt error killed its program and failed the rollout.
Breaking
oktrace (the failure stays visible on the call record). Previously the rollout failed with the stashed error even on a clean exit.Verification
uv run pytest -q tests/v1— passed; live E2E tests skipped withoutPRIME_API_KEYuv run ruff check/uv run ruff format --checkE2E repro against a local vLLM (
vllm serve Qwen/Qwen3-0.6B --max-model-len 4096 --enable-auto-tool-choice --tool-call-parser hermes), one dummy task whose toolset returns a ~49KB payload so the turn after the tool call exceeds the 4k window (one-off fixture, not part of this PR):main)ProviderError: upstream 400: This model's maximum context length is 4096 tokens…ok=false,stop_condition=errorok=true,stop_condition=agent_completed, the overlong call still recorded with itsProviderErrorontrace.callsSame task against the OpenAI API (
gpt-5.6-luna,reasoning_effort="none"), payload ~1.2M tokens: the API rejects with400 code="context_length_exceeded": "Input tokens exceed the configured limit of 922000 tokens…"— the program catches it and the rollout completes withok=true,stop_condition=agent_completed, the failed call recorded ontrace.calls.Same task against OpenRouter directly (
openai/gpt-4o-mini, payload ~500k tokens): the 400 wraps the upstream error but relays its raw body ("This model's maximum context length is 128000 tokens…",provider_error_code: context_length_exceeded) — caught, rollout completes withok=true,stop_condition=agent_completed.Also probed
deepseek/deepseek-v4-flashon Prime Inference with the same task at growing payloads:ok=true), so the window is large;400 "Invalid request."that carries no context-length markers — the harness correctly treats it as a non-overflow provider error, re-raises, and the rollout fails with the realProviderError. The clean-exit path cannot trigger on this provider until it relays the upstream context-length message (this also blinds every marker-based overflow detector: our loops, pi/prime-agent, Codex).🤖 Generated with Claude Code
Note
Medium Risk
Behavior change: clean harness exit after a model-call failure now yields an ok trace instead of failing the rollout; marker-based overflow detection can miss generic provider 400s without length wording.
Overview
Context overflow is now treated as a normal budget stop for the null harness (which has no compaction): model
BadRequestErrors are classified via provider-specific markers inis_context_overflow, and matching failures exit the program instead of crashing.Rollout behavior changes when the harness finishes successfully after a failed model call: the driver no longer re-raises the stashed
session.error, so the trace can beokwith the failure still on the call record. Harnesses that raise on the error still fail the rollout through the existing exception path.RolloutSession.errordocs are updated to describe this split.Reviewed by Cursor Bugbot for commit 426b36e. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Summary
Follow-up to #2453: the null harness has no compaction, so the relayed overlong prompt error killed its program and failed the rollout.
Breaking
oktrace (the failure stays visible on the call record). Previously the rollout failed with the stashed error even on a clean exit.Verification
uv run pytest -q tests/v1— passed; live E2E tests skipped withoutPRIME_API_KEYuv run ruff check/uv run ruff format --checkE2E repro against a local vLLM (
vllm serve Qwen/Qwen3-0.6B --max-model-len 4096 --enable-auto-tool-choice --tool-call-parser hermes), one dummy task whose toolset returns a ~49KB payload so the turn after the tool call exceeds the 4k window (one-off fixture, not part of this PR):main)ProviderError: upstream 400: This model's maximum context length is 4096 tokens…ok=false,stop_condition=errorok=true,stop_condition=agent_completed, the overlong call still recorded with itsProviderErrorontrace.callsSame task against the OpenAI API (
gpt-5.6-luna,reasoning_effort="none"), payload ~1.2M tokens: the API rejects with400 code="context_length_exceeded": "Input tokens exceed the configured limit of 922000 tokens…"— the program catches it and the rollout completes withok=true,stop_condition=agent_completed, the failed call recorded ontrace.calls.Same task against OpenRouter directly (
openai/gpt-4o-mini, payload ~500k tokens): the 400 wraps the upstream error but relays its raw body ("This model's maximum context length is 128000 tokens…",provider_error_code: context_length_exceeded) — caught, rollout completes withok=true,stop_condition=agent_completed.Also probed
deepseek/deepseek-v4-flashon Prime Inference with the same task at growing payloads:ok=true), so the window is large;400 "Invalid request."that carries no context-length markers — the harness correctly treats it as a non-overflow provider error, re-raises, and the rollout fails with the realProviderError. The clean-exit path cannot trigger on this provider until it relays the upstream context-length message (this also blinds every marker-based overflow detector: our loops, pi/prime-agent, Codex).🤖 Generated with Claude Code
Changes since #2456 opened
CONTEXT_OVERFLOW_MARKERSconstant in the null harness to use provider-specific context overflow detection phrases [426b36e]openai.BadRequestErrortoopenai.APIStatusErrorand added HTTP status code validation (400 or 413) to theis_context_overflowfunction's overflow detection logic, which now requires both a matching context overflow marker in the error details and one of the specified status codes [f00bf5d]