feat: relay overlong prompt errors to the harness - #2453
Merged
Conversation
Interception rewrote overlong prompt failures into a synthesized "context_length" body and stopped the trace. Relay them like any other provider failure instead: the harness receives the provider's original message with a deterministic 400, so it can recover (e.g. compact and retry) or fail like any other provider error. Nothing catches OverlongPromptError after that, so remove the type: raise a plain ProviderError with status 400 at the two sites that produced it (renderer pre-flight overflow, Responses context_length_exceeded), drop the phrase-sniffing from model_error, and drop "context_length" from Trace.is_truncated - nothing stops with it anymore. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 27, 2026
mikasenghaas
marked this pull request as ready for review
August 27, 2026 18:26
Contributor
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This changes production handling of overlong prompts across the renderer, interception server, harness-visible HTTP responses, and trace status: failures that were previously recorded as clean context-length truncations become relayed provider errors. Because this alters existing rollout and reporting semantics across multiple components, human review is appropriate. You can add or adjust custom eligibility rules. Learn more. |
mikasenghaas
requested review from
hallerite and
xeophon
and removed request for
hallerite
August 27, 2026 19:35
xeophon
approved these changes
Aug 27, 2026
mikasenghaas
added a commit
that referenced
this pull request
Aug 27, 2026
## Summary - catch a context-overflow error in the null harness program and return cleanly instead of crashing - trust a clean harness completion: the rollout no longer re-raises a stashed model-call error after the program exits successfully - a program that dies on the error still surfaces it through the harness exception path, and the failed call stays recorded on the trace either way Follow-up to #2453: the null harness has no compaction, so the relayed overlong prompt error killed its program and failed the rollout. ## Breaking - A harness program that completes cleanly after a failed model call now produces an `ok` trace (the failure stays visible on the call record). Previously the rollout failed with the stashed error even on a clean exit. ## Verification - `uv run pytest -q tests/v1` — passed; live E2E tests skipped without `PRIME_API_KEY` - `uv run ruff check` / `uv run ruff format --check` E2E repro against a local vLLM (`vllm serve Qwen/Qwen3-0.6B --max-model-len 4096 --enable-auto-tool-choice --tool-call-parser hermes`), one dummy task whose toolset returns a ~49KB payload so the turn after the tool call exceeds the 4k window (one-off fixture, not part of this PR): | | rollout | trace | | --- | --- | --- | | before (`main`) | failed: `ProviderError: upstream 400: This model's maximum context length is 4096 tokens…` | `ok=false`, `stop_condition=error` | | after (this PR) | completed | `ok=true`, `stop_condition=agent_completed`, the overlong call still recorded with its `ProviderError` on `trace.calls` | Same task against the OpenAI API (`gpt-5.6-luna`, `reasoning_effort="none"`), payload ~1.2M tokens: the API rejects with `400 code="context_length_exceeded": "Input tokens exceed the configured limit of 922000 tokens…"` — the program catches it and the rollout completes with `ok=true`, `stop_condition=agent_completed`, the failed call recorded on `trace.calls`. Same task against OpenRouter directly (`openai/gpt-4o-mini`, payload ~500k tokens): the 400 wraps the upstream error but relays its raw body (`"This model's maximum context length is 128000 tokens…"`, `provider_error_code: context_length_exceeded`) — caught, rollout completes with `ok=true`, `stop_condition=agent_completed`. Also probed `deepseek/deepseek-v4-flash` on Prime Inference with the same task at growing payloads: - a 706k-token prompt is accepted (the run completes normally, `ok=true`), so the window is large; - past the window (~1M tokens and up), the gateway rejects with a generic `400 "Invalid request."` that carries **no context-length markers** — the harness correctly treats it as a non-overflow provider error, re-raises, and the rollout fails with the real `ProviderError`. The clean-exit path cannot trigger on this provider until it relays the upstream context-length message (this also blinds every marker-based overflow detector: our loops, pi/prime-agent, Codex). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Behavior change: clean harness exit after a model-call failure now yields an ok trace instead of failing the rollout; marker-based overflow detection can miss generic provider 400s without length wording. > > **Overview** > **Context overflow** is now treated as a normal budget stop for the null harness (which has no compaction): model `BadRequestError`s are classified via provider-specific **markers** in `is_context_overflow`, and matching failures **exit the program** instead of crashing. > > **Rollout behavior** changes when the harness finishes successfully after a failed model call: the driver **no longer re-raises** the stashed `session.error`, so the trace can be **`ok`** with the failure still on the call record. Harnesses that **raise** on the error still fail the rollout through the existing exception path. > > `RolloutSession.error` docs are updated to describe this split. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 426b36e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ## Summary > > - catch a context-overflow error in the null harness program and return cleanly instead of crashing > - trust a clean harness completion: the rollout no longer re-raises a stashed model-call error after the program exits successfully > - a program that dies on the error still surfaces it through the harness exception path, and the failed call stays recorded on the trace either way > > Follow-up to #2453: the null harness has no compaction, so the relayed overlong prompt error killed its program and failed the rollout. > > ## Breaking > > - A harness program that completes cleanly after a failed model call now produces an `ok` trace (the failure stays visible on the call record). Previously the rollout failed with the stashed error even on a clean exit. > > ## Verification > > - `uv run pytest -q tests/v1` — passed; live E2E tests skipped without `PRIME_API_KEY` > - `uv run ruff check` / `uv run ruff format --check` > > E2E repro against a local vLLM (`vllm serve Qwen/Qwen3-0.6B --max-model-len 4096 --enable-auto-tool-choice --tool-call-parser hermes`), one dummy task whose toolset returns a ~49KB payload so the turn after the tool call exceeds the 4k window (one-off fixture, not part of this PR): > > | | rollout | trace | > | --- | --- | --- | > | before (`main`) | failed: `ProviderError: upstream 400: This model's maximum context length is 4096 tokens…` | `ok=false`, `stop_condition=error` | > | after (this PR) | completed | `ok=true`, `stop_condition=agent_completed`, the overlong call still recorded with its `ProviderError` on `trace.calls` | > > Same task against the OpenAI API (`gpt-5.6-luna`, `reasoning_effort="none"`), payload ~1.2M tokens: the API rejects with `400 code="context_length_exceeded": "Input tokens exceed the configured limit of 922000 tokens…"` — the program catches it and the rollout completes with `ok=true`, `stop_condition=agent_completed`, the failed call recorded on `trace.calls`. > > Same task against OpenRouter directly (`openai/gpt-4o-mini`, payload ~500k tokens): the 400 wraps the upstream error but relays its raw body (`"This model's maximum context length is 128000 tokens…"`, `provider_error_code: context_length_exceeded`) — caught, rollout completes with `ok=true`, `stop_condition=agent_completed`. > > Also probed `deepseek/deepseek-v4-flash` on Prime Inference with the same task at growing payloads: > > - a 706k-token prompt is accepted (the run completes normally, `ok=true`), so the window is large; > - past the window (~1M tokens and up), the gateway rejects with a generic `400 "Invalid request."` that carries **no context-length markers** — the harness correctly treats it as a non-overflow provider error, re-raises, and the rollout fails with the real `ProviderError`. The clean-exit path cannot trigger on this provider until it relays the upstream context-length message (this also blinds every marker-based overflow detector: our loops, pi/prime-agent, Codex). > > 🤖 Generated with [Claude Code](https://claude.com/claude-code) > > <!-- CURSOR_SUMMARY --> > --- > > > [!NOTE] > > **Medium Risk** > > Behavior change: clean harness exit after a model-call failure now yields an ok trace instead of failing the rollout; marker-based overflow detection can miss generic provider 400s without length wording. > > > > **Overview** > > **Context overflow** is now treated as a normal budget stop for the null harness (which has no compaction): model `BadRequestError`s are classified via provider-specific **markers** in `is_context_overflow`, and matching failures **exit the program** instead of crashing. > > > > **Rollout behavior** changes when the harness finishes successfully after a failed model call: the driver **no longer re-raises** the stashed `session.error`, so the trace can be **`ok`** with the failure still on the call record. Harnesses that **raise** on the error still fail the rollout through the existing exception path. > > > > `RolloutSession.error` docs are updated to describe this split. > > > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 426b36e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> > <!-- /CURSOR_SUMMARY --><!-- Macroscope's changelog starts here --> > #### Changes since #2456 opened > > - Updated `CONTEXT_OVERFLOW_MARKERS` constant in the null harness to use provider-specific context overflow detection phrases [426b36e] > - Changed exception handling from `openai.BadRequestError` to `openai.APIStatusError` and added HTTP status code validation (400 or 413) to the `is_context_overflow` function's overflow detection logic, which now requires both a matching context overflow marker in the error details and one of the specified status codes [f00bf5d] > <!-- Macroscope's changelog ends here --> > <!-- Macroscope's pull request summary ends here --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
added a commit
that referenced
this pull request
Aug 31, 2026
## Summary - add an optional `CompactionConfig` to the bash harness: compact into a handoff summary at `summarize_at_tokens`, or when 16k tokens remain below the model context window - the program discovers the window itself from the provider's `/models` card (via `models.list()`), so the interception server gains a stateless `GET /v1/models` relay (one route serves every dialect) - truncate a tool result over 20KB to its head and tail with a warning naming the original size - compact reactively on an attributed 400/413 overflow error, from the current state; a rejected checkpoint falls back to the last state that passed a threshold check (which holds a full reserve of room by definition), and an empty or tool-calling reply is resampled - after three failed checkpoint attempts the program ends the run cleanly — still a trainable sample - an overflow with no history beyond the task propagates: nothing to reclaim The RLM harness gets the same policy in #2459, stacked on this PR so this one can merge before nano-rlm does. Together with #2453 (merged) and #2459 this supersedes #2448. ## Breaking - None on its own: `compaction` is a new optional field, unset by default. ## Verification - `uv run pytest -q tests/v1` — passed; live E2E tests skipped without `PRIME_API_KEY` - Tool truncation checked by exact match: a synthetic 55KB output produces the byte-identical expected head/tail + warning header; short outputs pass through untouched. Terminal-Bench 2 e2e: 8 tasks, local vLLM `poolside/Laguna-XS-2.1` at 32k (glm45 reasoning + glm47 tool parsers), `compaction = {}` so the threshold auto-discovers to `32768 − 16384 = 16384`. Trace analysis: - 8/8 rollouts end `ok=true` with scoring run (1 solved); no harness errors, no truncation stops. - The threshold triggers exactly where designed: 7/8 episodes compacted with peak contexts of 17.0k-23.4k tokens; the one episode that stayed at 15.4k never compacted. - 20KB tool truncation visible in 3 episodes' traces (`Warning: truncated output` with original size). - Laguna puts its entire checkpoint reply in the reasoning channel, so under the summaries-are-content-only rule every compaction on this model exercises the failure path end to end: three resampled checkpoint attempts, then the program ends the run cleanly - the rollout stays a trainable sample. Summary carry-over itself was demonstrated on models that answer in the content channel (Qwen3-0.6B at 4k/8k, deepseek-v4-flash on tb2): every non-final branch ends with the checkpoint prompt followed by the summary, and the next branch opens with `[system, framed summary]`. - A final combined verification run on a content-channel model is pending. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > The shared agent loop now permanently truncates large tool outputs and, when compaction is enabled, replaces long histories with model-generated summaries—both can change task outcomes; the new models relay is ancillary and failures only disable auto-thresholds. > > **Overview** > Adds an **opt-in** context compaction path for the bash agent: `BashHarnessConfig` gains `CompactionConfig` (`summarize_at_tokens` optional), which forwards `--compaction` / `--summarize-at-tokens` into the shared minimal/bash chat program. > > When compaction is on, the program uses a new **`Compactor`** to compact before context blows up: it can trigger on usage crossing a threshold (explicit token limit, or auto from the model card with a 16k reserve), on `finish_reason=length`, on post-tool estimated token growth, or reactively on provider 400/413 overflow. Compaction asks the model for a plain-text handoff summary (`tool_choice=none`), rebuilds history as system + framed summary user message, falls back to the last “good” checkpoint on overflow, and **ends the loop cleanly** on `CompactionFailed` instead of crashing. Threshold discovery calls `models.list()` against the intercepted base URL. > > **Tool results are always middle-truncated** at ~20KB (head/tail + warning) via `bound_tool_message` before they enter the transcript, including interception rewrites. > > The interception server adds **`GET /v1/models`** as an authenticated upstream relay (30s timeout) so the in-container SDK can read provider context-window fields without recording a model turn. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e1c80ac. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Add tool truncation and auto-compaction to `BashHarness` agent loop > - Adds `CompactionConfig` to `BashHarnessConfig` and passes `--compaction` and `--summarize-at-tokens` flags to the spawned program in [harness.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-1544e2ee3426241d8c97940a3e8daf25b2eeeeb95f8abb27e88bed4af420e609) > - Introduces the `Compactor` class in [program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36) that detects context exhaustion, builds a summary checkpoint, and rebuilds the conversation; auto-discovers a token threshold from the provider's `/v1/models` endpoint when none is given > - Truncates large tool outputs to `TOOL_OUTPUT_MAX_BYTES` (20,000) with head/tail preservation via `truncate_tool_output` before they enter the conversation > - Proxies `GET /v1/models` through the interception server in [server.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-84e2f8d027d609e8760ef6d533535ec9d99dfc6384d5ddb6111b4001b2010f35) so the program can query the provider for context window size > - `chat()` now returns the full completion object instead of the first message, and the main loop ends cleanly on `CompactionFailed` > - Risk: `chat()` signature and return type changed in [program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36); any callers expecting the first message instead of the full completion will break. Tool outputs over 20,000 bytes are silently truncated mid-conversation. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 27e1c91.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
context_lengthbody and stopping the traceOverlongPromptErrortype andmodel_errorphrase-sniffingProviderError(status_code=400)at the renderer pre-flight and Responsescontext_length_exceededsitesFirst half of the compaction stack; the harness-side recovery lands on top in the follow-up PR.
Breaking
"context_length"is gone as a stop condition, soTrace.is_truncatedno longer reports it.Verification
uv run pytest -q tests/v1— passed; live E2E tests skipped withoutPRIME_API_KEYuv run ruff check/uv run ruff format --check🤖 Generated with Claude Code
Note
Replace
OverlongPromptErrorwithProviderError(status_code=400)OverlongPromptErrorclass and its heuristic phrase-based detection from errors.py;model_errornow always returnsProviderErrorwith an explicit status code.TrainClient.get_responsenow raisesProviderError(status_code=400)instead of wrapping inOverlongPromptError.response_from_wiremapscontext_length_exceededandinvalid_promptto status 400, rate limits to 429, and everything else to 502 viamodel_error.OverlongPromptErrorbranches in both streaming and non-streamingInterceptionServer.requesthandlers; these errors now flow through the genericRolloutErrorpath.Trace.is_truncatedno longer treatscontext_lengthas a truncation condition.context_lengthtrace stop or custom 400 body; they surface as genericProviderErrorfailures with the provider's status. Callers checking forOverlongPromptErrorspecifically or relying onTrace.is_truncatedfor context-length stops need to update.Macroscope summarized fe0dad7.
Note
Medium Risk
Breaking behavior for rollouts and any code checking
OverlongPromptErrororis_truncatedfor context-length stops; core model-call error handling changes but is scoped to overlong-prompt semantics.Overview
Overlong prompts are no longer treated as a special “clean truncation” path inside interception.
OverlongPromptErroris removed fromverifiers.v1.errors, along withmodel_error’s phrase-based context-length detection.Renderer pre-flight overflow in
TrainClientand Responsescontext_length_exceedednow surface asProviderErrorwith HTTP 400 (using the renderer’sOverlongPromptErroronly at the catch site in train). The interception server drops dedicated handlers that stopped the trace with"context_length"and a syntheticrollout stopped: context_lengthbody; those failures follow the normalRolloutError/ProviderErrorHTTP relay with the provider message.Trace.is_truncatedno longer treatscontext_lengthas a stop condition. Harnesses that relied on automatic trace stop or that exception type must handle 400 provider failures explicitly (e.g. compaction in a follow-up).Reviewed by Cursor Bugbot for commit fe0dad7. Bugbot is set up for automated code reviews on this repo. Configure here.