Skip to content

feat: relay overlong prompt errors to the harness - #2453

Merged
mikasenghaas merged 1 commit into
mainfrom
feat/relay-overlong-errors
Aug 27, 2026
Merged

feat: relay overlong prompt errors to the harness#2453
mikasenghaas merged 1 commit into
mainfrom
feat/relay-overlong-errors

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Aug 27, 2026

Copy link
Copy Markdown
Member

Summary

  • relay an overlong prompt to the harness like any other provider failure
  • return the provider's original error message with a deterministic 400
  • stop synthesizing a context_length body and stopping the trace
  • remove the now-unused OverlongPromptError type and model_error phrase-sniffing
  • raise a plain ProviderError(status_code=400) at the renderer pre-flight and Responses context_length_exceeded sites

First half of the compaction stack; the harness-side recovery lands on top in the follow-up PR.

Breaking

  • Interception no longer stops a trace on an overlong prompt. A harness that does not recover (e.g. by compacting) fails the rollout like any other provider error.
  • "context_length" is gone as a stop condition, so Trace.is_truncated no longer reports it.

Verification

  • uv run pytest -q tests/v1 — passed; live E2E tests skipped without PRIME_API_KEY
  • uv run ruff check / uv run ruff format --check

🤖 Generated with Claude Code

Note

Replace OverlongPromptError with ProviderError(status_code=400)

  • Removes OverlongPromptError class and its heuristic phrase-based detection from errors.py; model_error now always returns ProviderError with an explicit status code.
  • Renderer preflight overflow in TrainClient.get_response now raises ProviderError(status_code=400) instead of wrapping in OverlongPromptError.
  • response_from_wire maps context_length_exceeded and invalid_prompt to status 400, rate limits to 429, and everything else to 502 via model_error.
  • Removes dedicated OverlongPromptError branches in both streaming and non-streaming InterceptionServer.request handlers; these errors now flow through the generic RolloutError path.
  • Trace.is_truncated no longer treats context_length as a truncation condition.
  • Behavioral Change: overlong prompts no longer produce a special context_length trace stop or custom 400 body; they surface as generic ProviderError failures with the provider's status. Callers checking for OverlongPromptError specifically or relying on Trace.is_truncated for context-length stops need to update.

Macroscope summarized fe0dad7.


Note

Medium Risk
Breaking behavior for rollouts and any code checking OverlongPromptError or is_truncated for context-length stops; core model-call error handling changes but is scoped to overlong-prompt semantics.

Overview
Overlong prompts are no longer treated as a special “clean truncation” path inside interception. OverlongPromptError is removed from verifiers.v1.errors, along with model_error’s phrase-based context-length detection.

Renderer pre-flight overflow in TrainClient and Responses context_length_exceeded now surface as ProviderError with HTTP 400 (using the renderer’s OverlongPromptError only at the catch site in train). The interception server drops dedicated handlers that stopped the trace with "context_length" and a synthetic rollout stopped: context_length body; those failures follow the normal RolloutError / ProviderError HTTP relay with the provider message.

Trace.is_truncated no longer treats context_length as a stop condition. Harnesses that relied on automatic trace stop or that exception type must handle 400 provider failures explicitly (e.g. compaction in a follow-up).

Reviewed by Cursor Bugbot for commit fe0dad7. Bugbot is set up for automated code reviews on this repo. Configure here.

Interception rewrote overlong prompt failures into a synthesized
"context_length" body and stopped the trace. Relay them like any
other provider failure instead: the harness receives the provider's
original message with a deterministic 400, so it can recover (e.g.
compact and retry) or fail like any other provider error.

Nothing catches OverlongPromptError after that, so remove the type:
raise a plain ProviderError with status 400 at the two sites that
produced it (renderer pre-flight overflow, Responses
context_length_exceeded), drop the phrase-sniffing from model_error,
and drop "context_length" from Trace.is_truncated - nothing stops
with it anymore.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mikasenghaas
mikasenghaas marked this pull request as ready for review August 27, 2026 18:26
@macroscopeapp

macroscopeapp Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This changes production handling of overlong prompts across the renderer, interception server, harness-visible HTTP responses, and trace status: failures that were previously recorded as clean context-length truncations become relayed provider errors. Because this alters existing rollout and reporting semantics across multiple components, human review is appropriate.

You can add or adjust custom eligibility rules. Learn more.

@mikasenghaas
mikasenghaas requested review from hallerite and xeophon and removed request for hallerite August 27, 2026 19:35
@mikasenghaas
mikasenghaas merged commit 2e8d186 into main Aug 27, 2026
13 checks passed
@mikasenghaas
mikasenghaas deleted the feat/relay-overlong-errors branch August 27, 2026 19:43
mikasenghaas added a commit that referenced this pull request Aug 27, 2026
## Summary

- catch a context-overflow error in the null harness program and return
cleanly instead of crashing
- trust a clean harness completion: the rollout no longer re-raises a
stashed model-call error after the program exits successfully
- a program that dies on the error still surfaces it through the harness
exception path, and the failed call stays recorded on the trace either
way

Follow-up to #2453: the null harness has no compaction, so the relayed
overlong prompt error killed its program and failed the rollout.

## Breaking

- A harness program that completes cleanly after a failed model call now
produces an `ok` trace (the failure stays visible on the call record).
Previously the rollout failed with the stashed error even on a clean
exit.

## Verification

- `uv run pytest -q tests/v1` — passed; live E2E tests skipped without
`PRIME_API_KEY`
- `uv run ruff check` / `uv run ruff format --check`

E2E repro against a local vLLM (`vllm serve Qwen/Qwen3-0.6B
--max-model-len 4096 --enable-auto-tool-choice --tool-call-parser
hermes`), one dummy task whose toolset returns a ~49KB payload so the
turn after the tool call exceeds the 4k window (one-off fixture, not
part of this PR):

| | rollout | trace |
| --- | --- | --- |
| before (`main`) | failed: `ProviderError: upstream 400: This model's
maximum context length is 4096 tokens…` | `ok=false`,
`stop_condition=error` |
| after (this PR) | completed | `ok=true`,
`stop_condition=agent_completed`, the overlong call still recorded with
its `ProviderError` on `trace.calls` |

Same task against the OpenAI API (`gpt-5.6-luna`,
`reasoning_effort="none"`), payload ~1.2M tokens: the API rejects with
`400 code="context_length_exceeded": "Input tokens exceed the configured
limit of 922000 tokens…"` — the program catches it and the rollout
completes with `ok=true`, `stop_condition=agent_completed`, the failed
call recorded on `trace.calls`.

Same task against OpenRouter directly (`openai/gpt-4o-mini`, payload
~500k tokens): the 400 wraps the upstream error but relays its raw body
(`"This model's maximum context length is 128000 tokens…"`,
`provider_error_code: context_length_exceeded`) — caught, rollout
completes with `ok=true`, `stop_condition=agent_completed`.

Also probed `deepseek/deepseek-v4-flash` on Prime Inference with the
same task at growing payloads:

- a 706k-token prompt is accepted (the run completes normally,
`ok=true`), so the window is large;
- past the window (~1M tokens and up), the gateway rejects with a
generic `400 "Invalid request."` that carries **no context-length
markers** — the harness correctly treats it as a non-overflow provider
error, re-raises, and the rollout fails with the real `ProviderError`.
The clean-exit path cannot trigger on this provider until it relays the
upstream context-length message (this also blinds every marker-based
overflow detector: our loops, pi/prime-agent, Codex).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Behavior change: clean harness exit after a model-call failure now
yields an ok trace instead of failing the rollout; marker-based overflow
detection can miss generic provider 400s without length wording.
> 
> **Overview**
> **Context overflow** is now treated as a normal budget stop for the
null harness (which has no compaction): model `BadRequestError`s are
classified via provider-specific **markers** in `is_context_overflow`,
and matching failures **exit the program** instead of crashing.
> 
> **Rollout behavior** changes when the harness finishes successfully
after a failed model call: the driver **no longer re-raises** the
stashed `session.error`, so the trace can be **`ok`** with the failure
still on the call record. Harnesses that **raise** on the error still
fail the rollout through the existing exception path.
> 
> `RolloutSession.error` docs are updated to describe this split.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
426b36e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ## Summary
>
> - catch a context-overflow error in the null harness program and
return cleanly instead of crashing
> - trust a clean harness completion: the rollout no longer re-raises a
stashed model-call error after the program exits successfully
> - a program that dies on the error still surfaces it through the
harness exception path, and the failed call stays recorded on the trace
either way
>
> Follow-up to #2453: the null harness has no compaction, so the relayed
overlong prompt error killed its program and failed the rollout.
>
> ## Breaking
>
> - A harness program that completes cleanly after a failed model call
now produces an `ok` trace (the failure stays visible on the call
record). Previously the rollout failed with the stashed error even on a
clean exit.
>
> ## Verification
>
> - `uv run pytest -q tests/v1` — passed; live E2E tests skipped without
`PRIME_API_KEY`
> - `uv run ruff check` / `uv run ruff format --check`
>
> E2E repro against a local vLLM (`vllm serve Qwen/Qwen3-0.6B
--max-model-len 4096 --enable-auto-tool-choice --tool-call-parser
hermes`), one dummy task whose toolset returns a ~49KB payload so the
turn after the tool call exceeds the 4k window (one-off fixture, not
part of this PR):
>
> | | rollout | trace |
> | --- | --- | --- |
> | before (`main`) | failed: `ProviderError: upstream 400: This model's
maximum context length is 4096 tokens…` | `ok=false`,
`stop_condition=error` |
> | after (this PR) | completed | `ok=true`,
`stop_condition=agent_completed`, the overlong call still recorded with
its `ProviderError` on `trace.calls` |
>
> Same task against the OpenAI API (`gpt-5.6-luna`,
`reasoning_effort="none"`), payload ~1.2M tokens: the API rejects with
`400 code="context_length_exceeded": "Input tokens exceed the configured
limit of 922000 tokens…"` — the program catches it and the rollout
completes with `ok=true`, `stop_condition=agent_completed`, the failed
call recorded on `trace.calls`.
>
> Same task against OpenRouter directly (`openai/gpt-4o-mini`, payload
~500k tokens): the 400 wraps the upstream error but relays its raw body
(`"This model's maximum context length is 128000 tokens…"`,
`provider_error_code: context_length_exceeded`) — caught, rollout
completes with `ok=true`, `stop_condition=agent_completed`.
>
> Also probed `deepseek/deepseek-v4-flash` on Prime Inference with the
same task at growing payloads:
>
> - a 706k-token prompt is accepted (the run completes normally,
`ok=true`), so the window is large;
> - past the window (~1M tokens and up), the gateway rejects with a
generic `400 "Invalid request."` that carries **no context-length
markers** — the harness correctly treats it as a non-overflow provider
error, re-raises, and the rollout fails with the real `ProviderError`.
The clean-exit path cannot trigger on this provider until it relays the
upstream context-length message (this also blinds every marker-based
overflow detector: our loops, pi/prime-agent, Codex).
>
> 🤖 Generated with [Claude Code](https://claude.com/claude-code)
>
> <!-- CURSOR_SUMMARY -->
> ---
>
> > [!NOTE]
> > **Medium Risk**
> > Behavior change: clean harness exit after a model-call failure now
yields an ok trace instead of failing the rollout; marker-based overflow
detection can miss generic provider 400s without length wording.
> > 
> > **Overview**
> > **Context overflow** is now treated as a normal budget stop for the
null harness (which has no compaction): model `BadRequestError`s are
classified via provider-specific **markers** in `is_context_overflow`,
and matching failures **exit the program** instead of crashing.
> > 
> > **Rollout behavior** changes when the harness finishes successfully
after a failed model call: the driver **no longer re-raises** the
stashed `session.error`, so the trace can be **`ok`** with the failure
still on the call record. Harnesses that **raise** on the error still
fail the rollout through the existing exception path.
> > 
> > `RolloutSession.error` docs are updated to describe this split.
> > 
> > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for
commit 426b36e. Bugbot is set up for
automated code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
> <!-- /CURSOR_SUMMARY --><!-- Macroscope's changelog starts here -->
> #### Changes since #2456 opened
>
> - Updated `CONTEXT_OVERFLOW_MARKERS` constant in the null harness to
use provider-specific context overflow detection phrases [426b36e]
> - Changed exception handling from `openai.BadRequestError` to
`openai.APIStatusError` and added HTTP status code validation (400 or
413) to the `is_context_overflow` function's overflow detection logic,
which now requires both a matching context overflow marker in the error
details and one of the specified status codes [f00bf5d]
> <!-- Macroscope's changelog ends here -->
>
<!-- Macroscope's pull request summary ends here -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas added a commit that referenced this pull request Aug 31, 2026
## Summary

- add an optional `CompactionConfig` to the bash harness: compact into a
handoff summary at `summarize_at_tokens`, or when 16k tokens remain
below the model context window
- the program discovers the window itself from the provider's `/models`
card (via `models.list()`), so the interception server gains a stateless
`GET /v1/models` relay (one route serves every dialect)
- truncate a tool result over 20KB to its head and tail with a warning
naming the original size
- compact reactively on an attributed 400/413 overflow error, from the
current state; a rejected checkpoint falls back to the last state that
passed a threshold check (which holds a full reserve of room by
definition), and an empty or tool-calling reply is resampled
- after three failed checkpoint attempts the program ends the run
cleanly — still a trainable sample
- an overflow with no history beyond the task propagates: nothing to
reclaim

The RLM harness gets the same policy in #2459, stacked on this PR so
this one can merge before nano-rlm does. Together with #2453 (merged)
and #2459 this supersedes #2448.

## Breaking

- None on its own: `compaction` is a new optional field, unset by
default.

## Verification

- `uv run pytest -q tests/v1` — passed; live E2E tests skipped without
`PRIME_API_KEY`
- Tool truncation checked by exact match: a synthetic 55KB output
produces the byte-identical expected head/tail + warning header; short
outputs pass through untouched.
Terminal-Bench 2 e2e: 8 tasks, local vLLM `poolside/Laguna-XS-2.1` at
32k (glm45 reasoning + glm47 tool parsers), `compaction = {}` so the
threshold auto-discovers to `32768 − 16384 = 16384`. Trace analysis:

- 8/8 rollouts end `ok=true` with scoring run (1 solved); no harness
errors, no truncation stops.
- The threshold triggers exactly where designed: 7/8 episodes compacted
with peak contexts of 17.0k-23.4k tokens; the one episode that stayed at
15.4k never compacted.
- 20KB tool truncation visible in 3 episodes' traces (`Warning:
truncated output` with original size).
- Laguna puts its entire checkpoint reply in the reasoning channel, so
under the summaries-are-content-only rule every compaction on this model
exercises the failure path end to end: three resampled checkpoint
attempts, then the program ends the run cleanly - the rollout stays a
trainable sample. Summary carry-over itself was demonstrated on models
that answer in the content channel (Qwen3-0.6B at 4k/8k,
deepseek-v4-flash on tb2): every non-final branch ends with the
checkpoint prompt followed by the summary, and the next branch opens
with `[system, framed summary]`.
- A final combined verification run on a content-channel model is
pending.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> The shared agent loop now permanently truncates large tool outputs
and, when compaction is enabled, replaces long histories with
model-generated summaries—both can change task outcomes; the new models
relay is ancillary and failures only disable auto-thresholds.
> 
> **Overview**
> Adds an **opt-in** context compaction path for the bash agent:
`BashHarnessConfig` gains `CompactionConfig` (`summarize_at_tokens`
optional), which forwards `--compaction` / `--summarize-at-tokens` into
the shared minimal/bash chat program.
> 
> When compaction is on, the program uses a new **`Compactor`** to
compact before context blows up: it can trigger on usage crossing a
threshold (explicit token limit, or auto from the model card with a 16k
reserve), on `finish_reason=length`, on post-tool estimated token
growth, or reactively on provider 400/413 overflow. Compaction asks the
model for a plain-text handoff summary (`tool_choice=none`), rebuilds
history as system + framed summary user message, falls back to the last
“good” checkpoint on overflow, and **ends the loop cleanly** on
`CompactionFailed` instead of crashing. Threshold discovery calls
`models.list()` against the intercepted base URL.
> 
> **Tool results are always middle-truncated** at ~20KB (head/tail +
warning) via `bound_tool_message` before they enter the transcript,
including interception rewrites.
> 
> The interception server adds **`GET /v1/models`** as an authenticated
upstream relay (30s timeout) so the in-container SDK can read provider
context-window fields without recording a model turn.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
e1c80ac. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

<!-- Macroscope's pull request summary starts here -->
<!-- Macroscope will only edit the content between these invisible
markers, and the markers themselves will not be visible in the GitHub
rendered markdown. -->
<!-- If you delete either of the start / end markers from your PR's
description, Macroscope will append its summary at the bottom of the
description. -->
> [!NOTE]
> ### Add tool truncation and auto-compaction to `BashHarness` agent
loop
> - Adds `CompactionConfig` to `BashHarnessConfig` and passes
`--compaction` and `--summarize-at-tokens` flags to the spawned program
in
[harness.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-1544e2ee3426241d8c97940a3e8daf25b2eeeeb95f8abb27e88bed4af420e609)
> - Introduces the `Compactor` class in
[program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36)
that detects context exhaustion, builds a summary checkpoint, and
rebuilds the conversation; auto-discovers a token threshold from the
provider's `/v1/models` endpoint when none is given
> - Truncates large tool outputs to `TOOL_OUTPUT_MAX_BYTES` (20,000)
with head/tail preservation via `truncate_tool_output` before they enter
the conversation
> - Proxies `GET /v1/models` through the interception server in
[server.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-84e2f8d027d609e8760ef6d533535ec9d99dfc6384d5ddb6111b4001b2010f35)
so the program can query the provider for context window size
> - `chat()` now returns the full completion object instead of the first
message, and the main loop ends cleanly on `CompactionFailed`
> - Risk: `chat()` signature and return type changed in
[program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36);
any callers expecting the first message instead of the full completion
will break. Tool outputs over 20,000 bytes are silently truncated
mid-conversation.
>
> <!-- Macroscope's review summary starts here -->
>
> <sup><a href="https://app.macroscope.com">Macroscope</a> summarized
27e1c91.</sup>
> <!-- Macroscope's review summary ends here -->
>
<!-- Macroscope's pull request summary ends here -->

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants