You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A streamed /v1/responses request whose prompt is longer than the context window is not answered the way OpenAI answers it. OpenAI opens the stream and ends it with response.failed, whose response.error.code is context_length_exceeded. Clients act on that event. The Codex CLI reads it as a full context window: it ends the turn and compacts its history before the next one. It reads an HTTP 400 as an invalid request that it does not retry or recover from.
On main the server answers such a request in one of three ways, depending on where the prompt is rejected. None of them reaches the client as a full context window.
The executor runs in worker processes (the default). The rejection arrives while the stream runs. guard_responses_stream sends error and response.failed, but with error.code: "server_error", and then re-raises. The re-raise aborts the connection mid-body (an incomplete chunked read). The Codex CLI keeps the code of response.failed and acts on it only when the stream ends cleanly. On a transport error it reports a stream error and retries the same request.
The executor runs in the server process. generate_async raises at once, and the server answers HTTP 400 with code: "context_length_exceeded".
Disaggregated serving. The context phase never streams, so the context worker rejects the request with its 400. The frontend relays that 400, also for a streamed request.
This change answers all three with the same stream: response.created, response.in_progress, error, then response.failed. The failed snapshot carries error.code: "context_length_exceeded" and the rejection text as error.message. Sequence numbers run from 0 to 3, and the stream ends normally.
classify_stream_termination recognizes an over-length RequestError by its text (the text is what crosses the executor IPC boundary) and returns the new cause context_length_exceeded. describe_stream_termination reports such a rejection by its message alone, the text the non-streamed 400 carries.
ResponsesStreamingProcessor.get_stream_failed_events sets error.code to context_length_exceeded for that cause and keeps server_error for every other cause. The SDK's ResponseError.code literal does not list context_length_exceeded (checked with openai 2.48 and 3.39). So the event is validated with server_error in that slot, and the real code is written into the serialized event by the same serializer as every other frame. For every other cause the frame is byte-identical to before.
guard_responses_stream ends the stream normally after the terminal events of an over-length rejection, instead of re-raising. Every other failure is re-raised as before.
rejected_responses_stream (new) streams a request that was rejected before generation: the opening pair, then the same failure events. OpenAIServer.openai_responses uses it when generate_async itself raises the rejection for a streamed request.
OpenAIDisaggServer._handle_exception(self, exception) keeps its one-argument signature: subclasses such as NeMo-RL's OpenAIDisaggServerAdaptor override it with that signature and delegate to super(), so it is still called, unchanged, for every failure of a wrapped request. The conversion happens in the caller, _wrap_entry_point: when the handler returns the context worker's 4xx context_length_exceeded envelope for a streamed Responses request, the route answers with the failure stream instead of that 400. An override that answers the failure itself (raises, or returns anything else) keeps its answer. The snapshot names the model the client asked for, because the frontend does not know the served name.
Unchanged:
A non-streamed request keeps HTTP 400 with code: "context_length_exceeded", which is how OpenAI rejects a non-streamed over-length request.
Chat Completions and Completions keep their current behavior, streamed or not.
A prompt longer than max_num_tokens with chunked prefill disabled is a different rejection ("The prompt length ... should not exceed max_num_tokens"). It keeps its current form, as on the 400 path.
This was found while serving DeepSeek-V4.1-Flash to the Codex CLI. The change does not depend on the model.
Test Coverage
CPU-only tests, in files that l0_cpu.yml already lists:
test_a_streamed_responses_request_fails_in_stream[in_worker|in_process] drives OpenAIServer.openai_responses with an executor that rejects the prompt in a worker (the rejection travels as an ErrorResponse through the real GenerationResult and error handler) or in-process (generate_async raises). It checks the four events, their numbering, the response id, the code and message, and that the body ends without raising.
test_a_non_streamed_responses_request_keeps_the_400[...] and test_other_rejections_of_a_streamed_request_are_unchanged[...].
test_disagg_route_fails_a_streamed_responses_request_in_stream and test_disagg_route_keeps_the_400_for_other_requests[...] (both drive _wrap_entry_point).
test_a_subclass_handler_is_called_for_an_ordinary_failure[worker_400|internal_error], test_a_subclass_handler_is_called_for_a_streamed_over_length_request, test_a_subclass_that_answers_the_over_length_failure_itself_keeps_its_answer and test_a_subclass_that_returns_its_own_answer_keeps_it[...]: a subclass overriding _handle_exception(self, exception) with one argument, as NeMo-RL does, is called for every failure, the streamed over-length request still fails in-stream, and an override's own answer is kept. With the earlier two-argument call these fail with TypeError.
the cause and detail of an over-length RequestError, and of a ValueError with the same text, which stays internal_error;
the failed code for each cause, and the frame of every other cause serialized like any other frame;
the guard ends an over-length stream normally and still re-raises other engine failures;
a rejection before generation streams the same frames as one raised while the stream runs.
On a build of current main, these two files and the two disaggregated Responses test files (tests/unittest/disaggregated/test_openai_server_stream_errors.py, tests/unittest/disaggregated/test_disagg_responses_api.py) pass: 147 tests.
Also checked outside the test suite: generated-input properties over the same functions (any chunking and number of frames already sent, wrapped rejection texts, request fields with floats and non-ASCII text), with planted bugs to confirm the checks can fail. An HTTP run under uvicorn with a reader that follows the Codex CLI's stream handling reports a full context window for both in-worker and in-process rejections. On main the same reader reports a stream error (incomplete chunked read) and an HTTP 400.
PR Checklist
PR description clearly explains what and why.
GitHub Bot Help
To see a list of available CI bot commands, please comment /bot help.
Streamed /v1/responses requests rejected for context length now produce a response.failed event with error.code: "context_length_exceeded" and end normally. The change covers rejection before generation, during streaming, and in disaggregated serving. Other streamed failures retain their existing handling. Non-streamed over-length requests retain HTTP 400.
No current review findings or severity counts were supplied. The author reports that 113 CPU-only tests passed across the specified Responses test files, compared with 91 on main. The author also reports generated-input checks and an HTTP run; these were outside the test suite and were not independently verified. The supplied CI comments report mixed pipeline outcomes, including failures and unstable runs.
QA Engineer Review
Two test files were modified. test_context_length_error.py covers streamed and non-streamed requests, in-process and worker rejection paths, and disaggregated routing. test_responses_streaming_events.py covers error classification, event serialization, and stream termination. Both files are listed in tests/integration/test_lists/test-db/l0_cpu.yml. Neither appears in the searched manual-QA lists. Coverage verdict: sufficient.
Per-File QA Perspective
tensorrt_llm/serve/openai_disagg_server.py — Verify that a streamed Responses request rejected by the disaggregated context worker receives an in-stream failure. Verify that other failures retain their existing handling.
tensorrt_llm/serve/openai_server.py — Verify that a pre-generation context-length rejection returns a failure stream for streamed Responses requests. Verify that non-streamed requests and other errors retain their existing handling.
tensorrt_llm/serve/responses_utils.py — Verify the context_length_exceeded code, terminal event sequence, and normal stream termination. Verify that other stream failures retain their existing classification and propagation.
tests/unittest/llmapi/apps/test_context_length_error.py — Covers in-process, worker, and disaggregated rejection paths, non-streamed behavior, and other rejection causes. Listed in tests/integration/test_lists/test-db/l0_cpu.yml; no entry appeared in the searched manual-QA lists.
tests/unittest/llmapi/apps/test_responses_streaming_events.py — Covers error classification, event serialization, stream termination, and propagation of other failures. Listed in tests/integration/test_lists/test-db/l0_cpu.yml; no entry appeared in the searched manual-QA lists.
…am with context_length_exceeded
A streamed /v1/responses request whose prompt exceeds the context window
now gets the stream OpenAI sends for it: response.created,
response.in_progress, error, then response.failed with error.code
context_length_exceeded, after which the stream ends normally. Clients
act on that event: the Codex CLI compacts its history on it, while it
treats an HTTP 400 as an invalid request and an aborted stream as a
transport error that it retries.
Before, the answer depended on where the executor rejected the prompt:
- in a worker process (raised while the stream runs): response.failed
carried server_error, and the re-raise then aborted the connection;
- in this process (raised by generate_async): HTTP 400;
- disaggregated serving (the context worker's 400): HTTP 400.
Non-streamed requests keep HTTP 400 with code context_length_exceeded.
Every other stream failure keeps server_error and the re-raise; its
response.failed frame is byte-identical to before.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.
No actionable comments were generated in the recent review.
Walkthrough
Streamed Responses requests with context-length errors now receive an in-stream response.failed event. The standard and disaggregated server paths use rejection-stream helpers. Other errors and non-streamed requests retain their existing handling.
Changes
Responses stream rejection handling
Layer / File(s)
Summary
Classify and serialize stream failures tensorrt_llm/serve/responses_utils.py, tests/unittest/llmapi/apps/test_responses_streaming_events.py
The stream utilities classify context-length errors and emit a response.failed event with the context_length_exceeded code. The stream ends normally for this rejection. Other failures retain their existing behavior. Tests cover classification, serialization, and stream termination.
Standard server handling tensorrt_llm/serve/openai_server.py, tests/unittest/llmapi/apps/test_context_length_error.py
The standard Responses entry point returns a rejected event stream for streamed context-length errors. Tests cover streamed and non-streamed requests and other rejection behavior.
Disaggregated server handling tensorrt_llm/serve/openai_disagg_server.py, tests/unittest/llmapi/apps/test_context_length_error.py
The disaggregated route converts recognized context-length error responses into rejected event streams for streamed Responses requests. Tests cover the route and exception-handler response handling.
No merge-blocking behavior has been established from the available evidence.
Pre-merge checks: 4 · 1
Failed checks: 1 warning
Check name
Status
Explanation
Resolution
Docstring Coverage
Docstring coverage is 61.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…
Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed checks: 4 passed
Check name
Status
Explanation
Title check
The title follows the repository template: a [None] ticket tag, a [fix] type, and a concise summary. It names the main change, which makes streamed Responses requests that exceed the context window fa…
Description check
The description follows the repository template. It explains the problem and the fix, gives the full behavior for the three rejection paths, and lists the unchanged behavior. It also gives a detailed …
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title follows the repository template: a [None] ticket tag, a [fix] type, and a concise summary. It names the main change, which makes streamed Responses requests that exceed the context window fail in-stream with the context_length_exceeded code. The changeset matches this: it classifies over-length RequestErrors, assigns the code in the failed event, and ends the stream normally.
Full details: Description check
Explanation
The description follows the repository template. It explains the problem and the fix, gives the full behavior for the three rejection paths, and lists the unchanged behavior. It also gives a detailed test coverage section and a checked PR checklist item. The template's 'Description' and 'Test Coverage' sections are present and filled in. The checklist contains only the first item, so the other template items are not addressed. These gaps are non-critical.
Full details: Docstring Coverage
Explanation
Docstring coverage is 61.11% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 18 functions across 5 files.
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
…xception handler for subclasses
OpenAIDisaggServer._handle_exception(self, exception) keeps the signature
it has on main. Subclasses override it with that signature (NeMo-RL's
OpenAIDisaggServerAdaptor), so the extra request argument would have made
every failed request raise TypeError there.
The in-stream answer for a streamed Responses request that the context
worker rejected for its length moves to the caller, _wrap_entry_point.
The handler, base or override, rules on the failure first, as it does for
every failure; only when it returns the worker's context_length_exceeded
envelope is that envelope re-shaped into the stream. An override that
answers the failure itself keeps its answer.
Tests drive the wrapped route instead of the handler, and a subclass
with a one-argument handler is checked for an ordinary failure and for a
streamed over-length Responses request.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
JunyiXu-nv
added a commit
to JunyiXu-nv/TensorRT-LLM
that referenced
this pull request
Nov 4, 2026
…h a verdict
A run of the skill on a clean machine, from the skill alone, could not
reach a passing cell. This adds what it missed:
- The Codex default profile's echo is measured (exact: codex-cli 0.155.1,
12 sessions against the fake server). Conformance was BLOCKED on it. The
contract template now asks for echo.
- The capture changes are named by PR (NVIDIA#21605, NVIDIA#21602, NVIDIA#20936), with where to
get a wheel that matches the base and which CPU suites to run.
- The wiring snippets use absolute paths, create their task, pass --build-id
and --matrix, and state which claims a wiring run must pass.
- The real run has a full serve command with the recorder in front, and
profiles/model/serving/ holds complete configs for GLM-5.3-NVFP4 and
DeepSeek-V4.1-Flash as their cells were served.
- The matrix template proposes starting thresholds, lists the units each
threshold needs, drops keys the verdict never reads, and names cell
directories.
- templates/full-tier-report.md: the full tier's steps and the five files it
writes.
- Smaller additions: restricting egress, a harness's own sandbox inside a
container, Codex over-length on main (PR NVIDIA#21060), where the GLM goldens
came from, test and gate hygiene, and the controls receipt command.
Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A streamed
/v1/responsesrequest whose prompt is longer than the context window is not answered the way OpenAI answers it. OpenAI opens the stream and ends it withresponse.failed, whoseresponse.error.codeiscontext_length_exceeded. Clients act on that event. The Codex CLI reads it as a full context window: it ends the turn and compacts its history before the next one. It reads an HTTP 400 as an invalid request that it does not retry or recover from.On main the server answers such a request in one of three ways, depending on where the prompt is rejected. None of them reaches the client as a full context window.
guard_responses_streamsendserrorandresponse.failed, but witherror.code: "server_error", and then re-raises. The re-raise aborts the connection mid-body (an incomplete chunked read). The Codex CLI keeps the code ofresponse.failedand acts on it only when the stream ends cleanly. On a transport error it reports a stream error and retries the same request.generate_asyncraises at once, and the server answers HTTP 400 withcode: "context_length_exceeded".This change answers all three with the same stream:
response.created,response.in_progress,error, thenresponse.failed. The failed snapshot carrieserror.code: "context_length_exceeded"and the rejection text aserror.message. Sequence numbers run from 0 to 3, and the stream ends normally.classify_stream_terminationrecognizes an over-lengthRequestErrorby its text (the text is what crosses the executor IPC boundary) and returns the new causecontext_length_exceeded.describe_stream_terminationreports such a rejection by its message alone, the text the non-streamed 400 carries.ResponsesStreamingProcessor.get_stream_failed_eventssetserror.codetocontext_length_exceededfor that cause and keepsserver_errorfor every other cause. The SDK'sResponseError.codeliteral does not listcontext_length_exceeded(checked with openai 2.48 and 3.39). So the event is validated withserver_errorin that slot, and the real code is written into the serialized event by the same serializer as every other frame. For every other cause the frame is byte-identical to before.guard_responses_streamends the stream normally after the terminal events of an over-length rejection, instead of re-raising. Every other failure is re-raised as before.rejected_responses_stream(new) streams a request that was rejected before generation: the opening pair, then the same failure events.OpenAIServer.openai_responsesuses it whengenerate_asyncitself raises the rejection for a streamed request.OpenAIDisaggServer._handle_exception(self, exception)keeps its one-argument signature: subclasses such as NeMo-RL'sOpenAIDisaggServerAdaptoroverride it with that signature and delegate tosuper(), so it is still called, unchanged, for every failure of a wrapped request. The conversion happens in the caller,_wrap_entry_point: when the handler returns the context worker's 4xxcontext_length_exceededenvelope for a streamed Responses request, the route answers with the failure stream instead of that 400. An override that answers the failure itself (raises, or returns anything else) keeps its answer. The snapshot names the model the client asked for, because the frontend does not know the served name.Unchanged:
code: "context_length_exceeded", which is how OpenAI rejects a non-streamed over-length request.max_num_tokenswith chunked prefill disabled is a different rejection ("The prompt length ... should not exceed max_num_tokens"). It keeps its current form, as on the 400 path.This was found while serving DeepSeek-V4.1-Flash to the Codex CLI. The change does not depend on the model.
Test Coverage
CPU-only tests, in files that
l0_cpu.ymlalready lists:tests/unittest/llmapi/apps/test_context_length_error.pytest_a_streamed_responses_request_fails_in_stream[in_worker|in_process]drivesOpenAIServer.openai_responseswith an executor that rejects the prompt in a worker (the rejection travels as anErrorResponsethrough the realGenerationResultand error handler) or in-process (generate_asyncraises). It checks the four events, their numbering, the response id, the code and message, and that the body ends without raising.test_a_non_streamed_responses_request_keeps_the_400[...]andtest_other_rejections_of_a_streamed_request_are_unchanged[...].test_disagg_route_fails_a_streamed_responses_request_in_streamandtest_disagg_route_keeps_the_400_for_other_requests[...](both drive_wrap_entry_point).test_a_subclass_handler_is_called_for_an_ordinary_failure[worker_400|internal_error],test_a_subclass_handler_is_called_for_a_streamed_over_length_request,test_a_subclass_that_answers_the_over_length_failure_itself_keeps_its_answerandtest_a_subclass_that_returns_its_own_answer_keeps_it[...]: a subclass overriding_handle_exception(self, exception)with one argument, as NeMo-RL does, is called for every failure, the streamed over-length request still fails in-stream, and an override's own answer is kept. With the earlier two-argument call these fail withTypeError.tests/unittest/llmapi/apps/test_responses_streaming_events.pyRequestError, and of aValueErrorwith the same text, which staysinternal_error;On a build of current main, these two files and the two disaggregated Responses test files (
tests/unittest/disaggregated/test_openai_server_stream_errors.py,tests/unittest/disaggregated/test_disagg_responses_api.py) pass: 147 tests.Also checked outside the test suite: generated-input properties over the same functions (any chunking and number of frames already sent, wrapped rejection texts, request fields with floats and non-ASCII text), with planted bugs to confirm the checks can fail. An HTTP run under uvicorn with a reader that follows the Codex CLI's stream handling reports a full context window for both in-worker and in-process rejections. On main the same reader reports a stream error (incomplete chunked read) and an HTTP 400.
PR Checklist
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.🤖 Generated with Claude Code
Dev Engineer Review
Streamed
/v1/responsesrequests rejected for context length now produce aresponse.failedevent witherror.code: "context_length_exceeded"and end normally. The change covers rejection before generation, during streaming, and in disaggregated serving. Other streamed failures retain their existing handling. Non-streamed over-length requests retain HTTP 400.No current review findings or severity counts were supplied. The author reports that 113 CPU-only tests passed across the specified Responses test files, compared with 91 on main. The author also reports generated-input checks and an HTTP run; these were outside the test suite and were not independently verified. The supplied CI comments report mixed pipeline outcomes, including failures and unstable runs.
QA Engineer Review
Two test files were modified.
test_context_length_error.pycovers streamed and non-streamed requests, in-process and worker rejection paths, and disaggregated routing.test_responses_streaming_events.pycovers error classification, event serialization, and stream termination. Both files are listed intests/integration/test_lists/test-db/l0_cpu.yml. Neither appears in the searched manual-QA lists. Coverage verdict: sufficient.Per-File QA Perspective
tensorrt_llm/serve/openai_disagg_server.py— Verify that a streamed Responses request rejected by the disaggregated context worker receives an in-stream failure. Verify that other failures retain their existing handling.tensorrt_llm/serve/openai_server.py— Verify that a pre-generation context-length rejection returns a failure stream for streamed Responses requests. Verify that non-streamed requests and other errors retain their existing handling.tensorrt_llm/serve/responses_utils.py— Verify thecontext_length_exceededcode, terminal event sequence, and normal stream termination. Verify that other stream failures retain their existing classification and propagation.tests/unittest/llmapi/apps/test_context_length_error.py— Covers in-process, worker, and disaggregated rejection paths, non-streamed behavior, and other rejection causes. Listed intests/integration/test_lists/test-db/l0_cpu.yml; no entry appeared in the searched manual-QA lists.tests/unittest/llmapi/apps/test_responses_streaming_events.py— Covers error classification, event serialization, stream termination, and propagation of other failures. Listed intests/integration/test_lists/test-db/l0_cpu.yml; no entry appeared in the searched manual-QA lists.