[glamr-agent]

Closes #17010

Overview

Concurrent requests with the same client ID can overwrite the frontend’s in-flight state and fail or receive another caller’s response. This fix gives each HTTP and gRPC request a server-generated internal ID while preserving client IDs for correlation and responses.

Failure behavior and recorded reproduction

Two concurrent requests that send the same client request id break each other. On /inference/v1/generate, two requests with x-request-id: 1 both fail with HTTP 500, and the frontend logs Unlinked response; response task was dropped or cancelled or Unlinked request; initiating request task was dropped or cancelled. A caller can also receive the other caller's stream.

The frontend tracks each in-flight request by its id, so it can route the response back to the caller. On the OpenAI, Anthropic and Responses routes, that id came from the x-dynamo-request-id or request-id header. On /inference/v1/generate and SGLang /generate, it came from x-request-id or the body request_id / rid. A second request with the same id silently replaced the first request's entry, so one response lost its caller. The defect is in the Dynamo frontend.

Reproduction	Without the fix	With the fix	Note
Exact failure on the reported setup	❌ Exact failure	✅ Passed	Live frontend with one vLLM worker on one A100, concurrent request pairs sharing a client id. Without the fix, 0 of 20 /inference/v1/generate requests sharing x-request-id: 1 succeeded, with HTTP 500 and the detached-stream errors. With the fix, 60 of 60 concurrent requests on the generate and chat routes returned 200 with their own output, in each of two runs.
Exact error in a test	❌ Exact error	✅ Passed	Rust route tests on CPU with an echo engine behind a real frontend pipeline. Without the fix, the concurrent route tests got HTTP 500, and the chat route logged the detached-stream receiver error. With the fix, all of them passed.

Key Changes

Core logic: approximately 90 added lines, excluding tests, documentation, comments, and whitespace.

• base.rs refuses duplicate in-flight IDs and releases entries when callers are dropped.
• logging.rs and HTTP/gRPC handlers generate internal IDs while native generate routes keep client IDs in responses and x_request_id trace fields.
• push_handler.rs preserves duplicate refusal as Conflict, allowing migration without marking a healthy worker down.

Request ID behavior and migration scope

The frontend now generates its own internal request id for every HTTP and gRPC request and never takes it from the client. Client ids are correlation values only: x-request-id is still logged and echoed, and the two native generate routes still return the client's request_id or rid in the response. The frontend also refuses a second in-flight request with an id it already holds, instead of replacing the first one, and frees the id when a caller goes away.

Case	Before	After
Two concurrent generate requests share x-request-id	HTTP 500 or crossed streams	Each gets its own output
Two concurrent chat requests share x-dynamo-request-id	HTTP 500	Each gets its own output
Client sends x-dynamo-request-id or request-id	Header became the internal id	Header ignored, one deprecation WARN
Single request with a unique x-request-id	Works	Unchanged

If a migration retry reaches the same worker while the first attempt is still waiting for its engine to create a response stream, the worker refuses the retry's duplicate request id. The refusal is typed Conflict, so the frontend retries the request on another worker and does not mark the refusing worker as down. The reinforcement learning operations page now describes client ids as correlation values, except the rid of native SGLang session requests, which SGLang uses as the session key.

The gRPC completion and tensor handlers also generate internal request ids. KServe still copies the client’s proto request id into unary and streaming responses independently of the internal context id.

Start reviewing at lib/runtime/src/pipeline/nodes/sources/base.rs.

Validation

This PR adds and updates tests in lib/runtime/tests/pipeline.rs, lib/runtime/src/logging.rs, lib/runtime/src/pipeline/network/ingress/push_handler.rs, lib/runtime/src/pipeline/network/egress/push_router.rs, lib/llm/src/http/service/generate.rs, lib/llm/src/http/service/sglang_generate.rs, lib/llm/tests/http-service.rs, lib/llm/tests/kserve_service.rs, lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs and tests/frontend/test_request_tracing_logs.py. The Rust tests run without a GPU, against echo engines or the mocker behind a real frontend pipeline. The Python test is an end-to-end run of the frontend with a mocker worker, also without a GPU.

Behavior checked	Checked by	Without the fix	With the fix
A second in-flight request with the same id is refused, and the first caller still gets its own output	New unit test	❌ Failed: the duplicate request hung until the 5 s timeout	✅ Passed
A caller that is dropped while its request is in flight frees its id, so the same id is accepted again	New unit test; it fails with the previous version of this PR's base.rs	❌ Failed: the retry was refused as a duplicate	✅ Passed
HTTP request spans mint a fresh id and ignore request-id and x-dynamo-request-id	New unit test	❌ Failed: the span id was the client's UUID	✅ Passed
Two concurrent /inference/v1/generate requests sharing x-request-id each get their own tokens, and the body echoes the client id	New unit test; also the reproduction above	❌ Exact error: HTTP 500 instead of 200	✅ Passed
/inference/v1/generate mints the internal id and keeps vLLM's x-request-id before body request_id precedence for the public id	One new and two changed unit tests; before this PR the changed tests expected the client value as the internal id	❌ Failed: the internal id was the client value	✅ Passed
Two concurrent SGLang /generate requests sharing rid each get their own output, and the worker envelope keeps the client rid	New unit test	❌ Exact error: HTTP 500 instead of 200	✅ Passed
SGLang /generate keeps client ids as correlation only	New unit test	❌ Failed: the internal id was not a minted UUID	✅ Passed
Concurrent chat, Responses and Anthropic requests sharing x-dynamo-request-id and request-id each get their own content	New integration test; also the reproduction above	❌ Exact error: HTTP 500 with the detached-stream receiver error	✅ Passed
The request_id annotation is a fresh UUID, not the client's x-dynamo-request-id	Changed integration test; before this PR it expected the header value echoed	❌ Failed: the annotation was the header value	✅ Passed
An ignored x-dynamo-request-id logs one deprecation WARN with a header field, and the logged request id is a fresh UUID	Changed end-to-end test, frontend with a mocker worker; before this PR it expected a must be a valid UUID WARN	❌ Failed: no WARN with the new text	✅ Passed
The sidecar testkit binds an injected fault to the next HTTP request without a client id	One changed integration test, run for the vLLM, SGLang and TensorRT-LLM sidecars	❌ Failed 3 of 3 runs: got 200 instead of the injected 400 with the old testkit	✅ Passed

The existing KServe tests test_infer_success and test_tensor_infer now submit concurrent requests with the same valid client UUID. Their engines check that the internal context id is a UUID different from the client id, and the tests check that each response still echoes the client id.

The tables were recorded on commit 453394da18. Later commits change how a worker's duplicate-id refusal is typed, add one unit test for it, and edit comments and docs. Locally, those changes were checked only by cargo test -p dynamo-runtime --lib on the new duplicate_request_id_is_a_conflict test and the changed pre_stream_invalid_request_does_not_quarantine_healthy_worker test (both pass) and by cargo clippy -p dynamo-runtime --all-targets -- -D warnings; everything else was checked only by CI.

The base merge keeps main’s tracing interest guard in generate_dispatch_span_uses_resolved_request_id and retains its separate internal and client ID assertions. The two runtime pipeline regression tests now bound their backend-arrival waits to five seconds so an early dispatch failure cannot hang them.

Issues and Ongoing Work

• Closes [BUG]: Concurrent requests with the same client-supplied request id interfere with one another #17010. The issue describes client-controlled IDs overwriting frontend in-flight state.
• feat(frontend): align response ids with Dynamo request ids #13349 aligns response body IDs with existing context IDs; this PR changes how ingress creates those context IDs.
• fix(frontend): emit request_end trace records on native generate endpoints #17036 adds request-end records on native generate routes; this PR preserves client correlation while changing internal identity.
• docs(observability): document the request identifier model and correlation rules #17451 documents the identifier model after this PR merges and depends on it.
• The frontend/runtime/observability review accepted vLLM engine ID reuse and gRPC ingress span/correlation headers as separate follow-ups. Those paths are not changed by the base merge.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill. The review used separate design/performance, validation/style, and adversarial reviews. This maintenance pass used static inspection and remote CI; it did not run local tests, builds, or benchmarks.

Summary by CodeRabbit

• New Features

  • Server-generated request IDs now provide unique tracking for each request, including concurrent requests sharing a client correlation ID.
  • Duplicate in-flight request IDs are rejected, while IDs can be reused after completion or cancellation.
  • Request ID responses are standardized across native SGLang and inference endpoints.

• Bug Fixes

  • Deprecated request ID headers are ignored while client correlation remains available through supported headers.

• Documentation

  • Clarified request ID behavior and endpoint-specific response fields.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a18007-b705-74bd-b97a-e59952660fc3.docs.buildwithfern.com/dynamo/dev

[glamr-agent]

factory: Initial validation snapshot.

Validation status: complete

Check	Result	Scope or reason
Checks the changed Rust crates with cargo check and Clippy.	Passed (exit 0)	Recorded command: cargo test -p dynamo-runtime --test pipeline frontend_. Tested revision 453394da18.
Starts Dynamo with vLLM on one GPU and sends a real request.	Passed (exit 0)	One A100, vLLM, Qwen/Qwen3-0.6B with dummy weights. Concurrent same-id requests: 0 of 20 succeeded without the fix, 60 of 60 with it, in each of two runs. Tested revision 453394da18.

The SGLang /generate route was checked only with in-process Rust tests, because no SGLang engine was available. The four test_disagg_* end-to-end tests were not run locally.

Detailed evidence — validation complete

Left out of the table because CI repeats them: building the changed source and importing its compiled extension (Check 1), lint and formatting with pre-commit (Check 2).

Evidence tally: [4/4 validated]

Advisory AI assessment: sound. This is an AI judgment, not an approval.

Validation result: pass — on 453394da18 the build, Clippy, lint and all planned Rust tests pass; the new dropped-caller test fails with the previous base.rs and passes with the new one; the test that passed without the fix is gone; the changed WARN test fails on main and passes on the new head; and on one A100 with vLLM, 0 of 20 concurrent same-id requests succeeded without the fix and 60 of 60 with it, in each of two runs.

Evidence audit: complete [4/4 validated] — the command report comes from recorded runs.

The head 49736a8da2 adds only comment edits on top of the tested 453394da18.

Commands and results [4/4 validated]

Generated from the commands recorded during this run.

Check 1

Builds the changed Dynamo source and confirms that Python can import its compiled extension.

Result: Passed (exit 0)

Command:

Not shown because the exact command contained private run data.

Check 2

Checks the changed files with the repository's fast lint and formatting commands.

Result: Passed (exit 0)

Command:

pre-commit run --files docs/fern/pages/use-cases/reinforcement-learning/operations-and-simulation.md lib/llm/src/http/service/generate.rs lib/llm/src/http/service/openai.rs lib/llm/src/http/service/sglang_generate.rs lib/llm/tests/http-service.rs lib/runtime/src/logging.rs lib/runtime/src/pipeline/error.rs lib/runtime/src/pipeline/nodes/sources/base.rs lib/runtime/tests/pipeline.rs lib/sidecar/testkit/src/control.rs lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs lib/sidecar/testkit/tests/support/process.rs tests/frontend/test_request_tracing_logs.py

Check 3

Checks the changed Rust crates with cargo check and Clippy.

Result: Passed (exit 0)

Command:

cargo test -p dynamo-runtime --test pipeline frontend_

Check 4

Starts Dynamo with vLLM on one GPU and sends a real request.

Result: Passed (exit 0)

Command:

/scratch/wi17010-r2/live/live.sh fixed

[glamr-agent]

No description provided.

[coderabbitai]

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 8e0e7233-a12a-45c9-836e-3ab7f604110b

📥 Commits

Reviewing files that changed from the base of the PR and between ebfcce7 and 49736a8.

📒 Files selected for processing: 13

• docs/fern/pages/use-cases/reinforcement-learning/operations-and-simulation.md
• lib/llm/src/http/service/generate.rs
• lib/llm/src/http/service/openai.rs
• lib/llm/src/http/service/sglang_generate.rs
• lib/llm/tests/http-service.rs
• lib/runtime/src/logging.rs
• lib/runtime/src/pipeline/error.rs
• lib/runtime/src/pipeline/nodes/sources/base.rs
• lib/runtime/tests/pipeline.rs
• lib/sidecar/testkit/src/control.rs
• lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs
• lib/sidecar/testkit/tests/support/process.rs
• tests/frontend/test_request_tracing_logs.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

HTTP inference routes now use server-generated internal request IDs and keep client-supplied IDs for correlation and worker envelopes. The pipeline rejects duplicate in-flight IDs and cleans up pending entries. Sidecar test helpers support requests without explicit request IDs.

Changes

Request identity and lifecycle

Layer / File(s)	Summary
Server-generated ingress IDs and correlation lib/runtime/src/logging.rs, lib/llm/src/http/service/openai.rs, lib/llm/src/http/service/generate.rs, lib/llm/src/http/service/sglang_generate.rs, lib/llm/tests/http-service.rs, tests/frontend/test_request_tracing_logs.py, docs/fern/pages/use-cases/reinforcement-learning/operations-and-simulation.md	HTTP request spans and request contexts use server-generated internal IDs. Deprecated request-id and x-dynamo-request-id headers are ignored. Tests and documentation describe client IDs as correlation values and document endpoint response-ID behavior.
Public IDs in worker envelopes and responses lib/llm/src/http/service/generate.rs, lib/llm/src/http/service/sglang_generate.rs	Generate and SGLang handlers pass public request IDs to worker envelopes and return public IDs in responses. Concurrent-request tests check that shared client IDs retain distinct internal contexts and independent outputs.
Duplicate in-flight request protection lib/runtime/src/pipeline/error.rs, lib/runtime/src/pipeline/nodes/sources/base.rs, lib/runtime/tests/pipeline.rs	Frontend.generate rejects duplicate in-flight request IDs. A guard removes pending entries when delivery fails or a caller is dropped. Tests cover rejection and later ID reuse.
Sidecar test requests without supplied IDs lib/sidecar/testkit/src/control.rs, lib/sidecar/testkit/tests/router_sidecar_mocker_integration.rs, lib/sidecar/testkit/tests/support/process.rs	Controller queues plans for requests with unknown IDs and consumes them in FIFO order. Sidecar tests use this queue and call Frontend.generate without an ID header.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 49736

HTTP routes now use server-generated internal request IDs, so concurrent requests sharing a client ID no longer collide. No concrete merge-blocking risk was found in the reviewed changes.

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		Issue #17010 requires server-minted internal IDs for affected HTTP routes, client IDs used only for correlation, duplicate in-flight requests rejected, caller cancellation cleanup, and regression test…
Out of Scope Changes check		The documentation, runtime lifecycle changes, route tests, integration tests, and sidecar testkit updates support the request-ID fix or its validation. No unrelated production behavior is demonstrated…
Title check		The title clearly identifies the main change: server-minted internal request IDs. It accurately covers the HTTP scope, although the gRPC reference may overstate the changes because the stated objectiv…
Description check		The description is thorough and follows the template's main sections: Overview, Key Changes, Validation, Issues and Ongoing Work, and the PR checklist. It names the linked issue (Closes #17010), descr…

Full details: Linked Issues check

Explanation

Issue #17010 requires server-minted internal IDs for affected HTTP routes, client IDs used only for correlation, duplicate in-flight requests rejected, caller cancellation cleanup, and regression tests. The reviewed changes implement these behaviors in logging.rs, generate.rs, sglang_generate.rs, openai.rs, base.rs, and the related tests. The documented gRPC exclusion matches the issue scope.

Full details: Out of Scope Changes check

Explanation

The documentation, runtime lifecycle changes, route tests, integration tests, and sidecar testkit updates support the request-ID fix or its validation. No unrelated production behavior is demonstrated. The changes do not extend the fix to the explicitly excluded gRPC routes.

Full details: Title check

Explanation

The title clearly identifies the main change: server-minted internal request IDs. It accurately covers the HTTP scope, although the gRPC reference may overstate the changes because the stated objectives keep gRPC routes out of scope.

Full details: Description check

Explanation

The description is thorough and follows the template's main sections: Overview, Key Changes, Validation, Issues and Ongoing Work, and the PR checklist. It names the linked issue (Closes #17010), describes the defect and fix, lists the changed files, and reports test results with and without the fix. Minor gaps: the template's Overview and Key Changes sections are expanded with long tables and HTML details blocks, and the Key Changes list omits several changed files named in the summary (for example generate.rs, openai.rs, sglang_generate.rs, and the sidecar testkit). The checklist item is checked. These gaps are non-critical, so the description is mostly complete.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[glamr-agent]

factory: @dynamo-ops please run full CI for 8081718

[dynamo-ops]

/ok to test 8081718

[glamr-agent]

factory: @dynamo-ops please run full CI for bb563a5

[dynamo-ops]

/ok to test bb563a5

[glamr-agent]

nursery: @dynamo-ops please run full CI for bd3bd12

[dynamo-ops]

/ok to test bd3bd12

[glamr-agent]

nursery: @dynamo-ops please run full CI for a2fb5bf

[dynamo-ops]

/ok to test a2fb5bf

[glamr-agent]

nursery: @dynamo-ops please run full CI for 971ce6b

[dynamo-ops]

/ok to test 971ce6b

[glamr-agent]

nursery: @dynamo-ops please run full CI for fe4eea1

[dynamo-ops]

/ok to test fe4eea1

[glamr-agent]

nursery: @dynamo-ops please run full CI for 060b6a7

[dynamo-ops]

/ok to test 060b6a7

[nnshah1]

Approving for frontend, runtime and observability. Verified on head 060b6a7: every route now mints its own id and a duplicate key is refused instead of replacing the in-flight entry; migration still re-dispatches through the router and a worker-side Conflict stays migratable; the dispatch spans keep x_request_id so client correlation is unchanged. All required checks green, no open threads. Still needs a docs codeowner for the RL page change. Follow-ups tracked separately: vLLM reusing the Dynamo id as its engine id, and gRPC ingress span/correlation header.

— Neelay + 🤖

[glamr-agent]

nursery: @dynamo-ops please run full CI for af4f31c

[dynamo-ops]

/ok to test af4f31c

[alec-flowers]

Feedback for GLAMR
It felt like there were maybe a few too many tests and also a few too many files touched. I imagine this could have been kept smaller and simpler.