[Aphoh]

Based directly on main (after #16983 merged). It does not depend on #16977 or #16978.

Split head b93c7e26fe: 368 trace/discovery/generate CPU tests, workspace all-targets check and strict Clippy pass. The GPU and campaign results below were measured on the earlier stacked branch that also contained #16977 and #16978.

Summary

Record request-end traces for every registered /generate format in the shared HTTP handler. This includes native SGLang, native vLLM, and external formats such as the zstd example, without adding tracing methods to the plugin API.

Adapt the tracing work in #16641 to the shared handler. Reuse existing request/session context and trace lifecycle; count logical output tokens across P/D and migration, preserve backend finish metadata, and avoid treating prefill completion as final request completion. Opaque vLLM multimodal payloads remain excluded from token-only replay even when routing falls back. The hint is frontend-local and does not change the worker wire protocol. When the model has a reasoning or tool-call parser and the frontend has its tokenizer, the request-end trace task decodes the finished output once, off the response path, and records reasoning_tokens, parsed tool-call names (no IDs or arguments), and finish_reason=tool_calls on a natural stop. The client does nothing extra; non-agent /generate rows now carry backend finish metadata too.

Part of #16640. Related: #16641, #16675.

Validation

Request-end parsing (33a0dd55f7): 90 request-trace and 260 discovery CPU tests, strict all-target Clippy, formatting, docs lint and changed-file pre-commit pass. Not yet run on GPU.

Current validation (2026-10-23): merged upstream fc9b9969cb, preserving live request/token accounting and SGLang HTTP connection retirement. Base head f3cf552c6f passes 326 CPU tests independently. Full-stack head 1d59b498a9 passes 374 CPU tests: 334 backend/sidecar/HTTP tests, 24 focused generate tests, 11 tracing integration tests, and five external codec tests. Strict all-target Clippy with Rust 1.96.1, formatting, ownership checks, and changed-file pre-commit pass. GPU tests were not rerun for this merge or the review fixes.

Route collision tests no longer create duplicate process-wide paths. Multimodal fallback fixtures assert their exact rejection causes. The vLLM worker-metadata fixture now enters with the migration PR that adds its field, so the base builds independently.

Live vLLM qualification (2026-10-22)

Fresh complete-stack head 5e09cf5333, release frontend and dev-profile Rust sidecar, vLLM 0.31.0, and PrimeIntellect/qwen3-moe-tiny on a B200: 156/156 checks passed (52 aggregated, 52 NIXL P/D, 52 processed-logprob/sampling-mask checks). Each configuration includes session/no-session trace probes with exact input/output counts and replay context; all 273 recorded request IDs are unique within their runs. All 48 disconnected requests received acknowledged engine Abort RPCs. The captured-wire audit verifies 91 P/D handoffs.

Native parity now checks full prompt-logprob JSON, including token labels, as well as tokens, scores, candidate order, masks, and finish reasons. This campaign fixed two existing vLLM issues in the base PR (#17088): valid one-token prompt frames were rejected, and native prompt-logprob token labels were omitted. Both regression tests failed before the fixes. Fresh CPU checks: 173 vLLM format/sidecar tests, 12 shared HTTP tests, strict Clippy, formatting, and changed-file hooks pass.

P/D uses two engine processes on one GPU; cross-GPU/node transport is unqualified. vLLM routed-expert transport, migration, and generate SSE remain unsupported. This is correctness validation, not a GPU performance claim.

Latest live expert qualification (2026-10-22)

The prerequisite is SGLang #44002, backported to 0.5.22 by #44212. A test image with its exact two added lines on the pinned 0.5.21 engine now returns prefill expert rows. The earlier default fused runner returned all-zero expert buffers, so earlier expert-byte checks proved transport consistency, not meaningful routing capture. New runs use flashinfer_trtllm_routed, assert four distinct valid expert IDs per token/MoE layer, and compare exact frontend bytes against captured worker responses.

Fresh rebuilt-stack results: patched P/D 33/33; aggregated 33/33; unpatched-engine compatibility 33/33. The four formerly failing max_new_tokens=1 cases (unary/SSE × logprobs) now pass. Prefill generates the sole token and holds all expert rows; decode does no model step and omits the field. Base commit 1be33b900a retains that complete prefill value only for a successful terminal response with one attempt-local completion token, using the existing codec validation/join. It does not synthesize missing decode data for intermediate, aborted, or multi-token responses. No plugin API or external zstd codec changes were needed.

The old simulator invented decode expert rows in this case. Correcting it reproduced the bug in the existing property laws before the fix. Base tests: 17 passed; restacked backend/HTTP/sidecar tests: 198 passed; external zstd tests: 5 passed, including inherited 512-case laws. Strict Clippy and changed-file hooks pass. The standalone example reused the validated lockfile because the repository's pinned Tokio version is now yanked; dependency pins are unchanged. Live runs used clean stack head c6c1c80284, fresh dev-profile wheel and sidecar, exact captured-worker expert-byte comparisons, and all 48 engine-confirmed abort/release checks passed. The test-image backport does not change Dynamo's SGLang dependency pin.

• Full LLM suite: 3,185 passed, 13 ignored. One migration tracing assertion failed on the first run; isolated and full reruns passed. Baseline comparison passed, so a baseline reproduction of that transient failure is not claimed.
• Backend/plugin suites: 351 passed. Strict all-target Clippy, cargo check, formatting, and changed-file pre-commit checks passed.
• An external HTTP host exercised SGLang, vLLM, and the zstd plugin with session/no-session requests and supported unary/SSE modes: 10 combinations passed, each with exactly one correct request-end record.
• Fresh dev-profile frontend wheel and Rust sidecar on B200, pinned SGLang v0.5.21/cu130 and PrimeIntellect/qwen3-moe-tiny: 33/33 aggregated and 33/33 P/D cases passed, including native parity, concurrency, malformed inputs, prompts, unary/SSE tracing, and 16 engine-confirmed abort/release checks in each mode.
• Two live worker-kill campaigns: 16/16 streams completed, eight migrated from the killed worker to the survivor. Independent captured-wire audits verified exact token order, prompt/output logprobs (two-ULP JSON tolerance), expert bytes, replay prefixes/budgets, and exactly one correct trace per request.
• This campaign caught a genuine trace-accounting race: a completed 256-token request could record 255 when a retiring router guard overwrote the shared tracker. Native generate now owns an authoritative logical-output counter for trace emission; shared tracker semantics and other endpoints are unchanged. The existing lifecycle test now reproduces the overwrite for both agent and non-agent records. Full LLM suite, strict Clippy, cargo check, formatting, and pre-commit passed again after the fix.

The stress campaign also exposed a pre-existing shared-runtime limitation: simultaneous duplicate public request IDs overwrite response correlation channels. The original handlers use the same IDs and unchanged runtime code. This PR preserves public IDs and does not claim duplicate-ID concurrency support.

Unpatched SGLang 0.5.21 omits prefill expert rows; full expert parity above uses the test-image backport. External zstd tracing is covered by real HTTP tests with mock workers; it still lacks live GPU qualification. vLLM live qualification is recorded above.

An additional 33/33 P/D cases passed with expert capture requested: the oracle explicitly verifies native prefill lacks expert rows and Dynamo omits the incomplete blob. This validates omission behavior, not full P/D expert-byte reconstruction.

Frontend performance validation

Three interleaved repetitions compare the same release stack with request-end tracing disabled/enabled, using four frontend CPU cores and a local JSONL sink. Measured CPU/request changes are +1.4% for 128-token inputs and +0.1% for 16K-token inputs; throughput stays within 1%. No extra agent-context metadata is supplied in this performance workload.

All 639,514 emitted records passed exact-count, unique-ID, input/output-token-count, and replay-hash consistency checks. This measures frontend tracing with mock workers; it does not qualify remote sinks, GPU serving, or sidecar overhead.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a19580-d10b-76e8-930a-d41466f9bc47.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 41275392979 (attempt 1) on b93c7e26fe

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41275392979. Updated on every full-CI run of this PR.}

[gongshaotian]

That's so cool, man.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Aphoh]

/ok to test 6b45e4b

[Aphoh]

/ok to test 0370b97

[Aphoh]

/ok to test 9dd9370

[Aphoh]

/ok to test b93c7e2