feat(sglang): support format-owned native generate migration by Aphoh · Pull Request #16977 · ai-dynamo/dynamo

Aphoh · 2026-10-21T03:55:19Z

Stacked on #17802, which adds the backend-neutral migration hooks; this PR holds only SGLang migration. It does not depend on #16978, #16997, or vLLM migration (#17766).

Summary

Enable migration for eligible native SGLang /generate requests through the plugin interface introduced by #16983. A format-owned continuation rebuilds the opaque engine request, reduces remaining token budgets, preserves prompt metadata, and rebases response counters across failed attempts.

Worker-advertised sampling defaults participate in admission. Stateful sampling, unresolved token budgets, unbounded budgets, and unsupported response modes remain non-migratable. Retries respect the configured sequence limit; exhausted budgets finish without another worker call. vLLM /generate migration remains disabled.

The property harness adds injected failures, repeated retries, migration-equivalence and admission laws, and an independent saturation oracle. HTTP integration tests exercise unary/SSE retries, sequence limits, and logical-request cache metrics.

Part of #16675. Part of #16674.

Validation

After restacking the reviewed base fixes: all 11 catalog HTTP integration tests, 21 SGLang unit tests, and 37 vLLM unit tests pass on the complete stack. Migration admission honors deferred validation and the original JSON media-type contract.
Earlier broad validation: 3,491 unit tests passed: 3,184 LLM, 57 backend-format, and 250 sidecar tests; 13 existing LLM tests ignored.
Earlier broad validation: 42 integration tests passed: seven catalog HTTP tests and 35 sidecar tests.
All five SGLang property laws and all three low-level Base64/counter properties run 512 cases each. Four pinned SGLang source-contract checks pass.
Strict all-target Clippy passed for the LLM, both format crates, catalog, both sidecars, and Python bindings. Changed-file hooks and formatting pass.

Incremental diff: 22 files, +1,344 / -163. No build-manifest or ownership-policy changes.

Live complete-stack qualification also passed on B200 with the rebuilt frontend wheel and Rust sidecar, pinned SGLang v0.5.21/cu130, and PrimeIntellect/qwen3-moe-tiny. Two campaigns killed one of two token-only sidecars after eight streams started: 16/16 streams completed, eight actually migrated from the killed worker to the survivor. Captured engine outputs verify exact token order, prompt/output logprobs (up to two ULPs for JSON round trips), routed-expert bytes, replay prefixes, remaining budgets, and exactly-once trace rows. All migrated requests returned 256 tokens. The tracing follow-up #16997 fixes a per-attempt counter race found by this campaign.

Migration uses --skip-tokenizer-init, matching its existing token-only admission rule. This is correctness qualification; no throughput or latency claim.

Frontend performance follow-up

Move disjoint prompt snapshots out of consumed first-attempt streaming chunks after rendering. Keep copy semantics for overlapping custom fields, P/D, retries, and unary assembly. No plugin API changes. The 512-case snapshot law, native merge/migration laws, HTTP tests, strict Clippy, and changed-file hooks pass.

Combined with #16983's reusable SSE buffer, the complete-stack logprob-heavy benchmark drops from 2.55 to 1.98 ms of frontend CPU/request (about 5% above the original baseline, down from 34%). This is a four-core release frontend with fixed mock workers, not an isolated measurement of this child PR or a GPU benchmark. Residual low-load short-request CPU cost remains (+61 µs/request, or 14%).

devin-ai-integration · 2026-10-21T04:57:10Z

✅ Dynamo PR CI passed — run 41409732522 (attempt 1) on `14c8b2e744`

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41409732522. Updated on every full-CI run of this PR.}

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Original stack head: c6c1c80. Refs: #16675, #16674, #16640 Fixes: #16762 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: d382dd1. Refs: #16675, #16674, #16640 Fixes: #16762 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: 5e09cf5. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: c541550. Remove duplicate vLLM sampling declarations and response-finalization plumbing. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: b2de7a8. Remove duplicate vLLM sampling declarations and response-finalization plumbing. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Remove RL-only sidecar rolling-upgrade shims; use current plugin projections and transparent sampling serialization. CPU validation: 338 tests, strict Clippy and pre-commit. Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, agent tracing, and vLLM prompt-logprob fixes. Remove duplicate sampling/finalization code and RL-only rolling-upgrade shims. Include the review fixes for route-test isolation and multimodal fallback fixtures. Preserve upstream live request/token metrics and SGLang HTTP connection retirement while removing the superseded generate handlers. Single-commit snapshot of #16983 -> #16977 -> #16978 -> #16997. Stack head: 1d59b49. Upstream parent: fc9b996. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

copy-pr-bot · 2026-10-26T08:29:39Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Aphoh · 2026-10-30T04:41:23Z

/ok to test d2f3969

Aphoh · 2026-11-01T03:46:11Z

/ok to test 3c596fd

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Retain migration snapshots after rendering the first aggregated attempt, moving disjoint fields out of the discarded chunk. Keep copy semantics for overlapping custom fields, P/D, retries, and unary assembly. Add a property law for nested custom snapshots. No plugin API changes. Refs: #16675, #16674 Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Keep the vLLM projection fixture aligned with the Worker field added by this PR. The base PR remains independently buildable. Refs #16675. Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

System One binds through the SGLang format with no engine info and never migrates a scoring branch; the crate's test format implements migrate. Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh · 2026-11-01T05:02:55Z

/ok to test 488723f

…enerate-migration Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh · 2026-11-02T07:58:22Z

/ok to test 14c8b2e

pull-request-size Bot added the size/XXL label Oct 21, 2026

github-actions Bot added feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Oct 21, 2026

This was referenced Oct 21, 2026

feat(sglang): add compressed generate format example #16978

Draft

refactor(frontend): unify native generate handlers behind format plugins #16676

Closed

Aphoh mentioned this pull request Oct 21, 2026

refactor(frontend): unify native generate handlers behind format plugins #16983

Merged

Aphoh force-pushed the warnold/sglang-generate-migration branch 3 times, most recently from 14d65fb to 1b66ee6 Compare October 21, 2026 06:49

Aphoh added this pull request to stack #16987 October 21, 2026 07:00

Aphoh force-pushed the warnold/sglang-generate-migration branch from 1b66ee6 to 7621e81 Compare October 21, 2026 08:46

Aphoh mentioned this pull request Oct 21, 2026

feat(frontend): trace requests across generate format plugins #16997

Draft

Aphoh force-pushed the warnold/sglang-generate-migration branch from 7621e81 to 5e0d8ee Compare October 22, 2026 03:01

Aphoh force-pushed the warnold/sglang-generate-migration branch from 5e0d8ee to 780fdff Compare October 22, 2026 04:51

Aphoh force-pushed the warnold/sglang-generate-migration branch 2 times, most recently from 3869390 to 154e998 Compare October 22, 2026 09:23

Aphoh force-pushed the warnold/sglang-generate-migration branch from 154e998 to f62c303 Compare October 22, 2026 10:07

Aphoh force-pushed the warnold/sglang-generate-migration branch 2 times, most recently from 4cb9de3 to 870a83a Compare October 23, 2026 07:48

Aphoh force-pushed the warnold/generate-format-plugins-base branch from 07c4505 to e15018c Compare October 26, 2026 08:26

Aphoh force-pushed the warnold/sglang-generate-migration branch from 870a83a to e4a0219 Compare October 26, 2026 08:29

Aphoh force-pushed the warnold/generate-format-plugins-base branch from e15018c to 9dd347e Compare October 28, 2026 06:34

Aphoh force-pushed the warnold/sglang-generate-migration branch from f7cd044 to 13d540f Compare October 28, 2026 06:36

Aphoh force-pushed the warnold/generate-format-plugins-base branch from 9dd347e to e721abe Compare October 29, 2026 04:31

Aphoh force-pushed the warnold/sglang-generate-migration branch from 13d540f to f046df7 Compare October 29, 2026 04:31

Aphoh force-pushed the warnold/generate-format-plugins-base branch from e721abe to e5b66e4 Compare October 30, 2026 04:18

Aphoh force-pushed the warnold/sglang-generate-migration branch from f046df7 to d2f3969 Compare October 30, 2026 04:41

Aphoh force-pushed the warnold/sglang-generate-migration branch from d2f3969 to 3c596fd Compare November 1, 2026 03:45

Base automatically changed from warnold/generate-format-plugins-base to main November 1, 2026 04:34

Aphoh and others added 5 commits November 1, 2026 12:34

feat(sglang): support format-owned native generate migration

3a034ba

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

test(generate): implement migrate on the passthrough test exchange

ba42f8d

Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh force-pushed the warnold/sglang-generate-migration branch from 3c596fd to 488723f Compare November 1, 2026 04:34

Aphoh removed this pull request from stack #16987 November 1, 2026 05:02

Aphoh mentioned this pull request Nov 1, 2026

feat(vllm): support native generate request migration #17766

Draft

Aphoh added this pull request to stack #17795 November 2, 2026 04:29

Aphoh mentioned this pull request Nov 2, 2026

feat(generate): add format-owned migration hooks for native generate #17802

Draft

Merge branch 'warnold/generate-migration-hooks' into warnold/sglang-g…

14c8b2e

…enerate-migration Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh removed this pull request from stack #17795 November 2, 2026 07:58

Aphoh changed the base branch from main to warnold/generate-migration-hooks November 2, 2026 07:58

jain-ria mentioned this pull request Nov 5, 2026

fix(sglang): keep SGLang's first-chunk input logprobs in the unary join #17370

Open

1 task done

Navigation Menu

Repository navigation

feat(sglang): support format-owned native generate migration - #16977

feat(sglang): support format-owned native generate migration#16977
Aphoh wants to merge 6 commits into
warnold/generate-migration-hooksfrom
warnold/sglang-generate-migration

Aphoh commented Oct 21, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

Aphoh commented Oct 30, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Aphoh commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Frontend performance follow-up

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Dynamo PR CI passed — run 41409732522 (attempt 1) on 14c8b2e744

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

Aphoh commented Oct 30, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Aphoh commented Oct 21, 2026 •

edited

Loading

devin-ai-integration Bot commented Oct 21, 2026 •

edited

Loading

✅ Dynamo PR CI passed — run 41409732522 (attempt 1) on `14c8b2e744`