feat(sglang): add compressed generate format example by Aphoh · Pull Request #16978 · ai-dynamo/dynamo

Aphoh · 2026-10-21T03:55:39Z

Based directly on main (after #16983 merged). It does not depend on #16977 or #16997.

Summary

Add an external SGLang catalog for workers that emit Base64(zstd(bytes)) in meta_info.routed_experts. Aggregated unary and SSE responses preserve the worker's encoded value unchanged. Only the prefill/decode join decompresses the two payloads, replaces decode's stale prompt rows with prefill's prefix, and recompresses the joined result for the client. Stock SGLang requires a producer change to emit this encoding.

Implement the existing SGLang ClientCodec trait from #16983. Override sequence joining for routed experts and inherit native Base64 joining for other fields and request handling. Missing or corrupt P/D sequence data follows the native omission policy. This PR changes only the standalone example; it makes no production API or testkit changes.

The standalone example extends the built-in catalog and is selected with DYN_GENERATE_FORMAT=sglang-zstd. The example owns a simulated worker that compresses engine output, so inherited property laws exercise compressed data throughout P/D and migration.

All changes are in the SGLang-owned format tree.

Part of #16675. Part of #16674.

Validation

Five standalone tests pass from a copy outside the repository with absolute path dependencies and DYN_GENERATE_FORMAT=sglang-zstd.
All five inherited property laws run 512 cases each with compressed workers. A separate 512-case property checks P/D output against an uncompressed byte oracle and verifies other sequence fields retain their native encoding.
Real localhost unary/SSE tests assert byte-for-byte passthrough, including opaque values that cannot be decoded. Missing and corrupt P/D payloads cannot expose stale expert data.
All 21 native SGLang unit tests and 11 catalog HTTP integration tests pass after the compatibility fixes and restack.
Strict all-target Clippy for the external example, SGLang format and catalog, changed-file hooks, formatting, whitespace and SGLang-only ownership checks pass.

Incremental diff: eight files, +384 / -17. CPU and simulated-worker validation only; no GPU or rebuilt-wheel qualification is claimed.

devin-ai-integration · 2026-10-21T05:07:28Z

✅ Dynamo PR CI passed — run 41275392215 (attempt 1) on `ae4af98afd`

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 41275392215. Updated on every full-CI run of this PR.}

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Original stack head: c6c1c80. Refs: #16675, #16674, #16640 Fixes: #16762 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: d382dd1. Refs: #16675, #16674, #16640 Fixes: #16762 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: 5e09cf5. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: c541550. Remove duplicate vLLM sampling declarations and response-finalization plumbing. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned format plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, generic agent tracing, and the one-token P/D expert preservation fix. Reuse SSE serialization buffers and move consumed prompt snapshots to remove measured frontend CPU overhead. Preserve one-token vLLM prompt logprobs and fill missing native token labels. Squash the complete #16983 -> #16977 -> #16978 -> #16997 stack without changing its tested source tree. Stack head: b2de7a8. Remove duplicate vLLM sampling declarations and response-finalization plumbing. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Remove RL-only sidecar rolling-upgrade shims; use current plugin projections and transparent sampling serialization. CPU validation: 338 tests, strict Clippy and pre-commit. Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Unify native SGLang and vLLM generate routes through backend-owned plugins and the shared HTTP lifecycle. Include SGLang migration, the external Base64+zstd codec example, agent tracing, and vLLM prompt-logprob fixes. Remove duplicate sampling/finalization code and RL-only rolling-upgrade shims. Include the review fixes for route-test isolation and multimodal fallback fixtures. Preserve upstream live request/token metrics and SGLang HTTP connection retirement while removing the superseded generate handlers. Single-commit snapshot of #16983 -> #16977 -> #16978 -> #16997. Stack head: 1d59b49. Upstream parent: fc9b996. Refs: #16675, #16674, #16640 Fixes: #16762, #17088 Co-authored-by: gongshaotian <gstian5555@outlook.com> Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

copy-pr-bot · 2026-10-26T08:29:39Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Aphoh · 2026-10-30T04:41:24Z

/ok to test bd951d8

Aphoh · 2026-11-01T03:46:12Z

/ok to test 9885500

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh · 2026-11-01T05:02:56Z

/ok to test ae4af98

pull-request-size Bot added the size/L label Oct 21, 2026

github-actions Bot added the docs label Oct 21, 2026

This was referenced Oct 21, 2026

refactor(frontend): unify native generate handlers behind format plugins #16676

Closed

feat(sglang): support format-owned native generate migration #16977

Draft

Aphoh changed the title ~~docs(sglang): add external Base64 and zstd generate codec example~~ feat(sglang): add compressed generate format example Oct 21, 2026

github-actions Bot added feat and removed docs labels Oct 21, 2026

Aphoh mentioned this pull request Oct 21, 2026

refactor(frontend): unify native generate handlers behind format plugins #16983

Merged

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 22611b0 to 3bca961 Compare October 21, 2026 05:31

Aphoh force-pushed the warnold/sglang-generate-migration branch from bdd1d9e to 4184504 Compare October 21, 2026 05:31

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 3bca961 to 8c8f72a Compare October 21, 2026 06:30

Aphoh force-pushed the warnold/sglang-generate-migration branch from 4184504 to 14d65fb Compare October 21, 2026 06:30

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 8c8f72a to b1d3c54 Compare October 21, 2026 06:49

Aphoh force-pushed the warnold/sglang-generate-migration branch from 14d65fb to 1b66ee6 Compare October 21, 2026 06:49

Aphoh added this pull request to stack #16987 October 21, 2026 07:00

Aphoh force-pushed the warnold/sglang-generate-migration branch from 1b66ee6 to 7621e81 Compare October 21, 2026 08:46

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from b1d3c54 to 8042d5e Compare October 21, 2026 08:46

Aphoh mentioned this pull request Oct 21, 2026

feat(frontend): trace requests across generate format plugins #16997

Draft

Aphoh force-pushed the warnold/sglang-generate-migration branch from 7621e81 to 5e0d8ee Compare October 22, 2026 03:01

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 8042d5e to 05ef3fc Compare October 22, 2026 03:01

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 05ef3fc to d87a60c Compare October 22, 2026 04:51

Aphoh force-pushed the warnold/sglang-generate-migration branch from 5e0d8ee to 780fdff Compare October 22, 2026 04:51

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from d87a60c to 80c1c33 Compare October 22, 2026 07:56

Aphoh force-pushed the warnold/sglang-generate-migration branch from 780fdff to 3869390 Compare October 22, 2026 07:56

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 80c1c33 to 60e2f34 Compare October 22, 2026 09:23

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 60e2f34 to b3e8f27 Compare October 22, 2026 10:07

Aphoh force-pushed the warnold/sglang-generate-migration branch from 154e998 to f62c303 Compare October 22, 2026 10:07

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from b3e8f27 to d008f44 Compare October 23, 2026 02:59

Aphoh force-pushed the warnold/sglang-generate-migration branch from f62c303 to 4cb9de3 Compare October 23, 2026 02:59

Aphoh force-pushed the warnold/sglang-generate-migration branch from 4cb9de3 to 870a83a Compare October 23, 2026 07:48

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from d008f44 to f50c73a Compare October 23, 2026 07:48

Aphoh force-pushed the warnold/sglang-generate-migration branch from 870a83a to e4a0219 Compare October 26, 2026 08:29

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from f50c73a to dcadd04 Compare October 26, 2026 08:29

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch 2 times, most recently from bd5ff03 to b8c26ef Compare October 28, 2026 06:36

Aphoh force-pushed the warnold/sglang-generate-migration branch 2 times, most recently from 13d540f to f046df7 Compare October 29, 2026 04:31

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from b8c26ef to 9fa9d00 Compare October 29, 2026 04:31

Aphoh force-pushed the warnold/sglang-generate-migration branch from f046df7 to d2f3969 Compare October 30, 2026 04:41

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 9fa9d00 to bd951d8 Compare October 30, 2026 04:41

Aphoh force-pushed the warnold/sglang-generate-migration branch from d2f3969 to 3c596fd Compare November 1, 2026 03:45

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from bd951d8 to 9885500 Compare November 1, 2026 03:45

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from 9885500 to eabc895 Compare November 1, 2026 04:34

Aphoh and others added 3 commits November 1, 2026 04:40

docs(sglang): add external Base64 and zstd generate codec example

2a8a24b

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

fix(sglang): preserve compressed expert payloads until the P/D join

ee907ba

Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

test(sglang): use the base testkit worker in the zstd example

ae4af98

Signed-off-by: William Arnold <warnold@nvidia.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

Aphoh force-pushed the warnold/sglang-generate-zstd-example branch from eabc895 to ae4af98 Compare November 1, 2026 05:02

Aphoh removed this pull request from stack #16987 November 1, 2026 05:02

Aphoh changed the base branch from warnold/sglang-generate-migration to main November 1, 2026 05:02

Navigation Menu

Repository navigation

feat(sglang): add compressed generate format example - #16978

feat(sglang): add compressed generate format example#16978
Aphoh wants to merge 3 commits into
mainfrom
warnold/sglang-generate-zstd-example

Aphoh commented Oct 21, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

Aphoh commented Oct 30, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Aphoh commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

devin-ai-integration Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Dynamo PR CI passed — run 41275392215 (attempt 1) on ae4af98afd

Uh oh!

copy-pr-bot Bot commented Oct 26, 2026

Uh oh!

Aphoh commented Oct 30, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

Aphoh commented Nov 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Aphoh commented Oct 21, 2026 •

edited

Loading

devin-ai-integration Bot commented Oct 21, 2026 •

edited

Loading

✅ Dynamo PR CI passed — run 41275392215 (attempt 1) on `ae4af98afd`