Skip to content

feat(sglang): upload native generate metadata - #14895

Draft
Aphoh wants to merge 1 commit into
warnold/sglang-metadata-upload-fallbackfrom
warnold/sglang-generate-metadata-upload
Draft

Aphoh wants to merge 1 commit into
warnold/sglang-metadata-upload-fallbackfrom
warnold/sglang-generate-metadata-upload

Conversation

@Aphoh

@Aphoh Aphoh commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • treat the complete top-level nvext object on native SGLang /generate requests as Dynamo-owned and remove it from the opaque request forwarded to SGLang
  • carry the typed nvext envelope in PreprocessedRequest.extra_args, enabling the OpenDAL metadata uploader added by feat(sglang): upload sidecar metadata with OpenDAL #11470
  • project Dynamo routing fields from nvext while preserving x-data-parallel-rank header precedence
  • retain native stream control metadata inline while moving routed experts, log-probability arrays, and other large metadata into the uploaded artifact
  • document native /generate metadata-upload configuration and response behavior

Stacked on #11470. Merge that PR first.

Validation

  • cargo +stable test -p dynamo-llm --lib sglang_generate --locked — 5 passed
  • targeted typed-nvext ownership, unknown-field rejection, and opaque-response tests — 4 passed
  • cargo +stable test --manifest-path lib/sidecar/sglang/Cargo.toml --locked — 43 passed
  • targeted native metadata-upload test verifies routed experts are absent inline and present in the decoded artifact
  • B200 end-to-end test using SGLang 0.5.16 and Qwen3-4B with the PR-built Python extension and sidecar:
    • streamed 8 tokens through native POST /generate
    • confirmed all of nvext remained Dynamo-owned and was not forwarded to SGLang
    • retained request ID, finish reason, and token counts inline while moving log-probability arrays into choice_0.msgpack.zst
    • forced a primary write failure and confirmed fallback_url received the artifact
    • decoded the artifact with Python zstandard and msgpack, verifying schema version 1 and terminal metadata
  • cargo +stable clippy -p dynamo-llm --lib --no-deps --locked with allowances for pre-existing Rust 1.98 lints
  • cargo +stable fmt --all -- --check
  • python3 docs/fern/scripts/docs_lint.py --scan docs — 0 errors (pre-existing warnings remain)
  • git diff --check

@copy-pr-bot

copy-pr-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Sep 15, 2026
Signed-off-by: William Arnold <7565007+Aphoh@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant