feat: SGLang DGD recipe for DeepSeek-V4-Flash on B200 - #8703
Closed
KrishnanPrash wants to merge 22 commits into
Closed
feat: SGLang DGD recipe for DeepSeek-V4-Flash on B200#8703KrishnanPrash wants to merge 22 commits into
KrishnanPrash wants to merge 22 commits into
Conversation
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
Signed-off-by: ayushag <ayushag@nvidia.com>
…el_type
Previously `PromptFormatter::from_mdc` activated the V4 / V3.2 native
formatters based on a substring match against `mdc.display_name`:
name_lower.contains("deepseek") && name_lower.contains("v4")
Two problems:
1. **Rename-fragile** — `display_name` is overwritten by
`--served-model-name`, so a valid DeepSeek-V4 checkpoint served as
`dsflash` (or any other operator-chosen alias) silently fell through
to the HF Jinja path and the native Rust formatter never fired.
2. **Loose** — `contains("v4")` mis-matched composite names like
`deepseek-v3.2-v4-foo`, short-circuiting to V4 before the V3.2
branch.
Rework detection to key off `config.json` `model_type`, which
DeepSeek-V4-Pro / V4-Flash both ship as `"deepseek_v4"` (verified on
the HF repo). The config value is set by the model author and survives
any rename. `display_name` is kept only as a fallback for MDCs that
don't have a loadable config.json (tokenizer-only deployments), and a
concrete-but-different config.json value now authoritatively suppresses
the fallback — a `model_type: "llama"` model served as
`deepseek-v4-flash` no longer gets the V4 formatter.
Also tightens the display-name fallback to an anchored match equivalent
to `^deepseek(?:[-_.])?v4(?:[-_.]|$)` (no `regex` crate dep), closing
the composite-name hole flagged as N5 in the V4 review.
Unit tests cover: canonical V4 variants (hyphen / underscore / dot /
concat separator), negative cases (composite / non-V4 / mis-prefixed),
config-primary precedence, and the V3.2 symmetric case.
`HFConfig::model_type` is a required field at parse time, so a missing entry fails deserialization and `get_model_info().ok()` drops the whole `ModelInfo` — we correctly fall back to the display-name heuristic. But `"model_type": ""` (rare, legal JSON) parses cleanly into `Some(String::new())`, hits the `Some(_) => false` arm in `is_deepseek_v4`, and *suppresses* the display-name fallback. A DeepSeek-V4 model served as `deepseek-v4-flash` with an empty `model_type` would silently miss the native formatter. Normalize empty strings to `None` via `.filter(|s| !s.is_empty())` so "no signal from config.json" behaves uniformly regardless of whether the field was absent or blank.
`to_json` serializes tool schemas, response-format bodies, and arbitrary message-content fallbacks through a custom `PythonFormatter`. The buffer was `Vec::with_capacity(64)` — too small for anything beyond a trivial value, and worse than `Vec::new()` for tiny values since it forces an up-front heap alloc that's immediately outgrown. For a ~5 KB tool schema (200-field array), 64 bytes triggers 6 sequential doublings (64→128→256→512→1024→2048→4096→8192), each a memcpy of the current contents. Callers hit this on every request that includes tools or a response_format, so the overhead is per-request on the hot path. Pre-size the buffer from one compact `serde_json::to_string` pass. The `PythonFormatter` adds exactly 1 byte per structural separator (`,`→`, ` and `:`→`: `), which bounds the final length by ~12.5% over compact — `compact_len + compact_len/8` is a tight upper bound that eliminates all reallocations in the formatted pass. A 256-byte floor keeps tiny payloads from under-allocating after empty-map round-trips. Adds a ~5 KB round-trip test (`test_to_json_handles_large_payload`) that pins: - Output parses back to the input (no truncation). - Python-style spacing survives deep nesting (no bare `",", / `":"`). All 7 deepseek_v4 unit tests pass including the new one.
…per call
`extract_tool_calls`, `extract_invokes`, `parse_parameters` each built a
regex pattern from `format!("...{}...", regex::escape(config.field))` and
`Regex::new`'d it on every invocation. On streaming hot paths this
recompiled three regexes per chunk — independently of what the chunk
actually contained — despite the config strings being effectively fixed
per backend (V3.2 and V4 are the only variants in practice).
Introduce a `OnceLock<RwLock<HashMap<DsmlRegexKey, Arc<DsmlRegexes>>>>`
module cache keyed on the six config strings, and look up (with a shared
read lock on the fast path) before falling through to compile-and-insert.
The cache has at most two entries for the process lifetime.
`extract_invokes` and `parse_parameters` now take `&DsmlRegexes` instead
of `&DsmlParserConfig`, so the inner call tree carries compiled regexes
rather than repeatedly hashing and re-looking-up the same entry.
All 19 DSML parser unit tests still pass unchanged — the cache is a pure
behavioral no-op.
…mat!
`render_tools` built the tool-section of the system prompt by running
four sequential `String::replace` passes over `TOOLS_TEMPLATE`:
TOOLS_TEMPLATE
.replace("{tool_schemas}", ...) // allocates a fresh String
.replace("{dsml_token}", ...) // allocates again
.replace("{thinking_start_token}", ...)
.replace("{thinking_end_token}", ...)
Each `replace` allocates a new `String` sized for the whole template —
and after the first call the template already contains the inlined
schemas, so each subsequent pass copies the full (potentially kB-scale)
payload. Four tool-schema-inlined copies per render.
Collapse into one `format!` with named arguments: a single allocation
sized by the macro from the final length. Four placeholders, one pass.
The standalone `TOOLS_TEMPLATE` const is no longer referenced and is
removed — the template body now lives inside the `format!` macro where
it's used.
Byte-identical output: all four fixture-driven tests in
`tests/deepseek_v4_encoding.rs` pass unchanged, including the
tools-present fixtures that exercise every placeholder.
`merge_tool_messages` cloned every input message at loop top (`let msg = msg.clone();`) regardless of which branch handled it: - `tool` role: extracts two fields into a fresh `tool_result` JSON — never pushes `msg`. The full clone was discarded. - `user` role: extracts the text content and preserves three named fields via per-field `v.clone()` into a fresh message — never pushes the original. The full clone was discarded. - other roles: passes `msg` through into `merged`. Only this branch actually needs to own the value. On long chat histories this cloned the entire conversation's JSON tree once per turn — tool calls with large payloads paid the worst cost. Iterate by reference (`for msg in messages` already binds `&JsonValue` from a slice), drop the top-level clone, and move the clone into the pass-through `else` branch where ownership is actually needed. The two "consecutive-user merge" branches are also tightened with let-chains to avoid the `merged.last_mut().unwrap()` unwrap that was guarded by the `can_merge` flag a few lines earlier — original malformed-data behavior (silent drop rather than fall-through) is preserved with an explicit comment. All 7 deepseek_v4 unit tests and all 4 fixture-driven integration tests pass unchanged — the output JSON is byte-identical for every shape the existing suite covers.
`encode_messages_with_options` used to scan the message list twice:
1. `drop_thinking_messages` computed `find_last_user_index(&input)` at
the top of its body to decide which pre-last-user messages to drop.
2. After drop (or instead of it), the encode loop computed
`find_last_user_index(&full)` again to pass into `render_message`.
Each `find_last_user_index` is a full O(n) reverse scan of a freshly-
allocated JSON message vec. On long chat histories this is two redundant
scans per request — one when drop runs, one always.
Dedup: hoist `find_last_user_index(&full)` to the single site in
`encode_messages_with_options` and thread both its input (used by drop
to decide) and its *output* (computed inline while iterating) through
`drop_thinking_messages`:
- Caller computes `let mut last_user_idx = find_last_user_index(&full)`
once on the post-merge/sort list.
- `drop_thinking_messages(full, last_user_idx)` now returns
`(Vec<JsonValue>, Option<usize>)` — the dropped list plus the
post-drop position of the same "last user/developer" message,
tracked as `out.len()` at the moment it's pushed.
- The encode loop reads the returned index directly — no rescan.
The post-drop index is mathematically equivalent to a full
`find_last_user_index(&dropped)` rescan: `drop_thinking_messages`
never drops or reorders the message at `last_user_idx`, and every
surviving message before it counts toward the new position by 1. Three
new unit tests pin this equivalence for the three cases that matter:
- `test_drop_thinking_messages_returns_post_drop_last_user_idx` —
shape where a pre-last-user `developer` is dropped, forcing the
trailing developer to shift from idx 3 to idx 2. Asserts the
returned index equals a full rescan.
- `test_drop_thinking_messages_preserves_idx_when_nothing_dropped` —
no-shift sanity: returned idx equals the input idx.
- `test_drop_thinking_messages_no_user_in_history` — edge case where
both pre- and post-drop indices are `None`.
All 10 deepseek_v4 unit tests and all 4 fixture-driven integration
tests pass unchanged.
Focused pass over the DeepSeek-V4 formatter + reasoning registration. No behavior changes are intended for any fixture-covered input; every existing unit and integration test in `deepseek_v4` and `reasoning` passes unchanged. All items come from the review of 88478b4..HEAD. Addressed: - Smell 4 / idiomaticity 4 — `to_json` now returns `Result<String>` instead of silently collapsing serialization / UTF-8 errors to `"{}"`. Cascaded through `render_tools`, `extract_visible_text`, `normalize_message_contents`, `render_tool_result_content`, the two `render_message` tool/response-format sites, `encode_arguments_to_dsml`, and the `impl OAIPromptFormatter::render` entry point. Errors now surface as `anyhow::Error` with context instead of corrupting the prompt. - Smell 7 — `value.as_str().unwrap().to_string()` (guarded a few lines earlier by `.is_string()`) replaced with a `match value { JsonValue:: String(s) => ... }` pattern so the invariant holds at the type level rather than by convention. - Smell 8 — magic `&["user","system","tool","latest_reminder", "direct_search_results"]` list inside `drop_thinking_messages` hoisted to module-scope `const KEEP_ROLES: &[&str]` with a doc comment explaining what the list means. - Smell 9 — `"[Unsupported {}]"` literal injected into user-visible prompts now goes through a shared `UNSUPPORTED_PLACEHOLDER_FMT` constant and emits a `tracing::warn!` at both call sites (user-content blocks and tool_result items), matching the existing `extract_visible_text` drop-path warn. - Smell 10 — `ReasoningParserType::DeepSeekV4` dedicated variant added, replacing the silent `Qwen` alias. Verified against deepseek-ai/DeepSeek-V4-Pro/encoding/encoding_dsv4.py: V4 uses the same `<think>` / `</think>` delimiters as Qwen today, so the new variant's match arm points at the same `BasicReasoningParser` config — but future V4-specific divergence (max-thinking mode, different tokens) now has a place to land without rippling through Qwen. - Smell 11 — the three-alias entries (`deepseek_v4` / `deepseek-v4` / `deepseekv4`) in the reasoning parser map now carry an inline comment explaining why (callers wire the name through from heterogeneous sources: `--dyn-reasoning-parser` flag, vLLM recipes, chat-template authors; accepting all three avoids a canonical-form requirement). - Smell 12 — `RESPONSE_FORMAT_TEMPLATE` const + `replace("{schema}", ..)` collapsed into a direct `format!("## Response Format:...{}", schema)` at the two system/developer call sites. One fewer indirection, one fewer allocation per render. - Smell 13 — `TOOL_CALLS_BLOCK_NAME` const now has a docstring explaining its role (one DSML tag is extracted; the others are inline because they appear exactly once). The summary's alternative — also constifying `INVOKE_BLOCK_NAME` / `PARAMETER_BLOCK_NAME` — would introduce consts used in exactly one place each, net-negative for readability. Deferred (tracked for follow-up commits): - Smell 2 (V3.2/V4 helper duplication). The V4 versions of `to_json`, `extract_visible_text`, `normalize_message_contents`, etc. are not byte-identical to V3.2 — V4's `to_json` uses `PythonFormatter` while V3.2 uses a character-scan that has a known escape-handling bug. Extracting a shared module would change V3.2 behavior; needs its own commit with V3.2 regression coverage first. - Smell 3 (`render_message` split into `render_assistant_body` / `render_user_body` / `append_transition_token`). Worth doing, but mechanically sizable; keeping this commit focused. - Smell 5 (test move: keep invariants in-module, move scenarios to integration). Opinionated; deferred. Tests: 10 unit + 4 integration + 91 parsers reasoning tests green.
Pure-style pass over the DeepSeek-V4 formatter. No behavior changes; all
10 unit + 4 integration tests still green.
- Idiom 1: `find_last_user_index` role matcher
`.map(|r| r == "user" || r == "developer").unwrap_or(false)`
→ `.is_some_and(|r| matches!(r, "user" | "developer"))`
(closer to the `Python` `is_some_and` sentinel semantics the file uses
elsewhere for last_user_idx comparisons.)
- Idiom 2: `resolve_thinking_mode` nested-`if let` over the same `args`
collapsed with 2024-edition let-chains; the outer scope `args` is
matched twice but it's a cheap reference comparison and the flatter
shape matches the rest of the file (e.g. the `can_merge` branches in
`merge_tool_messages` already use this form).
- Idiom 3: `.and_then(|v| v.as_str())` / `.as_array()` / `.as_bool()` /
`.as_array_mut()` closures across ~30 sites replaced with
`JsonValue::as_str` / `as_array` / `as_bool` / `as_array_mut` fn
pointers. No closure allocation, one less turbofish-free indirection.
- Idiom 5: `anyhow::bail!("Unknown role: {}", other)` →
`anyhow::bail!("Unknown role: {other}")` (2024-edition captured
identifier; matches `"{text_so_far}"` / `"{idx}"` usage elsewhere in
the file).
- Idiom 6: `#[inline]` on `ThinkingMode::as_str` (two-branch match
returning `&'static str`) and `task_token` (seven-branch match
returning `&'static str`). Both are on the per-message render hot path
and smaller than the call overhead; inlining lets LLVM fold them into
caller string-equality checks.
Deferred from the idiomaticity list:
- Idiom 4 (`to_json` → `Result<String>`) — already landed in the
previous cleanup commit alongside Smell 4.
- Idiom 7 (`extract_visible_text` → `Cow<'_, str>`). The text-only path
already avoids re-allocating (`text.clone()` is a `String::clone`,
which is still O(n) but unavoidable without a `Cow`). Wrapping in
`Cow<'_, str>` would change the return type through
`normalize_message_contents` / `render` ripple and intermittently save
one clone per message. Worth its own focused commit with a benchmark.
Tests: 10 unit + 4 integration, all green.
Signed-off-by: ayushag <ayushag@nvidia.com>
Introduces a `CASE.<N>` / `CASE.<family>.<N>` taxonomy for parser test
categories:
- new `lib/parsers/README.md` — crate landing page (parser families,
request-flow diagram, how-to-add-a-parser)
- new `lib/parsers/TESTING.md` — 16 generic + 2 XML-family + 1
Harmony categories, with per-category definitions, applicability,
and example anchors
Adds 4 DSML pinning tests that document current behavior on truncated
V4 output:
- `CASE.5` missing outer `</｜DSML｜tool_calls>` fence — whole block
silently dropped (same structural class as Kimi K2 pre-recovery)
- `CASE.4` missing inner `</｜DSML｜invoke>` fence — call silently
dropped
- `CASE.4` malformed JSON in `string="false"` param — falls back to
raw string (intentional, pins the `unwrap_or_else` behavior)
Annotates every existing V4 test with a `/// CASE.<N>` doc comment
naming the category it pins, across 4 files (dsml parser tests,
parsers registry, reasoning registry, streaming e2e). Adds a coverage
manifest block above the V4 test section listing what's still NOT
covered so future contributors see the gaps without reading the full
test file:
- `CASE.5` mid-stream truncation recovery (TODO)
- `CASE.4` parameter-close and middle-invoke-bleed variants
- `CASE.11` tool_choice auto/required/named/none
- `CASE.12` `FinishReason::Length`
- `CASE.14` empty-content / null at e2e
- `CASE.15` duplicate calls (universal gap)
- `CASE.16` regression (V4 is brand new, no bugs filed yet)
No production code changes. All 389 dynamo-parsers tests + 4 V4
encoding golden tests + 24 V4 streaming tests pass.
Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
…t const strings Addresses biswapanda's review comments on #8665: - L389 "split into separate funcs" — render_message's 221-line match is split into six per-role helpers: render_system_role, render_developer_role, render_user_role, render_latest_reminder_role, render_assistant_role, plus append_response_format / append_tools_section to dedup the sys/developer response-format and tools blocks. render_message itself stays as a slim dispatcher that handles the reasoning-effort prefix and the transition-token tail. - L400 "const strings" — hard-coded `<tool_result>` / `</tool_result>` tags extracted to TOOL_RESULT_OPEN / TOOL_RESULT_CLOSE consts alongside the other wire-format constants. The response-format preamble (previously duplicated inline in the system and developer branches) becomes RESPONSE_FORMAT_PREAMBLE with a {} placeholder. Already addressed by prior commits on the branch (no-op here): - L32 "Should be ReasoningParserType::DeepSeekV4" — the three deepseek_v4 aliases now map to ReasoningParserType::DeepSeekV4. - L899 "hard coded path" — the `/home/ayush-lab/...` absolute path in the test has been removed. - L266 "clippy manual_pattern_char_comparison" — fixed in 77602e4 by using a char array. Pure refactor — behavior is identical. All 4 V4 encoding golden tests + 7 V4 streaming tests pass unchanged. Clippy clean on dynamo-llm. Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
The fixture's expected_output carried a finish_reason field that the streaming test harness never reads — finish_reason validation is hardcoded against the stream chunks themselves, not the expected_output struct. None of the other 6 deepseek-v4 fixtures have it. Drop it for consistency. Addresses CodeRabbit comment 2 on #8665. Signed-off-by: Keiven Chang <keivenc@nvidia.com> Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Adds a DynamoGraphDeployment recipe for serving DeepSeek-V4-Flash via SGLang on Blackwell (B200) GPUs with Dynamo frontend. Recipe: - Dockerfile.dsv4-sglang: multi-stage build layering Dynamo runtime onto lmsysorg/sglang:deepseek-v4-blackwell base - sglang-dgd.yaml: DGD manifest with TP4, flashinfer_mxfp4 MoE, EAGLE MTP 3/4 speculative decoding, V4 tool call + reasoning parsers Validated: 11/12 e2e tests passing (tool calling, reasoning, streaming, multi-tool, mixed param types, special chars, no-param tools).
github-actions
Bot
added
feat
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a DynamoGraphDeployment recipe for serving DeepSeek-V4-Flash via SGLang on Blackwell (B200) GPUs with Dynamo frontend. Includes a multi-stage Dockerfile and DGD YAML.
recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang): Builds dynamo runtime from source (V4 parsers + routed_experts fix), layers ontolmsysorg/sglang:deepseek-v4-blackwellrecipes/deepseek-v4-flash/sglang/sglang-dgd.yaml): Frontend + decode worker, TP4, flashinfer_mxfp4 MoE, EAGLE MTP 3/4Key SGLang args
--moe-runner-backend flashinfer_mxfp4(MXFP4 MoE kernels for FP4 expert weights)--speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4--dyn-tool-call-parser deepseek_v4 --dyn-reasoning-parser deepseek_v4--chunked-prefill-size 4096 --disable-flashinfer-autotuneEnv vars
SGLANG_JIT_DEEPGEMM_PRECOMPILE=0+SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1(bypass deep_gemm API mismatch)PYTHONPATH=/workspace/sglang/python:...(fix sglang namespace package shadowing)Dependencies
codex/deepseek-v4-parsers(PR chore(frontend): Add DeepSeek V4 parser support + Test Cases #8665 — V4 formatter + parsers)44fb33afromishan/investigate-tmrw(PR [INVESTIGATE] sglang + DSv4: kwarg workaround breaks live — needs diagnosis #8671 —return_routed_expertskwarg fix)Image