Harden DeepSeek V4 DSpark DP2 serving

[benmyles]

Motivation

Note

This is an internal integration draft in the shiftup-ai/sglang fork. It is not an upstream submission. The branch will be refined, dependency-linked, and revalidated before any separate upstream PR is considered.

This draft collects a set of correctness, throughput, and operability changes developed while running DeepSeek V4 Flash with DSpark, DP2, HiCache, varied structured-output grammars, and mixed cached/uncached prefill traffic.

The main bottlenecks were:

• repeated and serialized grammar compilation/traversal on the scheduler thread;
• unnecessary distributed synchronization between two co-located DP schedulers;
• prefill batches routing to eager execution even when their shapes were graph-safe;
• sparse-prefill allocations and output copies that prevented stable graph capture;
• insufficient visibility into scheduler, grammar, prefill-graph, and HiCache behavior;
• lifecycle edge cases that could turn a clean shutdown into noisy secondary NCCL failures.

The optimized paths are intentionally fail-closed. When an explicitly enabled native, graph, grammar, or VMM path cannot satisfy its invariants, startup or the affected request fails instead of silently selecting a slower implementation.

Important

This is a large integration draft so the complete, soak-tested patch set is visible in one place. It can be split into smaller review units after maintainers confirm the preferred boundaries.

Companion XGrammar dependency

DSpark constrained decoding in this branch requires the companion XGrammar draft, which adds batched draft-tree traversal and the persistent compiler/serialization behavior used by the cache. SGLang checks for BatchGrammarMatcher.batch_traverse_draft_tree and fails startup when the API is unavailable; it does not silently return to scalar traversal. The companion change is now linked above; its final dependency revision must be pinned in SGLang before this PR is made ready for review.

Guided review

Tip

A practical review path: follow the numbered sections below in order. They trace one production request from HTTP ingress to a DP scheduler, through grammar and DSpark verification, into the synchronized prefill/decode execution paths, and finally through HiCache, metrics, and shutdown. The source links are pinned to the reviewed commit, so they will not drift if the draft branch moves.

Important

This SGLang change consumes the native batch API in the companion XGrammar draft. Review the SGLang grammar pipeline and that PR as one correctness boundary: SGLang deliberately fails startup when DSpark constrained decoding is enabled without the required XGrammar API.

At a high level, the patch replaces repeated scheduler-thread setup and coarse distributed coordination with bounded, explicitly owned fast paths. The key design rule is consistent throughout: an enabled optimized path must prove its geometry and lifetime assumptions; it must not silently continue on a different path after those assumptions fail.

Review map

Step	Reviewer question	Primary code
1	Does an explicitly routed API batch reach one DP scheduler atomically?	Tokenizer batching, DP dispatch
2	Can varied grammars compile concurrently without duplicate work or unbounded memory?	In-memory grammar cache, in-flight dedup and LRU, persistent cache
3	How is constrained DSpark verification overlapped without advancing the wrong FSM?	Grammar pipeline, worker integration, single FSM barrier
4	Does native DP2 synchronization preserve collective semantics and fail closed?	Python contract, C11 exchange, scheduler integration
5	When may both DP ranks enter a full-prefill CUDA graph?	Shared admission, graph admission, capture setup
6	Are replay buffers, TBO child metadata, sparse workspaces, and model outputs lifetime-safe?	Replay population, DSV4 metadata, TBO replay views, sparse outputs
7	Do HiCache control, scheduler health, and process teardown have explicit ownership?	HiCache gate, scheduler health, ordered teardown
8	Can operators tell which path actually ran and where time went?	Runtime metrics, metric emission, HiCache metrics
9	Which tests and gates protect each risk surface?	Grammar tests, native DP2 tests, prefill/TBO tests

flowchart LR
    A["HTTP batch<br/>routed_dp_rank = N"] --> B["Tokenizer<br/>one batch envelope"]
    B --> C["DP controller<br/>one ZMQ message"]
    C --> D["Scheduler rank N"]

    D --> E["Grammar cache<br/>memory → in-flight → disk/compile"]
    E --> F["DSpark grammar pipeline<br/>batched CPU traversal"]
    D --> G["DP2 scheduler exchange<br/>POSIX shm + futex"]
    G --> H["Shared verify/prefill tier"]

    H --> I["Prefill graph admission"]
    I --> J["Captured transformer body"]
    J --> K["Eager logits tail"]
    F --> K

    J --> L["DSV4 attention<br/>TBO + stable workspaces"]
    L --> M["HiCache"]

    D --> N["Per-rank metrics"]
    M --> N
    D --> O["Fresh scheduler snapshot"]

Loading

---

1. Preserve request intent from HTTP ingress to one DP rank

Start with the smallest behavioral change because it determines the GPU batch shapes seen by everything that follows.

The tokenizer’s batching policy recognizes a batch whose members all name the same routed_dp_rank. It keeps that batch together even under DP attention. _send_batch_request then wraps the tokenized items in one batch envelope.

At the controller, _dispatch_atomic_routed_batch validates that every item names the same live rank and sends the entire list in one ZMQ message. That one-message detail is the point of the change: if 64 requests arrive as 64 messages, the scheduler can wake after an arbitrary prefix and fragment the intended prefill into several smaller launches. One message lets the destination scheduler enqueue the complete API batch before forming work.

The adjacent load-budget refresh throttle prevents a different burst pathology. Snapshot refreshes are limited to once per 20 ms, so the controller’s speculative increments are not overwritten by the same stale scheduler snapshot after every request.

before: API batch → item 0 → scheduler wakes → small prefill
                    item 1
                    ...
                    item 63

after:  API batch → [item 0 ... item 63] → one scheduler receive → one scheduling opportunity

Note

Atomic delivery applies only when the route is explicit and uniform. An unrouted or mixed-rank batch keeps the general load-balanced per-request behavior; it is not misrepresented as an atomic single-rank batch.

Reviewer lens: ingress and DP routing

• Confirm out-of-range or inactive explicit ranks are rejected.
• Confirm all per-request timing metadata survives the batch envelope.
• Confirm mixed routes do not accidentally select the first request’s rank.
• Confirm the load-budget throttle preserves speculative accounting during a burst.

The focused contract is exercised by the atomic routing tests.

---

2. Turn grammar compilation into a bounded, shared service

Structured-output traffic may contain many syntactically different schemas, but it also contains duplicates that arrive concurrently or differ only in JSON whitespace. This step removes work at three levels.

2.1 Canonical keys and bounded process memory

BaseGrammarBackend initialization makes worker count, cache bytes, and cache entries explicit positive limits. JSON key normalization parses and re-emits JSON in a compact canonical form, so formatting differences do not create separate cache entries.

The lookup path has three outcomes:

1. A memory hit returns a request-local matcher copy and moves the prototype to the MRU end.
2. A matching compile is already in flight, so the request attaches to the shared future.
3. The caller becomes the owner and submits exactly one compile.

The owner registers its completion callback outside the cache lock because Future.add_done_callback() may invoke synchronously for an already-finished task. This avoids the callback-under-lock self-deadlock covered by the regression test.

Insertion and eviction remove the in-flight marker, cache the immutable prototype, account its memory, and evict from the LRU head until both hard limits are satisfied. Every request receives its own matcher copy, so concurrent requests never share mutable FSM state.

2.2 Process-safe persistent compiled grammars

PersistentXGrammarCache scopes its directory by XGrammar version, tokenizer digest, compiler identity, and policy. An entry from a different tokenizer or compiler configuration therefore cannot be mistaken for a compatible grammar.

get_or_compile takes a per-entry file lock before reading or compiling. The first process publishes either a serialized grammar or an authenticated “compile locally” marker; peers then deserialize or compile according to that recorded policy.

The adaptive policy in lines 271–337 compares measured native compile time with estimated deserialization time. If a grammar is too large for the cache or cheaper to compile locally by the configured margin, the cache stores a compact marker rather than repeatedly paying a slower deserialize.

Publication uses temporary files plus os.replace, so readers see an old complete entry or a new complete entry—not a partial write. Global byte accounting and pruning are protected by a separate lock; pruning skips entries currently locked by another compiler and refuses to report success while still above the hard limit.

Finally, XGrammarGrammarBackend owns the bounded native compiler and persistent cache, while all grammar dispatch types pass through the same cache contract.

flowchart TD
    A["normalized grammar key"] --> B{"memory LRU hit?"}
    B -->|yes| C["copy matcher for request"]
    B -->|no| D{"same key compiling?"}
    D -->|yes| E["attach to shared future"]
    D -->|no| F["one owner future"]
    F --> G{"persistent entry?"}
    G -->|serialized grammar| H["validate + deserialize"]
    G -->|local-compile marker| I["compile locally"]
    G -->|missing| J["compile once"]
    J --> K{"deserialize cheaper?"}
    K -->|yes| L["atomic serialized publication"]
    K -->|no| M["atomic authenticated marker"]
    H --> N["bounded memory LRU"]
    I --> N
    L --> N
    M --> N
    N --> C

Loading

Caution

Corrupt persistent entries are rejected; they are not quietly deleted and recompiled. A malformed cache artifact must be visible as a constrained-decoding failure rather than hiding integrity or compatibility problems.

The main tests are in-flight dedup/deadlock/LRU coverage and persistent round-trip, corruption, policy-marker, and concurrent-accounting coverage.

---

3. Overlap DSpark grammar work without weakening token correctness

This is the most accuracy-sensitive cross-component path in the PR. Its job is to build one allowed-token mask for every DSpark draft node while the GPU is busy with target verification, then apply those masks before acceptance.

3.1 Allocate the pipeline once and require the native dependency

DSparkGrammarPipeline.init checks for the companion XGrammar batch API and CUDA at startup. It constructs:

• one persistent native BatchGrammarMatcher;
• one persistent CUDA copy stream;
• two pinned-host/device mask slots; and
• events that describe draft-copy completion, mask readiness, and safe slot reuse.

There is no scalar traversal fallback. Missing native support, invalid geometry, or a non-CUDA deployment fails immediately.

3.2 Start the CPU dependency before target verification

begin validates the verify tensor, rotates to the next buffer slot, and enqueues a nonblocking device-to-host copy of the draft tokens. It records an event instead of waiting on the host, leaving the CPU copy and any prior slot reuse to overlap the target forward.

The DSpark worker calls this immediately after constructing verify_ids_2d, then launches target verification. See the worker’s verify sequence.

3.3 Advance the previous FSM exactly once, then traverse this draft

finish first invokes the scheduler grammar barrier. That barrier resolves the prior batch’s asynchronously copied accepted tokens and advances each request FSM before the new tree is traversed.

The implementation then:

1. waits for this step’s draft-token D2H event;
2. resolves only the requests carrying active XGrammar matchers;
3. supplies per-request root positions for reasoning transitions;
4. calls the native batched draft-tree traversal once;
5. enqueues the resulting mask H2D copy; and
6. makes the current compute stream wait on mask readiness before logits are filtered.

The result processor’s advance_grammar_fsm is the single FSM-advance authority. Its idempotence marker permits either eager advancement in the overlap barrier or later advancement in a non-overlap result path, never both. When a grammar terminates within an accepted speculative run, the retained-prefix logic drops the suffix before it can be committed to KV or emitted.

The worker’s early result copy uses a dedicated CUDA stream plus one background materialization worker. This moves tolist() and event waiting off the scheduler hot path while retaining timing for queue, copy, and conversion phases.

sequenceDiagram
    participant S as Scheduler
    participant C as Copy stream
    participant T as Target GPU
    participant X as XGrammar CPU pool
    participant A as Acceptance

    S->>C: begin: draft ids D2H
    S->>T: launch target verify
    T-->>S: target verify enqueued
    S->>S: grammar barrier advances prior committed tokens
    C-->>X: pinned draft ids ready
    X->>X: one batched tree traversal
    X->>C: masks H2D
    C-->>T: mask-ready event
    T->>T: apply masks to target logits
    T->>A: accept legal prefix
    A->>C: accepted tokens D2H for next barrier

Loading

Reviewer lens: grammar correctness

• The prior accepted run advances the matcher before the next draft is traversed.
• Reasoning-mode transitions select the correct root within the proposed block.
• Two buffer slots cannot be reused while a GPU consumer still reads one.
• A grammar-terminating token is retained; only the speculative suffix after it is discarded.
• Every missing event, unsupported grammar wrapper, incomplete native traversal, or absent async result is a hard error.

The Python truncation contract is covered by the speculative grammar result tests, and the production gate below exercises varied schemas through the full GPU path.

---

4. Replace fixed-geometry DP2 scheduler coordination with a native exchange

The Python scheduler previously paid distributed-collective overhead to exchange seven small integers between two co-located ranks. This PR adds a purpose-built C11 transport for the exact DSpark geometry, while retaining the general distributed implementation for deployments that do not explicitly enable it.

4.1 The native ABI is intentionally narrow

The Python loader and runtime validator require:

• ABI version 2;
• a unique session identifier;
• a positive timeout;
• metrics enabled;
• the exact DP2/attention-TP1/CP1 geometry; and
• alternate MLP synchronization flags explicitly disabled.

Manager creation and exchange bind each process to one rank and expose separate MLP-info and verify-tier channels. A handle cannot cross a process fork. Any native error becomes a Python exception.

4.2 The C path publishes, waits, and fails as one shared protocol

The shared layout places each rank on separate cache lines and keeps two sequence slots so the writer can progress without overwriting the peer’s current read. Futex helpers and layout validation verify magic, ABI geometry, and lock-free atomics before traffic begins.

sglang_dp2_sync_exchange performs a release-publish of the local payload, wakes the peer, and waits for the matching peer sequence. During the wait it checks:

• a peer-published shared error;
• dead or replaced peer ownership;
• local and peer sequence monotonicity; and
• the absolute timeout.

A fatal condition is published to shared state and wakes both ranks, so one side cannot continue alone. Close and unlink clear ownership, wake a waiting peer, and let the final process unlink the shared-memory object.

4.3 Scheduler decisions remain collective decisions

MLPSyncBatchInfo.all_gather uses the native payload only when the strict feature is enabled. It reduces graph eligibility across both ranks and reconstructs global token counts, forward modes, TBO eligibility, and graph admission. Raw batch preparation treats an idle rank as a real participant with zero local tokens, ensuring asymmetric production traffic does not strand the active rank at the next collective.

Warning

Once SGLANG_DSPARK_DP2_SHM_MLP_SYNC=1, there is no Gloo/NCCL fallback. Missing library, wrong ABI, incompatible geometry, peer death, sequence divergence, or timeout aborts the path. The generic implementation remains available only when this specialized feature was never enabled.

The native test suite compiles the C source with strict warnings, rejects invalid ranks, proves timeout behavior, runs two-process repeated exchange, checks payloads/sequences, and verifies that Python loads the packaged ABI.

---

5. Make full-prefill CUDA graphs a synchronized DP decision

A CUDA graph cannot be chosen independently by two ranks that will later enter the same DSV4/DeepEP collectives. The admission path therefore starts in the scheduler, before either rank pads or executes.

5.1 Agree on eligibility and one token bucket

Local prefill eligibility allows only supported prefill modes and bounds request count for the full backend. Idle ranks are provisionally permissive because DP padding can turn them into valid participants once a peer contributes a prefill.

After the seven-field exchange, shared admission selects the first captured bucket large enough for the global maximum token count and refuses more than 4× padding. Every rank receives the same decision and bucket.

5.2 Capture stable addresses and an eager tail

PrefillCudaGraphRunner.init owns stable token buffers, request-axis metadata, backend selection, TBO eligibility, DSpark auxiliary outputs, and the captured attention-metadata registry.

For breakable and full backends, only the transformer body is captured. The LM head and logits processor remain eager. That boundary avoids capturing a large (request_slots, vocabulary) output and lets the tail consume the live request metadata rather than zero-length padding slots.

can_run_graph checks embeddings, target-verify mode, TBO synchronization, hidden-state mode, inactive ranks, the scheduler’s shared decision, captured token bounds, padding ratio, and per-bucket request slots.

Important

After DP admission says “graph,” a rank-local rejection is not allowed to route just that rank eager. _reject_graph raises because graph/eager divergence across ranks can deadlock collectives or silently corrupt the batch.

5.3 Use zero sentinels to make request geometry capture-stable

Capture preparation places all captured tokens in request slot 0 and fills the remaining fixed request axis with zero-length sentinels. At replay, load_batch copies the live batch into stable storage, normalizes mixed prefill to the captured extend mode, and explicitly clears the unused request-slot tail so stale lengths from a prior replay cannot become work.

Recursive output trimming preserves tuples and lists while slicing only tensors whose first dimension is the captured token bucket. Body replay temporarily replaces the inner layer forward with graph replay, then runs the outer model and eager logits tail against current request metadata.

shared DP decision
      │
      ├── not graph-safe ──► both ranks eager
      │
      └── graph-safe
             │
             ├── pad tokens to shared captured bucket
             ├── pad request axis with zero-length sentinels
             ├── refresh stable metadata/buffers
             ├── replay captured transformer body
             └── trim token outputs + run eager logits tail

The focused graph tests cover padding-factor admission, per-bucket request slots, and nested output trimming.

---

6. Make DSV4 attention, TBO, and sparse prefill graph/lifetime-safe

The graph runner provides stable top-level inputs; DSV4 attention must provide equally stable internal metadata and workspaces.

6.1 Refresh captured metadata in place

DSV4’s captured metadata objects retain ownership of tensors whose addresses are embedded in a graph. The breakable-graph hooks build one capture object and refresh it in place for replay, preventing temporary Python objects from freeing graph-referenced storage.

The prefill planner adds explicit output bounds checks and handles sentinel-heavy graph batches in the device planning path. Invalid plan geometry traps instead of writing beyond its token allocation.

6.2 Preserve TBO’s fixed token boundary and request axis

TBO metadata dispatch refreshes each child backend from the parent replay view. The prefill split divides live request lengths at a fixed flat-token boundary; a request with no tokens in a child becomes a true zero sentinel. Child replay views preserve the captured request axis while giving each child only its logical token range.

Capture and replay preparation in the TBO plugin use the same split and synchronize the feature gate across DP ranks. The dedicated TBO tests check fixed-boundary layouts, request geometry, zero sentinels, and DP/phase gates.

6.3 Reserve sparse-prefill outputs before allocator fragmentation

Sparse prefill remains an eager path for shapes outside dense graph coverage, but it no longer allocates its largest outputs on every layer call. SparsePrefillOutputWorkspace reserves the maximum output, max-logit, and LSE tensors before graph capture and returns bounded slices. This removes the need to find a fresh contiguous allocation—roughly 1 GiB for a full TP2 chunk—after variable decode traffic fragments the CUDA allocator.

The DSV4 backend creates that workspace once and the sparse forward path reuses both KV scratch and output tensors. Dequantization writes directly into the combined workspace, avoiding torch.cat. The SGL kernel wrapper’s caller-owned output contract requires all three outputs together and invokes the new into-variant.

Finally, Q normalization/RoPE aliases input and output safely, and the model uses that in-place path to remove another eager-prefill allocation without changing arithmetic.

Reviewer lens: CUDA lifetime and accuracy

• Every pointer captured by CUDA remains owned for the graph’s entire lifetime.
• Replay mutates tensor contents, not captured addresses or shapes.
• Sentinel requests carry zero logical work and cannot reuse stale metadata.
• Planner output bounds are checked on every write-producing path.
• Sparse workspace slices reject capacity or geometry changes.
• The eager tail observes only raw requests and trimmed token-axis outputs.
• In-place Q normalization/RoPE is safe because each vector is fully loaded before the first store.

The GPU accuracy table below validates dense/sparse plans, sparse outputs, in-place Q normalization/RoPE, DSpark auxiliary outputs, and captured-versus-eager results.

---

7. Give HiCache, health checks, VMM, and shutdown explicit ownership

These changes turn several “best effort” operational edges into auditable contracts.

HiCache control and teardown

The one-shot HiCache gate is configured at scheduler initialization. The internal-state handler accepts an eviction only when the control is armed, the value is positive, HiCache is active, and the scheduler is idle. It consumes the arm before execution and uses the existing write-back eviction path so device leaves are preserved on host first.

The original shutdown failure is fixed by giving LogicalHostPool.destroy() the same ownership interface expected by HostPoolGroup.destroy(). Unified radix cache release can therefore tear down heterogeneous host-pool groups without an attribute error interrupting the distributed cleanup.

Non-inference scheduler health

/health_scheduler does not submit a token-generation request. It requires one fresh load snapshot for every expected DP rank and returns 503 during startup, shutdown, read failure, missing/duplicate ranks, stale snapshots, or implausible future timestamps. This detects a dead scheduler without adding inference traffic or depending on a model forward.

Ordered process and distributed cleanup

The tokenizer SIGTERM path drains active requests, stops the subprocess watchdog before expected child exits, sends a shutdown request, and waits for every tracked child to exit cleanly.

Each scheduler forwards shutdown to the detokenizer, exits its loop, releases host resources, reaches a CPU-group barrier, and only then destroys distributed groups in the scheduler finally. This prevents rank 0 from closing TCPStore while rank 1’s NCCL heartbeat is still alive. The watchdog’s clean-exit API distinguishes an expected zero exit from a crash or timeout.

Strict same-host VMM transport

POSIX file-descriptor export and SCM_RIGHTS exchange/import make the same-host VMM path explicit. Invalid spans, missing descriptors, duplicate metadata, peer failure, or mapping errors raise. When this path is selected, an export failure does not quietly change the memory-sharing mechanism.

Note

The shutdown fix addresses both the primary LogicalHostPool.destroy exception and the secondary NCCL/TCPStore noise it caused. The absence of that noise is a consequence of completing cleanup in rank order, not log suppression.

---

8. Make every optimized path observable per rank

The metrics are not decorative; they are how an operator proves the intended path is active under real mixed traffic.

Scheduler metric definitions add accumulated phase time, call counts, and rolling maxima. Grammar, DSpark, and prefill definitions distinguish memory/in-flight/disk/local-compile/new-compile resolutions, persistent-cache phases, DSpark grammar outcomes, CUDA graph admissions, graph shapes, and useful versus executed tokens.

The corresponding emitters live in:

• scheduler phase and prefill execution methods;
• grammar cache and DSpark pipeline methods;
• HiCache backup, scheduler-phase, and pending-operation methods; and
• the scheduler’s timed loop phases.

The native DP2 wrapper separately records total exchange, peer wait, arrival skew, and post-latest-arrival timing in _SyncMetrics.

Tip

Read these as per-rank series. Sum counters/rates across DP ranks for whole-endpoint throughput; compare rank-labelled gauges and histograms to diagnose imbalance. The expected aggregation is documented in the production metrics reference.

The runtime metrics test uses an isolated Prometheus registry and verifies exact metric names, labels, and values for scheduler, prefill, grammar, and HiCache signals.

---

9. Finish with executable contracts and operator documentation

The change is large, but the focused tests line up cleanly with its risk surfaces:

Risk surface	Executable contract
Grammar canonicalization, duplicate compile coalescing, completed-future deadlock, byte/entry LRU	Base grammar backend tests
Persistent cache round trip, corruption rejection, adaptive marker checksum, concurrent ledger recovery	Persistent XGrammar cache tests
Atomic same-rank batches versus mixed-rank dispatch	DP routing tests
Native ABI, invalid rank, fatal timeout, two-process exchange	DP2 native tests
Scheduler presence, freshness, skew, and route registration	Health endpoint tests
Speculative grammar truncation at termination	Result processor tests
Full-prefill padding, request-slot bounds, nested output trimming	Prefill graph tests
TBO fixed split, child geometry, sentinels, synchronized gates	DSV4 TBO tests
Logical host-pool group shutdown	HiCache teardown regression
Child success, crash, timeout, and watchdog validation	Watchdog tests
Metric names, labels, counters, gauges, and histograms	Runtime metric test

For operators and integrators, the patch also documents:

• DSpark prerequisites, checkpoint geometry, and constrained decoding;
• grammar, native DP2, HiCache, and prefill validation environment controls;
• the non-inference scheduler health endpoint; and
• per-rank production metric aggregation.

Suggested final review checklist

• A uniform explicit route remains one batch through tokenizer and DP controller.
• Grammar prototypes are bounded and copied before request-local mutation.
• One normalized key creates at most one in-flight compile per process.
• Persistent cache scope includes every compatibility dimension.
• Corrupt cache entries, absent XGrammar APIs, and invalid DSpark geometry fail visibly.
• The grammar FSM advances exactly once before the next draft-tree traversal.
• A grammar-terminating speculative suffix is neither cached nor emitted.
• Native DP2 exchange is used only for DP2/TP1/CP1 and never falls back after enablement.
• Both ranks agree on graph admission, token bucket, and TBO participation.
• Captured pointers remain owned and sentinel metadata is cleared on every replay.
• Sparse workspace and planner bounds are enforced before writes.
• HiCache gate operations are armed, one-shot, idle-only, and write-back preserving.
• Health checks cover every DP rank without inference.
• Watchdog, detokenizer, host pools, and distributed groups shut down in dependency order.
• Metrics prove which path ran and retain a DP-rank label where aggregation matters.
• Focused unit/native tests and the GPU gates below cover correctness, pressure, and recovery.

Note

The accuracy, performance, soak, and current-main verification sections below are the evidence for this implementation. The generated Cursor summary remains at the bottom as an independent high-level roadmap.

Relationship to open upstream work

Note

2026-07-27 audit of all 3,907 open sgl-project/sglang PRs, using text and changed-path searches. Relevant large PRs were expanded; shared files alone were not counted.

Executive takeaway

No open PR duplicates this integration or its shm/futex DP2 exchange, persistent XGrammar cache, DSpark batch-grammar pipeline, atomic routed batch, or /health_scheduler.

Area	Open upstream PRs	Compare / reviewer action
Grammar service	sgl-project#32085, sgl-project#22196, sgl-project#16874	Direct subsets: cgroup-aware workers, entry LRU, and TP0 compilation. This draft additionally canonicalizes/deduplicates work, bounds bytes and entries, and persists XGrammar. Combine sgl-project#32085's sizing with these semantics.
Constrained speculation	sgl-project#15465, sgl-project#31728, sgl-project#25361, sgl-project#26508, sgl-project#27165, sgl-project#28680, sgl-project#28943, sgl-project#30155, sgl-project#31534, sgl-project#32509	Complementary: scalar traversal, rollback/stop-token fixes, FSM/output trimming, and DFlash/EAGLE masks. This draft adds native batch traversal and a prior-result FSM barrier for DSpark.
DP sync / routing	sgl-project#23011, sgl-project#26016, sgl-project#32209, sgl-project#32187, sgl-project#31706, sgl-project#26186, sgl-project#26612, sgl-project#31170	sgl-project#23011 optimizes the general collective; this draft replaces it only for strict DP2 and fails closed. Others fix padding/health or choose a rank; this draft atomically transports that rank's batch.
DSV4 prefill graphs	sgl-project#30420, sgl-project#29985, sgl-project#30825, sgl-project#30206, sgl-project#31686, sgl-project#31698, sgl-project#31100, sgl-project#32477, sgl-project#31689	Strong shared surface: capture, cached-prefix/multi-request shapes, WAR events, DP extents, and padding writes. This draft adds DP-agreed buckets, fixed request axes, eager logits, and DSV4/TBO refresh.
DSpark / TBO / kernels	sgl-project#30720, sgl-project#31422, sgl-project#32374, sgl-project#31432, sgl-project#28842, sgl-project#32194, sgl-project#32319, sgl-project#32526, sgl-project#32183, sgl-project#32467	Mostly complementary verify, host-sync, TBO, metadata, and planner work. sgl-project#30720 is the largest conflict (22 files), but is GLM-5.2/ROCm versus DSV4/Blackwell DP2 here. Preserve sgl-project#32183/sgl-project#32467 first.
Sparse prefill	sgl-project#27276, sgl-project#25400, sgl-project#31888, sgl-project#23741	Upstream fixes CP/ragged/kernel geometry; this draft adds preallocated outputs, caller-owned FlashMLA buffers, and in-place norm/RoPE.
HiCache	sgl-project#31715, sgl-project#31668, sgl-project#31713, sgl-project#31887, sgl-project#32214, sgl-project#30393, sgl-project#32388, sgl-project#31883, sgl-project#31884, sgl-project#28507, sgl-project#29859, sgl-project#26649	Complementary, with metric conflicts: eviction, sidecar/SWA/L3/Mooncake ownership, and tier metrics. This draft adds idle-only control, LogicalHostPool.destroy, teardown, and path counters. Consolidate metric names.
Lifecycle / load	sgl-project#16484, sgl-project#24124, sgl-project#28599, sgl-project#32523	Closest shutdown, drain, and load-publication work. This draft orders process-group teardown and validates every DP snapshot; compose around one owner/schema.

Rebase correctness work first, consolidate grammar/metric contracts, then rebase graph/TBO work before the DP2 and DSpark fast paths; rerun all included gates with fast paths asserted active.

Modifications

Constrained decoding and grammar compilation

• Canonicalize grammar cache keys and replace the unbounded cache with byte- and entry-bounded LRU accounting.
• Coalesce concurrent compilation of an identical grammar into one in-flight future.
• Avoid a callback-under-lock deadlock when a compilation future is already complete.
• Add a process-safe persistent XGrammar cache with scoped keys, atomic publication, checksums, corruption rejection, size accounting, and adaptive deserialize-versus-local-compile selection.
• Add a batched DSpark grammar pipeline with native draft-tree traversal, double-buffered masks, asynchronous host-to-device transfer, target-verification overlap, and fail-closed invariants.

DSpark and DP2 scheduling

• Keep explicitly routed HTTP batches atomic through tokenizer and DP-controller dispatch.
• Use compact ragged verification and synchronize graph-safe verify tiers between DP ranks.
• Add a C11 POSIX shared-memory/futex exchange for the fixed co-located DP2 scheduler geometry, with ABI validation, stale-peer detection, timeouts, cleanup, and detailed timing metrics.
• Keep idle ranks participating in required collectives so asymmetric traffic cannot strand the active rank.

DeepSeek V4 prefill and kernels

• Add full-prefill CUDA graph admission, request-slot padding, output lifetime management, nested-output trimming, and eager-reference validation.
• Capture DeepSeek V4 prefill TBO and DSpark auxiliary work in the graph-safe path.
• Reuse stable sparse-prefill workspaces and extend the pinned FlashMLA sparse-prefill API to write into a caller-owned output tensor.
• Perform Q normalization/RoPE in place, preserve dynamic KV scale handling, and add sentinel/padding checks around captured batches.

HiCache, health, and lifecycle

• Add LogicalHostPool.destroy() and make host-pool-group teardown safe, fixing the shutdown exception that otherwise interrupts distributed cleanup.
• Track HiCache backup, pending-operation, and scheduler-phase metrics.
• Add an explicit one-shot, idle-only HiCache gate control.
• Add /health_scheduler, a non-inference health check that verifies each DP rank is present and publishing a fresh scheduler snapshot.
• Improve controller, scheduler, process-group, and watchdog shutdown ordering.
• Freeze the warmed scheduler object graph into Python's permanent GC generation.
• Make explicitly enabled POSIX VMM initialization fail closed.

Observability, documentation, and tests

• Add per-rank scheduler-phase, prefill-graph, grammar-cache, DSpark-grammar, HiCache, and native-DP2 metrics.
• Document DSpark launch requirements, scheduler health, all new environment controls, and correct per-rank metric aggregation.
• Add regression coverage for grammar deduplication/deadlock/LRU behavior, persistent-cache integrity, atomic DP routing, scheduler health, native DP2 synchronization, prefill TBO/graph padding, HiCache teardown, runtime metrics, and watchdog cleanup.

Accuracy Tests

The GPU results below were collected from the same patch set on its deployment-base integration commit using a 2x NVIDIA B300 DP2 configuration. The current upstream/main port has additionally passed the current-main unit, native, formatting, and documentation checks listed below. An exact-current-commit GPU replay remains before this draft is marked ready.

Validation	Result
GSM8K gate	1,319 examples, 96.74% accuracy, 0 errors, 0 truncations; gate threshold 96%
Structured-output gate	64 varied schemas; all outputs parsed and matched canonical JSON
Cross-request isolation	32 concurrent long-context seeded request groups plus 16 probes; no cross-request context contamination, including HiCache host round trips
Prefill CUDA correctness	Dense/sparse prefill plan, sparse output, in-place Q norm/RoPE, and DSpark shared-prefill output matched eager references
Mixed-load stability	12 decode holders per rank plus 16K-token prefill; minimum free device memory 9.4576 GiB

Speed Tests and Profiling

Production-shaped replay, identical request distribution and correctness gates:

Metric	Control	Optimized	Change
Output throughput	1,476.81 tok/s	1,967.96 tok/s	+33.26%
Request throughput	28.08 req/s	37.47 req/s	+33.44%
Request latency p50	—	—	-17.92%
Request latency p95	—	—	-15.37%
Request latency p99	—	—	-20.05%
Energy per request	—	—	-16.33%

Additional measurements:

• Full-prefill graph coverage: 86.57% versus 0% for the control; dense graph coverage through 11,616 tokens, with sparse/eager execution at 11,648 and 16,384 tokens.
• Structured-output throughput gate: 10,946.54 output tok/s.
• Continuous soak: 2h03m41s, 260 samples, 0 violations, no restart/OOM/fatal, maximum 61 active requests per rank, and both queues drained.
• Native DP2 exchange, 20,000 iterations: p50 2.89 us, p99 11.59 us, post-latest-arrival p50 2.89 us, post-latest-arrival p99 7.11 us, mean 5.58 us.

Current upstream/main verification

• pre-commit over all 73 changed files: passed, including Ruff, Black, isort, codespell, ClangFormat, secret detection, and CI registry validation.
• Focused new/changed regression suite: 79 passed, 1 skipped, 24 subtests passed.
• Broader constrained-decoding and non-GPU DSpark compatibility suite: 191 passed, 56 skipped, 21 subtests passed.
• Native DP2 tests compile the C implementation with strict warnings and exercise ABI checks, invalid ranks, fatal timeout behavior, and two-process repeated exchange.
• python3 -m compileall and git diff --check: passed.
• Mintlify mint validate: passed.
• Mintlify mint broken-links: no broken links.

Remaining before ready for review

• Finish the linked companion XGrammar change, then update the SGLang dependency pin to its final revision.
• Repeat the GPU correctness, accuracy, and production-shaped replay on the exact current-main PR commit.
• Split the integration draft if maintainers prefer separate grammar, DP2, prefill, and lifecycle PRs.

Checklist

• Format code with the repository pre-commit hooks.
• Add unit tests for new behavior and regressions.
• Update relevant documentation.
• Provide accuracy and speed benchmark results.
• Follow the SGLang code style guidance.

Review and Merge Process

This remains a draft until the companion dependency and exact-commit GPU replay are complete. No merge or CI escalation is requested yet.

---

Note

High Risk
Touches distributed DP2 synchronization, full-prefill CUDA graph capture/replay, hierarchical cache eviction, and grammar compilation caches—any invariant bug can cause silent wrong tokens, stuck ranks, or startup failure. Companion XGrammar API is mandatory for DSpark structured output.

Overview

Title

Harden DeepSeek V4 DSpark DP2 serving: grammar, prefill graphs, native sync, and lifecycle

Intent

This draft integrates correctness and throughput work for DeepSeek V4 Flash + DSpark on co-located DP2 with HiCache, structured output, and mixed prefill/decode traffic. It moves grammar work off the hot scheduler path, replaces NCCL-heavy DP coordination where geometry is fixed, admits full-prefill CUDA graphs for graph-safe shapes, and tightens shutdown and observability. Enabled fast paths are fail-closed (startup or request failure instead of silent fallback).

Details

• Grammar: Byte/entry-bounded LRU, normalized keys, in-flight compile dedup, persistent on-disk XGrammar cache with adaptive compile vs deserialize; DSpark batched draft-tree traversal with overlapped mask transfer (requires companion XGrammar batch_traverse_draft_tree).
• DP / routing: Atomically deliver explicitly routed HTTP batches to one DP rank; optional native shared-memory/futex scheduler exchange for DP2 + attention TP1 + CP1 (SGLANG_DSPARK_DP2_SHM_*).
• DSV4 prefill: Full-prefill graph bucket keyed by (batch_size, num_tokens); TBO child padding for fixed request axis; sparse-prefill fixed output workspace and caller-owned FlashMLA outputs; metadata lifetime pinned for graph capture; prefill planner bounds checks and MTP-uniform fast-path guards in JIT kernels; in-place Q norm/RoPE alias safety.
• DSpark scheduling: Overlap grammar barrier via async prefetch; exclude one-token logprob prefill from premature decode merge; validate_dflash_request generalized for DSpark overlap rules.
• HiCache / ops: One-shot idle-only device eviction via SGLANG_HICACHE_GATE_CONTROL + /set_internal_state; backup and scheduler-phase metrics.
• Health / shutdown: /health_scheduler (fresh per-DP load snapshots, no inference); graceful shutdown through controller, detokenizer, scheduler with ordered dist teardown; controller raises on non-zero scheduler exit; strict POSIX VMM fd export for same-host DP.
• Docs / metrics: DSpark section in speculative decoding docs; env vars for grammar, DP2 shm, prefill eager validation; production metrics for scheduler phases, prefill graphs, grammar cache, DSpark grammar, HiCache, DP2 sync.

Roadmap for Reviewers

1. xgrammar_persistent_cache.py + base_grammar_backend.py + xgrammar_backend.py — Cache LRU, inflight futures, disk cache fail-closed behavior (core correctness for constrained decoding).
2. dspark_grammar_pipeline.py (if in branch beyond diff snippet) + batch_result_processor.py grammar barrier — DSpark overlap path vs legacy copy_done + tolist.
3. single_node_dp2_sync.py + dp_attn_adapter.py — Native DP2 exchange geometry and fallback policy.
4. deepseek_v4_backend.py + tbo_backend.py + two_batch_overlap.py — Prefill graph capture/replay, TBO padding, child metadata refresh (largest graph-risk surface).
5. c_plan.cuh + sparse_prefill_utils.py — Planner bounds and fixed sparse output buffers.
6. scheduler.py — Phase metrics, HiCache gate, shutdown barrier, DSpark batch merge exclusions.
7. http_server.py + data_parallel_controller.py — Health and atomic routed batches.
8. Docs (speculative_decoding.mdx, environment_variables.mdx, production_metrics.mdx) — Skim for operator accuracy.

Diagrams

  HTTP batch (routed_dp_rank=N)
       |
       v
  DP Controller --[single ZMQ msg]--> Scheduler N  (atomic batch)
       |
  DP2 DSpark (optional SGLANG_DSPARK_DP2_SHM_MLP_SYNC)
       |
  Rank0 <---- POSIX shm/futex ----> Rank1
       |                              |
  draft propose + grammar batch       verify tier sync
       |                              |
       v                              v
  Target verify (CUDA graph or eager prefill bucket)

^{Reviewed by Cursor Bugbot for commit 18cc7bf. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 18cc7bf. Configure here.}

[cursor]

Idle ranks veto DP prefill TBO

High Severity

prepare_all_gather now sets local_can_run_tbo=False for every non-is_extend_without_speculative batch, including IDLE. DP aggregation takes the min of those votes, so an idle peer vetoes TBO on the active rank. Asymmetric DP2 prefill therefore loses TBO even though the new unit test expects idle peers to vote true.

Additional Locations (1)

• test/registered/unit/batch_overlap/test_deepseek_v4_prefill_tbo.py#L175-L181

 

^{Reviewed by Cursor Bugbot for commit 18cc7bf. Configure here.}

[cursor]

Decode CUDA graph TBO disabled

Medium Severity

capture_one_batch_size now returns immediately unless the mode is extend-without-speculative. Decode CUDA graph capture still calls this hook whenever --enable-two-batch-overlap is set, so decode graphs are captured on the non-TBO path for every model, not only DeepSeek V4.

Additional Locations (1)

• python/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py#L945-L946

 

^{Reviewed by Cursor Bugbot for commit 18cc7bf. Configure here.}

[cursor]

Grammar keys not fully canonical

Low Severity

_normalize_cache_key re-serializes JSON with compact separators but omits sort_keys=True. Equivalent schemas that only differ in key order still miss both the in-process LRU and the persistent XGrammar cache, so identical grammars can recompile.

 

^{Reviewed by Cursor Bugbot for commit 18cc7bf. Configure here.}