Harden DSpark and add SM120 PCIe serving support

[voipmonitor]

Final release follow-up (2026-07-12, adf15cadb)

The synchronized release sweep completed all 40 TP2/TP4 x backend x MTP/DSpark cases with valid decode, coding, and 8k/64k/128k prefill artifacts. Standard MTP0 remained at or above the v9 baseline across the latency-sensitive C1 cells.

The final follow-up changes only the environment launcher: TP4 lucifer-cutlass + dspark could leave 777 MiB free while the first real FlashInfer MoE prefill requested a 764 MiB transient workspace. Its default GPU_MEMORY_UTILIZATION is now 0.94 instead of 0.9465. The exact final image profiled 1,523,880 KV tokens and completed C1/C64 plus 128k prefill without OOM.

Final image: voipmonitor/vllm:fathomless-firmament-ds4-v10-vllmadf15ca-b12x90172a5-fi2cba2f7-cu132-20260712
Digest: sha256:4f07aefe2aa15f66d5fd90580eaa2553926aea0c3fa46bc227610e90c668c9f0

Final clean-cache correction (2026-07-12, 5dd080a70)

This section supersedes the earlier event-lifetime conclusion below while retaining the exploration history.

Fresh, isolated compilation caches showed that the apparent event/stream fixes were warm-cache false positives:

• the clean immutable 07f877827 image completed C1 and then hung at C64;
• fresh-event, completion-retired, inner-indexer-off, outer-indexer-off, main-compressor-off, and all-post-GEMM-off variants also hung at C64;
• both TP workers stopped in CapacityBasedVerificationManager._flush_draft_token_capacity_copy() while their GPUs remained at 100% utilization;
• capacity-off, fixed-shape mask verification, and the debug TP consistency barrier did not hang;
• forcing compact varlen dispatch to PIECEWISE graphs still hung, ruling out FULL-graph replay and the attention backend as the primary cause.

The failure was the non-canonical TP control input for compact verification. Each TP rank independently read back its draft-capacity tensor and selected a physical verifier graph. Any rank-local difference could therefore select different token shapes and eventually deadlock a shape-sensitive TP collective.

The production fix broadcasts the small active capacity vector from TP rank 0 before the existing asynchronous D2H copy. Every rank now trims the same requests and selects the same physical graph. This keeps all current pre/post-GEMM auxiliary streams, FULL CUDA graphs, Lucifer/B12X backends, and dynamic K behavior enabled; it adds no global CUDA synchronization and does not serialize attention.

Clean-cache validation on TP=2 Lucifer CUTLASS + B12X indexer/all-reduce:

Test	Result
C1 after cold capture	245.2 tok/s
C64 after the C1 graph transition	2700.5 tok/s
recovery C1	241.3 tok/s
120-second C64 soak	2751.4 tok/s

The soak ended with running=0, waiting=0, and error=0; capacity telemetry showed compact batches continuing on cg=FULL. The complete capacity test module passed in the pinned runtime image (22 passed), and Ruff format/check plus git diff --check passed.

The communication investigation remains unchanged: NCCL kernels account for roughly 16% of the observed C64 GPU interval before overlap. Pair-local P2P reached 56.4 GB/s, NCCL 39.47 GB/s, and B12X DMA 48.67 GB/s, but the end-to-end B12X-DMA gain was only about 1.4% in the three-run C128 comparison. Communication tuning is therefore secondary to the now-fixed TP graph-shape correctness issue.

Summary

This PR consolidates the DeepSeek-V4-Flash-DSpark work developed and validated on RTX 6000 Pro Blackwell (SM120), rebased onto dev/fathomless-firmament. It combines correctness hardening, optional DSpark experiments, the SM120 PCIe serving stack from upstream vLLM PR vllm-project#47979, and an environment-driven launcher that makes the tested configurations reproducible.

The production default remains conservative: DSpark uses probabilistic fixed-depth K=5 verification. Capacity-aware verification, dynamic physical draft depth, the rowwise-FP8 draft head, alternative all-reduce paths, and SP/async-TP are opt-in.

Latest validated follow-up (2026-07-12)

Commits e24d8ae through 07f8778 complete the profiler-driven DSpark performance and correctness pass. This section supersedes the earlier exploratory capacity and performance numbers below.

What changed

• Fixed-width draft paths no longer synchronize the host to recover draft IDs.
• Variable-length draft IDs piggyback on the asynchronous model output and use one final D2H event per step.
• Safe post-GEMM auxiliary-stream overlap is restored with explicit producer/consumer ordering.
• Buffered InstantTensor is the launcher default.
• CUDA graph sizes are capped at the physical verifier width.
• VLLM_DSPARK_CAPACITY_ACTIVATION_BATCH_SIZE gates compact verification at the profiled load knee.
• C1-C16 replay exact uniform K5 target graphs without confidence, STS, capacity allocation, or capacity readback work.
• C32+ uses compact varlen verification and load-aware physical draft depth.
• Exact uniform graph descriptors use a separate dispatch map. This prevents their intermediate token counts from splitting generic varlen buckets; after the fix, every observed C32/C64 compact step used a FULL graph and no cg=NONE fallback remained.
• B12X graph-padding rows receive a safe non-zero indexer sequence-length sentinel.
• Capacity-copy flush ordering now preserves pending updates on no-draft steps without reintroducing low-load synchronization.
• Post-GEMM attention overlap remains enabled. A vLLM capture scope now covers descriptor prewarms and the actual CUDA graph capture, retaining every startup event generation that may be referenced by a later graph artifact. Runtime eager invocations receive fresh, non-recycled event handles without Python-side retention. This fixes the fresh-process K5 -> K3 replay hang without adding a per-step synchronization or serializing the attention branches.
• Global and DCP top-k mapping sanitize stale CUDA-graph padding request slots before addressing block tables; padding-row lengths remain zero and inert.

The TP=2 auto-profiled saturation budget was 160 draft tokens, corresponding to an activation knee of 32 requests at K=5. At C64 the controller selects physical K=3 and typically reduces 320 possible draft rows to roughly 95-105 retained rows plus 64 bonus rows.

Final TP=2 ctx0 decode

Path	C1	C8	C16	C32	C64
fixed K5 Lucifer baseline	237.2	770.9	1186.1	1764.6	2614.6
always-on compact varlen	193.7	739.3	1149.6	1856.4	2708.9
load-aware compact varlen	239.5	764.5	1180.1	1799.6	2718.2

Two additional 30-second C64 runs measured 2719.1 and 2738.0 tok/s. Three-run mean is 2725.1 tok/s with sample SD 11.2. Five coding runs measured median 323.0, mean 327.5, and max 359.2 tok/s, with zero CJK runs.

PCIe/NCCL conclusion

Pair-local P2P measured 56.4 GB/s. On the exact 6 MiB serving tensor, NCCL measured 39.47 GB/s and B12X DMA 48.67 GB/s. Despite the 23.3% isolated collective gain, B12X DMA improved end-to-end C128 by only 0.72%. Forcing 16 NCCL channels improved C64 by 1.71% while costing about 2.7% KV capacity. Torch traces identify target MoE verification, not NCCL, as the dominant cost, so both remain optional diagnostics.

Additional validation

• focused event, capacity, and CUDA padding tests: 30 passed
• Ruff, compileall, and git diff --check passed
• controlled same-cache C1 -> C64 A/B reproduced the hang with shape-keyed, completion-retired, ephemeral-only, and actual-capture-only lifetime variants
• the final outer capture scope completed C1 -> C64 -> C1 at 248.1/2727.5/245.3 tok/s; a 120-second C64 soak measured 2743.4 tok/s with a healthy server and no CUDA/NCCL errors
• three independent 30-second C64 runs
• five coding runs, zero CJK output
• 22k/28k eviction marker tests passed before and after eviction
• 1024-token long code output had zero replacement and zero CJK characters

What changed

DSpark correctness and determinism

• Decouple the DSpark draft Gumbel stream from target acceptance/recovery randomness and preserve a deterministic per-request sampling state across padded replays.
• Harden DSpark metadata and TP sampling-state handling, including stale request-slot writes.
• Mask cache-restored tokens out of the DFlash/DSpark draft context.
• Skip scheduler lookahead allocation for running prefill chunks.
• Preserve the DeepSeek-V4 SWA MLA kv_quant_mode through merge/unify.
• Fix DCP=1 sparse-MLA verifier causal lengths.
• Add the missing B12X auxiliary-stream ordering required by DSpark verification.
• Retain CUDA-graph backbone outputs for the lifetime of graph replay instead of allowing their storage to be reused.
• Keep non-causal draft attention off unsupported FlashInfer full-graph paths and prefer FlashAttention on SM100f where required.

These changes address the classes of failures seen during long-context and full-KV-cache testing: stale draft slots, nondeterministic padded-capacity state, races between the B12X auxiliary stream and the verifier, and graph outputs whose lifetime ended before downstream consumers completed. Review follow-up also makes capacity compaction out-of-place, fails closed for block-unaligned restored prefixes, validates that confidence-head weights really loaded, and hardens finite-value/configuration checks.

Capacity-aware and variable-length verification

• Add reusable speculative-capacity bookkeeping and DSpark-specific capacity state.
• Support padded-capacity allocation with per-request logical lengths.
• Keep variable-length verification on full CUDA graphs when the selected attention/indexer path supports it.
• Add online SPS/STS profiling and load-aware physical-depth control.
• Add configuration, warmup, telemetry, and focused tests for the new paths.

Masked capacity verification remains non-default because CUTLASS still executes the padded target computation. Compact varlen verification now has a validated load-aware mode: exact fixed-width K5 graphs are used below the activation knee, while C32+ uses physically compact target batches.

SM120 post-GEMM stream safety

The first correctness pass serialized FlashInfer SM120 post-GEMM indexer and
compressor work. That diagnostic removed long-prefill illegal accesses and
proved that the original implementation had a missing producer/consumer edge,
but it also cost roughly 13% at C1 and several percent at C64.

The final implementation keeps the overlap. GEMM fan-out and post-GEMM work use
separate event sets. Three progressively stricter A/B tests were required:

1. Capture-only event isolation was insufficient because DS4 attention can run
   eagerly between CUDA graph segments.
2. Retaining one eager event set per physical token shape appeared to pass with
   one warm AOT artifact, but a fresh-cache C1 -> C64 run reproduced the hang
   (253.9 tok/s followed by 0.0).
3. Reusing a shape generation only after a non-blocking completion event also
   failed (251.4 followed by 0.0). Completion therefore does not imply that an
   event handle is no longer owned by a captured graph artifact.

A controlled same-cache run with non-recycled event handles completed at
239.2/2745.2 tok/s, proving that permanent event-handle reuse was the trigger.
The first production attempt retained events only when CUDA reported the current
stream as capturing and released other eager wrappers after enqueue. That passed
as an overlay but a clean immutable image changed the AOT schedule and reproduced
the C64 hang at 254.9/0.0 tok/s. Adding scopes only around the explicit
torch.cuda.graph blocks also failed at 250.9/0.0.

The missing lifetime begins earlier: CudaGraphManager.capture() runs descriptor
prewarm forwards before each capture, and multi-stream custom ops may create event
handles there that later graph artifacts retain. The production fix therefore
marks the entire manager capture phase, including prewarms, with a context-local
vLLM capture scope. Standard, breakable, and direct FULL capture blocks also carry
the scope for correctness outside the normal manager. Captured/prewarm event
wrappers are retained with their owning modules. Runtime eager wrappers remain
private to one invocation and are released after enqueue; pending
cudaEventDestroy is asynchronous, matching
torch.cuda.Stream.wait_stream().

A retain-everything diagnostic passed but grew RSS from 16.19 to 16.24 GiB during
a 120-second C64 run, so it was rejected. The bounded outer-scope implementation
completed C1 -> C64 -> C1 at 248.1/2727.5/245.3 tok/s and sustained C64 for
120 seconds at 2743.4 tok/s. The server remained healthy and its log contained no
CUDA/NCCL errors. Runtime code cannot append to the retained capture-event list
after the outer scope exits.

Global synchronization remains rejected. The fix adds no steady-state barrier
and does not disable parallel attention. B12X large-tensor DMA also remains
explicit opt-in: its standalone collective is faster, but the end-to-end gain
was below 1% and an earlier integrated long-prefill path produced an illegal
memory access.

Standard-path CUDA-graph padding and v15 parity

During final v9/v15 parity validation, standard MTP-off serving exposed a separate correctness bug in the capacity-integration plumbing. CUDA-graph capture initializes the persistent is_padding buffer as padding for the full capture shape. The standard non-speculative path did not clear active rows when VLLM_MOE_SKIP_PADDING=0, yet slot mapping always consumed the buffer. Active tokens consequently received PAD_SLOT_ID, KV writes were skipped, and long generation could become CJK/garbled while short-context throughput was distorted.

The fix now passes an input-padding mask to slot mapping only when speculative/capacity trimming or MoE padding actually consumes it. Those paths refresh the live rows; ordinary decode ignores the stale persistent buffer and avoids an extra fill kernel. In the same final audit:

• variable-length indexer metadata is selected only when DSpark varlen capacity control is genuinely active, not merely because the binary supports the kernel;
• post-GEMM overlap is enabled for non-speculative, causal MTP, and DSpark serving; capture-private event generations keep graph-shape changes from reusing an in-flight event handle;
• stale top-k padding request indices are redirected to a safe block-table row before masked output lengths make those graph-only rows inert.

After every A/B/C server had fully loaded before any client started, three isolated TP=2 A8 C1 runs measured 141.3, 141.6, and 141.5 tok/s; the v15 control was 141.7 tok/s. The current implementation produced normal English/code output and passed six focused padding, indexer, and stream-policy regression tests inside the pinned image. Earlier 132-133 tok/s observations were invalid concurrent-load measurements, not a code regression.

Causal MTP parity follow-up

A synchronized v9 comparison found a standard-MTP regression that MTP-off canaries could not expose: TP=2 B12X A8 MTP2 measured 196.0 tok/s versus the published v9 217.4 tok/s, while acceptance was unchanged (0.6548 versus 0.6559). Re-running the new image through the exact old v9 launcher produced 198.6 tok/s, proving that the loss was in the code path rather than the helper or arguments.

Two safe optimizations were restored without weakening the DSpark guards:

• causal MTP verification may use the now-independent post-GEMM event/aux-stream set; DSpark remains serialized;
• draft hidden-state reconstruction uses b12x_mhc_post when the B12X mHC path is active, while retaining the newer final-state reuse that avoids duplicate reconstruction.

The same TP=2 A8 canary progressed from 196.0 to 205.8 and then 208.9 tok/s; coding median reached 220.6 tok/s. Disabling the new B12X fused all-reduce + RMSNorm path reduced C1 to 201.2 tok/s, so that fusion is beneficial and remains enabled. The remaining small delta is shared with the Fathomless MTP orchestration path and is evaluated by the immutable-image TP=2/TP=4 release sweep rather than hidden in launcher defaults.

Optional rowwise-FP8 draft head

• Add a rowwise-FP8 DSpark draft lm_head implementation and loader integration.
• Keep it behind VLLM_DSPARK_FP8_DRAFT_HEAD=1.
• Add focused loader and numerical tests.

This was retained because it can help bandwidth-bound systems, but on RTX 6000 Pro it was approximately neutral at C1 and only about 1.2% faster at C64. It is therefore not the default.

SM120 PCIe serving stack

This branch includes the rebased changes from upstream vLLM PR vllm-project#47979:

• SM120 sequence-parallel threshold entries and scoped divisibility checks.
• Full FlashInfer native spec-decode verification CUDA graphs.
• PCIe-safe symmetric-memory barriers and stream ordering.
• Two-shot symmetric-memory all-reduce support with topology/runtime guards.
• Opt-in custom all-reduce on PCIe-only topologies, with the two-stage choice stored per communicator instead of leaking through a process-global environment mutation.
• Workspace growth synchronization and fused-op event ordering.

Sequence parallelism and async TP are useful for V1 paths, but DSpark currently requires the V2 model runner and this vLLM revision explicitly rejects SP under V2. The launcher therefore refuses that unsupported combination instead of silently claiming it is active.

The validated Lucifer image also needs FlashInfer PR vllm-project#3871 plus the canonical SM120 DSV4 topk=256 fixes: decode PR vllm-project#3817 and prefill PR vllm-project#3896. DSpark TP=2 exercises both shapes during warmup and serving, so the release image pins all three FlashInfer PRs outside this vLLM diff.

Reproducible DS4 launcher

serve-ds4-flash.sh is now an environment-driven launcher suitable for installation in an image as /usr/local/bin/serve-ds4-flash.sh.

It provides:

• MODE=mtp0|mtp2|mtp3|dspark
• BACKEND=b12x-a16|b12x-a8|b12x-a8-dglin|lucifer-default|lucifer-cutlass
• ALLREDUCE_MODE=b12x|vllm-custom|vllm-custom-2stage|nccl
• INDEXER_BACKEND=auto|b12x|native
• B12X_PCIE_DMA=0|1, default off after the integrated long-prefill failure
• DSPARK_DRAFT_ATTENTION_BACKEND=auto|B12X_MLA_SPARSE|FLASHINFER_MLA_SPARSE_DSV4|FLASHMLA_SPARSE_DSV4 for explicit draft-only experiments
• automatic CUDA graph sizing from MAX_NUM_SEQS, capped so generated capture sizes never exceed the declared graph maximum
• optional explicit graph-capture patterns
• all capacity, dynamic-depth, FP8-draft-head, rejection-sampling, and SPS/STS controls as opt-in environment variables
• explicit environment reset between B12X and Lucifer profiles so inherited variables cannot contaminate a run
• the OpenAI usage, request-ID, prompt-token-detail, memory-profile, and FlashInfer-autotune serving defaults used by the published benchmarks
• DRY_RUN=1 for exact command inspection without starting a server

The launcher does not hard-code GPU IDs. Container orchestration owns GPU placement.

Development history and rejected approaches

The following branches were implemented or measured and are intentionally not production defaults:

• Adaptive/load-aware K: lowering logical K works and preserves graph shapes, but the CUTLASS verifier uses a masked padded target pass. It does not skip the corresponding target compute, so fixed K=5 remained the best result through the intended C1-C128 range.
• Block rejection verification: synthetic verification microbenchmarks improved, but end-to-end DSpark did not; draft/target work and scheduling dominated.
• Old draft-pass compaction (former local PR dspark: draft-pass cleanup (-10% draft time) + upstream rejection-utils consolidation with opt-in block verification #71): it targeted an earlier architecture that no longer exists after the V2/capacity refactor. Its still-relevant safety checks are covered by the current implementation and tests.
• Rowwise-FP8 draft head (former local PR dspark: rowwise-fp8 draft lm_head (VLLM_DSPARK_FP8_DRAFT_HEAD, opt-in) #73): retained as the opt-in implementation in this branch, but not made default because the RTX gain was negligible at low concurrency.
• B12X versus native indexer: true variable-length verification requires the B12X indexer path. Lucifer defaults to its native indexer because forcing the hybrid path was not a general performance win.
• SP/async TP for DSpark: unavailable because DSpark is V2-only and SP is rejected by V2 in this base. The helper exposes SP only for compatible non-DSpark modes.
• Alternative collectives: vLLM custom all-reduce, two-stage custom all-reduce, symmetric-memory, and NCCL paths remain selectable. The validated default is B12X oneshot for small decode tensors plus PyNCCL for large prefill tensors. Automatic B12X DMA was rejected as a default after an integrated DSpark long-prefill illegal-memory-access failure, even though its standalone shape/replay tests passed.
• Global CUDA synchronization, permanent stream serialization, shape-keyed reuse, completion-retired reuse, ephemeral-only lifetime, and retain-everything: each was useful diagnostically. Reuse and ephemeral-only variants still hung under controlled fresh-AOT C1 -> C64 transitions; retain-everything passed but leaked host memory during sustained serving. The production fix keeps overlap, retains events only for the bounded outer capture/prewarm phase, and uses non-recycled ephemeral events at runtime.
• Long-prefill shape/cache hypotheses: classic allocator mode, lower KV allocation, compile endpoint 7,936, scheduled-token reshaping, draft-layer KV removal, no-op speculator/model loading, and disabling auxiliary outputs did not remove the regression. A same-shape MTP-off target stayed near 14k tok/s, narrowing the fault to the spec-config-enabled target stream path.
• Separate Q/K/V, FP32 draft head, fake-FP8 projection input, reference attention, and greedy draft sampling: earlier bring-up experiments either reduced acceptance, reduced throughput, or produced no correctness improvement. The validated path uses fused QKV, the normal head precision, probabilistic draft sampling, and model-config target layers [40, 41, 42].
• Concurrent load and benchmark: starting a benchmark while another model was loading produced invalid 132-133 tok/s outliers. Reproduction scripts must finish loading every server in a wave before any client starts.

Performance context

On the current TP=2 Lucifer/CUTLASS fixed-K5 path, one synchronized validation sweep measured:

C1	C8	C16	C32	C64	coding median
219.15	745.20	1130.37	1677.41	2481.46	286.32

The final Docker/wiki sweep is intentionally kept outside this PR description so published numbers are tied to an immutable image digest and synchronized launch procedure.

Validation

• git diff --check origin/dev/fathomless-firmament...HEAD
• bash -n serve-ds4-flash.sh
• Ruff format/check for every review-touched Python file
• focused config/SP tests: DSpark finite-value validation and speculative CUDA-graph SP rounding
• PCIe gate/barrier tests: 7 passed
• DRY_RUN=1 for all 20 combinations of four modes and five backend profiles
• event-pool, capacity, FlashInfer/B12X padding, indexer, and stream regression tests: 30 passed
• final capture-scope and non-recycled event-lifetime subset: 8 passed, 3 skipped, plus Ruff and git diff --check
• Lucifer CUTLASS + B12X oneshot/PyNCCL E2E prefill after full server load: 64k 14,297 tok/s; five repeated 128k measurements 13,274-13,443 tok/s, with no crash across ten 128k requests including warmups
• focused no-GPU test run in the pinned CUDA image: 8 passed, 1 skipped across:
  • tests/v1/spec_decode/test_dspark_fp8_draft_head.py
  • tests/v1/spec_decode/test_dflash_cudagraph_lifetime.py
  • tests/v1/attention/test_deepseek_v4_dspark_metadata.py
• repeated coherent short- and long-context E2E DSpark output on TP=2 and TP=4 during development

The broad test_arg_utils.py invocation was also attempted in a container without GPU access. Its platform-autodetection cases fail because CUDA is unavailable in that test container; this is an environment limitation rather than a branch regression.

Related work

• B12X compatibility PR: Handle plain CuTe compile callable with DSL options b12x#28
• Upstream SM120 PCIe serving stack: [Perf] SM120 PCIe serving stack: SP/async-TP enablement, FlashInfer spec-decode FULL cudagraphs, and PCIe-safe multi-GPU comms vllm-project/vllm#47979
• FlashInfer graph-safe multi-token decode: Fix: graph-safe uniform multi-token decode on FA2 tensor-core path flashinfer-ai/flashinfer#3871
• FlashInfer DSV4 topk-256 decode: fix(sparse-mla-sm120): add TOPK=256 decode-dsv4 instantiation for DeepSeek-V4 DSpark flashinfer-ai/flashinfer#3817
• FlashInfer DSV4 topk-256 prefill: fix(sparse-mla-sm120): add topk=256 prefill-dsv4 dispatch (fixes #3828) flashinfer-ai/flashinfer#3896
• This supersedes the useful portions of local PRs dspark: draft-pass cleanup (-10% draft time) + upstream rejection-utils consolidation with opt-in block verification #71 and dspark: rowwise-fp8 draft lm_head (VLLM_DSPARK_FP8_DRAFT_HEAD, opt-in) #73; those historical PRs can remain closed.

Summary by CodeRabbit

• New Features

  • Added DSpark capacity-based verification controls via --dspark-capacity-verification-mode, plus SPS-curve profiling, dynamic draft depth, and online temperature calibration.
  • Introduced optional FP8 draft-head acceleration (when supported).
  • Enabled variable-length speculative decoding for CUDA graphs and added request token-limiting (max_req_tokens), plus extended PCIe-safe comm controls.

• Bug Fixes

  • Improved CUDA-graph capture-size rounding for sequence-parallelism and spec decoding.
  • Fixed padding/masking and token-index edge cases across slot mapping, attention metadata propagation, and sampler token ID clamping.

[ormandj]

Same-hardware comparison against current upstream main, using independently tuned profiles for the same C1-C8 / near-1M workload constraint.

Hardware: 2x RTX PRO 6000 Max-Q, 300 W/card, PCIe 4.0 x16, TP2, DeepSeek-V4-Flash-DSpark, fixed probabilistic K=5. Benchmark: temperature 0, context 0, 30 s/cell, 1,024 output tokens.

C	current main d973cce3ca + five minimal carries	this PR, Lucifer+B12X
1	187.29	195.03
2	283.00	293.83
4	490.12	435.67
8	621.64	637.54

C1 triplicate medians: current main PyNCCL 184.84 tok/s; this PR PyNCCL 188.62; this PR B12X 193.96. B12X adds 2.83% within this PR and accounts for approximately 60% of its total measured C1 advantage over current main. Current main is 2.00% behind when both profiles use PyNCCL.

Exact cold prefill, current / this-PR PyNCCL / this-PR B12X: 8k 8,274 / 8,437 / 8,635; 64k 8,238 / 8,587 / 8,673; 128k 7,507 / 7,834 / 7,880 tok/s.

Cold 960,164-token prompt, zero cached tokens: current main 284.221 s TTFT (3,378 prompt tok/s); this PR 273.377 s (3,512 prompt tok/s). Current main was 3.82% lower in prompt throughput. Reported KV pools: current main 1,399,974 tokens at max context 1,048,576; this PR 1,052,723 at max context 1,032,192.

No measured cell had a request error, capacity timeout, or inference-time JIT warning. Benchmark SHA256: fa227030012a8b55545af6b6a50fa4adcbdff8d003bb16469dc8e2de024ed0c0.

[voipmonitor]

Follow-up 399eae7d9 makes the Lucifer CUTLASS DSpark memory default topology-aware. TP2 keeps 0.9465, which profiles the 7.89 GiB required for 262,144 KV tokens; TP4+ keeps 0.94, which preserves headroom for the 764 MiB first-prefill FlashInfer MoE workspace. This is a launcher default correction, not a backend or feature workaround.

Dry-run checks produce TP2 --gpu-memory-utilization 0.9465 and TP4 0.94. The exact final image started TP2 DSpark with no override, profiled 262,144 GPU KV tokens, and measured C1 239.95 tok/s plus post-JIT 64k prefill 6,892 tok/s, with no CUDA/NCCL/Xid failure.

Published image: voipmonitor/vllm:fathomless-firmament-v16-vllm8f86f42-b12xfe06f49-fi801d57a-cu132-20260714

Manifest: sha256:7a0ed4f956bc2f753fd8c67d32d4ee7358e71922794a471abdb9ae6513cabc54

[voipmonitor]

#88 is now fully superseded by reviewable PRs against the current dev/fathomless-firmament head (e4879e1). Do not merge this branch.

Canonical replacements:

• perf: avoid fixed-width MTP draft host synchronization #97: fixed-width MTP/DSpark draft output without the host synchronization
• fix(spec decode): isolate probabilistic draft Gumbel streams #98: independent probabilistic draft Gumbel stream
• fix(scheduler): skip lookahead allocation for running prefill chunks #99: no lookahead allocation for running prefill chunks
• fix(kv cache): preserve MLA quantization metadata when unifying page sizes #100: preserve MLA KV quantization metadata during page-size unification
• fix(ds4): harden sparse verifier metadata for DCP1 and padded replays #101: DS4 sparse verifier metadata, DCP1 causal lengths, and padded replay safety
• fix(ds4): isolate auxiliary-stream events across CUDA graphs #102: graph-safe DS4 auxiliary-stream event lifetime and overlap
• perf(spec decode): return variable draft ids through async output #103: variable-length draft IDs through the existing async output transfer; merge perf: avoid fixed-width MTP draft host synchronization #97 first
• fix(spec decode): harden DSpark and DFlash edge paths #105: remaining DSpark/DFlash correctness fixes, including the block-table shift race, prefix-cache fail-closed path, deterministic TP state, and captured-output lifetime
• perf(dspark): add opt-in rowwise FP8 draft head #106: opt-in rowwise-FP8 DSpark draft head
• perf(dspark): add load-aware compact verification capacity #107: opt-in load-aware compact verification capacity; merge fix(spec decode): harden DSpark and DFlash edge paths #105 first

Related canonical quantization work is #104, replacing closed #92.

The following #88 content is intentionally not carried into core FF:

• the PR47979 sequence-parallel / torch symmetric-memory PCIe experiment: sequence parallelism is unavailable to the V2 runner used by DSpark in this stack, and the symmetric-memory transport was not the selected canonical B12X path;
• image-specific launch scripts, InstantTensor defaults, and model-specific KV/workspace reservations: these belong in the public image/helper repository, not a vLLM feature PR;
• broad SM120 stream serialization experiments: these were diagnostic workarounds and are replaced by the targeted event-lifetime fixes in fix(ds4): isolate auxiliary-stream events across CUDA graphs #102/fix(spec decode): harden DSpark and DFlash edge paths #105;
• B12X DMA/NCCL tuning experiments: measured end-to-end gains were secondary and they remain separate from DSpark correctness.

The combined #97-#107 tree has been merged locally in dependency order without conflicts. Closing #88 removes the conflicting mixed branch as a possible merge source while retaining its investigation history here.