[SM70][Draft] Extend Qwen3.8 DFlash2 concurrency paths

[yangzhuxinyzx]

Purpose

Extend the Qwen3.8-27B NVFP4 q7 DFlash2 operator stack from B1 toward B2/B4/B8 while preserving explicit rollback and exact quality gates.

• Tile NVFP4 QPN2 in independent eight-row CTAs and admit only measured positive shapes through M<=32. M=64 remains on TurboMind.
• Add request-major q8 grouped Flash-V100 verification for B2/B4/B8, preserving B1 q8/q16 and grouped sparse-page4. The new route has a native ABI capability check and remains default-off behind VLLM_FLASH_V100_DFLASH2_BATCHED_GROUPED_VERIFY=1.
• Admit uniform decode-only batches to the existing compact DFlash2 rejection kernel. The compact route remains behind its existing default-off switch.
• Add repeatable operator benchmarks and update the migration control document.

Test Plan

• Build Flash-V100 from source with CUDA 12.8, Torch 2.10, and sm_70 code generation.
• Compare batched grouped output against concatenated per-request grouped output for B2/B4/B8, separate and interleaved KV layouts.
• Replay a B4 CUDA graph while changing every request sequence length.
• Re-run existing B1 q8/q16 FP32, graph, fixed-interleaved, and sparse-page4 coverage.
• Compare compact rejection against dense rejection for B2/B4/B8, q3/q7, top-p 1.0/0.95, and temperature 0.6/1.0.
• Race QPN2 and TurboMind at M=8/16/32/64 across all Qwen3.8 projection shapes.
• Run targeted Python route tests, Ruff, clang-format, and git diff --check.

Test Result

• Flash-V100 source build: PASS.
• Batched grouped bitwise-per-request plus dynamic-sequence CUDA graph: 7 passed.
• Existing grouped q8/q16 regression: 25 passed.
• Grouped sparse-page4 regression: 2 passed.
• Grouped Python routing/workspace/default and ABI gates: 10 passed.
• Compact rejection dense equivalence matrix: 24 passed.
• QPN2 and DFlash2 contract tests: 24 passed.
• Ruff and format checks: PASS.

QPN2 weighted operator saving across the 64-layer Qwen3.8 projection mix is 3.115 ms at M=16 and 2.430 ms at M=32. M=64 regresses by 4.601 ms, so production dispatch deliberately falls back there. All QPN2 rows are finite with relative-L2 about 3.3e-4 to 5.5e-4 versus FP32 and cosine approximately one.

Grouped-vs-XQA CUDA graph speedup at 16K is 5.18x/5.48x/5.50x for B2/B4/B8. At 1K it is 0.77x/3.00x/3.22x; the B2 short-context loss is why batched grouped verification is not promoted by default. Batched grouped output is bitwise equal to per-request grouped output; max difference versus the accepted XQA route is 1.53e-5 at 1K and 3.81e-6 at 16K.

Compact rejection saves 0.415/0.921/1.106 ms at B2/B4/B8 versus dense top-k/top-p plus rejection, with exact accepted lengths and valid tokens.

Scope limitation

This is operator evidence, not an endpoint throughput claim. The fully QUASAR checkpoint is not present on this host. Real layer-55 TP4 weights from the local mixed NVFP4 checkpoint cover both MLP shapes; deterministic native-E2M1 tensors cover the remaining attention and GDN shapes. A matched fully QUASAR TP4 endpoint matrix is still required before changing the two default-off concurrency switches.

[yangzhuxinyzx]

Endpoint concurrency follow-up at 4507907928 is complete.

Contract: four V100-SXM2-32GB GPUs, TP4, local mixed-NVFP4 27B target with official BF16 LM head, q7 probabilistic DFlash2, FP8 E5M2 target KV, FP16 draft KV, Flash-V100/FlashQLA, FULL_AND_PIECEWISE graphs, official temp=1/top-p=.95/top-k=20, sixteen fixed SPEED-Bench 1K prompts x 512 output tokens.

B	output tok/s	vs B1	ideal efficiency	p50 TTFT	p50 TPOT	accepted length
1	187.77	1.000x	100.0%	318.83 ms	4.67 ms	4.16
2	175.26	0.933x	46.7%	393.04 ms	10.33 ms	4.33
4	247.13	1.316x	32.9%	457.74 ms	14.28 ms	4.32
8	362.53	1.931x	24.1%	1147.63 ms	18.62 ms	4.49

B2 is 6.7% below B1; do not promote the batched grouped verifier without a matched grouped-off endpoint arm. The first cold B2 was 139.26 tok/s with 11.46 s p99 TTFT due batch-specific sampling JIT and is not the steady baseline.

FULL target and DFlash graph audit records hit B2/q8=16, B4/q8=32, and B8/q8=64 on all ranks. Logs hit QPN2 M<=32, FlashQLA decode, FP8 E5M2 KV, compact rejection, and request-major grouped verification. All official rows completed 16/16 full-length outputs with no errors or empty text.

Caveats: the nominal prefix warmup recorded zero cache hits, so this is 1K+512 endpoint scaling rather than pure decode. The source overlay also lacked optional exact D256 prefill ops, so these are not final prefill/TTFT numbers. Greedy B1/B8 was byte-identical on 2/8 prompts and otherwise diverged into coherent text; this passes text health only, not semantic-quality equivalence. The fully QUASAR checkpoint remains unavailable.

[yangzhuxinyzx]

Follow-up: exact channel-FP8 QPN8 concurrency paths and endpoint evidence

The explicit gates use the steady B1 result of 187.77 output token/s:

• B2 gate: 300.43 token/s (80% efficiency)
• B4 gate: 525.76 token/s (70% efficiency)
• B8 gate: 901.30 token/s (60% efficiency)

Retained default-off candidates:

• Exact M=9..16 two-row-tile QPN8 path.
• Exact M=17..32 native dense QPN8 path, with chunked M16 fallback for the gated projection.
• Strict allowlisting to the measured Qwen3.8 TP4 channel-FP8 projection shapes.

Same-contract endpoint results:

Route	B2 token/s	B2 efficiency	B4 token/s	B4 efficiency
steady baseline	175.26	46.7%	247.13	32.9%
exact M16 + chunked M32	253.95	67.6%	309.06	41.2%
exact M16 + native dense M32	-	-	322.68	43.0%

The B2 row improves 44.9% over the old B2 result; native M32 improves B4 by 30.6%. All endpoint rows completed 16/16 requests and 512 output tokens per request, with no empty output or replacement characters. B2 acceptance was 47.62% with mean accepted length 4.33; native-M32 B4 was 51.68% / 4.62.

Actual-checkpoint operator validation covers M=17/18/24/31/32 and every production dense split. All outputs are bitwise equal to concatenated M8 calls (maximum difference zero), including CUDA Graph replay. The final M=17/M=32 gate repeats all four dense projection shapes and passes.

Rejected experiments:

• Moving the channel scale to the epilogue changed acceptance-length distribution and was removed.
• Draft temperature scale 0.85 did not improve native-M32 B4 and remains off.
• Native M64 was operator-exact but regressed B8 endpoint throughput to 283.94 token/s, 21.7% below the steady baseline; it was removed.

The throughput gates remain open. Further row tiling is not enough: B4 still needs roughly another 22 ms removed from an observed approximately 57 ms speculative round, while B8 requires a structural scheduling or deployment change. The next branch should separate target, rejection/logits, draft, and host cost without CUPTI, then test request/stream partitioning or replica topology. Two Nsight attempts crashed in cuptiActivityFlushAll during multiprocess shutdown and produced no usable report.

Additional localization and collective follow-up:

• The MRV2 diagnostic gate now admits dflash/dspark as well as mtp. Stable synchronized medians put target forward at 37.64/47.41 ms for B2/B4, target sample plus state at 1.29/1.54 ms, draft at 7.50/9.05 ms, and total GPU at 46.65/58.10 ms. Target forward is about 81% of both intervals.
• q3 B8 was rejected at 249.14 token/s versus the q7 steady 362.53 token/s; its shorter 3.19-token mean emission cannot amortize the target round.
• A default-off M16/M32 TP4 push all-reduce extension is bitwise equal to current custom-order output across 128-collective graph replays. M16 improves 18.45 to 11.03 us per collective and M32 26.78 to 18.36 us. M64 regressed and was removed.
• Combined QPN8 plus push-AR endpoint results are B2 258.04 token/s (68.7%) and B4 317.11 token/s (42.2%). B2 is 1.6% above exact M16. B4 has lower raw throughput than the prior 322.68 row but also lower acceptance length (4.39 versus 4.62); normalized round rate improves about 3.4%. Both completed 16/16 requests with healthy output. The switch stays default-off.
• A third stop-only Nsight attempt still crashed in cuptiActivityFlushAll without a report. No further capture-end variants are planned.

[yangzhuxinyzx]

Follow-up: exact channel-FP8 QPN8 concurrency paths and endpoint evidence

The explicit gates use the steady B1 result of 187.77 output token/s:

• B2 gate: 300.43 token/s (80% efficiency)
• B4 gate: 525.76 token/s (70% efficiency)
• B8 gate: 901.30 token/s (60% efficiency)

Retained default-off candidates:

• Exact M=9..16 two-row-tile QPN8 path.
• Exact M=17..32 native dense QPN8 path, with chunked M16 fallback for the gated projection.
• Strict allowlisting to the measured Qwen3.8 TP4 channel-FP8 projection shapes.

Same-contract endpoint results:

Route	B2 token/s	B2 efficiency	B4 token/s	B4 efficiency
steady baseline	175.26	46.7%	247.13	32.9%
exact M16 + chunked M32	253.95	67.6%	309.06	41.2%
exact M16 + native dense M32	-	-	322.68	43.0%

The B2 row improves 44.9% over the old B2 result; native M32 improves B4 by 30.6%. All endpoint rows completed 16/16 requests and 512 output tokens per request, with no empty output or replacement characters. B2 acceptance was 47.62% with mean accepted length 4.33; native-M32 B4 was 51.68% / 4.62.

Actual-checkpoint operator validation covers M=17/18/24/31/32 and every production dense split. All outputs are bitwise equal to concatenated M8 calls (maximum difference zero), including CUDA Graph replay. The final M=17/M=32 gate repeats all four dense projection shapes and passes.

Rejected experiments:

• Moving the channel scale to the epilogue changed acceptance-length distribution and was removed.
• Draft temperature scale 0.85 did not improve native-M32 B4 and remains off.
• Native M64 was operator-exact but regressed B8 endpoint throughput to 283.94 token/s, 21.7% below the steady baseline; it was removed.

The throughput gates remain open. Further row tiling is not enough: B4 still needs roughly another 22 ms removed from an observed approximately 57 ms speculative round, while B8 requires a structural scheduling or deployment change. The next branch should separate target, rejection/logits, draft, and host cost without CUPTI, then test request/stream partitioning or replica topology. Two Nsight attempts crashed in cuptiActivityFlushAll during multiprocess shutdown and produced no usable report.

Additional localization and collective follow-up:

• The MRV2 diagnostic gate now admits dflash/dspark as well as mtp. Stable synchronized medians put target forward at 37.64/47.41 ms for B2/B4, target sample plus state at 1.29/1.54 ms, draft at 7.50/9.05 ms, and total GPU at 46.65/58.10 ms. Target forward is about 81% of both intervals.
• q3 B8 was rejected at 249.14 token/s versus the q7 steady 362.53 token/s; its shorter 3.19-token mean emission cannot amortize the target round.
• A default-off M16/M32 TP4 push all-reduce extension is bitwise equal to current custom-order output across 128-collective graph replays. M16 improves 18.45 to 11.03 us per collective and M32 26.78 to 18.36 us. M64 regressed and was removed.
• Combined QPN8 plus push-AR endpoint results are B2 258.04 token/s (68.7%) and B4 317.11 token/s (42.2%). B2 is 1.6% above exact M16. B4 has lower raw throughput than the prior 322.68 row but also lower acceptance length (4.39 versus 4.62); normalized round rate improves about 3.4%. Both completed 16/16 requests with healthy output. The switch stays default-off.
• A third stop-only Nsight attempt still crashed in cuptiActivityFlushAll without a report. No further capture-end variants are planned.

TP4-only NVFP4 verifier follow-up:

• The q7 B8/M64 synchronized MRV2 phase probe produced 43 stable full-batch
  rounds: target forward 54.103 ms, target sample/state 2.024 ms, draft
  12.041 ms, and total GPU 68.290 ms. Target forward is 79.2% of the interval.
• VLLM_SM70_NVFP4_QPN2_M16_NATIVE is a default-off two-row-CTA route that
  reuses packed MLP weights while preserving split-K and FP32 reduction order.
  Real layer-55 TP4-rank-0 gate/up improves 72.30 to 66.76 us and down
  38.06 to 31.31 us, for a projected 0.786 ms saved per target round. M9,
  M15, and M16 are bitwise equal to concatenated M8 calls for both projections.
• Two single-instance TP4/B2 endpoint runs measured 271.46 and 270.20
  token/s. The final source-matched row is 270.20 token/s, 72.0% efficiency
  relative to the fixed 187.77-token/s TP4/B1 baseline. It completed 16/16
  requests and all 8,192 output tokens with no errors or malformed text;
  acceptance was 45.30% and mean accepted length 4.17.
• Rejected and removed: exact NVFP4 M32 (down projection regressed to
  140.94 us), q5 B8 (335.71 token/s, 7.4% below q7), and a QPN8
  single-accumulator variant (acceptance-normalized B2 rate regressed about
  0.6% and reduction order changed).
• Gates remain open at 300.43/525.76/901.30 token/s for B2/B4/B8. The next
  direction remains a single TP4 instance: MRV2-local verifier microbatching
  or request partitioning. Generic DBO does not apply to this dense TP4/DP1
  ModelRunnerV2 path.