[Kernel][SM70] Improve Qwen3.8 no-MTP concurrency

[yangzhuxinyzx]

Purpose

Improve Qwen3.8-Flash-Next NVFP4 TP4/V100 no-MTP concurrency using measured batch kernels. Frozen C1 reference: 70 tok/s; C4/C8/C16 targets: 238/420/728 aggregate tok/s. Integration base: 45a58ab6749096248dc15b1263bdf5faf51f5c70.

This is an AI-assisted WIP checkpoint. Human line-by-line review and quality admission are required before promotion or merge. It is not a release-ready speed claim.

Current checkpoint: cc22156c2b (DCO-signed). Grouped-MoE implementation remains 94dd55c899; the latest native experiment adds opt-in small-message push admission and rejects unsafe batch block-count overrides. Integration was fetched before publication (54478b9d289e); this experiment deliberately retains its frozen base rather than silently incorporating newer main changes. Synchronization and clean-wheel validation remain required before merge.

The measured prepared-scale C1 is 80.657 tok/s and must be protected within 1%; the 70 tok/s reference is only the fixed concurrency-efficiency denominator, not permission to regress single-request performance.

Scope and related PRs

• Extend direct native-NVFP4 expert dispatch, W13/SwiGLU and W2/reduction fusion, router, sparse-QSA launch geometry, shared gate, and TP4 push collectives to measured batch shapes.
• Preserve native NVFP4 weights, FP16 activation inputs, existing prefill, and no-MTP sampling settings.
• Experimental raw E4M3 scale residency saves memory, with exhaustive load-time checks for both strict prefill scales and effective HMMA decode scales. Default off.
• Dynamic M2--M16 QPN dispatch is independent of raw storage. Default off pending full-model quality admission.
• Add raw-scale workspace cleanup, native edge-encoding tests, and a portable real-checkpoint single-layer audit script.
• Add expert-grouped multi-row W13/SiLU with intra-CTA Split-K and grouped W2 reusing the same integer plan. Now built into _C, with fake ops and production apply dispatch behind VLLM_SM70_NVFP4_MOE_GROUPED_DECODE=1 (default off). Local shape and CPU metadata gates preserve single-token, prefill/mixed/verify fallbacks; no TP/KV/max-seqs/chunk binding. This differs from the rejected split singleton/repeated W2 implementation and avoids repeated planning.
• Related [Perf][SM70] Reduce Qwen3.8 no-MTP token latency #481 focuses on single-request token latency. W13/shared-gate implementations overlap and need consolidation; do not merge both blindly or count savings twice. Its PLE/QSA epilogues are reuse candidates, not accepted batch gains here. [SM70][Draft] Extend Qwen3.8 DFlash2 concurrency paths #476 is the separate DFlash2 concurrency campaign; this PR does not change that model's contract.

Test Plan

1. Real-checkpoint MoE loader/apply comparison and eager/CUDA Graph checks before any additional endpoint run.
2. Native E4M3 encoding, subnormal, signed-zero, NaN, global-slot layout, and effective-scale admission tests.
3. Focused route and cleanup regressions.
4. Finish C1 full-model mismatch localization; run coding, tool-call, structured-output and long-output quality scores before default enablement.
5. Only after sufficient operator gains, one paired fixed-workload C1/C4/C8/C16 throughput gate. No repeated full-model runs for noise-floor changes.

Test Result

• Single-HC TP4 trace identifies a 10,752-byte intermediate reduction missing the push whitelist: local down/up GEMMs are faster, but pull communication and publication erase the saving. Profiled service is diagnostic, not endpoint latency.

• New VLLM_SM70_TP4_PUSH_ALLREDUCE_SMALL_MESSAGES stays default off. It admits captured, fully-connected TP4 SM70 FP16 aligned messages within the existing 80-KiB buffer; no model/maxseq/chunk/KV binding and no communicator ABI change. When enabled, ordinary collectives use the smallest covering grid, including 40 instead of 80 CTAs for the 80-KiB payload. Flag-off geometry and sum2 admission/geometry remain unchanged.

• Fixed an unsafe pre-existing tuning override: block count below ceil(bytes/2048) could leave output tails unwritten. The undersized override now falls back to the established safe launch.

• New four-rank native gate passes 13 message sizes, three graph orders, eight cycles for each of random/signed-zero/special-value inputs. Finite outputs are bitwise equal to a rank-ordered FP32/FP16 oracle; poisoned outputs, tail canaries and rank skew are covered. Includes an intentionally undersized block override. Not a model-quality score pass. Existing CPU allocator/dispatch suite: 21 passed.

• Extended native gate: 64 cycles per pattern, four graph sequences, 13 sizes, all four ranks passed, including interleaved ordinary/sum2 calls with shared push storage. v4 DSO SHA256: 348b782113785d374d397362b37cc93dc06f445d595b7cd4a0d5e5f3fdaf3888.

• Foreign-worker-contaminated timing attempts were discarded. Valid v4 HC hot-single-pair results were M4 31.235 -> 27.187 us, M8 31.872 -> 28.843 us, M16 33.781 -> 32.486 us. Rotating 16 distinct allocations of the same real weights instead gives 31.370 -> 31.198, 32.018 -> 33.462, and 33.947 -> 36.520 us: the apparent benefit does not survive a layer-like weight working set. This is a cache-footprint microbenchmark, not an actual 16-layer model or endpoint result. Raw timing outliers are retained.

• HC sharding is rejected for production; no endpoint run or promotion for this version. Keep generic push opt-in, and next screen fused local pointwise/disjoint-publication work with the rotating-weight case. No new accepted endpoint gain or model-quality pass is claimed.

• Rebuilt native _C; 52 focused admission/native tests passed.

• 108 CPU regression tests passed, 6 GPU-only cases skipped in that CPU run. These overlap the focused suite and are not a summed test count.

• Four timing-accounting regression tests passed.

• All applicable staged pre-commit hooks passed, including mypy, formatting, header checks and DCO sign-off. WIP implementation checkpoint: d20a077bf4.

• New grouped MoE tests: 22 GPU tests passed, including M1--M16 changing-route graph replay, poisoned buffers, all-singleton/repeated/invalid experts, exact grouped W2 versus native W2, and same-split W13 in both physical layouts. This is separate operator evidence, not a model-score pass.

• Actual layer-0/rank-0 weights, all 512 experts: raw/prepared outputs bitwise equal at M1--M16 and M784 prefill after separating storage from dispatch. Independent dynamic-QPN mode also matches between storage formats at M1--M16. Synthetic activations; this is not a model-quality score.

• Earlier boundary audit: all 30 combinations of layers 0/23/47, ranks 0/3 and M1/4/8/16/784 passed.

• Initial raw-on M3/M7 differences (up to 2.38e-7) were traced to an unintended TurboMind-to-QPN dispatch change. That coupling is removed.

• Remaining full-model C1 completion-hash difference is unresolved. Autotuned prefill reduction is a hypothesis, not a confirmed cause. Raw storage stays off.

Timing correction — targets remain unmet

Legacy fixed-width get_output() blocking wait excludes processing between receives and overestimates throughput. Original raw records are retained. Corrected aggregate uses emitted tokens divided by summed consecutive eligible engine timestamp intervals; input, seed and frozen 70 tok/s reference are unchanged.

C	Prepared-scale candidate tok/s	Raw-scale experiment tok/s	Raw fixed-70 efficiency	Target efficiency
4	212.48	213.11	76.1%	85%
8	357.37	363.28	64.9%	75%
16	536.38	540.67	48.3%	65%

These paired diagnostics predate the latest admission fix. Raw saves about 1.73 GiB/worker but adds only 1.66%/0.80% aggregate throughput at C8/C16 in that pair. Both arms use identical pinned custom-AR/FlashQLA sidecars; this is not a clean-wheel gate. The original request-level baseline and fixed-width steady-state results remain distinct contracts.

Evidence and risks

HC follow-up: no candidate admitted

Offline complete-graph attribution splits dense service into HC projections 3.558 ms, GDN inputs 2.023 ms and other roles, 9.334 ms total inside the old model graph (excludes outside-graph LM-head/sampling). HC postops add 1.074 ms. It also corrects the old residual interpretation: service overlap is 1.334 ms and no-kernel gaps 1.929 ms in the complete graph. Wall minus service is not a closed CPU/GPU-idle decomposition.

New benchmark-only FP16 tensor-core HC fusion and batched output-sharding screens all fail the performance gate. Actual layer-0 attention-HC weights, synthetic inputs, CUDA graphs, full Mix chain including both projections/epilogues/communications as applicable:

M16 experiment	Matched baseline	Candidate
Packed up/mix only	33.726 us	40.970 us
Packed full chain, best screened split20	34.571 us	42.443 us
TP4 shard both ends, existing reductions	33.992 us	41.787 us
TP4 shard down only	33.285 us	43.504 us

These are operator screens, not full-model latency. Changed-input/poisoned-workspace graph replay matches candidate eager results, but candidate-versus-control reductions differ and model scores remain untested. A vector-load follow-up had foreign GPU interference and unstable samples; its timings are not admissible. The distributed screens check process exclusivity before/after each timing group. No performance or quality promotion, new default, runtime dispatch or endpoint run results from this work. All staged pre-commit hooks pass, and task-owned GPU workers/locks are released.

This batch screen borrows the output-sharding idea from related #481, not its M1 kernel or communicator ABI. Local GEMM, scatter and collective costs must be broken down before a fused compute/publication design; do not repeat these losing implementations unchanged. Raw JSON/log names and commands are retained in the worklog.

Guarded production engine result

The source-build control/candidate/control run completed on the fixed TP4/V100 no-MTP 8K/256 workload. Same native _C SHA256 76f106f86f7e7bdf5f8a51b64378fee7ee09ba8a6d3cb51e699a944985711858, prepared scales, FP32 GDN state, FP16 KV, prefix/Mamba align, 256K max length, 2048 prefill chunk, max-seqs16. Only grouped decode differs.

C	Disabled baseline tok/s (bracket mean step)	Enabled tok/s	Change	Fixed-70 efficiency / target
1	81.493	81.435	-0.07%	reference
4	214.108	214.535	+0.20% (route unchanged)	76.62% / 85%
8	359.942	362.981	+0.84%	64.82% / 75%
16	540.951	584.568	+8.06%	52.19% / 65%

C16 complete step: disabled 29.566/29.589 ms, enabled 27.371 ms; 2.207 ms actual saving. Full-graph M8/M16 route hits confirmed. C1 protected; concurrency targets remain unmet. This is not a clean-wheel or model-quality gate. All task workers exited and GPUs were released.

Production loader/apply audit on real weights confirms M8/M16 hits, exact M1/M4/M8 and M17/M32/M784 fallbacks, M16 max abs 2.38419e-7 and relative L2 1.38547e-4, and exact graph/eager repetition. Mixed-NVFP4/dispatch CPU suites, the final 33-test dispatch suite and the 22 native GPU tests pass (CPU suites overlap, not additive). All applicable staged pre-commit hooks pass. No new quantization is introduced, but C16 FP32 association differs. Final CPU-integer metadata hardening fails closed for tensor metadata; the engine experiment predates this defensive check and whitespace-only CUDA formatting, neither changes the screened host-integer route.

All C1 completion hashes agree. Multi-request completions vary even between the two disabled controls; the cause and quality effect are not established. Quality scores, including actual decode-path rather than prefill-only perplexity, remain mandatory before enablement. No quality-pass or release-ready claim.

Latest grouped-MoE screen (benchmark-only)

• Actual layer-0/rank-0 weights, all 512 experts, synthetic activations; 48 captured C16 route patterns. Alternating graph A/B, 16 complete calls per graph to amortize Python replay submission.
• Grouped W13 split8 + grouped W2: mean complete MoE call 164.404 -> 112.939 us (-31.30%); sum across the 48 route patterns 2.470 ms saved. All 48 improved. This crosses the >=2 ms microbenchmark screen gate, not an actual full-model latency or throughput gate.
• The same layer's weights are reused with all 48 routing patterns. Do not describe it as a measured 48-layer round or convert it into released tok/s.
• Grouping and grouped W2 preserve same-split arithmetic. Split8 changes C16 FP32 association: max output absolute error 1.90735e-6, relative L2 1.97128e-4; no model-quality scores have passed yet.
• This was the admission screen before the guarded production integration above. Default dispatch is still unchanged; quality scores and all 238/420/728 targets remain unaccepted.

Detailed commands, artifact names, binary SHA256s, prior rejected variants, overlap with #481 and all pending gates are in docs/design/sm70_qwen38_nomtp_concurrency.md.

Portable diagnostic: benchmarks/kernels/verify_sm70_nvfp4_moe_raw_storage.py --model <checkpoint-dir> --layers 0,23,47 --ranks 0,3 --tokens 1,4,8,16,784 --out raw-storage.json (one idle SM70 GPU).

No model files, generated libraries, profiler captures, environment caches or private credentials are included. Keep Draft; do not merge until quality, throughput, clean-wheel and human-review gates pass.