Conversation
github-actions
Bot
added
the
documentation
This was referenced Sep 12, 2026
Keep EnvBool and EnvStr parsing, defaults, overrides, and dynamic reads in a lightweight dedicated registry. Explicitly merge its fields into scheduler environment metadata with the existing encoding and sorted keys. Restore the shared environment registry to upstream main.
yyihuang
changed the title
yyihuang
marked this pull request as ready for review
September 12, 2026 11:59
yyihuang
requested review from
Alisehen,
AniZpZ,
BBuf,
Edwardf0t1,
FlamingoPg,
Fridge003,
HaiShaw,
JustinTong0323,
OrangeRedeng,
b8zhong,
ch-wan,
hnyls2002,
ispobock,
mmangkad,
sogalin,
wisclmy0611,
xiezhq-hermann and
zijiexia
as code owners
September 12, 2026 11:59
Collaborator
Author
|
/tag-and-rerun-ci |
Collaborator
Author
|
/rerun-failed-ci Resume unfinished checks for 05b536a. PR Test Base run 34711186291 was cancelled while several CPU shards had already passed; no replacement run exists for this head. Preserve successful jobs, including the completed MLX and lint workflows. |
Collaborator
Author
|
/rerun-failed-ci |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Add an explicit
flashinfer_alphamoeMoE backend for the supported Qwen3-Next FP8 and GLM-5.2 NVFP4 checkpoints, plus a router-only mode that retains Triton expert compute. Shared backend registration, weight handling, workspace lifetime and runtime tracing are kept in one integration.Final W8 compute dependency validation
Published W8 source: f37119c07d56. The tested CUDA hash is unchanged; the accompanying test-text edit documents FP32 scale reassociation.
Current W8-head CI: pre-commit, API/docs checks and Build Docs passed. PR Test skipped its GPU matrix pending run-ci authorization; no CI GPU pass is claimed. The appendix below is historical.
FlashInfer W8 compute #4287 supplies the compute dependency. This pair uses SGLang runtime
5407ec1a7dfee227a408702addcc15007ec7f126and W8 CUDA SHA256237a9b44549fa82fb3f7833ee51df76dd08d0927fc22674bc62c77c2182429b1. The SGLang production implementation is unchanged.Final Qwen correctness (v46)
4xGB300,TP4/EP1; Qwen/Qwen3-Next-80B-A3B-Instruct-FP8@c5f5f263bdd5cc134092897864e8905d8fe7b928. Fixed AlphaMoE router: stock Triton experts B vs final W8 C (independent W8 attribution).
Independent audit: B=C=1259/1314 (95.8143%), delta 0; 7 gains/7 losses. Unchanged gates pass: both>=0.95,C-B>=-0.005. Five-shot GSM8K questions/prompts/references and wire/SDK inputs match. Each side: 1314 successful HTTP responses; 0 failures/retries/invalid/empty answers. Empty answers count wrong; paired outputs/HTTP receipts retained.
Runtime and CUDA identities are above. Dataset SHA256:
3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14.Four TP ranks: actual router kernels in B/C;
kernel_alpha_moe_w8a8_up_downin C. Real requests: H2048/I128/E512/k10/BM8,z1. C kernel-M discrete sets (count;min,max): decode(49;8,512), eager(97;515,911), prefill(28;64,16384). Exact sets/token counts retained; graph M denotes real-request replay geometry. Integrity passes.API B/C: 62.166970/50.689964s; harness 1448.686871s; physical turnaround 1460.407082s. Correctness wall times, not serving throughput or kernel timing.
Separate GB300 sanitizer replays preserve real eager inputs/layout: M1/H2048/I128/E512/k10/BM8,scale1,rank0/layer0,z1. Synccheck: 0 errors (23.510655s physical); racecheck: 0 hazards/errors/warnings (25.081664s). Scope excludes z2, graphs and other ranks. Capture: 545.017671s physical, 536.014468s harness, 513.267672s startup, 3.942568s request; no new accuracy/performance run.
Retained earlier integration measurements and review history
The unchanged record below retains earlier W8 accuracy and serving measurements, NVFP4 results and CI history. Its current/final-source wording applies to the recorded revisions. The new fixed-router W8 result above has separate attribution and no new serving-throughput measurement.
Code ready for review. Fresh Qwen E2E passed its accuracy gate. The preceding GLM/NVFP4 implementation failed accuracy and performance; the final FP32 accumulation repair passes full-model accuracy and request/kernel integrity but fails serving performance. The backend remains opt-in and is not automatically selected. Review-ready status does not assert model correctness, throughput improvement or release readiness.
Dependencies:
Current SGLang head:
05b536ae075e56145c3e80b9bd6a4bec58b6265b. The Qwen and preceding NVFP4 runs used5407ec1a7dfee227a408702addcc15007ec7f126with combined FlashInferc6407025a445d0d6c3bcfd28a7326456b17e4387. The only intervening changes are two CI test fixtures; the completepython/sglangproduction tree is identical (fb8de627afc329699e8b1b9b87b3df89ee0de438). The final FP32 GLM run uses the same SGLang runtime and FlashInfer unionf5c95353d3723360c55c6af313afd09bbd8bfdac; its NVFP4 files match public head17d0ba9ab8fc43cafbd0981944013f622a9f99fc. Shipping also requires the dependency APIs to land in an appropriate released FlashInfer pin.Fresh validation status — 2026-09-12
Qwen3-Next-80B-A3B-Instruct-FP8, TP4/EP1, identical 1314 held-out five-shot GSM8K requests:
All three have zero invalid answers. Paired prompts/raw outputs, all 45 serving records and actual CUDA symbols on all four TP ranks were independently audited. Both candidate comparisons retain accuracy ≥0.95 and delta ≥−0.005; W8A8 isolated against the fixed fused router also passes (−3/1314). Five-repeat median serving speedups at C32/C64/C128 are router-only 0.9162/0.9127/0.8891×, combined versus stock 1.0560/1.0629/0.9388×, and W8A8 versus the fixed fused router 1.1514/1.1640/1.0600×. Overall serving promotion fails because regressions remain. Full per-repeat absolute Performance, Speedup, latency and configuration records are in the router PR and the W8A8 PR. Physical execution totaled8550.42s; first-start-to-final-completion8774.24s including the preserved timeout/resume. GSM8K API runtimes were110.65/54.85/49.01s, respectively.
GLM-5.2-NVFP4: the preceding NVFP4 revision
85de99d87cb9e86076296d29264d53f09cafb748in FlashInferc6407025a445d0d6c3bcfd28a7326456b17e4387recorded 1267/1314 → 1249/1314, a −1.3699-percentage-point accuracy change that fails the unchanged −0.5-point allowance. The original empty-response guard also left both quality phases unsealed; empty predictions remain graded incorrect. The later diagnostic completed all 30 serving executions and four-rank GPU traces without replacing those original scores. C1/C8/C16 median paired serving speedups were 0.3422/0.1974/0.1543×, with zero wins at every concurrency. The full configuration, all 15 paired performance rows, response diagnostics and physical timing follow below. The current FP32 repair in FlashInfer #4340 at17d0ba9ab8fc43cafbd0981944013f622a9f99fccompleted its full GLM A/B and independent audit: 1254 → 1255 correct of 1314; accuracy passes and performance fails, with complete results below. Its passing scoped compilation/import CI is not GB300 numerical evidence; the Qwen results above are unchanged.Current test-only CI repairs add AlphaMoE to the existing dispatcher-contract test while preserving its LoRA rejection, and make the existing MLX shutdown fixture model the actual receive→process→return sequence. Remote CPU validation passed39 tests and4 subtests (21.73s test runtime,33.21s physical step). Apple Silicon MLX CI run #34711186190 passed. The scheduler-mixin check took 11 s; the CI job took 401 s including setup. The model E2E stages in that workflow were skipped, so this result is test coverage rather than model-accuracy evidence.
Current Qwen per-repeat Performance and attribution
Model
Qwen/Qwen3-Next-80B-A3B-Instruct-FP8@c5f5f263bdd5cc134092897864e8905d8fe7b928, TP4/EP1/DP1, BF16 activations from the FP8 checkpoint, Triton attention, KV dtype auto, chunked prefill 16384, memory fraction 0.7, full decode and breakable prefill graphs. Speculative decoding and shared-expert fusion are disabled. Only the stated router/compute path changes. Measurements below are the same audited 45 serving executions reused for three paired comparisons, not additional runs.Router only: stock Triton → fused router + Triton
Five fixed-workload repetitions run per server variant and are paired by repeat ID and seed. They are not five independent server launches. All repeats are included. Throughput speedup is candidate output tokens/s divided by baseline output tokens/s; values below 1 are regressions. TTFT, TPOT and request E2E columns are per-repeat medians in milliseconds; lower is better.
Workload: 1024 input / 512 output tokens, 1,024 requests per repeat.
Performance gate: FAIL. Every concurrency must have median paired speedup ≥ 1, and at least one must win all five paired repetitions.
W8A8 attribution: fixed fused router + Triton → fixed fused router + W8A8
Five fixed-workload repetitions run per server variant and are paired by repeat ID and seed. They are not five independent server launches. All repeats are included. Throughput speedup is candidate output tokens/s divided by baseline output tokens/s; values below 1 are regressions. TTFT, TPOT and request E2E columns are per-repeat medians in milliseconds; lower is better.
Workload: 1024 input / 512 output tokens, 1,024 requests per repeat.
Performance gate: PASS. Every concurrency must have median paired speedup ≥ 1, and at least one must win all five paired repetitions.
Combined path: stock Triton → fused router + W8A8
Five fixed-workload repetitions run per server variant and are paired by repeat ID and seed. They are not five independent server launches. All repeats are included. Throughput speedup is candidate output tokens/s divided by baseline output tokens/s; values below 1 are regressions. TTFT, TPOT and request E2E columns are per-repeat medians in milliseconds; lower is better.
Workload: 1024 input / 512 output tokens, 1,024 requests per repeat.
Performance gate: FAIL. Every concurrency must have median paired speedup ≥ 1, and at least one must win all five paired repetitions.
The final FP32 NVFP4 source is public FlashInfer
17d0ba9ab8fc43cafbd0981944013f622a9f99fc, tested in the combined revisionf5c95353d3723360c55c6af313afd09bbd8bfdac. Both NVFP4 implementation files match the public head exactly. SGLang runtime5407ec1a7dfee227a408702addcc15007ec7f126and public head05b536ae075e56145c3e80b9bd6a4bec58b6265bhave identical completepython/sglangtrees; their only differences are two CI test files.The full run and an independent audit of its retained raw records are complete. Accuracy passes the unchanged ≥0.92 / delta ≥−0.005 gate; serving performance fails at every concurrency. The extra correct answer does not establish a statistically significant improvement or attribute the earlier regression solely to accumulation precision. The preceding revision's failed evaluation remains below, unchanged.
2026-09-12 final FP32 NVFP4: accuracy PASS, performance FAIL
Model:
nvidia/GLM-5.2-NVFP4ataec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa. SGLang:5407ec1a7dfee227a408702addcc15007ec7f126; FlashInfer:f5c95353d3723360c55c6af313afd09bbd8bfdac.Comparison: stock MoE backend → AlphaMoE NVFP4 MoE. TP4 / EP1 / DP1, execution mode
graph, speculative decoding disabled, shared-expert fusion disabled on both sides.TP4/EP1 with speculative decoding and shared-expert fusion disabled on both sides; the current model registration also tests MTP, which is outside this AlphaMoE integration's supported scope.
moe_runner_backendflashinfer_trtllmflashinfer_alphamoeattention_backenddsadsaprefill_attention_backendNoneNonedecode_attention_backendNoneNonekv_cache_dtypefp8_e4m3fp8_e4m3chunked_prefill_size1638416384mem_fraction_static0.90.9cuda_graph_config{'decode': {'backend': 'full', 'bs': [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 512, 'tc_compiler': 'eager'}, 'prefill': {'backend': 'breakable', 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 2048, 'tc_compiler': 'eager'}}{'decode': {'backend': 'full', 'bs': [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 512, 'tc_compiler': 'eager'}, 'prefill': {'backend': 'breakable', 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 2048, 'tc_compiler': 'eager'}}GSM8K uses the repository's retained five-shot chat scorer (
sglang.test.simple_eval_mixed_prefix_gsm8k.GSM8KEval), all 1,314 held-out examples (the first five of the 1,319-example split supply the examples), temperature 0, top-p 1, and a 2,048-token generation limit. Both variants receive the same prompts and references.Accuracy gate: PASS. Both accuracies must be at least 0.92, and candidate minus baseline must be ≥ −0.005. The candidate gained 21 questions and lost 20; a passing accuracy gate does not assert bitwise equality or zero accuracy loss.
End-to-end kernel verification: PASS. This additionally requires complete server/request evidence and actual GPU kernel traces for each AlphaMoE variant in this comparison.
Performance
Five fixed-workload repetitions run per server variant and are paired by repeat ID and seed. They are not five independent server launches. All repeats are included. Throughput speedup is candidate output tokens/s divided by baseline output tokens/s; values below 1 are regressions. TTFT, TPOT and request E2E columns are per-repeat medians in milliseconds; lower is better.
Workload: 8192 input / 512 output tokens, 4 × concurrency requests per repeat.
Performance gate: FAIL (complete measurements). Every concurrency must have median paired speedup ≥ 1, and at least one must win all five paired repetitions.
Memory and execution evidence
Memory values below are resident-device snapshots after health/model/graph startup or after the named phase. They include model, KV cache and allocator reservations; they are not peak measurements or isolated CUDA Graph allocations.
Post-health GSM8K dispatch/capture receipts observed execution modes:
decode_graph_replay,eager,prefill_graph_replay. Startup capture and fixed-workload performance requests are excluded from this coverage.Observed AlphaMoE runtime shapes below come from those request receipts. Kernel M is the submitted kernel geometry, or the registered capture geometry when a real request replayed that graph. Dispatch M includes graph padding; real tokens are the actual request tokens before that padding. Each column lists its observed values separately, not a Cartesian product of supported shapes. An inclusive range contains only consecutive values that were all observed; missing values are not inferred. The stock baseline has no AlphaMoE kernel shape records.
decode_graph_replayalphamoe_nvfp4_aligned_moeeageralphamoe_nvfp4_aligned_moeprefill_graph_replayalphamoe_nvfp4_aligned_moeGPU execution witness: PASS. After unprofiled measurements, one stored real GSM8K prompt was replayed with a 32-token limit under SGLang's GPU/CUPTI profiler. This request is excluded from accuracy and performance. Required actual CUDA kernel symbols in all four TP traces:
kernel_alpha_moe_nvfp4_up_down.Harness invocation wall time: 6427.43 s (includes server startup, evaluation, optional timing/profiling and cleanup; excludes scheduler queue time). GSM8K runtime and per-request serving latency are reported separately above.
Final request integrity and physical timing
The auditor reconstructed every score from the canonical question, five-shot prompt, final successful HTTP response and returned SDK content. Both sides cover the same 1,314 held-out IDs. Empty final answers count as incorrect; failed or missing final requests cannot pass request integrity.
Full managed execution and first-start-to-completion turnaround: 6484.477 s; model harness: 6427.427 s. The serving rows measure completed HTTP workloads; startup, compilation, model loading and profiler collection are outside those rows. The GSM8K API times and per-request latencies have their separate boundaries above. All per-question paired outputs, HTTP receipts, serving records, post-health runtime shapes and four-rank device profiles are retained.
2026-09-12 preceding NVFP4 revision: accuracy FAIL; complete serving and response diagnostics
These results belong to the preceding NVFP4 implementation, public revision
85de99d87cb9e86076296d29264d53f09cafb748, contained in the combined FlashInfer revisionc6407025a445d0d6c3bcfd28a7326456b17e4387, on 4×GB300. Its recorded full-model accuracy fails the unchanged gate, and its complete serving measurements fail the performance gate. The current FP32 accumulation repair, public revision17d0ba9ab8fc43cafbd0981944013f622a9f99fc, is a different implementation. The final FP32 run is reported separately above and passes accuracy while failing performance. No result in this section establishes correctness or performance of that repair. The eight selected diagnostic responses and later serving/profile measurements remain separate from the original 1,314-question evaluation.Model:
nvidia/GLM-5.2-NVFP4ataec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa. SGLang:5407ec1a7dfee227a408702addcc15007ec7f126; FlashInfer:c6407025a445d0d6c3bcfd28a7326456b17e4387.Comparison: stock MoE backend → AlphaMoE NVFP4 MoE. TP4 / EP1 / DP1, execution mode
graph, speculative decoding disabled, shared-expert fusion disabled on both sides.TP4/EP1 with speculative decoding and shared-expert fusion disabled on both sides; the current model registration also tests MTP, which is outside this AlphaMoE integration's supported scope.
moe_runner_backendflashinfer_trtllmflashinfer_alphamoeattention_backenddsadsaprefill_attention_backendNoneNonedecode_attention_backendNoneNonekv_cache_dtypefp8_e4m3fp8_e4m3chunked_prefill_size1638416384mem_fraction_static0.90.9cuda_graph_config{'decode': {'backend': 'full', 'bs': [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 512, 'tc_compiler': 'eager'}, 'prefill': {'backend': 'breakable', 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 2048, 'tc_compiler': 'eager'}}{'decode': {'backend': 'full', 'bs': [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 512, 'tc_compiler': 'eager'}, 'prefill': {'backend': 'breakable', 'bs': [4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], 'full_prefill_max_req': None, 'full_prefill_prefix_chunk_tokens': None, 'max_bs': 2048, 'tc_compiler': 'eager'}}GSM8K uses the repository's retained five-shot chat scorer (
sglang.test.simple_eval_mixed_prefix_gsm8k.GSM8KEval), all 1,314 held-out examples (the first five of the 1,319-example split supply the examples), temperature 0, top-p 1, and a 2,048-token generation limit. Both variants receive the same prompts and references.The recorded accuracy regression fails the required gate. Full response diagnostics remain separate.
The original full evaluation recorded every answer but failed its empty-response guard. Its original scores below are unchanged. The later diagnostic replayed four selected original prompts sequentially and recorded complete HTTP responses, then measured serving and GPU execution using restarted servers. Selected responses do not replace, rescore or validate the full evaluation.
Recorded accuracy change: -1.3699 percentage points. Acceptance still requires both accuracies ≥ 0.92, candidate minus baseline ≥ −0.005, and complete paired request evidence. Empty final answers are graded incorrect. The original harness additionally stopped on empty text; that stop does not erase the measured accuracy regression. This diagnostic makes no passing accuracy claim.
Four selected original prompts were replayed sequentially with the same generation settings. An empty final answer can coexist with successful HTTP completion when the reasoning uses the token budget. These observations diagnose response termination and do not replace the original high-concurrency outputs.
lengthstopstopstoplengthstopstopstopPerformance
Five fixed-workload repetitions run per server variant and are paired by repeat ID and seed. They are not five independent server launches. All repeats are included. Throughput speedup is candidate output tokens/s divided by baseline output tokens/s; values below 1 are regressions. TTFT, TPOT and request E2E columns are per-repeat medians in milliseconds; lower is better.
Workload: 8192 input / 512 output tokens, 4 × concurrency requests per repeat.
Performance gate: FAIL (complete measurements). Every concurrency must have median paired speedup ≥ 1, and at least one must win all five paired repetitions.
Memory and execution evidence
Memory values below are resident-device snapshots after health/model/graph startup or after the named phase. They include model, KV cache and allocator reservations; they are not peak measurements or isolated CUDA Graph allocations.
Post-health GSM8K dispatch/capture receipts observed execution modes:
decode_graph_replay,eager,prefill_graph_replay. Startup capture and fixed-workload performance requests are excluded from this coverage.Observed AlphaMoE runtime shapes below come from those request receipts. Kernel M is the submitted kernel geometry, or the registered capture geometry when a real request replayed that graph. Dispatch M includes graph padding; real tokens are the actual request tokens before that padding. Each column lists its observed values separately, not a Cartesian product of supported shapes. An inclusive range contains only consecutive values that were all observed; missing values are not inferred. The stock baseline has no AlphaMoE kernel shape records.
decode_graph_replayalphamoe_nvfp4_aligned_moeeageralphamoe_nvfp4_aligned_moeprefill_graph_replayalphamoe_nvfp4_aligned_moeGPU execution witness: PASS. After unprofiled measurements, one stored real GSM8K prompt was replayed with a 32-token limit under SGLang's GPU/CUPTI profiler. This request is excluded from accuracy and performance. Required actual CUDA kernel symbols in all four TP traces:
kernel_alpha_moe_nvfp4_up_down.This invocation's physical wall time: 5814.13 s (includes server startup, evaluation, optional timing/profiling and cleanup; excludes scheduler queue time). GSM8K runtime and per-request serving latency are reported separately above.
The invocation time above is the follow-up diagnostic/serving invocation only; the original full GSM8K evaluation ran earlier. Full-evaluation runtime shapes come from the original post-health request trace; the four-rank device profile comes from the later diagnostic. Neither is a replacement correctness evaluation.
Original evaluation plus follow-up physical execution: 8935.65 s; turnaround from first start to final completion: 9829.03 s, including intervening gaps.
Modifications
--moe-runner-backend flashinfer_alphamoe, selecting W8A8 or NVFP4 according to the supported checkpoint quantization format.SGLANG_FLASHINFER_ALPHAMOE_ROUTER_ONLY=1with--moe-runner-backend triton. This isolates the router while leaving expert compute on Triton.torch.compileand speculative decoding are rejected.Supported integration scope
Qwen/Qwen3-Next-80B-A3B-Instruct-FP8@c5f5f263bdd5cc134092897864e8905d8fe7b928nvidia/GLM-5.2-NVFP4@aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aaBoth use separate shared experts. An execution plan with eight routed entries plus one forced shared slot does not mean the model has routed top-k nine. Such hand-authored plan fixtures do not establish production-shape coverage.
The table specifies admission scope, not fresh accuracy coverage. Dynamic M coverage must come from actual evaluation requests. Startup/capture buckets are not live-request observations.
Code verification on 2026-09-12
The changed-file pre-commit suite passed on this SGLang source. A real environment-field check passed 111 assertions covering defaults, dynamic reads, overrides, cleanup and metadata export, with the environment restored afterward. These checks did not run GPU inference.
E2E acceptance status
Five workload repetitions within one baseline/candidate server deployment are not five independently restarted deployments. No new numerical tolerance or relaxed accuracy gate is introduced. Kernel unit tests, random tensor oracles, smoke tests and microbenchmarks are engineering diagnostics and do not replace real-model accuracy.
Historical Qwen E2E correctness and performance
The following tables are restored reports from 2026-08-08, not results reproduced during this delivery. The source revisions and environments differ from the current candidate. Full raw per-request/per-repeat artifacts have not been revalidated in this session; rounded values are preserved as reported. These historical tables are separate from the current-head results above.
Historical model:
Qwen/Qwen3-Next-80B-A3B-Instruct-FP8atc5f5f263bdd5cc134092897864e8905d8fe7b928, SGLang v0.5.16, 4×GB300, TP4/EP1, Triton baseline versus combined flashinfer-ai/flashinfer#4339 router + flashinfer-ai/flashinfer#4287 W8A8. This does not isolate either kernel.The candidate reached the old
accuracy >= 0.95anddelta >= -0.005gates exactly. This is an observed drop and a boundary pass, not proof of identical outputs or no accuracy loss.Serving: 1,024 requests per workload, 1,024 input / 512 output tokens, three workload repeats per concurrency. Throughput columns are reported means; speedup is the reported mean of paired repeat ratios, which need not equal a ratio of rounded means.
TTFT was reported worse at all three concurrencies; exact TTFT/TPOT rows were not restored. Prefill CUDA Graph allocation was reported as 2.76 → 58.04 GB/GPU. This is a historical graph-allocation report, not a newly measured process-memory peak. Individual baseline/candidate repeat values were not restored, so a full repeat table cannot be reconstructed. These are three repeats within one server deployment per backend, not three independently restarted deployments.
The fixed geometry was
E512/H2048/I_local128/routed_top_k10/BM8, with shared experts separate. Dynamic M was not traced; the workload does not establish any particular M bucket.The existing historical source/import/adapter checks on another SGLang main revision were not full model evaluations and do not fill the current E2E gap.
Historical GLM NVFP4 E2E correctness and performance
The following tables are restored reports from 2026-08-08, not results reproduced during this delivery. The source revisions and environments differ from the current candidate. Full raw per-request/per-repeat artifacts have not been revalidated in this session; rounded values are preserved as reported. These historical tables are separate from the current-head results above.
Historical model:
nvidia/GLM-5.2-NVFP4ataec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa, 4×GB300, TP4/EP1,flashinfer_trtllmbaseline versus the AlphaMoE NVFP4 integration. The candidate included per-expert ModelOpt scale plumbing and a shared CUDA 13.3 attention-header compatibility fix. It was not raw flashinfer-ai/flashinfer#4340 atc0913e5060a6ddda6aaeb606e0f7bedeecfc3147.The old accuracy gate passed: both scores exceeded 0.92 and the candidate drop was within 0.005. This establishes only the reported benchmark threshold result for that scale-patched integration. It does not establish identical outputs, general numerical equivalence, or correctness of the raw NVFP4 PR.
Historical serving used 8,192 input / 512 output tokens at C1/C8/C16. All 30 baseline/candidate workload executions were reported complete. There were five workload repeats within one server deployment per backend; these are not five independently restarted deployments. Speedup below is candidate throughput / baseline throughput; larger than one favors the candidate.
“Not restored” means the visible recovery report lacked that row's absolute baseline/candidate durations. The reported ratios have not been independently recomputed from raw artifacts. No absolute measurements have been inferred from them.
Performance regressed by approximately 3.0×, 4.95× and 6.37× in runtime. The historical overall acceptance failed on performance, despite complete workloads and a passing accuracy threshold.
Runtime reports recorded
E256/H6144/I_local512/routed_top_k8/TP4/EP1, separate shared experts, and 278 eager M values spanning 513–16384 after health. They did not establish decode-graph kernel coverage. Exact TTFT/TPOT and memory comparison rows were not restored.Validation status and reviewer notes
Qwen's completed validation passed its accuracy gates; its full per-repeat results still fail overall performance promotion. The preceding NVFP4 implementation failed the recorded full-model accuracy gate and the complete serving-performance gate. The FP32 NVFP4 repair's full GLM A/B and independent audit pass accuracy and request/kernel integrity; serving performance fails at all three concurrencies. The restored August tables remain historical and do not replace these revision-specific September results.
Review the two quantization-specific scale/layout paths, router-only isolation, admission checks, workspace and graph lifetime, and runtime attribution. The FP32 NVFP4 repair's complete current-revision correctness and per-repeat performance results are included above; its substantial serving regressions remain a performance-promotion failure. Qwen's completed results remain attached to their tested source and are not replaced by NVFP4 diagnostics.
CI States
Latest PR Test (Base): ❌ Run #34711186291
Latest PR Test (Extra): ❌ Run #34711186221
Latest PR Test (AMD ROCm 10): ❌ Run #34711186444