[WIP][LoRA] BF16 support for the experimental_sgl_trtllm MoE-LoRA path & lora expert_outer_shared

[yushengsu-thu]

Summary

Adds BF16 support to the experimental_sgl_trtllm MoE-LoRA path. Previously this path was FP8/NVFP4-only and a bf16 checkpoint crashed at launch with AssertionError: experimental_sgl_trtllm LoRA currently requires FP8 block quant.

The fused MoE-LoRA CUDA kernels (trtllm_fp8/fp4_block_scale_routed_moe_lora) are not dtype-templated and no bf16 variant exists; the bf16 base kernel (trtllm_bf16_routed_moe) is monolithic with no LoRA hooks. This PR follows the same strategy the FP4 path used when it hit the missing-unfused-cubin wall: a decomposed pipeline — with the two NvFP4 quant stages removed, so it is bf16 end-to-end and needs zero new device kernels (it reuses moe::dev::permute / moe::dev::activation / moe::dev::finalize and the trtllm-gen Gemm2 bf16 cubins that the plain bf16 path already uses).

Phase 2 (included): the bf16 MoE op is wired into the existing two-stream LoRA overlap (install_two_stream_overrides, master switch SGLANG_EXPERIMENTAL_LORA_OPTI=1) — the gate_up LoRA delta runs on the shared side stream concurrent with the op's routing + permute + gate_up GEMM, joining via lora_ready_event right before the activation kernel (its only consumer), exactly like the FP8/FP4 O1 overlaps.

FP8/FP4 paths are byte-for-byte untouched (additive only). Weight layout untouched (consumes the same shuffled + BlockMajorK bf16 tensors as trtllm_bf16_moe).

Files changed (what & why)

file	change
python/sglang/jit_kernel/trtllm_lora_temp/data/csrc/trtllm_fused_moe_kernel_launcher.cu (+412)	New Bf16LoraLauncher + sgl_trtllm_bf16_routed_moe_lora wrapper + TVM FFI export. Pipeline: routing -> moe::dev::permute (bf16 gather) -> gate_up as raw Gemm2::Runner(Bf16,Bf16,Bf16, shuffled, BlockMajorK) (K=hidden, N=2*inter, gated-interleaved out) -> moe::dev::activation (de-interleaves on read, adds gate_up_lora_delta pre-SwiGLU, captures activation_lora_input, bf16 out, no quant) -> down Gemm2::Runner (K=inter, N=hidden) -> finalize (or returns the triple for the shared bf16 lora-finalize). lora_ready_event (waited right before activation) / gemm2_done_event (recorded after the down GEMM) for the two-stream overlap.
python/sglang/jit_kernel/trtllm_lora_temp/core.py (+71), __init__.py	trtllm_bf16_routed_moe_lora Python binding. Finalize reuses the existing dtype-agnostic bf16 trtllm_fp8_block_scale_moe_lora_finalize (the FP4 path already shares it).
python/sglang/srt/lora/trtllm_lora_temp/lora_layer.py (+28/-1)	bf16 (unquantized) detection before the FP8 asserts -> minimal FlashInferTrtllmBf16MoeQuantInfo; dispatch routes the bf16 quant-info to the new fused fn. FP8 asserts now gate only quantized checkpoints.
python/sglang/srt/lora/trtllm_lora_temp/lora_dispatch.py (+152)	fused_experts_none_to_experimental_sgl_trtllm_bf16_lora: mirrors the FP4 flow — virtual-experts gate_up delta via the dtype-agnostic merged_experts_fused_moe_lora_add, bf16 hidden fed directly (no per_token_group_quant_fp8), do_finalize=True, down-LoRA merged after with fuse_sum_all_reduce. This is the single-stream variant (prefill / non-decode batches, and the fallback of the two-stream wrapper).
python/sglang/srt/lora/trtllm_lora_temp/moe_overlap.py (+182)	Phase 2: fused_experts_none_to_experimental_sgl_trtllm_bf16_lora_two_stream — bf16 sibling of the FP8/FP4 two-stream MoE dispatches (O1-bf16). Side-stream gate_up delta + lora_ready_event join before activation; down-LoRA stays serial on the main stream (the down/finalize overlap is bench-verified net-negative on FP8/FP4 and corrupts base decode under cuda-graph replay, so it is intentionally not wired — gemm2_done_event=0).
python/sglang/srt/lora/trtllm_lora_temp/__init__.py (+16/-5)	Save/patch the bf16 fn alongside FP8/FP4 in install_two_stream_overrides (+ get_original_bf16_moe_lora_func for the non-decode fallback).
python/sglang/srt/lora/backend/base_backend.py (+13/-3)	Generic MoE-LoRA cuda-graph buffer init: read gemm1/gemm2_weights when the quant-info is the bf16 one, and don't assume 3-D expert weights (bf16 BlockMajorK-prepared weights are 4-D [E, N, K/128, 128]).

Verification (GB300 / sm_103, Qwen/Qwen3-30B-A3B-Instruct-2507 bf16, TP4/EP4, dummy rank-16 full-coverage adapter, --lora-use-virtual-experts --lora-backend triton, allreduce-fusion force-disabled)

• Launch: clean (no FP8 assert, no illegal memory access); first launch pays the module JIT rebuild (~7 min incl. nvcc), then stable.
• Coherence: greedy decode with lora_path=alpha and base both coherent — and token-for-token identical to the same command run with --moe-runner-backend triton (the verified bf16 reference path). Re-verified after phase 2 (two-stream active).
• Prefill logprob vs triton (same 39-token prompt): lora mean abs diff 0.105 / base 0.063 — the lora-path delta is the same order as the pure backend-numerics delta (trtllm-gen vs triton bf16 GEMM accumulation over 48 MoE layers), i.e. no anomalous error.
• Perf (bench_one_batch_server, in=out=2048):

bs	lora decode tok/s — phase 2 two-stream	phase 1 single-stream	triton path	two-stream vs triton
16	2613.6	2421.8	2161.5	+20.9%
32	4771.0	4484.2	3817.2	+25.0%
64	8324.2	7835.6	6128.2	+35.8%

Two-stream adds +7.9% / +6.4% / +6.2% over phase-1 single-stream (ITL 6.61/7.14/8.17 → 6.12/6.71/7.69 ms); no-lora cells are unchanged (base path unaffected). lora/no-lora decode ratio: 57.1/56.5/53.7% (triton) → 63.9/66.2/68.7% (phase 1) → 68.9/70.4/73.2% (phase 2).

Flow

Diagram style/reference: sgl-project#26602 (the FP8 two-stream original). Differences from the FP8 flow are marked in-diagram with DIFF.

Which diagram applies when: the side-stream overlaps (O7 qkv / O8 o_proj / O9 merged-column / O1 MoE) are installed by install_two_stream_overrides() under the master switch SGLANG_EXPERIMENTAL_LORA_OPTI=1 and fire per batch only for decode-shaped batches (tokens <= SGLANG_TWO_STREAM_MAX_TOKENS, default 256). Prefill and large batches always take the single-stream flow; decode takes the two-stream flow.

1) Normal LoRA (routed-expert)

(A) Single-stream flow (prefill / non-decode batches; also the pipeline structure itself)

sequenceDiagram
    autonumber
    participant M as Main CUDA stream

    Note over M: --- Attention block ---
    M->>M: qkv_proj quant_method.apply
    M->>M: sgemm_lora_a_fwd (qkv shrink)
    M->>M: qkv_lora_b_fwd (expand + atomic-add)
    M->>M: FlashAttention (fmha)
    M->>M: o_proj quant_method.apply
    M->>M: sgemm_lora_a_fwd (o_proj shrink)
    M->>M: sgemm_lora_b_fwd (expand + atomic-add)
    M->>M: tensor_model_parallel_all_reduce

    Note over M: --- MoE block --- NEW bf16 path ---
    M->>M: merged_experts_fused_moe_lora_add<br/>(gate_up shrink + expand)<br/>→ gate_up_lora_delta
    Note over M: DIFF vs FP8, no per_token_group_quant_fp8 step
    M->>M: sgl_trtllm_bf16_routed_moe_lora<br/>routing → permute bf16 hidden
    M->>M: gate_up GEMM as raw Gemm2 runner<br/>(bf16 weights, K hidden, N 2x inter)<br/>DIFF, replaces the fused-act PermuteGemm1
    M->>M: activation kernel (bf16 in, bf16 out)<br/>de-interleave + SwiGLU(+gate_up_lora_delta)<br/>→ writes activation_lora_input
    Note over M: DIFF vs FP8, no post-activation quant, stays bf16
    M->>M: down GEMM (Gemm2 runner) → finalize → moe_result
    M->>M: merged_experts_fused_moe_lora_add<br/>(down LoRA shrink + expand, atomic-add to moe_result)
    M->>M: downstream all_reduce (in qwen3_moe)

Loading

(B) Two-stream decode fast path (phase 2, this PR)

Attention overlaps (O7/O8) are the source PR's, unchanged by this PR; the MoE par block (O1-bf16) is new.

sequenceDiagram
    autonumber
    participant M as Main CUDA stream
    participant S as LoRA side stream

    Note over M,S: --- Attention block (same as the FP8 PR) ---
    par O7: shrink on side
        S->>S: sgemm_lora_a_fwd<br/>(qkv shrink)
    and base GEMM on main
        M->>M: qkv_proj quant_method.apply
    end
    S-->>M: join
    M->>M: qkv_lora_b_fwd (expand + atomic-add)
    M->>M: FlashAttention (fmha)
    par O8: shrink on side
        S->>S: sgemm_lora_a_fwd<br/>(o_proj shrink)
    and base GEMM on main
        M->>M: o_proj quant_method.apply
    end
    S-->>M: join
    M->>M: sgemm_lora_b_fwd (expand + atomic-add)
    M->>M: tensor_model_parallel_all_reduce

    Note over M,S: --- MoE block --- NEW bf16 two-stream (O1-bf16) ---
    par O1-bf16: gate_up LoRA on side
        S->>S: merged_experts_fused_moe_lora_add<br/>(gate_up shrink + expand)<br/>→ gate_up_lora_delta
    and main: bf16 op routing/permute/GEMM
        M->>M: sgl_trtllm_bf16_routed_moe_lora<br/>routing → permute bf16 hidden<br/>DIFF, no per_token_group_quant_fp8
        M->>M: gate_up GEMM as raw Gemm2 runner<br/>(bf16 weights, K hidden, N 2x inter)<br/>DIFF, replaces the fused-act PermuteGemm1
    end
    S-->>M: join (lora_ready_event) right before activation
    M->>M: activation kernel (bf16 in, bf16 out)<br/>de-interleave + SwiGLU(+gate_up_lora_delta)<br/>→ writes activation_lora_input<br/>DIFF, no post-activation quant
    M->>M: down GEMM (Gemm2 runner) → finalize → moe_result
    M->>M: merged_experts_fused_moe_lora_add<br/>(down LoRA, atomic-add to moe_result, serial on main)
    M->>M: downstream all_reduce (in qwen3_moe)

Loading

2) Shared-outer (TML) LoRA — where the shared-expert LoRA enters

The --experts-shared-outer-loras adapter (TML layout) is not a separate dispatch path — it is a
sub-mode of the two merged_experts_fused_moe_lora_add calls above, toggled by
experts_shared_outer_loras_a/b. The shared-expert LoRA is merged into the same gate_up / down
LoRA deltas as the routed-expert LoRA, via virtual-experts, but routed differently:

• routed-expert LoRA: per-expert weights, virtual_id = topk_id + lora_id * num_experts, EP-masked
  (non-owned topk_ids slots dropped to -1 on ranks > 0).
• shared-outer LoRA (virtual_experts.py): num_experts_for_weight = 1 (one weight set),
  topk_ids zeroed → routed by LoRA-id (base = 0), and never EP-masked (every rank applies
  the shared expert). In the bf16 dispatch it is _a = shared_outer on gate_up and _b = shared_outer
  on down; use_direct_expand_add silently falls back for the shared-outer expand.

The attention block is identical to Normal LoRA above (shared-outer only changes the MoE block,
marked DIFF); both flows are shown end-to-end from the attention block for parity with §1.

This is for standard attention (the Qwen3-30B bf16 model here): experts_shared_outer_loras is read only inside FusedMoEWithLoRA (the MoE expert LoRA layer), so the attention LoRA kernels (sgemm_lora_a/b, qkv_lora_b) are byte-identical to Normal LoRA. MLA models (Kimi / DeepSeek-V3.1) carry an extra kv_b attention-side LoRA correction (deepseek_mla_correction) not drawn here — also not gated by experts_shared_outer_loras.

(A) Single-stream

sequenceDiagram
    autonumber
    participant M as Main CUDA stream

    Note over M: --- Attention block ---
    M->>M: qkv_proj quant_method.apply
    M->>M: sgemm_lora_a_fwd (qkv shrink)
    M->>M: qkv_lora_b_fwd (expand + atomic-add)
    M->>M: FlashAttention (fmha)
    M->>M: o_proj quant_method.apply
    M->>M: sgemm_lora_a_fwd (o_proj shrink)
    M->>M: sgemm_lora_b_fwd (expand + atomic-add)
    M->>M: tensor_model_parallel_all_reduce

    Note over M: --- MoE block, bf16 shared-outer (TML) LoRA ---

    Note over M: gate_up LoRA is routed-expert plus shared-outer, into ONE gate_up_lora_delta
    M->>M: merged_experts_fused_moe_lora_add (gate_up)<br/>shared_outer_a True, shared_outer_b False, mul_routed_weight False
    Note over M: routed path, shrink A then expand B, expert-id routed<br/>virtual_id is topk_id plus lora_id x num_experts, EP-masked
    Note over M: DIFF shared-outer A, num_experts_for_weight 1,<br/>topk_ids zeroed so routes by LORA-id (base 0),<br/>never EP-masked, summed into the SAME gate_up_lora_delta
    M->>M: gate_up_lora_delta, shape tokens x top_k x 2 inter

    M->>M: trtllm_bf16_routed_moe_lora<br/>routing → permute (bf16 gather) → gate_up GEMM (raw bf16)
    M->>M: activation de-interleave + SwiGLU(+gate_up_lora_delta)<br/>→ writes activation_lora_input
    M->>M: down GEMM → finalize → moe_result

    Note over M: down LoRA is routed-expert plus shared-outer, atomic-add to moe_result
    M->>M: merged_experts_fused_moe_lora_add (down)<br/>shared_outer_a False, shared_outer_b True,<br/>mul_routed_weight True, fuse_sum_all_reduce True
    Note over M: DIFF shared-outer B (down expand), lora-id routed,<br/>not EP-masked, direct-expand-add disabled for shared-outer
    M->>M: downstream all_reduce (in qwen3_moe)

Loading

(B) Two-stream

sequenceDiagram
    autonumber
    participant M as Main CUDA stream
    participant S as LoRA side stream

    Note over M,S: --- Attention block (same as the FP8 PR) ---
    par O7: shrink on side
        S->>S: sgemm_lora_a_fwd<br/>(qkv shrink)
    and base GEMM on main
        M->>M: qkv_proj quant_method.apply
    end
    S-->>M: join
    M->>M: qkv_lora_b_fwd (expand + atomic-add)
    M->>M: FlashAttention (fmha)
    par O8: shrink on side
        S->>S: sgemm_lora_a_fwd<br/>(o_proj shrink)
    and base GEMM on main
        M->>M: o_proj quant_method.apply
    end
    S-->>M: join
    M->>M: sgemm_lora_b_fwd (expand + atomic-add)
    M->>M: tensor_model_parallel_all_reduce

    Note over M,S: --- MoE block, bf16 shared-outer (TML) LoRA, two-stream (decode, O1-bf16) ---
    par O1-bf16: gate_up LoRA (routed + shared-outer) on side
        S->>S: merged_experts_fused_moe_lora_add (gate_up)<br/>shared_outer_a True, shared_outer_b False, mul_routed_weight False<br/>→ ONE gate_up_lora_delta
        Note over S: routed path: shrink A → expand B, expert-id routed<br/>virtual_id = topk_id + lora_id x num_experts, EP-masked
        Note over S: DIFF shared-outer A: num_experts_for_weight 1,<br/>topk_ids zeroed → routes by LORA-id (base 0), never EP-masked,<br/>summed into the SAME gate_up_lora_delta
    and main: bf16 base MoE routing / permute / gate_up GEMM
        M->>M: sgl_trtllm_bf16_routed_moe_lora<br/>routing → permute (bf16 gather)
        M->>M: gate_up GEMM (raw Gemm2 runner, bf16, K hidden, N 2x inter)
    end
    S-->>M: join (lora_ready_event) right before activation
    M->>M: activation kernel (bf16 in/out)<br/>de-interleave + SwiGLU(+gate_up_lora_delta)<br/>→ writes activation_lora_input
    M->>M: down GEMM (Gemm2 runner) → finalize → moe_result
    Note over M: down LoRA (routed + shared-outer) stays SERIAL on main<br/>(down/finalize overlap intentionally not wired — gemm2_done_event=0)
    M->>M: merged_experts_fused_moe_lora_add (down)<br/>shared_outer_a False, shared_outer_b True,<br/>mul_routed_weight True, fuse_sum_all_reduce True
    Note over M: DIFF shared-outer B (down expand): lora-id routed,<br/>not EP-masked, direct-expand-add disabled for shared-outer
    M->>M: downstream all_reduce (in qwen3_moe)

Loading

Notes

• The down-LoRA/finalize overlap (gemm2_done_event) is plumbed in the launcher but intentionally NOT wired in the dispatch — on FP8/FP4 it was bench-verified net-neutral-to-negative and corrupted the base decode path under cuda-graph replay.
• Branch history includes a reverted earlier attempt (f2f98194d/bb812e6ce, a global triton routing that was withdrawn); net diff vs base is the implementation above only.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #27513652422
Latest PR Test (Extra): ❌ Run #27513652342

[Copilot]

Pull request overview

Adds BF16 (unquantized) support to the experimental_sgl_trtllm MoE-LoRA path by introducing a decomposed, bf16 end-to-end pipeline (permute → raw gate_up GEMM2 → LoRA-aware activation → down GEMM2 → finalize) that reuses existing TRTLLM-gen bf16 cubins and existing permute/activation/finalize kernels, while keeping existing FP8/FP4 paths additive/untouched.

Changes:

• Add a new BF16 MoE-LoRA launcher + TVM FFI export in the fused MoE kernel source.
• Add Python binding for trtllm_bf16_routed_moe_lora and route dispatch to the new BF16 LoRA function via a minimal FlashInferTrtllmBf16MoeQuantInfo.
• Make CUDA-graph MoE buffer sizing robust to BF16 BlockMajorK-prepared 4D expert weight shapes.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
python/sglang/srt/lora/trtllm_lora_temp/lora_layer.py	Detect BF16/unquantized checkpoints and dispatch to the BF16 experimental TRTLLM MoE-LoRA path.
python/sglang/srt/lora/trtllm_lora_temp/lora_dispatch.py	Add BF16 experimental TRTLLM MoE-LoRA dispatch implementation (single-stream).
python/sglang/srt/lora/backend/base_backend.py	Generalize CUDA-graph MoE buffer init to handle BF16 quant-info naming + 4D BlockMajorK expert weights.
python/sglang/jit_kernel/trtllm_lora_temp/data/csrc/trtllm_fused_moe_kernel_launcher.cu	Implement BF16 decomposed MoE-LoRA pipeline and export sgl_trtllm_bf16_routed_moe_lora via TVM FFI.
python/sglang/jit_kernel/trtllm_lora_temp/core.py	Add Python wrapper for the new BF16 routed MoE-LoRA kernel.
python/sglang/jit_kernel/trtllm_lora_temp/__init__.py	Export trtllm_bf16_routed_moe_lora from the jit-kernel package.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[yushengsu-thu]

Qwen3.5-35B-A3B-FP8 / Qwen3.5-35B-A3B-FP8-expert_shared — perf results

GB300 (sm_103), TP4/EP4, single node, bench_one_batch_server 2048/2048, commit 526e0ae22, flashinfer 0.6.11.post1. All four configs on the same base Qwen3.5-35B-A3B-FP8 and same pod, so the only variables are LoRA mode + rank.

• perf(lora): remove GPU-CPU sync barrier (.item()) in MoE LoRA path #2 normal = yanbin jybsuper/qwen35_35b_lora_alpha (real, rank 16, per-expert 2D, 256 experts).
• [lora] Enable piecewise CUDA graph (PCG) and breakable CUDA graph (BCG) with LoRA #3/[WIP][LoRA] BF16 support for the experimental_sgl_trtllm MoE-LoRA path & lora expert_outer_shared #4 expert_shared = auto-generated dummy shared-outer adapter (--experts-shared-outer-loras; gate/up lora_A=[1,r,H], down lora_B=[1,H,r]). Shared-outer engagement confirmed in the server log on all 4 ranks. Random weights → perf only, no accuracy/KL.

All three columns are throughput (tok/s, higher = faster); e2e tok/s = bs × (in+out) / e2e_latency = bs × 4096 / latency. Ratio = lora ÷ the no-lora arm of the same run (each run's own no-lora arm is listed directly above its config block, so every ratio is verifiable in-table; <100% = lora slower). no-lora arms reproduce within ~2% across runs (prefill/extend noisier).

config	bs	prefill/extend tok/s	decode tok/s	e2e tok/s
no-lora · run A	16	27707	3515	6236
no-lora · run A	32	30701	6117	10200
no-lora · run A	64	31320	10625	15868
normal r16 (real)	16	23453 (85%)	2761 (79%)	4942 (79%)
normal r16 (real)	32	22787 (74%)	4922 (80%)	8097 (79%)
normal r16 (real)	64	24107 (77%)	8517 (80%)	12585 (79%)
no-lora · run B	16	27935	3506	6230
no-lora · run B	32	31226	6095	10200
no-lora · run B	64	31424	10660	15917
expert_shared r16 (dummy)	16	22902 (82%)	2561 (73%)	4609 (74%)
expert_shared r16 (dummy)	32	22703 (73%)	4525 (74%)	7546 (74%)
expert_shared r16 (dummy)	64	22627 (72%)	7768 (73%)	11564 (73%)
no-lora · run C	16	31166	3512	6314
no-lora · run C	32	31424	5979	10044
no-lora · run C	64	31549	10372	15613
expert_shared r32 (dummy)	16	22393 (72%)	2527 (72%)	4542 (72%)
expert_shared r32 (dummy)	32	22449 (71%)	4411 (74%)	7372 (73%)
expert_shared r32 (dummy)	64	22772 (72%)	7719 (74%)	11528 (74%)

Notes

• On this FP8 35B hybrid (GDN + MoE), shared-outer is not faster than normal per-expert (decode ~73–74% vs ~79–80% of the no-lora ceiling), and rank 16 ≈ rank 32 in shared-outer mode.
• The no-lora cell uses the default MoE backend (ceiling); lora cells use experimental_sgl_trtllm, so the % bundles "backend switch + LoRA" vs the default ceiling — same convention as the regression pack. The normal-vs-expert_shared comparison is unaffected (both use the same backend/denominator).
• Cross-check vs recorded GB300 reference (ac51ef5ed, 2026-06-06): no-lora ceiling 3482/6041/10603 and normal-r16 +LoRA 2780/4983/8591 — this run matches within ~1–2%.
• Caveat: the real adapter also adapts in_proj_qkvz (GDN) + lm_head; the dummy adapts only q/k/v/o/gate/up/down_proj. The dummy covers fewer modules yet is slower, which strengthens the finding.

[yushengsu-thu]

Qwen3-30B-A3B-Instruct-2507-BF16 / Qwen3-30B-A3B-Instruct-2507-BF16-expert_shared — perf matrix (decode)

Measured on Qwen/Qwen3-30B-A3B-Instruct-2507 (bf16), GB300 (sm_103), TP4/EP4, experimental_sgl_trtllm MoE backend, trtllm_mha, --lora-use-virtual-experts, bench_one_batch_server in=out=2048. Exercises this PR's bf16 MoE-LoRA path + the O1-bf16 two-stream overlap (commit 526e0ae). Two model/adapter configs:

• A. normal — model pack Qwen3-30B-A3B-Instruct-2507-BF16: per-expert LoRA, no --experts-shared-outer-loras. Dummy adapters at rank 16 and rank 32.
• B. shared-outer — model pack Qwen3-30B-A3B-Instruct-2507-BF16-expert_shared: --experts-shared-outer-loras, real adapter yushengsu/lora-diff-Qwen3-30B-A3B-Instruct-2507 (r=32). Verified correct: teacher-forced logprob KL vs the vLLM/trainer reference = 0.0056 ≈ noise floor 0.0042; gsm8k base 0.950 / lora 0.940.

single = single-stream MoE-LoRA dispatch; two = O1-bf16 two-stream overlap (SGLANG_EXPERIMENTAL_LORA_OPTI=1). fus = --enable-/--enforce-disable-flashinfer-allreduce-fusion.

no-LoRA baseline (default fused backend), decode tok/s (bs16/32/64): fus ON 3932.5 / 6991.4 / 11757.9 · fus OFF 3782.0 / 6757.8 / 11437.3

decode tok/s (value = lora; % = lora ÷ no-LoRA at the matching fusion)

config	bs	single, fus ON	two, fus ON	single, fus OFF	two, fus OFF
A. normal r16	16	2160.9 (54.9%)	2701.8 (68.7%)	2110.5 (55.8%)	2642.2 (69.9%)
A. normal r16	32	4005.9 (57.3%)	4887.1 (69.9%)	3920.4 (58.0%)	4762.5 (70.5%)
A. normal r16	64	7067.5 (60.1%)	8445.4 (71.8%)	6897.2 (60.3%)	8306.3 (72.6%)
A. normal r32	16	2073.8 (52.7%)	2594.9 (66.0%)	2046.2 (54.1%)	2483.9 (65.7%)
A. normal r32	32	3897.6 (55.7%)	4723.5 (67.6%)	3791.9 (56.1%)	4558.2 (67.5%)
A. normal r32	64	6939.0 (59.0%)	8316.4 (70.7%)	6784.4 (59.3%)	8095.8 (70.8%)
B. shared-outer r32	16	1932.3 (49.1%)	2365.6 (60.2%)	1894.9 (50.1%)	2314.5 (61.2%)
B. shared-outer r32	32	3577.2 (51.2%)	4278.6 (61.2%)	3487.9 (51.6%)	4207.6 (62.3%)
B. shared-outer r32	64	6185.9 (52.6%)	7424.1 (63.1%)	6070.4 (53.1%)	7265.5 (63.5%)

Takeaways

• Two-stream (O1-bf16) is the dominant win, across all configs: +~25% (normal r16), +~20% (normal r32 / shared-outer) over single-stream — confirms the two-stream overlap added in this PR pays off on bf16.
• allreduce fusion ON vs OFF barely moves the LoRA cells (<1.5pt at fixed stream); fusion ON is end-to-end clean on this path (no graph-capture illegal access) — verified at capture + warmup + bs64 bench + profiler.
• Rank ordering r16 > r32 > shared-outer (lower rank lighter; the shared-outer triton branch + wider all-linear coverage cost a bit more).
• All cells coherent. The shared-outer absolute gap vs normal mixes module coverage (real all-linear vs the dummy's q/k/v/o/gate/up/down) with the shared-outer layout.
• No new sglang code is needed for the shared-outer config — --experts-shared-outer-loras already flows through the bf16 path added here (dispatch passes it to merged_experts_fused_moe_lora_add; mem_pool expert_dim=1 layout is dtype-agnostic). This matrix is informational; the code is at 526e0ae.

[yushengsu-thu]

merged: sgl-project#28953

[yushengsu-thu]

get merged: jybsuper@21ef24a