[diffusion] Add the dynamic LoRA delta inside the second GEMM

[mickqian]

Motivation

A dynamic (unmerged) LoRA computes its update as a full [tokens, out] tensor and then handles it in separate passes:

• scales it by alpha / rank, by the adapter strength, and by the request's LoRA scale, one pass each (two of them multiply by 1.0 in the usual case);
• adds it to the base output in a fourth pass;
• for row-parallel layers, adds the bias in another pass.

On a fast GPU these memory passes cost more than the base GEMM next to them. At real layer shapes on B300, the adapters CI and MiniMax-H3 use add the following on top of the base linears:

• Wan2.1 1.3B: +147%
• Z-Image-Turbo: +107%
• H3: +38%

This cost matters more now that --lora-merge-mode auto keeps adapters dynamic when a BF16 merge would round most of their update away (#43385), as with the H3 Turbo LoRA.

Modifications

• quality="lossless" (the default). The second LoRA GEMM accumulates the update straight onto the base output (addmm, beta = 1), so the [tokens, out] delta is never materialized. Before that, the combined scale (alpha/rank × strength × request scale) multiplies the rank-wide intermediate x @ Aᵀ as a tensor op, not as addmm's alpha.
  • Why not alpha: the compiled forwards read the request scale as a symbolic float, which a keyword argument would freeze at its first value. CI caught exactly that in test_request_lora_scale_preserves_delta_offset_and_base.
  • Stacked q/k/v (or gate/up) adapters take one addmm per group, on column views of the output.
  • A row-parallel layer's bias goes into the base GEMM's epilogue, on rank 0, as RowParallelLinear does without LoRA.
  • An FP32 adapter on BF16 activations keeps FP32 GEMMs and adds once.
• quality="exact". The reference chain is unchanged except that multiplications by a unit scale are skipped. x * 1.0 is exact, so the output stays the same bit for bit.
• The four dynamic forwards (replicated, column-, row-parallel, nn.Linear) now share one _add_lora_delta, so the two tiers are implemented once.
• The fused-kernels page lists the new request-gated fusion.

Accuracy Tests

• Layer level, at the real shapes. Each layer kind the Cseti Wan, elusarca Z-Image and Larry H3 Turbo adapters touch, with their ranks, on B300:

  • At exact, every layer is torch.equal to main.
  • At lossless, every layer passes the lossless-tier admission gate (quality_tier_admission.assert_error_no_worse_than_reference) against the FP64 math, with main as the baseline.
  • The error never grows. Where the separate bias pass goes away it shrinks: Wan's output projection goes from RMS 2.34e-3 to 1.93e-3 and its FFN down-projection from 5.70e-3 to 4.67e-3.

• New unit test. test_lora_dynamic_delta_in_gemm.py (CPU) checks:

  • the exact tier against the reference chain bit for bit, across unit and non-unit alpha, strength and request scale, for 2D and stacked adapters;
  • the lossless tier against the admission gate for 2D, stacked and FP32 adapters;
  • the row-parallel bias for both tiers;
  • that the default quality takes the lossless path.

  With it, the LoRA unit tests pass on B300 (91 passed).

• End to end. MiniMax-H3 T2VA at Ulysses 2 with the Larry Turbo v4 LoRA, --lora-merge-mode dynamic, 9 steps, same seed:

  • The MP4 at quality="exact" is byte-identical to main (2fa46923e49d).
  • At the default quality the video is a different but equally sharp sample (33d18ed0e431, main gives 266717abee3e), as any lossless-tier change gives on H3.
  • CI's consistency cases pin exact, so their goldens are unaffected.

Speed Tests and Profiling

LoRA cost per DiT forward, B300. Each value is the sum over the layers the adapter touches of layer(x) minus the base linear alone, isolated CUDA-event timings:

Model (adapter rank, tokens)	main	this PR, exact	this PR, lossless (default)
Wan2.1 1.3B 480p (128, 32760)	+76.3 ms (+143%)	+51.6 ms (+96%)	+29.6 ms (+55%)
Z-Image-Turbo 1024² (32, 4352)	+28.8 ms (+99%)	+22.0 ms (+76%)	+7.2 ms (+25%)
MiniMax-H3 T2VA, Ulysses 2 (64, 18871 per rank)	+174.4 ms (+38%)	+105.5 ms (+22%)	+71.1 ms (+15%)

What remains in the lossless column is reading x again for the first LoRA GEMM and one read-modify-write of the output in the second. That is the floor for an update applied next to a cuBLAS GEMM.

End to end, 2×B300. MiniMax-H3 T2VA at Ulysses 2 with the Larry Turbo v4 LoRA, 9 steps, client latency of the two warm requests per arm:

--lora-merge-mode / quality	main	this PR
merge	13.40 / 13.39 s	—
dynamic, default quality	14.59 / 14.62 s	13.99 / 13.79 s
dynamic, quality="exact"	14.62 / 14.60 s	14.40 / 14.19 s

At the default quality, a dynamic adapter now costs 0.5 s (+3.7%) over merged weights instead of 1.2 s (+9.0%).

The follow-up commit (scale on the rank-wide rows instead of alpha) leaves this adapter's output unchanged: its combined scale is 1.0, so the multiplication is skipped. A rerun after it gave the same MP4s at both qualities (33d18ed0e431, 2fa46923e49d) and 13.78 / 13.78 s at the default quality.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #38055503503
Latest PR Test (Extra): ❌ Run #38055503258
Latest PR Test (AMD ROCm 10): ❌ Run #38055503360

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
sglang-doc	🟢 Ready	View Preview	Oct 10, 2026, 6:18 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[mickqian]

/tag-and-rerun-ci