[mickqian]

Motivation

FUSED_LINEAR_GELU is in the "lossless" tier, and quality has defaulted to "lossless" since #42370, so the fused linear+GELU epilogue mounts by default. With a dynamic (unmerged) LoRA, the site's linear is a BaseLayerWithLoRA. Its .weight and .bias forward to the base layer, and it passed every duck-typed guard. So fused_linear_gelu_tanh(x, proj_in.weight, proj_in.bias) multiplied the base weight only and silently dropped the adapter's delta.

This affects every GELU site that serves a dynamic LoRA:

• LTX-2 two-stage, stage 2, always: the distilled LoRA is applied dynamically on 1370 layers;
• LTX-2.3 in original device mode, which is the default on H100/A100;
• dynamic LoRA serving on Wan, FLUX, HunyuanVideo, GLM-Image, Qwen-Image and LongCat.

CI did not catch it:

• [diffusion] Name the quality tiers after what they guarantee #42370 pinned every consistency-checked case to quality="exact";
• the site unit tests used plain nn.Linear;
• nightly does run LTX-2.3 TI2V at the default tier with a dynamic LoRA on H100, but it measures latency only;
• the Wan2.2-A14B LoRA case uses layerwise offload, so the fusion never mounted.

Modifications

fused_linear_gelu_site.py:

• can_use_linear_gelu takes the module forward while a LoRA wrapper adds output (lora_active). A disabled adapter, or a merged one without an output offset, keeps the fused path, because the base weight the epilogue reads already holds the update.
• The mount-time guard judges a LoRA wrapper by the layer it wraps. Before, the wrapper hid that layer's quantization, skip_bias_add and multi-rank gather_output.
• The module docstring still said the path mounts only for quality="high". It now describes the lossless tier.

Tests:

• test/unit/test_quality_fusion_lora.py: every fusion in _QUALITY_FUSION_HANDLERS either runs on a site whose linears carry a dynamic adapter, or is listed with the reason it cannot drop an adapter's output. So a new fusion has to pick a side. On a site, the mounted output must equal the unfused forward and differ from the base weight alone. The file also pins the per-call guard to the adapter state and the static guard to the wrapped layer.
• New 1-GPU server case wan2_1_t2v_1_3b_dynamic_lora_default_quality_1gpu: Wan2.1-T2V-1.3B serves the Arcane-Jinx adapter unmerged, at 480x272, 9 frames, 8 steps. The adapter carries ffn.0 (= ffn.fc_in) LoRA weights on all 30 blocks. The case needs no golden: the default-quality output must match a quality="exact" request to the same server, and differ from a request with the adapter off. Two DiffusionTestCase fields express these checks for any case: reference_quality and run_lora_effect_check.
• The case runs in the NVIDIA 1-GPU PR suite. It is skipped on AMD, whose PR lane parametrizes the same case list, because its thresholds are calibrated on NVIDIA only.
• The case's baseline is intentionally absent from perf_baselines/h100.json. Its first CI run fails and prints the block to add (the usual two-step seeding).

This PR and "[Fix] diffusion: Keep the linear-GELU verdict cache out of torch.compile traces" both edit can_use_linear_gelu. They merge cleanly in either order. The adapter check sits above that PR's compile branch, so it also applies while tracing; keep that order if either one is rebased.

Accuracy Tests

LTX-2 two-stage on 2xH200, against a stage-2-only exact reference: stage 1 runs at lossless and stage 2 is forced to exact. Stage-1 latents are bitwise identical across arms.

	PSNR vs reference	high-frequency energy vs reference
main	23.4–24.0 dB	13–15% lower (blurrier)
this PR	31.3–34.1 dB	within 1%

quality="exact" output is bitwise unchanged.

New server case on H200, per key frame. The case passes at mean_abs_diff ≤ 3.0 and PSNR ≥ 30 dB against quality="exact", and mean_abs_diff ≥ 12.0 against LoRA off.

	vs quality="exact"	vs LoRA off	result
this PR	mean_abs_diff 0.73 / 1.41 / 1.55, PSNR 45.9 / 42.1 / 41.3 dB	mean_abs_diff 15.0–16.2	pass
this PR with only the guard reverted to main	mean_abs_diff 8.35–8.75, PSNR 25.7–26.6 dB	mean_abs_diff 9.1–12.6	fail

The case takes 79–88 s on H200 (also re-run on this branch merged with the compile-guard PR: same numbers, pass).

Unit tests: test_quality_fusion_lora.py 5/5 pass on GPU. On main, test_linear_gelu_static_guard_reads_the_wrapped_layer fails: the wrapper hides gather_output on a TP2 layer.

Speed Tests and Profiling

Only requests with an active dynamic LoRA change path: their GELU sites run unfused. LTX-2 two-stage e2e on 2xH200 went from 7.34 s to 7.51 s. Requests without a LoRA, or with a merged one, keep the fused epilogue.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ✅ Run #39910057241
Latest PR Test (Extra): ❌ Run #39910057315
Latest PR Test (AMD ROCm 10): ❌ Run #39910057320

[mickqian]

/tag-and-rerun-ci