[diffusion] cache LoRA-merged weights in files the page cache can hold - #36062
Merged
Conversation
mickqian
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
ping1jing2 and
yichiche
as code owners
August 23, 2026 09:46
Merging an adapter in place copy-on-writes the checkpoint mapping, and the LoRA wrapper clones a second full base snapshot for unmerging: on MiniMax-H3 the two together peak at 75.3 GiB of anonymous host memory, the host budget reads 0.0 GiB, every pin plan collapses to zero, and a real 32 GB machine dies. All of it while every VRAM number stays green. Three changes, measured together at a 32 GiB host convention: - The startup adapter merges into a one-layer device temporary (same device and fp32 policy as the in-place path, so the bytes are identical) and the result lands in a per-layer cache file that is mapped straight back. The parameter ends up file-backed, and the offload managers classify it as mapped on their own. - With merges going through the cache the base storage is never written, so the unmerge snapshot becomes a zero-copy view instead of a 38 GB clone; a later in-place merge materializes the clone first. - A second start with the same (base, adapters, strengths, alphas) combination hits the cache and skips the merge entirely. Peak anonymous host memory 75.3 -> 19.2 GiB; host budget back to 18.4 GiB; DiT and text-encoder pins return. Multi-GPU and dynamic set_lora keep the in-place path. SGLANG_DIFFUSION_DISABLE_LORA_MERGE_CACHE restores it everywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mickqian
force-pushed
the
feat/lora-merged-store
branch
from
August 23, 2026 14:18
b0c23a8 to
587534f
Compare
mickqian
changed the title
mickqian
force-pushed
the
feat/lora-merged-store
branch
2 times, most recently
from
August 24, 2026 03:33
4ed9755 to
f7ba194
Compare
Review feedback: LoraMergeCache.get/put mutated the caller's parameter (a query with a side effect), and the pipeline's merge-via-cache did the layer's work — merge math and the merged/cpu_weight state transition. The cache is now a pure vault: get(name, shape, dtype) and put(name, merged) take and return tensors, touch no nn state, and dropping a mismatched combination is a documented contract instead of a hidden side effect. The layer gains compute_merged_weight() (same device and fp32 policy as the in-place merge, so the bytes are identical) and install_merged_weight() (the one place the cached-merge state transition happens). The pipeline's part shrinks to orchestration: get, else put(compute), else fall back in place, then install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mickqian
force-pushed
the
feat/lora-merged-store
branch
from
August 24, 2026 03:36
f7ba194 to
45b3486
Compare
longxin9715
added a commit
to longxin9715/sglang
that referenced
this pull request
Aug 24, 2026
…n-transport1 * 'main' of https://github.com/sgl-project/sglang: (326 commits) [diffusion] feat: cache LoRA-merged weights in files the page cache can hold (sgl-project#36062) [diffusion] Speed up LingBot high-quality VAE decode (sgl-project#36024) [diffusion] Honor XDG cache for model overlays (sgl-project#36019) Support streaming session on NPU (sgl-project#32597) fix(xpu): read enable_deterministic_inference from the config bag (sgl-project#36149) xeon ci fail fast strategy change (sgl-project#36146) [diffusion] Fix Hunyuan QKV pack indexing at production video shapes (sgl-project#36009) [diffusion] Refresh quality and BCG benchmark skills (sgl-project#36016) [MoE] Gather the cutlass MoE activation and its scales in one launch (sgl-project#34915) [diffusion] feat: add plain component weight overrides (sgl-project#36086) [diffusion] feat: support loading mixed w4a8 text encoders (sgl-project#36037) [diffusion] Default Hunyuan VAE to tiled decode (sgl-project#36012) fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models) (sgl-project#33057) chore: move cuda_vmm_utils.py under srt/utils/ (sgl-project#36053) [Intel XPU] Add xpu pass for biased_topk and hash_topk (sgl-project#33323) [CPU] Fix NUMA/core binding for DP ranks (sgl-project#32856) [Fix] Harden FlashAttention CUDA graph metadata bounds (sgl-project#35454) [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (sgl-project#33354) [diffusion] Fuse LongCat-Image QKNorm and interleaved RoPE (sgl-project#35995) [diffusion] Keep LongLive2 components resident on large GPUs (sgl-project#35993) ... # Conflicts: # python/sglang/srt/multimodal/processors/base_processor.py # python/sglang/srt/server_args.py
saturn-acc
pushed a commit
to saturn-acc/sglang
that referenced
this pull request
Aug 31, 2026
…an hold (sgl-project#36062) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
jakki-amd
pushed a commit
to jakki-amd/sglang
that referenced
this pull request
Sep 9, 2026
…an hold (sgl-project#36062) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Serving a distilled LoRA (e.g. lightx2v's 4-step MiniMax-H3-Turbo) under layerwise offload currently costs the host two full copies of the DiT: the in-place merge copy-on-writes the checkpoint mapping (61.7 GB becomes anonymous memory), and the LoRA wrapper clones a second full base snapshot for unmerging (38 GB more). Measured peak: 75.3 GiB of anonymous host memory —
host memory availablereads 0.0 GiB, every pin plan collapses to zero, and a real 32 GB machine dies. Every VRAM number stays green throughout (#36051 adds the CI guard for exactly this shape).Three changes, verified end to end on the turbo adapter at a 32 GiB host convention:
set_lora) materializes the clone first, preserving the old semantics.Measured (RTX 4090, MiniMax-H3 + 4-step turbo LoRA, 12 GiB cap + 32 GiB host convention)
Cache: 38 GB, 208 layers, one safetensors file per layer under
SGLANG_DIFFUSION_CACHE_ROOT/lora_merge_cache/<key>/. Multi-GPU and dynamicset_lorakeep the in-place path.SGLANG_DIFFUSION_DISABLE_LORA_MERGE_CACHErestores it everywhere.🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ✅ Run #32687067781
Latest PR Test (Extra): ❌ Run #32687067711
Latest PR Test (AMD ROCm 7.2): ⏳ Run #32687067896