[yueming-yuan]

Motivation

An adapter may target any subset of the members SGLang packs into one fused LoRA module: q_proj/k_proj/v_proj -> qkv_proj, gate_proj/up_proj -> gate_up_proj, q_a_proj/kv_a_proj_with_mqa -> fused_qkv_a_proj_with_mqa, Inkling wq_du/wk_dv/wv_dv/wr_du -> qkvr. The loader concatenated the members into the fused key at normalization time, assuming specific members are present. On Qwen3-0.6B with sglang-miles:

adapter targets	before
q, o	KeyError: ...self_attn.v_proj.lora_A.weight
k, v	ValueError: Cannot find target module name
up, down	ValueError: Cannot find target module name

A trainer could only work around this by exporting zero A/B pairs for untrained members, with widths computed from its own copy of the engine's layout.

Modifications

Members are no longer concatenated in LoRAAdapter. The memory pool writes each member into its own slot of the fused buffer:

• A: member i goes to rows [i*r, (i+1)*r); B: member i goes to the member's row range of the local output. Absent members are left zero, so the fused module computes exactly the partial adapter.
• The per-member geometry (full rows, TP-local offsets, TP slice incl. KV-head replication and MLA first_output_dim) comes from the *WithLoRA wrapper that already owns the fused layout: lora_b_member_full_rows(), lora_b_member_local_offsets(), slice_lora_b_member(). slice_lora_b_weights is rebuilt from the same methods, so there is one definition per wrapper. packed_modules_mapping is not used; it is absent on Llama/Qwen2/Qwen3 and does not match the LoRA modules on Qwen3.5 / NemotronH.
• LORA_FUSED_MEMBERS in lora/utils.py is the single member table; params_mapping and stacked_rank derive from it.
• An adapter that already carries the fused key (qkv_proj, fused_qkv_a_proj_with_mqa, ...) is split by the same geometry; carrying both the fused and the per-member form is rejected.
• MoE (routed / shared expert) gate/up keep the existing stacked path.

Behavior: an adapter covering all members loads byte-identical buffers to the old concatenating loader (unit test, all TP ranks, max_lora_rank = r and r+1). Partial adapters that raised before now load. An MLA adapter with a fused fused_qkv_a_proj_with_mqa key is newly accepted.

Tests

test/registered/unit/lora/test_lora_fused_member_load.py (CPU, 37 cases) on fake QKV / MergedColumn / Inkling qkvr / MLA wrappers:

• complete adapter == old concatenating loader (oracle copied from sglang-miles@4076a68579), every TP rank, max_lora_rank in {r, r+1}
• every member subset loads, absent members are zero
• fused-key adapters (shared and stacked A) match their per-member form; fused + per-member is rejected
• LoRAAdapter keeps dense members separate and still stacks expert gate/up

E2E on H200, Qwen3-0.6B, r=8: each adapter is compared against the same SGLang running the PEFT-merged weights (merge_and_unload). Mean |logprob diff|, rows = LoRA run, columns = merged model:

	full	q_o	k_v	up_down
full	0.046	0.610	0.545	0.650
q_o	0.613	0.038	0.212	0.243
k_v	0.543	0.226	0.045	0.236
up_down	0.649	0.258	0.241	0.037

The diagonal is bf16 rounding of the merged weights; every LoRA run matches only its own merged model. The full adapter's logprobs are identical (==) to sglang-miles before this change.

Not in this PR

Fused modules whose members are not plain row blocks of B, and MoE gate/up, keep the current path: GDN in_proj_qkvz / in_proj_ba, Mamba in_proj, routed/shared-expert gate_up_proj.

Consumer

radixark/miles#4101 drops the trainer-side zero-fill and exports only the trained members.

---

CI States

Latest PR Test (Base): ❌ Run #38660217031
Latest PR Test (Extra): ❌ Run #38660216994
Latest PR Test (AMD ROCm 10): ❌ Run #38660217049

[jybsuper]

LGTM