Serve LoRA adapters on six models (5/9) by dghosef · Pull Request #822 · mstar-project/mstar

dghosef · 2026-10-22T03:34:08Z

Based on main. Must not merge before #1082: this is the first PR that serves LoRA under TP, and without #1082's fence a TP rank can apply a teardown between broadcast and admit and pick a different slot than the other ranks. Pending GPU testing: the rows marked "pre-rebase head" ran before the rebase onto current main; I'll rerun them and update the table.

What this does

Turns on per-request LoRA for six models:
- Qwen3.5: every linear of the text stack, lm_head, embed_tokens, and the vision tower in its own pool.
- Orpheus and Higgs Audio.
- The Qwen3-Omni Thinker: attention, dense-layer MLPs and lm_head. An adapter on the MoE experts or router is refused.
- The Qwen3-TTS Talker.
- Chatterbox's T3, whose guidance rows run the request's adapter too.
A resources: lora: block in the serving YAML turns it on. Without one, every model runs exactly as before.
Qwen3.5 takes Hub adapters in every layout PEFT writes for it, including Tinker's split delta-net q/k/v and unembed_tokens.
docs/models.rst lists the models left out (Whisper, BAGEL, diffusion and flow models, ...) with the reason for each. It still marks Qwen3-Omni, Qwen3-TTS and Chatterbox not validated: they miss module_parity's merged/hf rule, which is stricter than bf16 noise (see the fp32-oracle rows).

How it works

Each model returns its specs from get_lora_specs (module_lora_spec over its weights root). Each weights root states its checkpoint rules once in checkpoint_rules(), and both the loader and the adapters use them.
Models declare no rows: the engine adds each step's LoRA rows from the step's segments. Chatterbox's uncond rows are segments of the same request, so they run its adapter.
Qwen3.5 vision prompts reach the LLM as token ids (-1 where an image row goes) plus the image rows, and preprocess embeds and splices them inside the step. So the token embedding is an ordinary target, and the shared layer's deferred embedding (module_lora_spec(deferred=), lora_deferred) is deleted.
A leased Qwen3.5 vision-encoder step drops the vision attention from its declaration, because the captured block loop plans its own. It still declares its frames, so the bound patch embed and merger have a plan to run base-only against.
The module_parity and GPU harnesses follow main:
- The Qwen3-Omni reference finds its text stack under transformers 5.19 and 5.22.
- Qwen3-TTS gets a transformers reference.
- References recompute RoPE in fp32.
- The Triton test commits each step after its forward.
- The merged reference checkpoint is merged into every file holding a tensor, so Chatterbox's multilingual T3 can't take the merge.

Tests

Test	Where	Result
`ruff check`	touched files	clean
CPU Core suite	this branch on `main` `53c47cb`, vs `main`	the modules this branch reaches (332 of 589): 7607 passed, 3 failed, the same 3 as `main`; 0 failing only on this branch
CPU: LoRA tests	each model's node through the generic path against merged weights (simulated TP where it shards), Hub key layouts, Qwen3.5 compiled vs eager and captured, a padded vision prefill with an embedding adapter, the conv patch embed compiled vs eager	225 passed, 3 skipped; after the harness fixes, `test_lora_models`/`graphs`/`pool` + `test_chatterbox_model`: 229 passed, 1 skipped
Chatterbox under guidance	H100, head `6e0d46c`, real checkpoint, step-0 max logit gap vs an fp32 merge at `cfg_weight` 0.5	LoRA (Triton, bf16) 0.20, same as the merge itself in bf16 (0.16–0.21); uncond rows forced to base would give 0.74–0.84. Served, graphs on or off: 0.23–0.28 (0.10–0.31 at `cfg_weight` 0)
GPU: LoRA kernels (`test_lora_triton_gpu.py`)	every Qwen3.5 target's delta bf16 vs fp32, delta-net rank shards, the 248320-wide LM head at TP4, one capture serving every adapter mix	26 passed (pre-rebase head); the step-order fix for its capture cases pending
Qwen3.5 vs HF + PEFT, 0.8B captured	2× H200; greedy, 48 tokens, vLLM's rule; 7 adapters + base, 6 text + 2 image prompts each	96/128 identical, 0 failures; same-code floor 50/64 (pre-rebase head)
Qwen3.5, 0.8B eager / captured vs eager	same	97/128 / 96/128 identical, 0 failures; eager floor 59/64 (pre-rebase head)
Qwen3.5, 9B captured / eager / captured vs eager / TP2 vs 1 GPU	released layout, Tinker, a vision adapter	51/64 / 52/64 / 54/64 / 57/64 identical, 0 failures (pre-rebase head)
Qwen3.5 LM head / embedding adapter (`pirate`, 4B), image prompts at c=1 vs c=3		pending
Qwen3.5 prefix cache with two adapters (`benchmark.lora.prefix_cache --endpoint chat`)		pending
Orpheus `module_parity`	Hub adapters jarvis, rsLoRA, bengali, sft	merged 11/15 (one cos 0.9969, two miss only the 10× control, one near-tie); hf 12/12 at cos ≥ 0.99999; served: all 200 (pre-rebase head)
Higgs Audio `module_parity`	2 synthetic adapters	merged 9/9 (cos ≥ 0.9994); hf 9/9 (cos ≥ 0.9997); served: 0 failures (pre-rebase head)
Qwen3-Omni `module_parity`	H200, transformers 5.22, eager	merged 0/9 (base rows fail too); hf 4/9 (adapter rows miss only the 10× control); fp32 oracle 7/9 (min cos: LoRA 0.9927, bf16 merge 0.889); per layer LoRA/merge error median 1.00
Qwen3-TTS `module_parity`	same	merged 4/9; hf 8/9; fp32 oracle 9/9; per layer median 1.00, max 1.39; served: all 200
Chatterbox `module_parity`	same	merged 6/9; hf 9/9; fp32 oracle 7/9 (the misses are layer 3's massive activation); per layer median 1.00, max 1.21; served: all 200
Retention vs vLLM 0.32.0 / SGLang 0.5.22, 0.8B	one H200 per engine, tok/s ÷ the same engine's LoRA-off tok/s, c=1 / 32 / 64, 3 rounds	base requests: mstar 99.2 / 95.0 / 98.9%, vLLM 90.5 / 90.6 / 86.8%, SGLang 74.8 / 83.9 / 83.1%. One adapter: 89.0 / 89.1 / 96.5%, vLLM 68.2 / 80.4 / 79.0%, SGLang 68.1 / 80.3 / 80.2%. Churn (3 adapters, 2 slots): 95.5 / 62.8 / 71.4%, vLLM 66.8 / 53.0 / 51.4%, SGLang 50.2‡ / 40.2‡ / 46.4‡ (pre-rebase head)
Retention, 9B	same, 2 rounds for competitors	base requests: mstar 100.0 / 99.8 / 100.4%, vLLM 100.5 / 97.2 / 95.7%. One adapter: 90.6 / 88.4 / 90.2%, vLLM 76.8 / 77.1‡ / 75.7‡, SGLang 63.1 / 72.1 / 68.5‡. Churn: 90.6 / 56.5 / 60.5%, vLLM 74.7 / 42.5 / 42.6% (pre-rebase head)
Churn latency	TTFT p50 at c=32 / 64	mstar holds a request until a slot frees: 199 / 401 ms (0.8B), 599 / 1046 ms (9B), vs vLLM's 48-63 ms (pre-rebase head)
Profile (`MSTAR_PHASE_TIMING`)	0.8B / 9B	0.8B is CPU-bound and LoRA's plan costs 0.05-0.07 ms a step; 9B is GPU-bound and an adapter step adds 0.5 ms (c=1) to 0.9 ms (c=64) of kernel time (pre-rebase head)

GPU rows: torch 2.12.1+cu132, triton 3.7.1, flashinfer 0.7.2, captured and compiled as served unless noted (the Chatterbox guidance row: torch 2.11.0+cu130, triton 3.6.0, Python graph runtime). Competitor cells count only with 0 failed requests; ‡ means every round of that engine failed some, and mstar had none.

AI assistance (Claude) was used for the code, tests and this description.

🤖 Generated with Claude Code

dghosef · 2026-10-22T03:44:00Z

Addressed in 2a974c1 and the description:

Vision replay (P1): fixed. The GPU vision-adapter results predate the rebase onto Qwen3.5: video input, matching transformers' Qwen3VLProcessor #427, so they ran the old leasing decision (batch size and patch count) and don't cover this code. The description now says the fix is untested on GPU.
Vision step with LoRA on: described. A leased step now declares the LoRA rows only.
GPU builds and commits: added (2× H200, torch 2.12.1+cu132, triton 3.7.1, flashinfer 0.7.2; lora/combined @ 39a8346b9, before the rebase). The Triton count was on this branch's rebased tree, before the fix.
CPU suite: the per-branch full suite is in Per-request LoRA adapters: the adapter pool, served from the serving config (3/9) #820, after the rebase: the same 3 signal-test failures as main, plus a capture-budget stub fixed in Per-request LoRA adapters: the adapter pool, served from the serving config (3/9) #820.

dghosef · 2026-10-22T03:58:00Z

Follow-up from a review of how the rebase interacts with main: d56b5d4 updates main's test_vision_block_loop_replays_exactly_when_leased (#427). Its placeholder step=object() became a real step declaring VISION_ATTN, since the replay decision now reads the step's declaration. 441ad7a lists Fish Audio S2-Pro (#492) among the models that take no adapters yet. Nothing else in main's 31 new commits was found to break the stack (read-only check; CI is the first run).

dghosef · 2026-10-23T18:17:37Z

/review

dghosef · 2026-10-23T18:20:37Z

/review

mstar-review · 2026-10-23T18:21:37Z

Automated review

🟡 No findings on e195b4f, light review

The code hasn't changed since the last summary (head is still e195b4f). The vision-replay P1 is still fixed, and the description edit fixed the last open item: it no longer says the text model gains logits(), and it now says the LM head delta runs on lm_head in LLMSubmodule._forward.

What the review checked

Description vs code: the 'Qwen3.5 details' section now names only Qwen3_5LanguageModel.embed() and puts the LM head delta in LLMSubmodule._forward, which matches the code.
Vision replay: _replay_block_loop replays only when the step doesn't declare VISION_ATTN. Qwen3.5: video input, matching transformers' Qwen3VLProcessor #427's test_vision_block_loop_replays_exactly_when_leased now passes a real SubmoduleStep that declares VISION_ATTN, where it used to pass object().
test_lora_graphs.py keeps its pytest.importorskip("safetensors") guard and runs on CPU only (F64, torch.device('cpu')), so it stays safe for the CPU Core job.
Earlier passes on this head traced these and found them sound: the deferred embedding delta in preprocess, the LoraStep in the leased block-loop region, Chatterbox CFG row order against PER_SEGMENT, the loader refactors for Higgs Audio, Qwen3-Omni, Qwen3-TTS and Chatterbox, and configs without a lora: block running unchanged.

Not reviewed: benchmark/lora/*.py (compare_engines, module_parity, phase_plan, prefix_cache, qwen3_5_parity, load, eager_site/sitecustomize.py), benchmark/lora/README.md, test/integration/test_lora_triton_gpu.py, test/modular/lora_module_helpers.py, test_lora_models.py; test_lora_graphs.py beyond its imports and first tests, test/modular/fixtures/*.json.

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

Posted by mistake

…atterbox The harness now has a transformers reference for the Qwen3-TTS Talker and keeps the references' RoPE in fp32, so the table's "no reference for the talker" and Chatterbox's open cosine-0.65 row no longer hold, and Higgs Audio's transformers pass predates the RoPE change. Layer by layer, an adapter's median error on all three models is within about 10% of a bf16 merge's; the misses are bf16 noise below the checks' tolerance. Orpheus is described as it measured: a few misses against merged weights, none against transformers + PEFT. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…sion - Qwen3-TTS's stack builds main's ResidualCodePredictor beside the Talker. - No compute scope around a forward: the pool's own plan guards deltas now. - The Qwen3.5 stack compiles without fullgraph (its per-layer cursor moves break the graph without LoRA too); the test pins that adapters add no break. - A replay's addressing is read before its commit, which clears the plan. - A text-only adapter is empty for the vision pool from its header, so its rows never wait: the test checks they take the zero slot with the pool full. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

A vision prompt now reaches the LLM as its per-position ids (-1 where an image or frame row goes) plus the packed vision rows. `preprocess` embeds the ids with `embed_tokens` and splices the rows in with one masked scatter, inside the planned step, so the token embedding is an ordinary LoRA target there as on the text walks. This removes the deferred embedding: `deferred=`, `Qwen3_5LanguageModel.embed`, `embed_ids`, `_sentinel_ids`, the `all()` gate in `preprocess` and the `lora_deferred` probes. The gate skipped the embedding delta for a whole step whenever a capture bucket padded it with a dummy row (three image prompts on the bs-4 bucket), so every real row lost its adapter's delta. A padding row now carries no ids and no rows, and adds nothing. Test: test_qwen3_5_vision_prompt_text_takes_its_embedding_delta_in_a_padded_step (three image prompts plus the bucket's padding row against merged weights); fails at 3e1e06b (672/928 elements off), passes here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`Qwen3_5ForCausalLM` and `Qwen3_5VisionModel` gain `checkpoint_rules()`; the two loaders read them, and the LoRA specs take them from the roots, so adapter placement can't drift from loading. `qwen3_5/lora.py` drops its explicit `rules=` and the private `_STACKED_PARAMS` import. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Drop `VISION_NAMES`. No serving config spells the old names, so the vision targets take `module_lora_spec`'s default names like every other target; the docs' `targets:` example lists them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

`model.embed` is a compiled region (dynamic in the patch count), and the bound patch conv's patch cutting and in-place delta are traced inside it. Inductor on CPU matches eager at 16 and 24 patches with an adapter row active. Passes at 3e1e06b too: coverage for a suspected bug that CPU Inductor does not show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Delete Qwen3-Omni's `_thinker_remap` / `_get_thinker_stacked_params` shims (the Talker and code-predictor loaders call `thinker_remap` and `thinker_stacked_params` directly), drop Qwen3-TTS's second code-predictor filter (its rules already drop those keys), and have the test helpers load every stack with `checkpoint_rules().load`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…F12) - compare_engines sets MSTAR_REQUIRE_ACCELERATOR_GRAPHS (the old MSTAR_REQUIRE_CUDA_GRAPHS is no longer read), and an mstar server whose log has a "Failed to capture" line is refused before any cell runs, instead of printing a warning after its cells counted. - Qwen3.5 now has a prefix cache, on by default: compare_engines turns it off to match vLLM and SGLang, and phase_plan turns it off because its load repeats prompts. - prefix_cache.py and the README no longer say Qwen3.5 keys no stream. - Internal branch names (`lora/gdn`) are gone from the docstrings. - main's `fail_if_graphs_required` docstring names the variable it reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

….F10, R1.F11) Blank lines before the "LoRA adapters" and "Python frontend" paragraphs, so RST stops merging them into the ones above. The adding_models example is OrpheusModel.get_lora_specs as written. The "Not yet" list names every registered family without LoRA specs. Higgs's docstring names its `prefill_audio` walk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1.F5) The acceptance harness's stand-in `preprocess` read `input_embeds` off a vision row; a row now brings its ids and image rows, so it splices them with the real `_vision_embeds`. The vocab-parallel embedding's collective moves from `prepare_inputs` into the step with it; the docstrings say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…(R2.F1) Qwen3.5 was the only user of a delta the node adds itself; its vision prompt now embeds inside the step, so every model either serves its embedding through the module's own forward (embeddings=True) or excludes it. Drop the parameter, the ModuleLoraMap field, the bind branch, the embeddings-check exemption, its test and its docs bullet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

….F2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

….F3) masked_scatter_ on CUDA allocates an int64 prefix sum the size of the broadcast embeddings (8*N*H bytes; aten/src/ATen/native/cuda/IndexKernel.cpp, masked_scatter__cuda) plus a contiguous N*H mask. The rows' positions come from a cumsum and searchsorted over the 1-D mask instead: O(N) scratch, no host sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vasilevklart

Chatterbox LoRA breaks under guidance (inline), rest checks out on H200.

vasilevklart · 2026-11-03T05:39:08Z

+    def get_lora_specs(self) -> list[LoraSpec]:
+        """Per-request adapters on T3: every linear of its Llama backbone and the speech
+        head (one row per segment: each stream's last token). Under classifier-free
+        guidance a request's cond and uncond rows both run its adapter, as one model


Served with guidance on, the LoRA audio leaves the merged-weights output within the first tokens (3/3, graphs on and off); with cfg_weight: 0 it matches exactly. Looks like the uncond rows run base

Couldn't reproduce a base uncond row, on CPU or on an H100. Step-0 max logit gap vs an fp32 merge, with guidance:

LoRA through the pool (Triton, bf16): 0.20

the fp32 merge itself in bf16: 0.16–0.21

uncond rows forced to base: 0.74–0.84

Served, graphs on or off: 0.23–0.28 with guidance, 0.10–0.31 without, and greedy parts after a near-tie either way. So it looks like bf16 rounding that guidance makes about 1.5× bigger. If you still have your run, step-0 logits against an fp32 merge would settle it.

Separately, module_parity's merged checkpoint could land in the multilingual T3 when the snapshot has both files, leaving t3_cfg on base. Fixed in 33c895b.

vasilevklart · 2026-11-03T05:39:09Z

+    ctx.set_padded_rids([*rids, *pad])
+    assert pool.admit(step, ctx).ok
+    pool.plan(step, ctx)
+    pool.commit(step, ctx)


nit: the GPU harnesses drifted from main: this fails 3/26 on H200 (a LoRA delta was called with no planned step, commit before the forward), module_parity crashes on enter_compute, prefix_cache.py on prefix_key_streams

Fixed in fb5f448. The Triton test now commits after the capture and after each replay, module_parity runs the step between plan and commit, and prefix_cache reads KVLabel.keys. 6e0d46c makes the prefix_cache preflight exit cleanly on non-LoRA models.

vasilevklart · 2026-11-03T05:39:09Z

-        # leased, the captured block loop plans its own attention per replay
-        if (piecewise_leases or {}).get(VISION_BLOCK_LOOP):
+        # Leased, the captured block loop plans its own attention per replay. A node
+        # serving adapters still declares its frames, so the engine adds their LoRA


nit: no row of a leased step runs a vision adapter (the base-only region refuses them in can_lease), so these frames only give the bound patch embed and merger a plan to run base-only against

Right, fixed the comment in aaa1639. The frames stay: without a planned step the bound patch embed and merger refuse to run.

…host reads of value-less tensors Found running mstar-project#822's test/integration/test_lora_triton_gpu.py under the emulator (which then reproduced the review's 3/26 "a LoRA delta was called with no planned step" failures on a CPU box): - header-only loading read every safetensors tensor as meta, so a test's own adapter file (its alpha scalars) lost its values; tensors under 64K elements are now read for real (a shapes-only checkpoint reads them as zeros); - torch.equal/allclose, f"{x:.3f}", .item() and friends on a value-less tensor, and host ops mixing one with real CPU tensors, read it as zeros, as every other host read does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…K.3) No row of a leased step runs a vision adapter (the base-only block loop refuses them in can_lease); the frames give the bound patch embed and merger a planned step to run base-only against, without which their deltas refuse to run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…on (K.2) - test_lora_triton_gpu: bucket_step no longer commits before the forward; the capture and each replay commit after it, as the engine does (the 3 capture cases refused: a delta called with no planned step). - module_parity: the pool has no compute scope; the mixed step runs between its plan and its commit. - prefix_cache: a model keys a prefix label through KVLabel.keys in kv_labels, not prefix_key_streams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…_labels never reads a config (K.2, review P3) keyed_labels on a bare instance crashed for models whose kv_labels read self.config (Cosmos3, V-JEPA 2) instead of exiting with the preflight's reason. Those models serve no LoRA, so the LoRA check now comes first; all 54 registered models now pass or exit cleanly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

- `kill -USR1 <pid>` prints every thread's Python stack (traceback, from a thread that sigwaits; the signal is blocked before torch starts its native threads). faulthandler now dumps only the faulting thread: dumping all of them from a signal handler crashed mid-dump while they ran. - mstar_sim.pytest_plugin (`-p mstar_sim.pytest_plugin --gpu-only`): runs the tests a CPU box skips for want of a GPU, and reports an assert that fails after a read of never-computed values as an expected failure; any other failure stays one. On mstar-project#822 it leaves exactly the review's 3/26. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#823's net change (its 9 commits, d057835..f852a65) applied as one commit on lora/3-models 7be9c60, which was rebased linearly onto main. Conflicts were with main's changes: ClientStreamInput's removal, CLIENT_STREAM_ROUTE, the data worker's ResultQueue, the API server's CPU pinning, the worker's per-message spans, and slot_wait_steps' removal from the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…-worker _resource_specs) #822 made _resource_specs a list of per-worker copies, each with its own adapter-info cache. _lora_ops read it as a dict, and a load or unload now updates every copy, so a reload under the same name is checked as its new file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…s its tensors The merge went into the first .safetensors that held the names, in rglob order. A full ResembleAI/chatterbox snapshot also holds the multilingual T3 under the same names, so the merge could land there and leave t3_cfg, the file the loader reads, as the base model, with no error. Every file holding a tensor is now merged, and a delta merged into none still fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#823's net change (its 9 commits, d057835..f852a65) applied as one commit on lora/3-models 7be9c60, which was rebased linearly onto main. Conflicts were with main's changes: ClientStreamInput's removal, CLIENT_STREAM_ROUTE, the data worker's ResultQueue, the API server's CPU pinning, the worker's per-message spans, and slot_wait_steps' removal from the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…-worker _resource_specs) #822 made _resource_specs a list of per-worker copies, each with its own adapter-info cache. _lora_ops read it as a dict, and a load or unload now updates every copy, so a reload under the same name is checked as its new file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#888's net change (its 19 commits, a3d22aa..e532a84, over old #823 f852a65) re-applied on the rebased #823 (lora/4-runtime-load), whose base, #822 on main, carries the reviewed #820/#821/#822. Conflicts came from three places: #822's review rounds (module_lora_spec(deferred=) gone, Qwen3.5 vision ids embedded in the step, the LoRA models loaded through their own checkpoint rules), main (per-worker _resource_specs copies, ResultQueue, CPU pinning, slot_wait_steps gone, device_loopback) and #823's fixes (LoraLoads.behind, refuse_non_adapter). Where main's reviewed LoRA code had settled a question #888 answered otherwise, #888's generalization is kept and expressed in main's semantics: - No `segment_rows` flag on nodes or region configs (main's R1.6 dropped lora_segment_rows: its one legal value said nothing). main's lora_rows becomes the generic with_segment_steps: a new step via replace() (R2.6), a step's own entry kept, none added without segments. NodeSubmodule.segment_keys replaces lora_keys; the engine's name stays engine_step. - Idle twins answer from the resource, not from request configs: Resource.idle_for (the pool's runs_adapters, inverted), so an adapter with nothing for a pool (its _empty set) leaves the step idle, as main's base-only choice did. The pool's guard against an idle step whose rows name an adapter stays. The "idle-capable spec must build request configs" engine check goes with it. - main's one_row_per_step regions captured under LoraBaseOnly are captured idle (BucketKey.idle, StepContext.idle from the bucket) and leased only to idle rows. - main's step-less warm-ups (pool.base_only) become Resource.outside_step, entered for the node's idle-capable resources. - main's module walk (stacks of any ModuleList, shape-mixed roles, unmatched `exclude`) runs inside #888's lazy declare(); _JoinedLoraMap gives a module to the node that places it before the most specific claimant, as main's linked maps did. - Request configs are built from the conductor's _node_resources() (the per-node instances), and the conductor keeps no adapter registry, so #823's per-worker spec copy fix has nothing left to update and its test goes. - Qwen3.5's leased vision step still returns None on a node without LoRA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

mstar-review Bot added area/benchmark model/many size/XL labels Oct 22, 2026

This was referenced Oct 22, 2026

Per-request LoRA adapters: the adapter pool, served from the serving config (3/9) #820

Merged

LoRA from a model's own modules (4/9) #821

Merged

Load and unload LoRA adapters at runtime (6/9) #823

Open

mstar-review Bot reviewed Oct 22, 2026

View reviewed changes

Comment thread mstar/model/qwen3_5/submodules.py Resolved

This comment has been minimized.

Sign in to view

mstar-review Bot added the type/feature label Oct 22, 2026

dghosef marked this pull request as draft October 22, 2026 03:41

dghosef force-pushed the lora/3-models branch from 05e30cf to 2a974c1 Compare October 22, 2026 03:42

dghosef force-pushed the lora/3-models branch from 441ad7a to 2aa1ea4 Compare October 22, 2026 04:05

dghosef force-pushed the lora/3-models branch from 2aa1ea4 to e195b4f Compare October 22, 2026 05:19

dghosef marked this pull request as ready for review October 22, 2026 17:40

This comment has been minimized.

Sign in to view

dghosef changed the title ~~[Model] Per-request LoRA on Qwen3.5, Orpheus, Higgs Audio, Qwen3-Omni, Qwen3-TTS and Chatterbox (3/4)~~ Per-request LoRA on Qwen3.5, Orpheus, Higgs Audio, Qwen3-Omni, Qwen3-TTS and Chatterbox (3/4) Oct 22, 2026

This comment has been minimized.

Sign in to view

vasilevklart previously requested changes Oct 26, 2026

View reviewed changes

dghosef force-pushed the lora/3-models branch from e195b4f to a63a63f Compare October 26, 2026 17:12

vasilevklart added this pull request to stack #884 October 26, 2026 18:17

dghosef force-pushed the lora/3-models branch from a63a63f to 795d283 Compare October 26, 2026 19:18

dghosef and others added 13 commits November 2, 2026 15:18

TP read tests: name and describe the in-step embedding collective (R2…

043c1b8

….F2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

dghosef force-pushed the lora/3-models branch from 07d5045 to 7be9c60 Compare November 3, 2026 01:10

mstar-review Bot added the area/model-shared label Nov 3, 2026

vasilevklart requested changes Nov 3, 2026

View reviewed changes

dghosef and others added 3 commits November 4, 2026 08:12

dghosef requested a review from vasilevklart November 4, 2026 19:06

dghosef changed the title ~~Per-request LoRA on Qwen3.5, Orpheus, Higgs Audio, Qwen3-Omni, Qwen3-TTS and Chatterbox (5/9)~~ Serve LoRA adapters on six models (5/9) Nov 4, 2026

NSagan271 mentioned this pull request Nov 4, 2026

Fix Qwen3-Omni and Cosmos3 decode steps that are prepared twice, and refuse a ViT LoRA that does nothing #1181

Open

Conversation

dghosef commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

How it works

Tests

Uh oh!

Uh oh!

This comment has been minimized.

dghosef commented Oct 22, 2026

Uh oh!

dghosef commented Oct 22, 2026

Uh oh!

This comment has been minimized.

dghosef commented Oct 23, 2026

Uh oh!

This comment has been minimized.

dghosef commented Oct 23, 2026

Uh oh!

mstar-review Bot commented Oct 23, 2026

Automated review

Uh oh!

vasilevklart left a comment

Choose a reason for hiding this comment

Uh oh!

vasilevklart Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

dghosef Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

vasilevklart Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

dghosef Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

vasilevklart Nov 3, 2026

Choose a reason for hiding this comment

Uh oh!

dghosef Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dghosef commented Oct 22, 2026 •

edited

Loading