Repository navigation
Data worker: prepare requests on a pool - #1109
Merged
Merged
Conversation
vasilevklart
approved these changes
Nov 2, 2026
| return max(1, min(4, self.left // 4)) | ||
|
|
||
| @property | ||
| def preprocess_workers(self) -> int: |
Collaborator
There was a problem hiding this comment.
nit: Sized from the default thread count, so a config with a higher preprocess_torch_threads oversubscribes the cpu, we can divide by the count _set_preprocess_threads
Collaborator
Author
There was a problem hiding this comment.
Done: CpuPlan.prepare_workers(threads) divides by the count the process runs, and the data worker passes torch.get_num_threads(), which _set_preprocess_threads has set by the time the API server is built. So preprocess_torch_threads: 8 with 8 CPUs left gives one worker (inline), and null (torch's default, every core) does too. preprocess_workers keeps the default-count value for the plan. Tests: test_a_higher_thread_count_shrinks_the_pool, test_the_pool_follows_the_threads_this_process_runs.
| appends handled behind the request, so anything else runs inline.""" | ||
| if not getattr(self.model, "PREPROCESS_PARALLEL", False) or self._client_edges: | ||
| return 1 | ||
| workers = (self._model_config or {}).get("preprocess_workers") |
Collaborator
There was a problem hiding this comment.
nit: preprocess_workers:, should we also mention it in serving.rst table next to tp_async_sched?
Collaborator
Author
There was a problem hiding this comment.
Added a preprocess_workers row under tp_async_sched, with a pointer to preprocess_torch_threads and adding_models.rst.
A config's preprocess_torch_threads above the plan's default put the pool over the CPUs left. The pool now divides by torch's thread count, which is set before the API server is built, and serving.rst gets a preprocess_workers row.
The pool now divides by torch's thread count, so the module fixes it at 2: 8 usable CPUs give 4 workers on any runner.
dghosef
added a commit
that referenced
this pull request
Nov 2, 2026
Since #1109 a model with PREPROCESS_PARALLEL keys prompts on the prepare pool, so the docstring names it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef
added a commit
that referenced
this pull request
Nov 2, 2026
…rf/split-4-shared main's #1109 prepares requests on a pool (Model.PREPROCESS_PARALLEL, preprocess_workers in the serving config, _prepare_pool / _release_prepared), the same thing this branch's preprocess pool did. main's is kept as the one mechanism, and this branch's copy is deleted: Model.PREPROCESS_WORKERS, the preprocess_workers() startup check and CPU cap, _prep_pool, _take_new_input / _take_input / _publish_waiting, the queue gauge's waiting_inputs, their docs rows (serving, adding_models, metrics) and their tests (the pool cases in test_data_worker_wakeup.py and test_serving_metrics.py). _positive_int goes back to entrypoint.py. Kept from this branch's data-worker work: the read_time / postprocess_s profiling of outputs (EncodedOutput, _encode's timing, postprocess_inline's per-row share), and the message thread's poll inside its try. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef
added a commit
that referenced
this pull request
Nov 2, 2026
V-JEPA2 opts into main's prepare pool (#1109): VJepa2Model declares PREPROCESS_PARALLEL (the AC model inherits it), and the measured pool sizes move from the classes' PREPROCESS_WORKERS (8 and 4) to the serving configs as preprocess_workers (configs/vjepa2.yaml 8, configs/vjepa2_ac.yaml 4), which main's data worker takes as given. The docs rows for the deleted PREPROCESS_WORKERS go with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dghosef
added a commit
that referenced
this pull request
Nov 2, 2026
…config (3/9) (#820) * [Distributed] One rank vote: CommGroup.agree and JointGroups.agree Six places reduced a per-rank value across the ranks, each its own way: all_reduce.py's _all_ranks_agree and ar_fusion.py's _all_ranks (a gloo MIN over one group), the graph runner's agree_across_ranks, and KVManager.post_warmup_validate and Worker._verify_tp_async_sched_agrees (a device all_gather per TP/SP axis read back to the host), and the worker's read-failure vote (all_reduce.py's flagged_on_any_rank, a gloo MAX). They now all go through CommGroup.agree, an all-reduce of a CPU int64 tensor over the group's cpu_group (the gloo twin, or the gloo side of XPU's mixed group), and JointGroups.agree, which runs it over the TP x SP block. None of these votes syncs the device any more; the two equality checks vote [v, -v] with MIN and report the min and max. agree raises ValueError for a multi-rank group with no cpu_group rather than reducing over the default world group. agree_across_ranks is gone: a piecewise runner built without a joint group gets a trivial one, so its votes and barriers need no None guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Worker] TP lockstep: the admit fence waits out a slow admit _await_admit_settled now waits for a TP lockstep step (tp_seq >= 0) however long its admit and plan take, warning every 10 s, instead of giving up after 10 s. This applies to every TP lockstep step, with or without MSTAR_TP_ASYNC_SCHED: a teardown landing inside the admit changes what it decides on that rank alone, and the ranks split. A resource admit may legitimately take longer than 10 s; the LoRA pool PR above this one is the first such resource (its admit can wait on a peer's adapter read). Any other step is still given up on after one interval. The GPU thread now sets launch_started_event from _execute_on_gpu_thread's outermost finally. It used to be set in a finally that a step whose plan future raised never reached, which with an unbounded wait would hang the fence. The wait also ends once the step's future is done: a GPU executor that fails to start completes its queued futures without running them, so the event is never set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Resources] LoRA adapter pool: per-request adapters in device slots LoraSpec / LoraAdapterPool: a fixed set of device slots (slot 0 the zero adapter), a pinned host tier read on the pool's own thread, and per-step addressing that groups each step's tokens by slot for the kernels (Triton shrink/expand/apply custom ops; a torch reference on CPU). Slots are held for a request's lifetime; a request that cannot get one is not ready, so a full pool is scheduler backpressure. Under TP every rank decides slots itself from state the leader's step order fixes, and an adapter read is agreed across the instance before it is installed. The engine brackets each planned step's preprocess and forward in StepRunner.compute_scope, so a delta called outside its step raises, and captures every bucket of a node serving adapters a second time under LoraBaseOnly, replayed by steps with no adapter row. Under TP the pool's load agreement is JointGroups.agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [API server][Conductor] Serve the serving config's LoRA adapters A resources: lora: block turns LoRA on. The OpenAI model field (every route) or a lora_adapter kwarg selects an adapter, as vLLM reads it; an unserved name is a 400, and /v1/models lists the adapters. The API server checks every configured adapter at startup and pins its weights' digest, which the conductor hands every worker; the conductor resolves resource specs with the YAML's overrides applied (serving_resources), as the workers do, and opens each request with its LoraReqConfig. Prefix-cache keys of a request on an adapter chain from a root naming its weights. The Rust frontend refuses a LoRA config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Diffusion] Read static LoRA merges through the shared checkpoint parser The diffusion merge's checkpoint parsing moves to mstar/engine/resources/lora/checkpoint.py, where the adapter pool reads adapters too; LoraSpec there becomes LoraFile, and the loader's _apply_stacked is public as apply_stacked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Docs][Benchmark] Per-request LoRA adapters: the resource, and its GPU checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Resources] LoRA: refuse a Kohya DoRA adapter as PEFT's is A Kohya-format DoRA checkpoint carries its magnitudes as `dora_scale` keys, which the shared checkpoint parser skipped, so the adapter was served (and statically merged into diffusion models) as plain LoRA, silently without its per-row rescaling. It is now refused by name like PEFT's `lora_magnitude_vector`. With nothing left to skip, the parser's None-for-skip path and its callers' checks are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Conductor] validate_deployment: LoRA declared beside the model's resources, autocast_dtype checked The pass declared the model's resources and its LoRA specs in one call, so a mistake in the `lora:` block (an unknown target, a model that serves no adapters, a bad value sized onto a second LoRA spec) skipped every `resources:` check and was labelled as get_node_resources failing. They are now separate steps: a declaration problem is its own line, and the second spec's sizing is left to the override step, which reports a bad value as `resources.lora: ...` with the rest. A `lora:` value that is not a mapping is refused as such, not as an unknown resource. LoraSpec.apply_yaml_overrides checks every value before it sets any, so a refused override leaves the spec as declared rather than holding the refused value for the next copy of the spec to raise again. `autocast_dtype` was refused only by each worker after spawn; the pass now parses it with the worker's own parser (parse_autocast_dtype). The API server read `resources:` before the conductor's pass for every deployment; a non-mapping block, or a `lora:` value that is not a mapping, now turns no LoRA on there (lora_enabled, served_lora_adapters, lora_specs), and adapter_prefix_roots checks for LoRA before it reads the KV specs, so a deployment without LoRA leaves the block to the pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_lora_pool: the KV control asserts the warning, not a throttle The control expected the 'No eviction candidate' warning once over three calls, but the warning has never been rate-limited, so CPU Core failed with 3 == 1. Assert that a KV refusal looks for a victim and warns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Worker] TP lockstep: the admit fence waits out a slow admit _await_admit_settled now waits for a TP lockstep step (tp_seq >= 0) however long its admit and plan take, warning every 10 s, instead of giving up after 10 s. This applies to every TP lockstep step, with or without MSTAR_TP_ASYNC_SCHED: a teardown landing inside the admit changes what it decides on that rank alone, and the ranks split. A resource admit may legitimately take longer than 10 s; the LoRA pool PR above this one is the first such resource (its admit can wait on a peer's adapter read). Any other step is still given up on after one interval. The GPU thread now sets launch_started_event from _execute_on_gpu_thread's outermost finally. It used to be set in a finally that a step whose plan future raised never reached, which with an unbounded wait would hang the fence. The wait also ends once the step's future is done: a GPU executor that fails to start completes its queued futures without running them, so the event is never set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lora_adapter is an engine kwarg: main's ignored-params log no longer warns on every LoRA request Main now logs the model_kwargs a model does not read; lora_adapter is read by the conductor for every model, like seed and prefix_cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * submit_request: check the generation kwargs before resolving the LoRA adapter Main's check runs first on a server with no state (pinned by test_submit_request_refuses_a_bad_seed_before_registering); the adapter lookup reads served_models, so it goes after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_lora_prefix_keys: the fake model gives main's data worker its partition topology Main's PreprocessWorkerThread reads get_partition_topology() for client streams at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ENGINE_KWARGS names LORA_ADAPTER_KWARG rather than repeating it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * warmup_compiles runs outside a planned step: say a LoRA delta refuses it Main's new hook (#927) runs after capture with no compute scope; the LoRA docs said the engine brackets every forward it runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * warmup_compiles: only an eager LoRA delta refuses a call outside a step The refusal is skipped while dynamo traces, so say a node serving adapters runs no adapted linear in warmup_compiles rather than that the delta refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Worker: release the admit fence on the step's completion, never give up on it (R1.2, R1.4, R1.5, R1.7) One release for "the step ended without launching": `_submit_gpu_step` sets the launch event when the step's future completes, run or not, in place of the outer `finally` and the `future.done()` poll. The GPU thread's cleanup scope now starts at the top of the body, so a raising plan future also releases the plan thread and publishes. The fence waits however long any step's admit takes, warning every `_ADMIT_FENCE_WARN_S` (renamed: it no longer times out). The give-up for non-lockstep steps bought nothing, since the main thread blocks on that step's result later in the same pass. The docstring now says what is fenced: every admit, with later per-request rows planning after it lifts. The fence test no longer depends on timing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Worker: correct the plan thread's commit-wait comment (R1.6) A lockstep admit can now legitimately run past 10 s, so the timeout no longer means the GPU thread died; it skips the pre-plan and exec plans N+1 inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: a step the GPU thread never runs still lifts the admit fence (R2.1) The fence now waits on the launch event with no limit, and the done-callback in Worker._submit_gpu_step is what sets it for a future that fails without running (BrokenThreadPool from a raising initializer). Pin that: the existing plan-raise case passes even with the release moved back into the function body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs/tests: drop stale spec_launch_started, de-parametrize the fence test (R2.2, R2.3) The async-worker skill named a local the fence change deleted; it now names the step's launch_started_event. The fence no longer reads tp_seq, so its test keeps one lone-rank case with a docstring that doesn't rest on lockstep. Rewrap the edited admit-failure docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: wait for a slot at the runner's admission, not in a queue of the pool's own (R1.1) The pool now answers Resource.ask / reserve / unreserve / grew_since, as RecurrentStatePool does: a request on an adapter waits in the runner's Admission, all or nothing with KV and the other resources that ask, so it holds no KV pages (or GDN slots) while it waits for a slot, and its node's readiness check parks instead of being rescanned every pass. A reservation counts slots (as the claims did) and pins none, so slot picks still follow from the slot table alone and TP ranks stay in lockstep with no journal; only the deciding rank (single GPU, or a lockstep leader) reserves. A slot is reserved only once the adapter's host copy is in, so the admit never waits for a read. Starvation is Admission's head-only rule: requests behind a cold adapter wait, so the held slots drain. Deleted: admit_retrieve's screen, the claims, aging, the drain-slot choice and the slot_wait_steps knob. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: guard deltas by the pool's own plan, not an engine compute scope (R1.4) A delta called outside a planned step only needs the pool to know it has no plan: it now clears its addressing at commit and rollback, and a delta called eagerly with none refuses. That deletes Resource.enter_compute/exit_compute, StepRunner.compute_scope, the engine's preprocess/forward split and the three graph-runner wraps, so engine.py, runner.py and base.py read as on main. The compile check now runs first, so a traced forward installs no guard on the pool's state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: know an adapter is empty for a pool from its header, before any admit (R1.6) A pool learned that an adapter has nothing for its targets only when an admit read it, so the first request on a text-only adapter waited at a vision pool for a slot it would never take, and its steps never took the base-only capture. The pool now reads it from each served file's header at startup (and from a runtime load's read), the same on every rank, skips preloading such adapters, and runs_adapters counts only adapters with something for the pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: the host tier holds what the ingests named, not what reads landed (R1.3) Each rank decided its host tier's drops when a read landed, against the requests still waiting at that moment, so the tiers diverged across a lockstep instance once more adapters were in rotation than cpu_capacity. The leader then reserved on its own copy while a follower's read had only just started, and the follower's admit held the whole instance for the read, digest and pinning. LoraAdapterStore.hold now decides membership at ingest, in ingest order, the same on every rank, and put only fills a held entry; an adapter a live request names is never dropped. A runtime load keeps its validated copy until the first request holds it, so it is not read twice. The eviction-time re-read and _wanted are gone: a reserved row's adapter stays held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: every device runs the custom ops; delete the two extra backend classes (R1.7) make_backend returned TorchLoraBackend off CUDA, which called the reference directly, so torch.compile on another accelerator traced its searchsorted / scatter_ / gather instead of taking one opaque node. It now returns LoraOpsBackend everywhere and imports triton_backend only to register the CUDA kernels; TorchLoraBackend and the empty TritonLoraBackend are gone. "torch" still selects the reference kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Delete the mstar.model.components.lora re-export shim (R1.10) It re-exported mstar.engine.resources.lora.checkpoint under a second import path; FLUX.2 and the diffusion LoRA tests now import from checkpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: check a served adapter from its header per spec, digest it once (R1.11) check_adapter_file ran a full worker read (fp32, padded to max_rank, sharded, SHA-256) once per LoRA spec, though the digest depends only on the file and how the spec converts it. It now checks each spec from the header (inspect_adapter, which covers the same refusals: no layers, rank, unmapped modules, misfits) and digests the file once per distinct convert_keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the FLUX.2 static merge refuses a DoRA adapter (R1.12) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenAI chat: name the adapter a lora_adapter kwarg picked in the response (R1.13) The response renamed model only when the model field named an adapter; one picked by the lora_adapter kwarg (extra_body) answered with the base model's name. It now names whatever adapter ServedModels.select resolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA admission: one reserve/unreserve helper; a base request waits behind a cold one (R1.1) _take and rollback reuse _reserve / _unreserve instead of repeating the counting. The KV-hold test now states the queue order main's admission gives: a base request asking KV after a request that waits for a slot waits behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: the base-only choice no longer meets per-rank empty-adapter reads (R1.6) An adapter's emptiness now comes from its file's header on every rank, so the case the test injected (one rank having read an adapter as empty) can't arise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: the pool counts live adapter requests in _live (R1.3) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: reserve a slot for a request only once its adapter's host copy is in (R2.2b) reserve() let a row through without its host copy when its adapter was resident in a slot or another row had reserved it. A reservation pins no slot, so a miss admitted first could evict that adapter, and the row's admit then waited on the GPU thread (every rank, under TP) for the re-read its ingest had started. Reserve only once the copy is in, or its read failed; a request on a resident adapter whose copy the tier dropped waits at admission for that re-read instead. Test: test_lora_pool::test_a_request_on_a_resident_adapter_waits_for_its_host_copy (fails before: r is admitted at once on the resident shortcut). The lockstep module now shares test_lora_pool's settle_reads helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: no rank waits on a read once a load has failed (R2.2b) A row reserved because its adapter's read failed is meant to fail at its admit. A request on the same adapter ingested before that admit started a retry, and the admit then waited for the retry on the GPU thread: the leader for its own, and under TP a follower whose read had failed too, while the leader waited in the agreement. The leader no longer retries a failed read while a row reserved on the failure waits for its admit (only the deciding rank reserves). And the load agreement is taken on where each rank's copy stands (failed < still reading < in, the lowest decides) before any rank waits: a rank still reading waits only when no rank has failed, and the ranks then agree the outcome as before. The pool's agree now takes the lowest of an int across the instance; one collective per installed adapter as before, a second only when a rank was still reading. Test: test_lora_tp_lockstep::test_no_rank_waits_on_a_retry_once_a_load_has_failed (before: both ranks wait out the gated retry; with only the agreement change too; with only the _hold change the follower waits alone and the barrier breaks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a queued request whose adapter read failed fails at once, not after a slot drains (R2.7) reserve() checked the free slots before the read error, so a request queued while its adapter was read, whose read then failed, sat at the head of admission until a slot drained (on one GPU holding back every newer request), only to fail at its admit. It is now let through at once, slots or not. An admit that finds no slot for such a row fails it instead of refusing the step (which would refuse every step it rides in until a slot drained): the full-pool refusal first agrees across the lockstep instance whether that adapter's read failed on any rank. Every rank reaches that refusal for the same miss, since all pick from the same slot table. The review's simpler fix (only reorder reserve's checks) would have reached the full-pool refusal with no slot free. Test: test_lora_pool::test_a_request_whose_read_fails_fails_without_waiting_for_a_slot (before: admit_waiting lets nothing through while a1 holds the slot). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a runtime load keeps no copy of the adapter outside the host tier (R2.3, R2.9) load_adapter kept its read in _loaded until a request named the adapter, a pinned copy per adapter, pool and rank with no bound: registering many adapters up front pinned gigabytes of host memory that cpu_capacity was meant to cap. The load now checks the file, pins its digest and drops the read; the first request on the adapter reads it again as it is ingested, on every rank at that ingest, and waits at admission rather than on the GPU thread. Costs one extra read per adapter loaded at runtime. _loaded is deleted, and _hold reads in whatever it newly holds (or retries). Also rewraps load_adapter_async's docstring, which had a broken line (R2.9). Test: test_lora_pool::test_a_runtime_load_keeps_no_copy_until_a_request_names_it (before: all three loads' copies stay alive). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a delta refuses on another thread than the one that planned its step (R2.4) The delta guard refused only while no step was planned. With the async worker (the default) the main thread runs step N+1's prepare_inputs while the GPU thread is between step N's plan and commit, so a delta called there ran against step N's rows, though the error text and the docs name prepare_inputs as the call that refuses. The pool records the planning thread in plan (a step's forward runs on it, a capture's included), and an eager delta on any other thread refuses too. Docs and the error text say so. Test: test_lora_graphs::test_a_delta_runs_only_between_its_steps_plan_and_commit gains the cross-thread case (before: the delta on another thread runs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: ask() does not ask again for a row it has already reserved (R2.5) Admission asks each node separately. For a pool serving two nodes, a request admitted on the first and asked on the second before its first admit was reserved twice: the second overwrote _reserved[rid] and left _reserved_count one count too high for good, a phantom slot in _spare_slots. ask() now answers None for a reserved row, as RecurrentStatePool.ask and KVManager.ask do. Unreachable today: every pool in the stack serves one node. Test: test_lora_pool::test_a_request_asked_again_by_a_second_node_is_reserved_once (before: r2 waits on the phantom count once r1 is gone). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: admitting a reserved row does not wake the waiting head (R2.8) _take gave the row's reservation back through _unreserve, which bumped _changes, so every pass that admitted any LoRA row re-ran the waiting head's whole reservation, KV's reserve/unreserve included, though turning a reservation into a hold frees nothing: the slot it counted is now held instead. Only a row placed on the zero slot (its adapter found empty at the admit) frees a count, and only that take bumps. No test: no behaviour changes but the retries skipped (tens of microseconds a pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say what backend "torch" does; the GPU kernel test registers Triton (R2.6) Since every device runs the custom ops, "torch" only skips importing triton_backend, and kernel registration is process-wide: once anything has imported it, a pool set to "torch" runs Triton too. The config comment, make_backend's docstring and the docs said "torch" forces the reference; they now say it registers no Triton kernel. test/integration/test_lora_triton_gpu.py imported only backend and ops, so run alone it compared the reference op with the reference function. It now imports triton_backend. GPU-unverified: run `pytest test/integration/test_lora_triton_gpu.py -v` on a CUDA box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenAI chat: resolve the adapter once, before submitting (R2.9) The chat handler resolved the request's adapter again after submit_request had, and outside the 400 mapping lora_kwargs gives: had the adapter stopped being served between the two calls (a runtime unload, #823), it raised after the request was submitted. The handler now resolves it with lora_kwargs before submitting and passes the result on, so the response names the adapter the submission carries. The router and Rust-parity test stubs bind APIServer.lora_kwargs, as they bind open_result_stream. Also reflows LoraSpec.apply_yaml_overrides' docstring, which ran past the file's width. No test: at this PR adapters are not unloaded at runtime, so the two resolutions cannot differ; test_lora_api's route test covers the response naming the adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: LoRA admission holds back newer requests on one GPU; host tiers can differ per rank (R2.1, R2.2a) R2.1: joining the runner's admission means a request waiting for a LoRA slot, or for its adapter's host copy, is first in KV's queue too on one GPU (KV asks for every request there), so it holds back every newer request on its node, base-model ones included, until a slot drains or the read lands. Under TP KV does not ask and base traffic flows. Letting a request pass a head that is short only on resources it does not ask would be a change to Admission's rules for every resource, so it is left for a decision of its own; the docs now state the behaviour and when it does not arise. R2.2a: the comments and docs said the host tier follows from the ingests alone, so a follower's read always starts at the leader's ingest. A tier also keeps what a live request names, and a follower takes removes from rank 0 and ingests from the conductor, unordered against each other: a remove that races an ingest can leave a follower without a copy rank 0 kept, and the next request on that adapter waits there for a whole read. No cheap exact fix (it needs ingests ordered against teardowns across the instance, or replicated tier membership); the pool, store and docs now say so, and the agreement before any wait is described as it now runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a request on a resident adapter whose read failed still counts a slot (R2.7) 8107c82 let every row whose adapter read had failed through admission at once, slots or not. A row whose adapter is still resident in a slot may yet run on it, so letting it skip the count could promise one slot to two rows, and their admit then hit the backstop refusal. Only a row whose adapter failed to read and is in no slot, which its admit fails whatever the slots, is let through now, as the review's fix said; one on a resident adapter is counted like any other. Test: test_lora_pool::test_a_request_on_a_resident_adapter_whose_read_failed_still_counts_a_slot (before: r is admitted beside y with the one slot already counted for y). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: wake a request waiting on an adapter that a miss evicts (R3.2) A request counted on a resident adapter whose re-read failed waits on the pool's free slots. When a miss's install evicts that adapter, nothing moved the pool's change counter (the evicting rows' takes no longer do), so the request waited for an unrelated slot release before it was let through to fail. The eviction now bumps the counter, once per eviction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: agree every miss's copy in one collective, failing failed reads before any slot is handed out (R3.1, R3.3a/c/e) A row let through on a failed read is reserved slots or not, so inside one admit its miss could take the last free slot ahead of a row that was counted on it; that row's step was then refused by the full-pool backstop. The admit now picks a candidate slot for every miss from the slot table at its start, agrees all the misses' copy states in one MIN collective (joint.agree takes the list as is), fails every adapter that failed on any rank before handing out a slot, then refuses if the misses left outnumber the slots, and only then waits for a follower's read (and agrees once more). _refuse_full and _load_miss are gone; an admit makes one collective with misses (two when a rank was still reading) instead of one per miss plus one on a full pool. ask shares _load_failed. Docs give the new collective count; the two-miss teardown test describes the current path, and its control admits row by row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say when _hold skips a failed read's retry (R3.3d) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA pool: read adapters on mstar.utils.threads.ThreadPool Main bans ThreadPoolExecutor under mstar/ (ruff TID251): a pool started from a pinned thread would share its core. ThreadPool starts its reader on the process mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: follow main's signatures in the tests; check KV fit on serving specs check_kv_pools_fit read the model's own specs and applied the YAML to them, so a LoRA deployment's `lora:` block was an unknown resource and the check returned without looking. It reads serving_resources, as every other spec reader does. The LoRA tests follow main: _kv_page_sizes keys only the caches some rank opens (node_groups, a named checkpoint), StepRunner.plan takes a phase, and _try_offload_cold_request takes the walk and reads scheduler.arrival. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Piecewise capture: refuse a LoRA step in a one_row_per_step region A one_row_per_step region (#977) declares and replays its step as one row named by the step's first request, so a LoRA pool planned in it would give every request's tokens that request's adapter. No model combines the two today; prepare_for_capture now refuses it with a ValueError naming the region and node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Nemotron duplex: check the session cap on the serving specs Resolve the specs with serving_resources, as check_kv_pools_fit does, rather than the declaration with the YAML overrides applied by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA serving: report a bad lora: key on a second spec as a ValueError size_lora_specs applies the lora: block to the LoRA specs keyed apart from lora before apply_yaml_overrides runs, and let a key no spec takes escape as a TypeError. Readers of the serving specs (check_kv_pools_fit) leave a ValueError to validation, so a typo on a two-spec model crashed them instead. It now converts the TypeError as apply_yaml_overrides does for the first spec, and validation's suppress around it needs only ValueError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA prefix roots: reads come from the request-preparation threads Since #1109 a model with PREPROCESS_PARALLEL keys prompts on the prepare pool, so the docstring names it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 2, 2026
dghosef
added a commit
that referenced
this pull request
Nov 2, 2026
* [Distributed] One rank vote: CommGroup.agree and JointGroups.agree Six places reduced a per-rank value across the ranks, each its own way: all_reduce.py's _all_ranks_agree and ar_fusion.py's _all_ranks (a gloo MIN over one group), the graph runner's agree_across_ranks, and KVManager.post_warmup_validate and Worker._verify_tp_async_sched_agrees (a device all_gather per TP/SP axis read back to the host), and the worker's read-failure vote (all_reduce.py's flagged_on_any_rank, a gloo MAX). They now all go through CommGroup.agree, an all-reduce of a CPU int64 tensor over the group's cpu_group (the gloo twin, or the gloo side of XPU's mixed group), and JointGroups.agree, which runs it over the TP x SP block. None of these votes syncs the device any more; the two equality checks vote [v, -v] with MIN and report the min and max. agree raises ValueError for a multi-rank group with no cpu_group rather than reducing over the default world group. agree_across_ranks is gone: a piecewise runner built without a joint group gets a trivial one, so its votes and barriers need no None guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Worker] TP lockstep: the admit fence waits out a slow admit _await_admit_settled now waits for a TP lockstep step (tp_seq >= 0) however long its admit and plan take, warning every 10 s, instead of giving up after 10 s. This applies to every TP lockstep step, with or without MSTAR_TP_ASYNC_SCHED: a teardown landing inside the admit changes what it decides on that rank alone, and the ranks split. A resource admit may legitimately take longer than 10 s; the LoRA pool PR above this one is the first such resource (its admit can wait on a peer's adapter read). Any other step is still given up on after one interval. The GPU thread now sets launch_started_event from _execute_on_gpu_thread's outermost finally. It used to be set in a finally that a step whose plan future raised never reached, which with an unbounded wait would hang the fence. The wait also ends once the step's future is done: a GPU executor that fails to start completes its queued futures without running them, so the event is never set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Resources] LoRA adapter pool: per-request adapters in device slots LoraSpec / LoraAdapterPool: a fixed set of device slots (slot 0 the zero adapter), a pinned host tier read on the pool's own thread, and per-step addressing that groups each step's tokens by slot for the kernels (Triton shrink/expand/apply custom ops; a torch reference on CPU). Slots are held for a request's lifetime; a request that cannot get one is not ready, so a full pool is scheduler backpressure. Under TP every rank decides slots itself from state the leader's step order fixes, and an adapter read is agreed across the instance before it is installed. The engine brackets each planned step's preprocess and forward in StepRunner.compute_scope, so a delta called outside its step raises, and captures every bucket of a node serving adapters a second time under LoraBaseOnly, replayed by steps with no adapter row. Under TP the pool's load agreement is JointGroups.agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [API server][Conductor] Serve the serving config's LoRA adapters A resources: lora: block turns LoRA on. The OpenAI model field (every route) or a lora_adapter kwarg selects an adapter, as vLLM reads it; an unserved name is a 400, and /v1/models lists the adapters. The API server checks every configured adapter at startup and pins its weights' digest, which the conductor hands every worker; the conductor resolves resource specs with the YAML's overrides applied (serving_resources), as the workers do, and opens each request with its LoraReqConfig. Prefix-cache keys of a request on an adapter chain from a root naming its weights. The Rust frontend refuses a LoRA config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Diffusion] Read static LoRA merges through the shared checkpoint parser The diffusion merge's checkpoint parsing moves to mstar/engine/resources/lora/checkpoint.py, where the adapter pool reads adapters too; LoraSpec there becomes LoraFile, and the loader's _apply_stacked is public as apply_stacked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Model] LoRA from a model's own modules The shared linears (column/row/QKV/merged parallel, FusedColumnLinear, the delta-net's fused in-projection, VocabParallelEmbedding, plain nn.Linear and patch convolutions) describe their own LoRA target. Binding swaps a module to its family's twin class, which adds the delta before the module's own gather or all-reduce; an unbound module keeps its class and its forward. The parallel linears' forwards are split into local compute and gather/reduce for that. module_lora_spec builds a node's LoraSpec by walking its weights root under the loader's CheckpointRules, so adapters are placed as weights are; NodeSubmodule.bind_node_resources binds the placed modules, and declared_step adds the pools' LoraStep for a node with lora_segment_rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Docs][Benchmark] Per-request LoRA adapters: the resource, and its GPU checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Docs] Declaring LoRA from a model's modules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Resources] LoRA: refuse a Kohya DoRA adapter as PEFT's is A Kohya-format DoRA checkpoint carries its magnitudes as `dora_scale` keys, which the shared checkpoint parser skipped, so the adapter was served (and statically merged into diffusion models) as plain LoRA, silently without its per-row rescaling. It is now refused by name like PEFT's `lora_magnitude_vector`. With nothing left to skip, the parser's None-for-skip path and its callers' checks are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Model] LoRA binding: refuse a spec not built by module_lora_spec; widen names outside a stack A pool whose spec was not declared with module_lora_spec would bind nothing, so the node would get no LoraStep and no base-only captures without saying so; binding now refuses it. A role outside every layer stack has no stack prefix, so the second name widening crashed on None instead of naming the clash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Conductor] validate_deployment: LoRA declared beside the model's resources, autocast_dtype checked The pass declared the model's resources and its LoRA specs in one call, so a mistake in the `lora:` block (an unknown target, a model that serves no adapters, a bad value sized onto a second LoRA spec) skipped every `resources:` check and was labelled as get_node_resources failing. They are now separate steps: a declaration problem is its own line, and the second spec's sizing is left to the override step, which reports a bad value as `resources.lora: ...` with the rest. A `lora:` value that is not a mapping is refused as such, not as an unknown resource. LoraSpec.apply_yaml_overrides checks every value before it sets any, so a refused override leaves the spec as declared rather than holding the refused value for the next copy of the spec to raise again. `autocast_dtype` was refused only by each worker after spawn; the pass now parses it with the worker's own parser (parse_autocast_dtype). The API server read `resources:` before the conductor's pass for every deployment; a non-mapping block, or a `lora:` value that is not a mapping, now turns no LoRA on there (lora_enabled, served_lora_adapters, lora_specs), and adapter_prefix_roots checks for LoRA before it reads the KV specs, so a deployment without LoRA leaves the block to the pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Test] LoRA targets: restore torch's dispatch-mode flags after the rank threads; a hand spec with no adapters On torch 2.9.1 (CI's pin) a dispatch mode entered on two rank threads at once leaves torch's module-level 'in dispatch mode' flags set, and dynamo then skips every later frame on that worker. The hand-written spec preloaded the YAML adapters and failed in the pool's constructor, before binding was reached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_lora_pool: the KV control asserts the warning, not a throttle The control expected the 'No eviction candidate' warning once over three calls, but the warning has never been rate-limited, so CPU Core failed with 3 == 1. Assert that a KV refusal looks for a victim and warns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [Worker] TP lockstep: the admit fence waits out a slow admit _await_admit_settled now waits for a TP lockstep step (tp_seq >= 0) however long its admit and plan take, warning every 10 s, instead of giving up after 10 s. This applies to every TP lockstep step, with or without MSTAR_TP_ASYNC_SCHED: a teardown landing inside the admit changes what it decides on that rank alone, and the ranks split. A resource admit may legitimately take longer than 10 s; the LoRA pool PR above this one is the first such resource (its admit can wait on a peer's adapter read). Any other step is still given up on after one interval. The GPU thread now sets launch_started_event from _execute_on_gpu_thread's outermost finally. It used to be set in a finally that a step whose plan future raised never reached, which with an unbounded wait would hang the fence. The wait also ends once the step's future is done: a GPU executor that fails to start completes its queued futures without running them, so the event is never set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * lora_adapter is an engine kwarg: main's ignored-params log no longer warns on every LoRA request Main now logs the model_kwargs a model does not read; lora_adapter is read by the conductor for every model, like seed and prefix_cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * submit_request: check the generation kwargs before resolving the LoRA adapter Main's check runs first on a server with no state (pinned by test_submit_request_refuses_a_bad_seed_before_registering); the adapter lookup reads served_models, so it goes after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_lora_prefix_keys: the fake model gives main's data worker its partition topology Main's PreprocessWorkerThread reads get_partition_topology() for client streams at construction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ENGINE_KWARGS names LORA_ADAPTER_KWARG rather than repeating it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * warmup_compiles runs outside a planned step: say a LoRA delta refuses it Main's new hook (#927) runs after capture with no compute scope; the LoRA docs said the engine brackets every forward it runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * warmup_compiles: only an eager LoRA delta refuses a call outside a step The refusal is skipped while dynamo traces, so say a node serving adapters runs no adapted linear in warmup_compiles rather than that the delta refuses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: refuse children a module runs through their weights itself The module walk decided capability by class and forward, so a linear its parent runs as F.linear over its weight, or folds at load, was listed as a target and an adapter on it validated and silently did nothing. A parent now names such children in `lora_unservable` (child -> reason); the walk merges them into the refused modules, so an adapter touching one is refused with the reason. Declared by absorbed MLA (its folded source projections), SigLIP's patch embedding (Command-A+), and EarTTS's attention, fusion and MoG head projections (Nemotron duplex). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: check declared unservable children exist, look them up by path A child named in `lora_unservable` that the module does not have was dropped silently, so a typo or a later rename lost the refusal and an adapter on the real child validated and did nothing. The walk now resolves each declared child and raises a ValueError naming the module and the child, as it does for the other model-supplied names. The declarations also no longer join the `exclude` patterns: each became an fnmatch pattern every later module was matched against (quadratic in layers, and literal names gained glob semantics). They are kept by exact path and looked up for the module and its ancestors before the `exclude` patterns are tried. The refusal test's table now names each case's parent module, rather than the test body picking it by case name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Command A+: refuse LoRA on the projections its decoder layer fuses On CUDA the language model fuses each decoder layer at load, and `fused_branches` then runs attention's qkv/o projections and the shared MLP's gate_up/down projections as slices of the layer's two GEMMs, never through their forward. A walk of the (unfused) meta root still listed them as targets, so an adapter on them would validate and do nothing. The layer now names them in `lora_unservable`. CPU, the only unfused case, refuses them as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA test: pin which MLA projections the walk serves, absorbed or not Absorbed MLA folds q_a_proj, kv_a_proj_with_mqa and kv_b_proj into its own weights at load, so it refuses them; unabsorbed it runs them, so they are targets. A meta walk of the shared MLAAttention at GLM-5.2's reduced config holds both sides of the condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say in the module walk's docstring that modules declare refusals too The module docstring listed only the call-site arguments as what a model states; a module's own `lora_unservable` is the other source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: Command-A+'s refusal says it fuses on CUDA; the doc names a declared path's rules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Worker: release the admit fence on the step's completion, never give up on it (R1.2, R1.4, R1.5, R1.7) One release for "the step ended without launching": `_submit_gpu_step` sets the launch event when the step's future completes, run or not, in place of the outer `finally` and the `future.done()` poll. The GPU thread's cleanup scope now starts at the top of the body, so a raising plan future also releases the plan thread and publishes. The fence waits however long any step's admit takes, warning every `_ADMIT_FENCE_WARN_S` (renamed: it no longer times out). The give-up for non-lockstep steps bought nothing, since the main thread blocks on that step's result later in the same pass. The docstring now says what is fenced: every admit, with later per-request rows planning after it lifts. The fence test no longer depends on timing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Worker: correct the plan thread's commit-wait comment (R1.6) A lockstep admit can now legitimately run past 10 s, so the timeout no longer means the GPU thread died; it skips the pre-plan and exec plans N+1 inline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: a step the GPU thread never runs still lifts the admit fence (R2.1) The fence now waits on the launch event with no limit, and the done-callback in Worker._submit_gpu_step is what sets it for a future that fails without running (BrokenThreadPool from a raising initializer). Pin that: the existing plan-raise case passes even with the release moved back into the function body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs/tests: drop stale spec_launch_started, de-parametrize the fence test (R2.2, R2.3) The async-worker skill named a local the fence change deleted; it now names the step's launch_started_event. The fence no longer reads tp_seq, so its test keeps one lone-rank case with a docstring that doesn't rest on lockstep. Rewrap the edited admit-failure docstring. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: wait for a slot at the runner's admission, not in a queue of the pool's own (R1.1) The pool now answers Resource.ask / reserve / unreserve / grew_since, as RecurrentStatePool does: a request on an adapter waits in the runner's Admission, all or nothing with KV and the other resources that ask, so it holds no KV pages (or GDN slots) while it waits for a slot, and its node's readiness check parks instead of being rescanned every pass. A reservation counts slots (as the claims did) and pins none, so slot picks still follow from the slot table alone and TP ranks stay in lockstep with no journal; only the deciding rank (single GPU, or a lockstep leader) reserves. A slot is reserved only once the adapter's host copy is in, so the admit never waits for a read. Starvation is Admission's head-only rule: requests behind a cold adapter wait, so the held slots drain. Deleted: admit_retrieve's screen, the claims, aging, the drain-slot choice and the slot_wait_steps knob. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: guard deltas by the pool's own plan, not an engine compute scope (R1.4) A delta called outside a planned step only needs the pool to know it has no plan: it now clears its addressing at commit and rollback, and a delta called eagerly with none refuses. That deletes Resource.enter_compute/exit_compute, StepRunner.compute_scope, the engine's preprocess/forward split and the three graph-runner wraps, so engine.py, runner.py and base.py read as on main. The compile check now runs first, so a traced forward installs no guard on the pool's state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: know an adapter is empty for a pool from its header, before any admit (R1.6) A pool learned that an adapter has nothing for its targets only when an admit read it, so the first request on a text-only adapter waited at a vision pool for a slot it would never take, and its steps never took the base-only capture. The pool now reads it from each served file's header at startup (and from a runtime load's read), the same on every rank, skips preloading such adapters, and runs_adapters counts only adapters with something for the pool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: the host tier holds what the ingests named, not what reads landed (R1.3) Each rank decided its host tier's drops when a read landed, against the requests still waiting at that moment, so the tiers diverged across a lockstep instance once more adapters were in rotation than cpu_capacity. The leader then reserved on its own copy while a follower's read had only just started, and the follower's admit held the whole instance for the read, digest and pinning. LoraAdapterStore.hold now decides membership at ingest, in ingest order, the same on every rank, and put only fills a held entry; an adapter a live request names is never dropped. A runtime load keeps its validated copy until the first request holds it, so it is not read twice. The eviction-time re-read and _wanted are gone: a reserved row's adapter stays held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: every device runs the custom ops; delete the two extra backend classes (R1.7) make_backend returned TorchLoraBackend off CUDA, which called the reference directly, so torch.compile on another accelerator traced its searchsorted / scatter_ / gather instead of taking one opaque node. It now returns LoraOpsBackend everywhere and imports triton_backend only to register the CUDA kernels; TorchLoraBackend and the empty TritonLoraBackend are gone. "torch" still selects the reference kernel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Delete the mstar.model.components.lora re-export shim (R1.10) It re-exported mstar.engine.resources.lora.checkpoint under a second import path; FLUX.2 and the diffusion LoRA tests now import from checkpoint. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: check a served adapter from its header per spec, digest it once (R1.11) check_adapter_file ran a full worker read (fp32, padded to max_rank, sharded, SHA-256) once per LoRA spec, though the digest depends only on the file and how the spec converts it. It now checks each spec from the header (inspect_adapter, which covers the same refusals: no layers, rank, unmapped modules, misfits) and digests the file once per distinct convert_keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: the FLUX.2 static merge refuses a DoRA adapter (R1.12) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenAI chat: name the adapter a lora_adapter kwarg picked in the response (R1.13) The response renamed model only when the model field named an adapter; one picked by the lora_adapter kwarg (extra_body) answered with the base model's name. It now names whatever adapter ServedModels.select resolved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA admission: one reserve/unreserve helper; a base request waits behind a cold one (R1.1) _take and rollback reuse _reserve / _unreserve instead of repeating the counting. The KV-hold test now states the queue order main's admission gives: a base request asking KV after a request that waits for a slot waits behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: the base-only choice no longer meets per-rank empty-adapter reads (R1.6) An adapter's emptiness now comes from its file's header on every rank, so the case the test injected (one rank having read an adapter as empty) can't arise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Test: the pool counts live adapter requests in _live (R1.3) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: reserve a slot for a request only once its adapter's host copy is in (R2.2b) reserve() let a row through without its host copy when its adapter was resident in a slot or another row had reserved it. A reservation pins no slot, so a miss admitted first could evict that adapter, and the row's admit then waited on the GPU thread (every rank, under TP) for the re-read its ingest had started. Reserve only once the copy is in, or its read failed; a request on a resident adapter whose copy the tier dropped waits at admission for that re-read instead. Test: test_lora_pool::test_a_request_on_a_resident_adapter_waits_for_its_host_copy (fails before: r is admitted at once on the resident shortcut). The lockstep module now shares test_lora_pool's settle_reads helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: no rank waits on a read once a load has failed (R2.2b) A row reserved because its adapter's read failed is meant to fail at its admit. A request on the same adapter ingested before that admit started a retry, and the admit then waited for the retry on the GPU thread: the leader for its own, and under TP a follower whose read had failed too, while the leader waited in the agreement. The leader no longer retries a failed read while a row reserved on the failure waits for its admit (only the deciding rank reserves). And the load agreement is taken on where each rank's copy stands (failed < still reading < in, the lowest decides) before any rank waits: a rank still reading waits only when no rank has failed, and the ranks then agree the outcome as before. The pool's agree now takes the lowest of an int across the instance; one collective per installed adapter as before, a second only when a rank was still reading. Test: test_lora_tp_lockstep::test_no_rank_waits_on_a_retry_once_a_load_has_failed (before: both ranks wait out the gated retry; with only the agreement change too; with only the _hold change the follower waits alone and the barrier breaks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a queued request whose adapter read failed fails at once, not after a slot drains (R2.7) reserve() checked the free slots before the read error, so a request queued while its adapter was read, whose read then failed, sat at the head of admission until a slot drained (on one GPU holding back every newer request), only to fail at its admit. It is now let through at once, slots or not. An admit that finds no slot for such a row fails it instead of refusing the step (which would refuse every step it rides in until a slot drained): the full-pool refusal first agrees across the lockstep instance whether that adapter's read failed on any rank. Every rank reaches that refusal for the same miss, since all pick from the same slot table. The review's simpler fix (only reorder reserve's checks) would have reached the full-pool refusal with no slot free. Test: test_lora_pool::test_a_request_whose_read_fails_fails_without_waiting_for_a_slot (before: admit_waiting lets nothing through while a1 holds the slot). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a runtime load keeps no copy of the adapter outside the host tier (R2.3, R2.9) load_adapter kept its read in _loaded until a request named the adapter, a pinned copy per adapter, pool and rank with no bound: registering many adapters up front pinned gigabytes of host memory that cpu_capacity was meant to cap. The load now checks the file, pins its digest and drops the read; the first request on the adapter reads it again as it is ingested, on every rank at that ingest, and waits at admission rather than on the GPU thread. Costs one extra read per adapter loaded at runtime. _loaded is deleted, and _hold reads in whatever it newly holds (or retries). Also rewraps load_adapter_async's docstring, which had a broken line (R2.9). Test: test_lora_pool::test_a_runtime_load_keeps_no_copy_until_a_request_names_it (before: all three loads' copies stay alive). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a delta refuses on another thread than the one that planned its step (R2.4) The delta guard refused only while no step was planned. With the async worker (the default) the main thread runs step N+1's prepare_inputs while the GPU thread is between step N's plan and commit, so a delta called there ran against step N's rows, though the error text and the docs name prepare_inputs as the call that refuses. The pool records the planning thread in plan (a step's forward runs on it, a capture's included), and an eager delta on any other thread refuses too. Docs and the error text say so. Test: test_lora_graphs::test_a_delta_runs_only_between_its_steps_plan_and_commit gains the cross-thread case (before: the delta on another thread runs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: ask() does not ask again for a row it has already reserved (R2.5) Admission asks each node separately. For a pool serving two nodes, a request admitted on the first and asked on the second before its first admit was reserved twice: the second overwrote _reserved[rid] and left _reserved_count one count too high for good, a phantom slot in _spare_slots. ask() now answers None for a reserved row, as RecurrentStatePool.ask and KVManager.ask do. Unreachable today: every pool in the stack serves one node. Test: test_lora_pool::test_a_request_asked_again_by_a_second_node_is_reserved_once (before: r2 waits on the phantom count once r1 is gone). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: admitting a reserved row does not wake the waiting head (R2.8) _take gave the row's reservation back through _unreserve, which bumped _changes, so every pass that admitted any LoRA row re-ran the waiting head's whole reservation, KV's reserve/unreserve included, though turning a reservation into a hold frees nothing: the slot it counted is now held instead. Only a row placed on the zero slot (its adapter found empty at the admit) frees a count, and only that take bumps. No test: no behaviour changes but the retries skipped (tens of microseconds a pass). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say what backend "torch" does; the GPU kernel test registers Triton (R2.6) Since every device runs the custom ops, "torch" only skips importing triton_backend, and kernel registration is process-wide: once anything has imported it, a pool set to "torch" runs Triton too. The config comment, make_backend's docstring and the docs said "torch" forces the reference; they now say it registers no Triton kernel. test/integration/test_lora_triton_gpu.py imported only backend and ops, so run alone it compared the reference op with the reference function. It now imports triton_backend. GPU-unverified: run `pytest test/integration/test_lora_triton_gpu.py -v` on a CUDA box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * OpenAI chat: resolve the adapter once, before submitting (R2.9) The chat handler resolved the request's adapter again after submit_request had, and outside the 400 mapping lora_kwargs gives: had the adapter stopped being served between the two calls (a runtime unload, #823), it raised after the request was submitted. The handler now resolves it with lora_kwargs before submitting and passes the result on, so the response names the adapter the submission carries. The router and Rust-parity test stubs bind APIServer.lora_kwargs, as they bind open_result_stream. Also reflows LoraSpec.apply_yaml_overrides' docstring, which ran past the file's width. No test: at this PR adapters are not unloaded at runtime, so the two resolutions cannot differ; test_lora_api's route test covers the response naming the adapter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Docs: LoRA admission holds back newer requests on one GPU; host tiers can differ per rank (R2.1, R2.2a) R2.1: joining the runner's admission means a request waiting for a LoRA slot, or for its adapter's host copy, is first in KV's queue too on one GPU (KV asks for every request there), so it holds back every newer request on its node, base-model ones included, until a slot drains or the read lands. Under TP KV does not ask and base traffic flows. Letting a request pass a head that is short only on resources it does not ask would be a change to Admission's rules for every resource, so it is left for a decision of its own; the docs now state the behaviour and when it does not arise. R2.2a: the comments and docs said the host tier follows from the ingests alone, so a follower's read always starts at the leader's ingest. A tier also keeps what a live request names, and a follower takes removes from rank 0 and ingests from the conductor, unordered against each other: a remove that races an ingest can leave a follower without a copy rank 0 kept, and the next request on that adapter waits there for a whole read. No cheap exact fix (it needs ingests ordered against teardowns across the instance, or replicated tier membership); the pool, store and docs now say so, and the agreement before any wait is described as it now runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a request on a resident adapter whose read failed still counts a slot (R2.7) 8107c82 let every row whose adapter read had failed through admission at once, slots or not. A row whose adapter is still resident in a slot may yet run on it, so letting it skip the count could promise one slot to two rows, and their admit then hit the backstop refusal. Only a row whose adapter failed to read and is in no slot, which its admit fails whatever the slots, is let through now, as the review's fix said; one on a resident adapter is counted like any other. Test: test_lora_pool::test_a_request_on_a_resident_adapter_whose_read_failed_still_counts_a_slot (before: r is admitted beside y with the one slot already counted for y). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: wake a request waiting on an adapter that a miss evicts (R3.2) A request counted on a resident adapter whose re-read failed waits on the pool's free slots. When a miss's install evicts that adapter, nothing moved the pool's change counter (the evicting rows' takes no longer do), so the request waited for an unrelated slot release before it was let through to fail. The eviction now bumps the counter, once per eviction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: agree every miss's copy in one collective, failing failed reads before any slot is handed out (R3.1, R3.3a/c/e) A row let through on a failed read is reserved slots or not, so inside one admit its miss could take the last free slot ahead of a row that was counted on it; that row's step was then refused by the full-pool backstop. The admit now picks a candidate slot for every miss from the slot table at its start, agrees all the misses' copy states in one MIN collective (joint.agree takes the list as is), fails every adapter that failed on any rank before handing out a slot, then refuses if the misses left outnumber the slots, and only then waits for a follower's read (and agrees once more). _refuse_full and _load_miss are gone; an admit makes one collective with misses (two when a rank was still reading) instead of one per miss plus one on a full pool. ask shares _load_failed. Docs give the new collective count; the two-miss teardown test describes the current path, and its control admits row by row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say when _hold skips a failed read's retry (R3.3d) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: make_lora_capable refuses with ValueError, as every other bind refusal (R1.15) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: lora_local_shape defaults to the weight's shape; only the embedding overrides it (R1.12) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a patch conv adds its delta at every output position, not only on one-patch inputs (R1.3) A bound patch conv fed a whole image or latent (the usual patch embed) raised on every forward, base-only ones included; its rows are now the output positions in token order, each reading the patch the conv read (a dropped remainder excluded). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: find a layer stack by its module list, not by the names layers/blocks (R1.4) A DiT's transformer_blocks (or any other stack) made one target per layer, so the YAML and every rows/names pattern had to list each layer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: linked specs leave a module to the spec that places it, not to the longest prefix (R1.6) A spec with no adapter roots beside one with a checkpoint prefix ignored all of its own modules (the prefixed spec claims everything at length 0), and the other then refused every adapter. Prefix length now only breaks ties between specs that both place a module, and picks which spec refuses one that none places. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a deferred embedding is served without embeddings=True, as the docstring offers (R1.7) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: refuse an exclude pattern that matches no LoRA-capable module (R1.8) As rows, names and deferred already are: a typo left the module served, so every request then failed at serve time on the delta guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: refuse a spec not built by module_lora_spec where specs are declared, not at worker bind (R1.16) declared_lora_specs runs in the conductor, the workers and validate_deployment, so a hand-written spec fails before any adapter check or worker start; the bind-time check it replaces is deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: drop declared_step's lora_segment_rows test, which binding already guarantees (R1.17) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: name the engine's step builder engine_step, apart from the node's declare_step (R1.14) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: the no-plan errors name the real causes: a None step, a region with no LoraStep, a call outside the step (R1.9) The delta guard sent an author whose declare_step returned None to prepare_inputs, and the addressing error still described the hand-written LoraStep this PR retires. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: say that nothing checks a piecewise region running adapted modules without its own LoraStep (R1.1) The comment and docs said its capture is refused; that held under the compute scope #820 deleted. A startup refusal of every region without a LoraStep would refuse regions that run no adapted module (Qwen3-TTS's depth loop runs the CodePredictor, outside the Talker root). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: declare the children main's models run through their weights (R1.2) Waypoint's MLPFusion (fc1/fc2), CosyVoice3's float32 LM head and Qwen3-Omni's code predictor (QKV in the unrolled depth loop, the stacked lm_head) read a linear's weight directly, so a spec walking them would bind a delta that never runs. The docs say why the declaration is the guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: ruff (line length, zip strict) on the round-1 fixes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test_lora_prefix_keys: build the fake model's spec map as module_lora_spec does (R1.16) declared_lora_specs now refuses a spec whose module_map is not a ModuleLoraMap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a module list whose entries differ in shape gives each entry a target (R2.1) Finding stacks by type made every ModuleList / Sequential a stack, so a list whose entries change width (UNet levels, vision stages, codec Sequentials) was refused with no declaration to opt out. A role whose entries differ in shape now falls back to one standalone target per entry, as such lists did before; docs say ModuleList or Sequential. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: drop the unreachable "step without segments" cause from the no-plan messages (R2.2) engine_step refuses a step without segments for a node with pools bound before the pool is reached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a split stack entry keeps its role for `rows` (R3.1) A stack whose entries differ in shape at a role gives each entry a target named by its path. `rows` now still matches those entries by the stack role, so a model's key does not depend on whether the entries agree in shape, and a `names` override on such a role, which cannot name several targets, is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA pool: read adapters on mstar.utils.threads.ThreadPool Main bans ThreadPoolExecutor under mstar/ (ruff TID251): a pool started from a pinned thread would share its core. ThreadPool starts its reader on the process mask. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: follow main's signatures in the tests; check KV fit on serving specs check_kv_pools_fit read the model's own specs and applied the YAML to them, so a LoRA deployment's `lora:` block was an unknown resource and the check returned without looking. It reads serving_resources, as every other spec reader does. The LoRA tests follow main: _kv_page_sizes keys only the caches some rank opens (node_groups, a named checkpoint), StepRunner.plan takes a phase, and _try_offload_cold_request takes the walk and reads scheduler.arrival. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Piecewise capture: refuse a LoRA step in a one_row_per_step region A one_row_per_step region (#977) declares and replays its step as one row named by the step's first request, so a LoRA pool planned in it would give every request's tokens that request's adapter. No model combines the two today; prepare_for_capture now refuses it with a ValueError naming the region and node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Nemotron duplex: check the session cap on the serving specs Resolve the specs with serving_resources, as check_kv_pools_fit does, rather than the declaration with the YAML overrides applied by hand. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA serving: report a bad lora: key on a second spec as a ValueError size_lora_specs applies the lora: block to the LoRA specs keyed apart from lora before apply_yaml_overrides runs, and let a key no spec takes escape as a TypeError. Readers of the serving specs (check_kv_pools_fit) leave a ValueError to validation, so a typo on a two-spec model crashed them instead. It now converts the TypeError as apply_yaml_overrides does for the first spec, and validation's suppress around it needs only ValueError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA prefix roots: reads come from the request-preparation threads Since #1109 a model with PREPROCESS_PARALLEL keys prompts on the prepare pool, so the docstring names it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: lora_unservable for main's code predictor and Wan2.2's compiled FFN #859 moved the Qwen3 talkers' code predictor into ResidualCodePredictor: its layers' qkv_proj now run through their forward, and the stacked lm_head is what the depth loop reads. Wan2.2's gelu_tanh_ffn runs the FFN's linear_in as an addmm over its weight when compiled on CUDA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA pool: no plan left behind a region's capture or a base-only step (R1.10) A piecewise region's capture plans outside any step, and its commit restored an enclosing plan the capture never saved, so its addressing stayed current: a region that declared no LoRA rows of its own replayed against it. And a base-only step left `active` False, which a compiled delta reads before it asks for the plan, so a later step without a LoraStep skipped its deltas silently. Every end of a step that does not hand an enclosing step's plan back now clears the plan and sets `active`, so the delta refuses in both forms until the next plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: the engine adds a bound node's rows wherever its step has segments (R1.6, R1.11, R1.13) engine_step raised for every step of a bound node that declared no segments (Bagel's), even on walks that run no adapted module, and refused a node's own LoraStep under a pool's key; binding refused a node without lora_segment_rows, whose one legal value left nothing to say. Now lora_rows adds each pool's LoraStep from the step's segments unless the step declares its own (kept), and adds none to a step with no segments, where an adapted module's delta refuses for want of a plan. The flag is gone. admit_declared reads the step the engine would plan, and engine_step has a test through a real NodeSubmodule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a piecewise region's step carries the node's rows (R1.2) engine_step added LoRA rows to the node's step only; a region's step is built by the piecewise runner from its config, and an attn_key region (which the config declares, so no model code can add to it) had no way to carry them. Its deltas read whatever plan was current. The runner now builds every region step (sizing, capture, per replay) through the same lora_rows, so a region's rows come from its own segments, as the node's do. A region replayed one row per step would give every request the first one's adapter (Qwen3.5's vision block loop): on a node serving adapters it is dropped from capture like a region whose resources cannot capture, and runs eagerly under the node's step, whose declare_step then returns its segments. The docs and the pool's refusals say so instead of asking models to declare region rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: the engine's step-less warm-ups run bound modules as a base-only step (R1.1) compile_regions and warmup_compiles run a node's forwards outside any planned step, after its modules are bound. Eagerly a bound delta refused there for want of a plan; traced, it skipped the eager check and asked the pool for an addressing it did not have. Qwen3.5's vision tower warms its patch embed and merger this way, so serving adapters failed the boot. The engine now runs both under each of the node's pools' base_only(): this thread holds a plan in which no row names an adapter, as a base-only step does, and the pool's own state comes back after. A compile traced there takes the base-only branch, so the first step with an adapter row compiles its own; warming a zero-adapter plan instead would need an addressing sized to each target's rows in each example, which a warm-up does not know. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: refuse the children modules run other than through their forward over the step's tokens (R1.3, R1.5, R1.9) The walk binds every LoRA-capable child a module does not name in lora_unservable, and a bound child its parent never runs through its forward silently drops its delta, or adds it to rows that are not the step's: - FastConformer's relative_k_proj and Chatterbox's linear_pos project position encodings (FastConformer's once, at bind, for a stream); - FastConformer's subsampling convs a ReLU follows run as F.conv2d over their weight on the Triton path; - Bagel's ViT patch embedding is converted at load to a linear over (kh, kw, C) patches, which a conv adapter's (C, kh, kw) factors do not fit; - OmniVoice's backbone runs a layer's projections, but for o_proj and down_proj, as F.linear over the fused weights apply_flashinfer leaves; the reference package builds those modules, so the mstar backbone names them per instance, whichever exist. Whisper's word-timestamp pass ran four projections as F.linear over their weights; it calls the modules now (the same ops unbound, test_whisper_alignment holds it to HF), so a bound one is never skipped there, rather than refusing LoRA on the decoder's attention for every walk. One parametrized case walks each annotated module, and the code predictor's from abff1d1, on the meta device. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * gelu_tanh_ffn: run a LoRA-bound linear_in through its forward (R1.4) Compiled on CUDA, the DiT FFN helper ran linear_in as an addmm over its weight, so a bound delta was never added; abff1d1 refused Wan2.2's linear_in for LoRA on every device to cover it. The helper now takes the fused epilogue only for an unbound linear_in and calls the module otherwise, and Wan2.2's FFN serves adapters again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: lease a one-row region only for a step whose rows run no adapter (R2.1) A one_row_per_step region on a LoRA node was dropped from capture at startup (logged at ERROR, and failing a start that requires accelerator graphs), so the region ran eagerly for every step, base-only ones included. Capture it as any region, and have PiecewiseAcceleratorGraphRunner.can_lease refuse a batch any of whose rows runs an adapter (LoraAdapterPool.runs_adapters, set at ingest and the same on every rank). A base-only step replays it with every row on the zero slot; a step with an adapter row runs the region eagerly under the node's own step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: refuse cross-attention's k/v projections, run over the encoder context (R2.2) CrossAttention.compute_kv and ParallelCrossAttention.compute_kv run k_proj and v_proj over the encoder's rows, not the step's tokens, so a delta bound to either would be addressed by the decoder's segments and land on the wrong rows. Both name them in lora_unservable, so an adapter touching them is refused at load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Whisper: the align walk declares its rows, so a LoRA delta has a plan (R2.3) Since the word-timestamp pass calls the decoder's projections as modules, a LoRA deployment's align step (base-only included) refused at its first projection: the walk declared no step, so no LoraStep addressed its rows. It now declares one segment per request over the tokens it teacher-forces, which are the rows the projections see, and no resource steps: the step is not captured or chunked, so nothing else reads it. The cross k/v projections it runs over the encoder output are refused for LoRA (R2.2). The docstring said the delta was never skipped; it now says where it adds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: lora_rows returns a new step instead of filling the caller's (R2.6) lora_rows added each pool's LoraStep with setdefault on the declared step's own dict, so a declare_step that returned one step object across calls kept the first call's rows on every later call. It now builds a new step with dataclasses.replace when it adds anything, leaving the declaration as given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * gelu_tanh_ffn: pin that a compiled LoRA-bound linear_in adds its delta (R2.5) A CUDA-only case (it skips on the CPU box): compiled, gelu_tanh_ffn runs linear_in as an addmm over its weight unless it is LoRA-bound (R1.4), so without that check the bound delta is dropped and the compiled output differs from the module's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: capture a one-row-per-step region base-only on a LoRA node (R3.1) A one_row_per_step region is leased only for steps with no adapter row (can_lease), yet it was captured under the region's own key, so the graph recorded the LoRA delta kernels and every replay planned an all-ZERO_SLOT addressing. Capture it under LoraBaseOnly instead: the graph records no delta, the replay's plan returns at once, and the pool's base-only guard raises if an adapter row ever reaches the lease. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA: a one-row region refuses a replay any of whose rows runs an adapter (R4 P3s) can_lease already refuses such a step; run now checks every row too, not only the first one the replay names. Also corrects the can_lease docstring (the region is captured base-only) and a test's wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * LoRA test: match the one-row region's own refusal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The data worker prepares one request at a time on its own thread. For Qwen3-Omni video that's 600 ms a clip, so at V2T c=4 a request sits behind the clips ahead of it and time in preprocess is most of its TTFT, and the output chunks of running requests wait behind the same thread. This moves media decode and
process_promptonto a pool for a model that says they can overlap.Change
Model.PREPROCESS_PARALLEL(default False) declares thatload_image/load_audio/load_videoandprocess_promptkeep no shared state. Qwen3-Omni sets it, no other model does.PreprocessWorkerThread._process_inputis split into_start_input(the thread's counters),_prepare_input(decode, processors, prefix keys; reads only what__init__fixed) and_finish_input(store, kwargs, send). Inline runs them in a row, as before._submit_prepareruns_prepare_inputon it and_release_preparedfinishes the oldest done request, one per pass, on the thread. Requests reach the conductor in arrival order whichever finishes first. Tensors are still stored and sent from the data worker's thread only._next_inputstakes nothing while it's full, so requests wait in the queue undecoded, and a release waits while the outbox is full.CpuPlan.preprocess_workerssizes it,min(8, cpus left // intra-op threads), 4 on 8 CPUs.preprocess_workers:in the serving config overrides it, 1 turns it off.Guard and fallback: no declaration, a model with client streams (an append has to follow its request), or one worker means the old inline path.
A request abandoned while it prepares: the API server drops its pending entry before it queues the abort, so
_release_preparedsees it as not pending and drops it unsent, as it would one still in the queue. The abort and READS_DONE that went out meanwhile are for a rid the conductor never saw, which it already handles.Does each stage release the GIL
Stage cost alone and how each scales across threads, 8 pinned cores, torch intra-op 2 (
dataworker_cost.py):image_processorload_video(decode, resize)video_processorDecode, resize, the video processor and the feature extractor release the GIL: the 2 and 4 thread columns are throughput against one thread. The one that holds it is the image processor, 2.3x at 4 threads, and it's 1.7 ms.
load_videostops at 2.5x because a clip is about 1.1 s of CPU across ffmpeg's and torch's threads, so 4 clips run out of the 8 cores rather than the GIL. A whole video preparation is 1.9x at 2 threads and 2.9x at 4, images 1.6x and 3.1x, audio 2.0x and 3.7x.On main
#1072 landed while this ran, so the gate below is against #1072 and the branch has since been rebased onto main. One more session per arm on main, same protocol, all arms sharing an Inductor cache that started from one cold compile:
Failed requests: 0. The 100-input set is 100/100 identical to main, and matches HF fp32 on 48/100, as main.
Gate against #1072
Failed requests: 0 of 7600.
With #1072 at the base the gate cells are V2T c=4, which goes from 1.65 to 3.11 tok/s and halves TTFT, and everything else, which doesn't move beyond the 1-2% between sessions. I2T c=16 TTFT p50 is 8 ms up (177 to 185) and p95 51 ms up (417 to 468).
V2T c=4 TTFT is 1222 ms against 708 at c=1, so it doesn't reach c=1. What's left isn't this thread: with the pool,
recv -> preprocess doneis 877 ms against 531 at c=1, and ingest to first chunk 302 ms against 152, where clips queue at the encoder and the Thinker. A clip is 1.5 s of CPU, so four at once are about all of the 8 cores. The pool's size and ffmpeg's threads barely move it:On a server that hasn't seen the clips vllm-omni is at 1011 ms and 3.66 tok/s here. The 592 ms it posts when the same clips ran at c=1 first comes from its processor cache.
API process CPU per request:
That's up 7-12 ms a request on the small inputs. It moved no wall clock in any cell, and I haven't established where it goes.
Correctness
Tests
ruff check .: passes.python -m pytest test/modular test/distributed test/chatterbox test/kokoro test/omnivoice test/sampling_test mstar/model/*/tests benchmark/worker_phases, xdist -n 8,OMP_NUM_THREADS=2): 9261 passed, 856 skipped, 49 failed. The same 49 fail on main, compared by id:test_loop_output_ownership(32),test_send_registration_failure(5),test_wan_vae_decode(4),test_z_image_native_equivalence(3), others 5.test_flex_attention_resource,test_waypoint_componentsandtest_waypoint_ditsegfault on this box on main too, so they run apart and aren't counted.test/modular/test_data_worker_prepare_pool.py, 20 tests: order, one release a pass, a full pool and a full outbox, failure in its slot, an abandoned request, the fallbacks, sizing.Hardware: 2 x B300 SXM6 (sm_103), torch 2.12.1+cu132, FlashInfer 0.7.2,
-O2, server on 8 cores, client on 4, prefix caching off. Two server sessions per arm, interleaved.Not in this PR