[Feature] Support Qwen3-TTS prefill-decode (PD) disaggregation by yurain26 · Pull Request #4934 · vllm-project/vllm-omni

yurain26 · 2026-07-07T05:13:02Z

Qwen3-TTS Prefill–Decode (PD) Disaggregation + Multi-token-prediction (MD) Disaggregation

See benchmark/BENCHMARK.md / README.md for details.

1. Description

PD disaggregation: the talker runs as a standalone prefill stage that produces the prefix KV via Mooncake and streams it to one or more decode stages; code2wav consumes the decode-stage output over a shared-memory connector. Same-host only, no cross-node RDMA required.
MD disaggregation: inside the decode worker, the code predictor (MTP) is decoupled from the AR talker and replicated (e.g. 2× MTP : 1× AR). Each replica owns an independent CUDA stream and a private CUDA-graph pool; requests are pinned to the same MTP replica across decode steps via crc32(req_id) % num_replicas.
Adds the Mooncake PD transfer patch vllm_omni/distributed/kv_transfer/mooncake_pd_patch.py; deploy configs 1P1D / 1P3D / 1P6D / 3P1D; example scripts and probe tools; orchestrator / scheduler / engine entrypoint changes; E2E test tests/test_pd_disaggregation.py.

vLLM v0.25.0　commit 8ccd02f85

2. Dataset

seed-tts-eval/en (English short-sentence TTS), 100 prompts per concurrency level (seed=0, reproducible).

which git-lfs || (apt-get update && apt-get install -y git-lfs && git lfs install)
cd /root/datasets && git clone https://github.com/BytedanceSpeech/seed-tts-eval.git
cd seed-tts-eval && git lfs pull
# model
modelscope download --model Qwen/Qwen3-TTS-12Hz-0.6B-Base --local_dir /root/models/Qwen3-TTS-12Hz-0.6B-Base

3. Data sampling (deterministic 100 prompts)

The benchmark loader uses random.Random(0).shuffle, so NUM_PROMPTS=100 always selects the same 100 rows. You can pre-extract and persist them:

python3 - <<'PY'
import pathlib, random
src = pathlib.Path("/root/datasets/seed-tts-eval/en/meta.lst")
dst = pathlib.Path("/root/datasets/seed-tts-100/en"); dst.mkdir(parents=True, exist_ok=True)
pw = dst/"prompt-wavs"
if not pw.exists():
    pw.symlink_to("/root/datasets/seed-tts-eval/en/prompt-wavs", target_is_directory=True)
rows = [l for l in src.read_text().splitlines() if l.strip() and not l.startswith("#")]
random.Random(0).shuffle(rows)          # same seed as the loader
(dst/"meta.lst").write_text("\n".join(rows[:100])+"\n")
print("locked", len(rows[:100]), "rows")
PY

Then use DATASET=/root/datasets/seed-tts-100 (only a symlink + a 100-row meta.lst; no re-download needed).

4. Parameters (Single vs 1P1D, both 2 GPUs)

min_tokens: 2 (single) avoids empty audio (#4962); stop_token_ids: [2150] (1P1D) enforces the codec EOS stop. The prefill-side min_tokens must be 1 (not 0): Mooncake only initiates the KV transfer when prefill ends with finish_reason='length'; setting 0 lets a first-token EOS end the step as stop and drop the transfer, so the decode side never receives the prefix KV.

5. Performance (`seed-tts-eval` en, 100 prompts/concurrency)

@ c=32: throughput +13.5%, RTF −14%; @ c=64: throughput +19%, RTF −12% (RTF still < 1.0).
@ c=8: single wins (PD cross-stage overhead is more noticeable at low concurrency).

6. Run flow

Prepare dataset: clone seed-tts-eval per §2 (git-lfs).
(Recommended) extract the deterministic 100-prompt subset: per §3 to build seed-tts-100; or skip — with DATASET=/root/datasets/seed-tts-eval + NUM_PROMPTS=100 the loader uses the same random.Random(0) seed and selects the same 100 rows, so results are identical.
Run the benchmark (below).

run_all_bench.sh iterates over each topology (single / 1p1d / 3p1d / 1p3d / 1p6d): ① start_pd.sh brings up the service and waits for /health → ② runs vllm bench serve --omni (openai-audio-speech backend, seed-tts dataset) per CONCURRENCIES level, saving the raw log / summary.txt / JSON / one self-describing check wav per level → ③ stop_pd.sh tears down the service and frees GPUs; finally it aggregates all topologies into a comparison table SUMMARY_ALL.txt (read this first).

cd /root/vllm-omni/examples/online_serving/qwen3_tts_pd/benchmark
MODEL=/root/models/Qwen3-TTS-12Hz-0.6B-Base \
DATASET=/root/datasets/seed-tts-eval LOCALE=en \
TOPOS="single 1p1d" CONCURRENCIES="8 32 64" NUM_PROMPTS=100 CHECK_AUDIO=0 \
bash run_all_bench.sh 2>&1 | tee results/cmp_run.log
# single topology only: bash start_pd.sh 1p1d | one-shot: bash bench_pd.sh 1p1d 8 50 zh | stop: bash stop_pd.sh 8091

Env vars MODEL / HOST / PORT / DATASET / VLLM_OMNI_DIR / RESULT_ROOT are all overridable; start_pd.sh auto-detects the vllm-omni repo root.

7. Test Plan

pytest tests/test_pd_disaggregation.py; manually deploy 1P1D via qwen3_tts_pd_1p1d.yaml and probe with probe_1p1d.py. E2E tests pass; probe_1p1d.py shows the audio_codes stream is bit-identical to the single-process baseline, with PD median TTFB within 3× of the baseline.

Base: main

chatgpt-codex-connector · 2026-07-07T05:13:07Z

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

yurain26 · 2026-07-07T05:41:23Z

@linyueqian @Sy0307 麻烦帮忙 review 一下这个 Qwen3-TTS PD disaggregation 的 PR，谢谢！

hsliuustc0106 · 2026-07-07T07:21:15Z

submit to main please

linyueqian

Thanks for the substantial work here. The M-P-N-D topology detection, per-prefill bootstrap bookkeeping, and the port-collision validation in pd_utils are all solid, and the deploy YAMLs are thoroughly annotated. I ran the PR end to end on one 8x NVIDIA H20 (141GB) node (Qwen3-TTS-12Hz-0.6B-Base, seed-tts-eval en, 50 prompts per cell, vllm bench serve --omni, concurrency 1/8/16/32) and reviewed the runtime changes. Requesting changes based on what I measured.

1. [Blocking] This PR halves non-PD Qwen3-TTS throughput at concurrency

single topology, default qwen3_tts.yaml, 1 GPU, identical prompt set:

c	base `d4a869fe` (req/s)	this PR (req/s)	delta
1	1.33	1.35	~0
8	3.83	2.13	-44%
16	4.49	2.17	-52%
32	4.45	2.15	-52%

Mean E2EL roughly doubles at c>=8 (1979 -> 3595 ms at c=8) and RTF at c=16 goes 0.72 -> 1.52, i.e. slower than real time. The cause is the _talker_mtp_forward change (see inline comment): reverting only that hunk on the PR head restores c=16 to 4.53 req/s. Every Qwen3-TTS deployment shares this code path, so this is a blocker independent of PD.

2. PD gains do not reproduce once the baseline is fair

Real 1p1d (3 GPUs: prefill + decode + code2wav; Mooncake handshake confirmed in logs and per-GPU memory residency):

c	req/s	mean TTFP (ms)	mean E2EL (ms)
1	1.11	272	899
8	2.50	289	3034
16	2.44	884	6145
32	2.43	1921	11111

And a tuned non-PD control on the base branch: talker and code2wav split across 2 GPUs, max_num_seqs: 128, gpu_memory_utilization: 0.6, max_num_batched_tokens: 4096 (the same caps the PD YAMLs use):

c	req/s	mean TTFP (ms)
1	1.45	148
8	6.18	344
16	7.32	712
32	8.78	677

Observations:

1p1d on 3 GPUs (2.4-2.5 req/s at c>=8) is slower than today's default single on 1 GPU (3.8-4.5 req/s), and about 3.6x slower than the 2-GPU non-PD control. The control also beats the absolute 1p6d number from the PR description (8.14 req/s on 8 GPUs) using 2 GPUs.
The benchmark table in the description compares PD against a single baseline that (a) runs on this branch, so it carries the regression from item 1, and (b) keeps max_num_seqs: 10 / gpu_mem 0.3 / 1 GPU while PD stages get 128-256 seqs / 0.6 / 3-8 GPUs. The single TTFP blowup at c>=16 in that table is queueing at the max_num_seqs cap, not a fundamental limit.
The one real effect I can confirm: PD keeps TTFP flatter under load (884/1921 ms at c=16/32 vs 1244/3328 ms for default single). But the 2-GPU non-PD control is better still (712/677 ms), and at c=1 the KV-transfer hop makes PD TTFP worse than baseline (272 vs 158 ms).

Suggestion: rerun the sweep with the baseline on the base branch, caps aligned, and a same-GPU-count non-PD control (independent replicas or a split-stage deploy). If 1pNd still wins under those controls at some N, that is the number worth advertising.

3. Correctness verification is currently not runnable

tests/test_pd_disaggregation.py is skipped at module level (see inline comment), so the stated test plan passes without executing an assertion.
probe_1p1d.py requires an AR-only NDJSON deploy; start_pd.sh references qwen3_tts_pd_ar_only_3p1d.yaml which is not in the PR, and the shipped 1p1d yaml emits audio, which the probe rejects by design. The manual test plan cannot be executed as written.
I spot-checked audio with whisper-small on the self-describing check wavs: intelligible speech in both PD and non-PD cells, occasional garbled samples in both (temp 0.9 on a cloned voice), so inconclusive. Given the probe docstring itself warns about the code-predictor KV path, a runnable seeded/greedy bit-equality check should land with this PR.

4. Environment and packaging

mooncake-transfer-engine 0.3.8.post1 wheels require glibc >= 2.34 and link against CUDA 12 userspace. On a glibc 2.32 host with a cu13 torch stack the connector cannot load at all (libcudart.so.12 -> GLIBCXX_3.4.29 -> GLIBC_2.34 chain); I could only bring PD up inside an Ubuntu 22.04 container after also apt-installing libcurl4 and libibverbs1. None of this is documented and there is no requirements/extras entry for mooncake. Please add an optional dependency and a supported-environment note.

Also: as @hsliuustc0106 said, please retarget to main; DCO is failing as well.

5. Comment rewrites

Roughly 40 existing comments across the scheduler, orchestrator, transfer adapter, runner mixin, and patch.py are rewritten into short [yurain] tags, deleting explanations that document invariants (the wire-schema asymmetry note in chunk_transfer_adapter, the prune-protection rationale in the runner mixin, the LogitsProcessor invariant, the issue #2274 reference in qwen3_code_predictor). Please revert all comment-only hunks; that will also shrink the diff substantially.

Happy to re-run the same sweep on the same hardware once the perf hunk is addressed.

linyueqian · 2026-07-07T18:07:42Z

-            # A torch.Generator is a single stream. Using one generator for a
-            # multi-row batch would make explicitly-seeded requests depend on
-            # other rows in the same scheduler step, so keep that path scalar.
+        if decode_batch_size > 1:


[Blocking] This makes every multi-row decode batch run the code predictor one request at a time, where the previous gate only did that for explicitly seeded requests without per-row generator support. Measured on one H20 (single topology, default yaml, identical prompts): c=8 3.83 -> 2.13 req/s, c=16 4.49 -> 2.17, c=32 4.45 -> 2.15. Reverting only this hunk on the PR head restores c=16 to 4.53. This path is shared by all Qwen3-TTS deployments, not just PD. If PD decode needs the scalar path for cross-request determinism, gate it on the PD decode role (e.g. kv_transfer_config.kv_role == "kv_consumer") or keep the previous generator-based condition.

linyueqian · 2026-07-07T18:07:42Z

+        )
+
+
+_patch_mooncake_connector_for_pd()


[High] This applies the Mooncake monkey patch at vllm_omni import time in every process whenever mooncake happens to be installed, including non-PD deployments and unrelated models, and it mutates upstream vllm classes globally (request_finished, _send_blocks, register_kv_caches, the bootstrap server). _init_pd_state already applies the patch when a PD topology is actually detected, which is the right scope. Suggest removing this import-time call.

linyueqian · 2026-07-07T18:07:42Z

+                    return ret_k if ret_k != 0 else ret_v
+            except Exception:
+                pass
+            return original_send_blocks(self_worker, remote_session, src_ptrs, dst_ptrs, lengths)


[High] The K/V-split pointer arithmetic assumes a contiguous [2, num_blocks, ...] layout, and any exception in the try block silently falls back to the unpatched single call. If the layout assumption is ever wrong (different attention backend, MLA, a future vllm layout change), this transfers the wrong half of the cache and surfaces as corrupted audio instead of an error. Suggest validating the layout explicitly and raising on mismatch, and documenting which vllm version this pointer math was written against.

linyueqian · 2026-07-07T18:07:42Z

+
+from vllm_omni.entrypoints.pd_utils import PDDisaggregationMixin
+
+pytestmark = pytest.mark.skip(reason="Temporarily skip PD entrypoint tests while PD config is being removed.")


[High] The whole module is skipped, so pytest tests/test_pd_disaggregation.py (the PR's test plan) passes without executing a single assertion. Either unskip the tests that apply to this implementation or drop the file. The docstring also references issue #1188 and a "v1908" migration that do not exist in this repo.

linyueqian · 2026-07-07T18:07:42Z

+                        merged_info[key] = sub
+                    else:
+                        merged_info[key] = value
+                request.additional_information = merged_info


[Medium] Behavior change for every async_chunk AR stage, not just PD: chunks previously replaced additional_information wholesale, now they are dict-merged one level deep, so a sub-key that disappears from a later chunk keeps its stale value. If the merge only needs to preserve omni_source_stage_id, consider carrying that single key instead of a general merge, or add a test pinning the intended semantics.

linyueqian · 2026-07-07T18:07:42Z

+            if len(non_pd_decode) == 1:
+                return non_pd_decode[0]
+            raise RuntimeError(
+                f"[Orchestrator] Ambiguous downstream stages for stage-{src_stage_id}: {candidates}"


[Medium] This now raises for a stage with more than one non-PD downstream, where the old code always routed to stage_id + 1. Any existing fan-out pipeline that worked with numeric adjacency will now crash at forward time. Consider falling back to stage_id + 1 with a warning, or auditing the shipped configs to confirm none are affected.

linyueqian · 2026-07-07T18:07:42Z

        if hasattr(sp, "min_tokens"):
            try:
-                sp.min_tokens = 1
+                sp.min_tokens = 0


[Medium] The removed comment said MooncakeConnector cancels the KV transfer unless finish_reason is "length"; min_tokens 1 -> 0 makes it possible for the prefill request to finish with reason "stop" at max_tokens=1. If the transfer still fires in that case, please note why (and keep the comment); if not, this reintroduces the bug the old value guarded against.

linyueqian · 2026-07-07T18:07:42Z

+                  "${SITE_PKG}/mooncake_transfer_engine-0.3.8.post1.dist-info" 2>/dev/null || true
+    fi
+    # 验证 mooncake import 正常（如果还失败就提前退出，省得跑 60 秒等 worker 死）
+    if ! python3 -c "from mooncake.engine import TransferEngine" 2>/dev/null; then


[Medium] Three issues in this script:

This preflight false-negatives on pip-CUDA stacks: bare python3 -c "from mooncake.engine import TransferEngine" fails on libcudart even when the real server (torch imported first) boots fine. Use python3 -c "import torch; from mooncake.engine import TransferEngine".

The site-packages mv self-heal above (with a sudo fallback) is not something a shipped example should do; pin the dependency properly instead.

Line 37: DEFAULT_VLLM_OMNI_DIR="$(find_repo_root)" runs under set -e before the VLLM_OMNI_DIR override is honored, so the script dies silently (no output at all) when placed outside a repo checkout. Also the ar_3p1d case references qwen3_tts_pd_ar_only_3p1d.yaml, which is not in the PR.

linyueqian · 2026-07-07T18:07:42Z

+| YAML | Topology | GPUs | Use |
+|------|----------|------|-----|
+| [`qwen3_tts_pd_1p1d.yaml`](../../../vllm_omni/deploy/qwen3_tts_pd_1p1d.yaml) | 1 prefill + 1 decode + 1 code2wav | 3 | Smallest configuration; matches the legacy 1P1D path exactly. Use this for the [probe script](#1-correctness--latency-probe-1p1d) and as the K=1 regression baseline. |
+| [`qwen3_tts_pd_mp1d.yaml`](../../../vllm_omni/deploy/qwen3_tts_pd_mp1d.yaml) | 2 prefill + 1 decode + 1 code2wav | 4 | Production M-P-1-D recipe with `pd_prefill_pick_strategy: round_robin`. |


[Low] This table, section 2, and the probe example (line 79) reference qwen3_tts_pd_mp1d.yaml and .codebuddy/plan/.../migration_inventory.md, neither of which exists in the PR; the troubleshooting section also references monkey_patch.py (now mooncake_pd_patch.py). Please reconcile the docs with what ships, and consider translating the shipped scripts' comments to English to match the rest of the repo.

…k figure

linyueqian · 2026-07-08T03:07:05Z

Follow-up on the "1p1d alone undersells PD, the value shows at 1pNd" point: we ran the shipped qwen3_tts_pd_1p6d.yaml unmodified on one 8x H20 (141GB) node, against the honest same-GPU-count baseline the earlier review asked for: 8 independent single replicas (default qwen3_tts.yaml, base branch d4a869fe, one per GPU) driven by 8 parallel bench clients. Same node, same container image, same dataset (seed-tts en, 50 prompts per client, 2 warmups per cell).

aggregate concurrency	8x single replicas (base)	1p6d PD (this PR)	throughput ratio
8	11.1 req/s, TTFP p50 146 ms	7.39 req/s, 230 ms	1.5x
16	18.9 req/s, 85 ms	8.49 req/s, 314 ms	2.2x
32	27.2 req/s, 121 ms	9.76 req/s (peak), 2023 ms	2.8x
64	28.4 req/s, 210 ms	8.88 req/s, 3478 ms	3.2x

Mean stream underrun is sub-second for both at every point (baseline 0.18-0.61 s, 1p6d 0.17-0.43 s), so continuity does not change the ranking.

A few observations:

Decode replication does scale, which confirms the direction of your argument: on our nodes the decode-side plateau goes 2.15 req/s (1 decode) to 4.4 (1p3d, 3 decodes) to 9.8 (1p6d, 6 decodes). But that gain is replication, not prefill/decode separation, and plain replication does it better: 8 replicas deliver 1.5x to 3.2x more throughput at every operating point, with median TTFP flat at 85-210 ms where 1p6d degrades to 2-3.5 s.
Under an SLO lens (TTFP p50 <= 500 ms and mean underrun <= 0.5 s), per-GPU goodput is 3.40 req/s/GPU for the replicated baseline (passes at C=32) vs 1.06 for 1p6d (last passing point c=16), a 3.2x gap. Our 1p6d absolute numbers are consistent with the table in the PR description (9.76 en here vs 8.14 zh there), so the PD-side measurement is not in question; the comparison target is.
The mechanism is what the earlier review noted: Qwen3-TTS prefill is a few percent of total compute, so there is very little for P/D separation to win, while decode (where TTS latency and continuity live) pays the KV-transfer and orchestration costs.

Two additional issues observed while running PD, both reproducible and worth investigating independently of the benchmark discussion:

Post-boot stall: on every PD topology we brought up (1p1d, 1p3d, 1p6d), the first request(s) after server start hit a 30-67 s time-to-first-packet outlier before the pipeline settles (for example 1p6d c=1 cell: median TTFP 254 ms, p99 67.7 s). Non-PD deployments on the same nodes do not show this.
Flaky 1p6d bring-up: one boot attempt failed with stage-6 (decode) hitting CUDA OOM during cudagraph warmup, with the worker having allocated 139.2 of 139.8 GiB despite gpu_memory_utilization: 0.6 in the yaml; an identical retry succeeded. Looks like a memory-profiling race during the sequential 8-stage bring-up.

Caveats for the record: en locale (zh prompt-wavs were not available on these nodes), 50 prompts per client, one repetition per cell, PD c=1 cells excluded from the figure due to the post-boot stall artifact, and two of eight baseline clients at C=64 reported anomalously low throughput (client-side skew suspected; the conservative sum is reported).

If there is a regime where the PD split itself (not decode replication) wins, the cleanest way to show it would be: baseline and PD on the same commit with the perf regression fixed, same GPU count on both sides, aligned max_num_seqs / gpu_memory_utilization / max_num_batched_tokens, and goodput under a stated TTFP+continuity SLO as the headline metric. Happy to re-run that matrix on the same hardware once the branch is updated.

linyueqian · 2026-07-08T04:00:28Z

@yurain26 here are the 4-GPU and 5-GPU replicated baselines you asked for, same protocol and node as the 8x run above (base branch d4a869fe, default qwen3_tts.yaml, one replica per GPU, parallel clients with 50 prompts each, 2 warmups per cell, seed-tts en, same container image).

4x single replicas (4 GPUs):

aggregate concurrency	agg req/s	median TTFP	mean underrun
4	5.39	146 ms	0.18 s
8	9.49	83 ms	0.25 s
16	13.62	124 ms	0.36 s
32	15.35	208 ms	0.61 s

5x single replicas (5 GPUs):

aggregate concurrency	agg req/s	median TTFP	mean underrun
5	6.78	145 ms	0.18 s
10	11.68	84 ms	0.25 s
20	16.94	~132 ms	0.36 s
40	19.07	~210 ms	0.60 s

Per-replica numbers are essentially identical across N=4/5/8 (about 1.35 / 2.35 / 3.39 / 3.83 req/s at per-instance c=1/2/4/8), so replication scales linearly on this node and you can extrapolate other N by multiplying.

For reference against your PD topologies on the same hardware class: 1p3d measured 4.4 to 4.6 req/s peak (4-GPU variant with code2wav co-located on the prefill GPU; your original 5-GPU layout should land close to that since the prefill GPU is mostly idle either way). So the current gaps are roughly 3x to 3.5x at matched GPU count. Happy to re-run any of these against your tuned branch when it is ready; the drivers are reusable as-is.

hsliuustc0106 · 2026-09-16T15:20:39Z

                logger.error(f"Error updating model intermediate buffer: {e}")
-            # Decode additional_information payloads (dictionary)
+            if model_buffer_applied is not None:
+                self._restore_qwen3_tts_pd_sampler_state(req_id, model_buffer_applied)


[P2] Same contract issue on the decode side: _restore_qwen3_tts_pd_sampler_state has five raise paths (missing req_state, invalid payload, bad tensor, missing generator for non-greedy, set_state failure) and both call sites here sit outside the surrounding try/except blocks, so a validation failure propagates out of _update_states → execute_model → EngineDeadError, killing every in-flight request on the replica. Please wrap per-request and mark just that request failed.

hsliuustc0106 · 2026-09-16T15:20:43Z

+            "pp_rank": int(getattr(self_worker, "pp_rank", 0)),
+            "addr": addr,
+        }
+        resp = requests.post(f"http://{host}:{port}/register", json=payload, timeout=5)


[P2] This synchronous requests.post(timeout=5) runs inside patched request_finished on the scheduler thread — an unreachable/slow bootstrap server stalls scheduling for up to 5s per finished request, and total failure is swallowed by the except below, so the first sign of trouble is decode pulls timing out at the 60s watchdog. Consider a background registration task with retry instead of inline registration.

hsliuustc0106 · 2026-09-16T15:20:47Z

+                pull_deadlines.pop(req_id, None)
+                blocks = pull_block_ids.pop(req_id, None)
+                if blocks:
+                    error_block_ids.update(blocks)


[P2] On expiry the request is reported finished and its blocks marked invalid so the scheduler can recompute — but the in-flight Mooncake pull task is not cancelled and keeps running. When the scheduler recycles those block ids for the recompute, a late-completing transfer can still be writing into them. Does the pull get cancelled (or the blocks fenced until the transfer settles) before the recompute can reuse them?

hsliuustc0106 · 2026-09-16T15:20:51Z

+        fetch_finished_recving_reqs._vllm_omni_wrapped = True  # type: ignore[attr-defined]
+        worker_cls.fetch_finished_recving_reqs = fetch_finished_recving_reqs  # type: ignore[assignment]
+
+    if not hasattr(worker_cls, "get_block_ids_with_load_errors"):


[P3] The hasattr guard means _vllm_omni_load_error_block_ids is only drained when the pinned vLLM does not already define get_block_ids_with_load_errors. On a pin where upstream defines it, this entire error path silently stops reporting load errors to the scheduler — worth an explicit version check (or an assertion) given the PR also needs to move from the v0.25.0 base to the repo's current pin.

hsliuustc0106

First pass from me on the core PD/MD paths (mooncake patch, scheduler KV-reclaim, orchestrator handoff, runner restore). The design is ambitious and mostly holds together — decode double-submit is idempotent, MTP replicas get private graph pools with correct stream ordering, KV-reclaim has a dedicated test, and the per-request handoff payload is one-shot. Findings inline: per-request raises on the orchestrator output-dispatch path and the runner restore path take down the whole process instead of one request (P2 ×2), inline bootstrap registration stalls the scheduler thread (P2), the pull watchdog leaves the in-flight transfer running while its blocks get recycled (P2), and the hasattr-guarded error drain silently disables on pins that already define get_block_ids_with_load_errors (P3).

Also from the thread + checks: buildkite is red with PR-caused failures (per @linyueqian's 08-30 analysis), DCO is failing, and the branch sits on a v0.25.0 base that no longer matches the repo pin — a rebase onto current main is needed before the next CI read means anything.

hsliuustc0106 · 2026-09-20T13:17:29Z

@yurain26 gentle ping — this high-priority PR has seen no updates for 22 days and review feedback is waiting. Could you share a status or ETA when you get a chance? Thanks!

vllm-omni-review-bot · 2026-09-23T06:13:16Z

Omni ReviewBot: CI is red on this head

@yurain26 required checks failed on f3b8ca9120bb:

buildkite/vllm-omni

Please fix the failure and push again; this note is updated in place when the head goes green or moves.

hsliuustc0106 · 2026-09-24T00:11:00Z

@yurain26 gentle ping — this high-priority PR has seen no updates for 25 days and review feedback is waiting. Could you share a status or ETA when you get a chance? Thanks!

hsliuustc0106 · 2026-09-27T00:17:59Z

@yurain26 gentle ping — this high-priority PR has seen no updates for 28 days and review feedback is waiting. Could you share a status or ETA when you get a chance? Thanks!

hsliuustc0106 · 2026-10-01T00:07:10Z

@yurain26 gentle ping — this high-priority PR has seen no updates for 32 days and review feedback is waiting. Could you share a status or ETA when you get a chance? Thanks!

vllm-omni-review-bot · 2026-10-02T12:13:49Z

Omni ReviewBot routing record

Assigned Strict on cursor (cursor-grok-4.6-high) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

vllm-omni-review-bot · 2026-10-02T13:17:48Z

Omni ReviewBot attempt record