From 3dcc0a618d9bc5bb35e51bb4930f5fd20a7cf461 Mon Sep 17 00:00:00 2001 From: noonghunna <10742901+noonghunna@users.noreply.github.com> Date: Wed, 19 Aug 2026 15:01:13 +0000 Subject: [PATCH] fix(qwen3.8): patch GDN+MTP async wild-write crash on dual-fast (#1052) On a Qwen3-Next hybrid (GDN linear attention) with the MTP drafter + prefix caching + async scheduling (vLLM v0.27.1 default), the engine died with a CUDA illegal memory access in gdn_attn.py (Xid 31 MMU fault, VIRT_WRITE) within 6-13k generated tokens of agent-shaped traffic. Cause: gpu_model_runner.py::synchronize_input_prep waits prepare_inputs_event (recorded BEFORE the forward + spec-decode fused-align postprocess) but not num_accepted_tokens_event (recorded AFTER it), so under async scheduling the next step's _update_states block-table mutation raced the still-in-flight postprocess -> stale state-block index -> wild GPU write. Fix: also wait num_accepted_tokens_event at the top of synchronize_input_prep, closing the write-after-read window. It relocates a wait align-mode already did later, so TPS is neutral. Scoped to spec decode (event None otherwise) and async (the method is a no-op without prepare_inputs_event). Delivered as an idempotent, anchor-checked install_script patch (vllm-gdn-mtp-async-spec-order, shared from the qwen3.6-27b tree like pr48375), wired on vllm/qwen38-27b-dual-fast, registered in patches.yml. Ours, not an upstream PR (diagnosis + reproducer posted on vllm#50021 and vllm#52873). Validated on the reference 2x3090 (v0.27.1): async-ON survived 3/3 boots to 33-36k generated tokens each (crossing ctx 32k + a compaction) where 5/5 unpatched arms died 6-13k; real delivery path confirmed; bench neutral (79.1 narr / 108.5 code / 1756 prefill >= baseline). Does NOT address the separate acceptance-collapse bug (vllm#52873). Co-Authored-By: Claude Opus 4.8 --- .../vllm-gdn-mtp-async-spec-order/README.md | 44 ++++++++++++ .../vllm-gdn-mtp-async-spec-order/install.sh | 7 ++ .../patch_gdn_mtp_async_spec_order.py | 70 +++++++++++++++++++ .../vllm/compose/dual/autoround-int4/mtp.yml | 5 ++ scripts/lib/profiles/patches.yml | 40 +++++++++++ 5 files changed, 166 insertions(+) create mode 100644 models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md create mode 100755 models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh create mode 100644 models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py diff --git a/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md new file mode 100644 index 000000000..ba3d944b5 --- /dev/null +++ b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md @@ -0,0 +1,44 @@ +# club-3090 fix — order async input-prep after the prev-step spec-decode postprocess + +**Bug:** our [club-3090#1052](https://github.com/noonghunna/club-3090/issues/1052) +· upstream [vllm#52873](https://github.com/vllm-project/vllm/issues/52873) + +the [vllm#50021](https://github.com/vllm-project/vllm/pull/50021) thread. On a +Qwen3-Next hybrid (GDN linear attention) with the MTP drafter **+** prefix caching +**+** async scheduling (vLLM v0.27.1 default), the engine dies with a CUDA illegal +memory access in `gdn_attn.py` (Xid 31 MMU fault, `VIRT_WRITE`) within 6–13k +generated tokens of agent-shaped traffic. Drafter-off **or** `--no-async-scheduling` +avoids it; `CUDA_LAUNCH_BLOCKING=1` suppresses it — i.e. a cross-stream race, not +an index bug. **Distinct from vllm#50021** (whose in-kernel bounds do NOT fix this — +built its head, crashed 2/2) and from vllm#43559/#48375 (the corruption pair). + +**Cause:** in `gpu_model_runner.py::synchronize_input_prep`, the next step waits +`prepare_inputs_event` before touching reused CPU tensors — but that event is +recorded at the END of input-prep, BEFORE the forward + spec-decode fused-align +postprocess (`_update_states_after_model_execute`, which records +`num_accepted_tokens_event`). So under async scheduling the next step's +`_update_states` block-table mutation raced the still-in-flight postprocess reading +those buffers → stale state-block index → wild GPU write. + +**Fix (2 lines):** also wait `num_accepted_tokens_event` (recorded AFTER the +postprocess) at the top of `synchronize_input_prep`, closing the write-after-read +window. It relocates a wait that align-mode already performed later inside +`_prepare_inputs`, so there is no net GPU stall. Scoped to spec decode (the event +is `None` otherwise) and async (the whole method is a no-op without +`prepare_inputs_event`). + +**Delivery:** idempotent, anchor-checked `install.sh` → `patch_gdn_mtp_async_spec_order.py`, +mounted at `/etc/club3090/gdn-async-order/` and invoked from the compose entrypoint +before serve. Anchor drift → the container **refuses to boot** (exit 1) rather than +serving unpatched. + +**⚠️ Attribution / status:** this is **our own fix, NOT an upstream PR** — no +upstream commit vendored here. Validated on the reference 2×3090 (v0.27.1) only, +not community-blessed. Evidence: async-ON survived **3/3 boots** to 33–36k generated +tokens each (crossing ctx 32k + a compaction) where **5/5 unpatched arms died at +6–13k**; TPS neutral (79.1 narr / 108.5 code / 1756 prefill ≥ baseline). It does +**not** address the separate acceptance-collapse bug (vllm#52873 / Bug A). + +**Drop trigger:** an upstream fix for this race lands AND the `vllm-stable` pin +moves past it → remove the mounts + entrypoint call + this dir; `patches.yml` row +retires. If we file it upstream, update `upstream.status` from `ours` to `open` +with the PR link. diff --git a/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh new file mode 100755 index 000000000..ceea0d059 --- /dev/null +++ b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +# club-3090 GDN+MTP async spec-order fix installer — runs in the container +# entrypoint before serve. Idempotent; refuses boot (exit 1) on anchor drift so a +# re-pinned image can't silently serve unpatched while the compose claims the fix. +# Inert unless spec decode is active (the guarded event is None otherwise). +set -u +python3 /etc/club3090/gdn-async-order/patch_gdn_mtp_async_spec_order.py diff --git a/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py new file mode 100644 index 000000000..44852260f --- /dev/null +++ b/models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +"""club-3090 fix — order async input-prep AFTER the previous step's spec-decode postprocess. + +Bug (our club-3090#1052, upstream vllm#52873 + the #50021 thread): on a +Qwen3-Next hybrid (GDN linear attention) with the MTP drafter + prefix caching + +async scheduling (vLLM v0.27.1 default), the engine dies with a CUDA illegal +memory access in gdn_attn.py (Xid 31 MMU fault, VIRT_WRITE) within 6-13k +generated tokens of agent-shaped traffic. + +Cause: in gpu_model_runner.py, `synchronize_input_prep` waits `prepare_inputs_event` +before the next step touches reused CPU tensors -- but that event is recorded at +the END of input-prep, BEFORE the forward + spec-decode fused-align postprocess. +So under async scheduling the next step's `_update_states` block-table mutation +raced the still-in-flight postprocess reading those buffers -> stale state-block +index -> wild GPU write. This waits `num_accepted_tokens_event` (recorded AFTER +the postprocess) as well, closing the write-after-read window. + +Scoped to spec decode (the event is None otherwise) and async scheduling (the +whole method is a no-op without prepare_inputs_event). An unrecorded event is +treated complete, so the first step is unaffected. NOT an upstream PR -- our own +fix; validated 3/3 boots on the reference 2x3090 (async-ON survived 33-36k +generated incl. a compaction where 5/5 unpatched died 6-13k), TPS-neutral. + +Idempotent; anchor-checked; exits non-zero on drift so the entrypoint can refuse +to boot a silently-unpatched configuration. +""" +import io +import sys + +TARGET = "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py" +MARKER = "club-3090 GDN+MTP async spec-order fix" +DEF = " def synchronize_input_prep(self):" +ANCHOR = " self.prepare_inputs_event.synchronize()\n" +INSERT = """ # club-3090 GDN+MTP async spec-order fix (our #1052 / upstream #52873 + + # PR#50021 thread). prepare_inputs_event above is recorded BEFORE the + # forward + spec-decode fused-align postprocess, so under async scheduling + # the next step's _update_states block-table mutation raced the + # still-in-flight postprocess -> stale state-block index -> wild GPU write + # (CUDA illegal access / Xid 31 VIRT_WRITE). Wait the postprocess event + # (recorded AFTER it) to close that write-after-read window. Scoped to spec + # decode (event None otherwise); an unrecorded event is complete, so the + # first step is unaffected. + if self.num_accepted_tokens_event is not None: + self.num_accepted_tokens_event.synchronize() +""" + +def main() -> int: + try: + src = io.open(TARGET, encoding="utf-8").read() + except OSError as e: + print(f"[gdn-async-order] REFUSE: cannot read target: {e}") + return 1 + if MARKER in src: + print("[gdn-async-order] already patched (idempotent no-op)") + return 0 + d = src.find(DEF) + if d < 0: + print("[gdn-async-order] REFUSE: anchor drift - def synchronize_input_prep not found") + return 1 + i = src.find(ANCHOR, d) + if i < 0: + print("[gdn-async-order] REFUSE: anchor drift - prepare_inputs_event.synchronize() not found in method") + return 1 + j = i + len(ANCHOR) + io.open(TARGET, "w", encoding="utf-8").write(src[:j] + INSERT + src[j:]) + print("[gdn-async-order] applied: input-prep now waits the prev-step spec-decode postprocess") + return 0 + +if __name__ == "__main__": + sys.exit(main()) diff --git a/models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml b/models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml index 562cfee05..2bc246579 100644 --- a/models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml +++ b/models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml @@ -152,6 +152,10 @@ services: - ../../../../../qwen3.6-27b/vllm/patches/w4a8-int8-act:/etc/club3090/w4a8:ro # Lets the MTP drafter attach on this hybrid (mamba/eagle block drop). - ../../../../../qwen3.6-27b/vllm/patches/vllm-pr48375-mamba-drop-eagle-block:/etc/club3090/pr48375:ro + # club-3090 GDN+MTP async spec-order fix — closes the async wild-write race + # (our #1052 / upstream #52873 + #50021 thread). Ours, not an upstream PR; + # validated 3/3 boots on the reference 2×3090. Anchor drift → boot refused. + - ../../../../../qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order:/etc/club3090/gdn-async-order:ro - ../../../../../../scripts/detect_nvlink.sh:/etc/club3090/detect_nvlink.sh:ro - ../../../cache/torch_compile:/root/.cache/vllm/torch_compile_cache - ../../../cache/triton:/root/.triton/cache @@ -204,6 +208,7 @@ services: fi bash /etc/club3090/w4a8/install.sh bash /etc/club3090/pr48375/install.sh || exit 1 + bash /etc/club3090/gdn-async-order/install.sh || exit 1 echo "[w4a8] VLLM_MARLIN_INPUT_DTYPE=$${VLLM_MARLIN_INPUT_DTYPE:- W4A16>} (W4A8=$${W4A8:-1})" AR="" [ "$${_NVLINK_ENABLED:-0}" = "1" ] || AR="--disable-custom-all-reduce" diff --git a/scripts/lib/profiles/patches.yml b/scripts/lib/profiles/patches.yml index ab51ea2c5..795e2a11e 100644 --- a/scripts/lib/profiles/patches.yml +++ b/scripts/lib/profiles/patches.yml @@ -360,6 +360,46 @@ patches: status: local-vendored drop_when: "vllm#48375 merges upstream AND the vllm-stable pin moves past it -> remove the 6 compose mounts + entrypoint calls + the patch dir; retire this row (re-run the #710 probe protocol on the new pin first)." + - id: vllm-gdn-mtp-async-spec-order + # club-3090's OWN fix (NOT an upstream PR) for the GDN+MTP async wild-write + # race: our #1052 / upstream #52873 + the #50021 thread. Shared from the + # qwen3.6-27b tree like pr48375. Wired on qwen3.8 dual-fast only for now + # (the slug it was validated on); broader MTP-compose rollout is a separate + # maintainer decision (the fix is engine-wide in gpu_model_runner.py). + model: [qwen3.8-27b] + files: + - models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh + - models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py + load_bearing_when: + - composes: + - vllm/qwen38-27b-dual-fast + reason: "Our own 2-line fix for the async wild-write crash (club-3090#1052 / vllm#52873 + the vllm#50021 thread): on a Qwen3-Next hybrid with MTP + prefix caching + async scheduling (vLLM v0.27.1 default), gpu_model_runner.py::synchronize_input_prep waits prepare_inputs_event (recorded BEFORE the forward + spec-decode fused-align postprocess) but not num_accepted_tokens_event (recorded AFTER it), so the next step's _update_states block-table mutation raced the still-in-flight postprocess -> stale state-block index -> wild GPU write (CUDA illegal access / Xid 31 VIRT_WRITE) within 6-13k generated tokens of agent traffic. The fix also waits the postprocess event, closing the write-after-read window; it relocates a wait align-mode already did later, so TPS is neutral. Distinct from vllm#50021 (its in-kernel bounds do NOT fix this - built its head, crashed 2/2) and from vllm#43559/#48375. On-rig validation 2026-08-19 (reference 2x3090, v0.27.1): async-ON survived 3/3 boots to 33-36k generated tokens each (crossing ctx 32k + a compaction) where 5/5 unpatched arms died 6-13k; real delivery path confirmed (switch.sh -> compose mount -> entrypoint install.sh logs '[gdn-async-order] applied'); bench 79.1 narr / 108.5 code / 1756 prefill >= baseline. Does NOT address the separate acceptance-collapse bug (vllm#52873 / Bug A)." + evidence: "models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md; club-3090#1052; vllm#52873; docs/UPSTREAM.md" + delivery: + dockerfile_bake: false + entrypoint_invoke: true + genesis: false + delivery_gaps: [] + delivery_mechanism: install_script + delivery_spec: + script: models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh + mounted_at: /etc/club3090/gdn-async-order + invoke: "bash /etc/club3090/gdn-async-order/install.sh" + invoked_before: vllm-serve + wired_at: [volumes, entrypoint] + note: "Idempotent, anchor-based (def synchronize_input_prep + the prepare_inputs_event.synchronize() line). MARKER is a unique comment string (the executable num_accepted_tokens_event.synchronize() call already exists elsewhere in the file, so it can NOT be the idempotency marker). Failure policy: anchor drift -> boot REFUSED (exit 1 in entrypoint), never a silent half-wire. Wired ONLY on vllm/qwen38-27b-dual-fast (the validated slug); the fix is engine-wide so it is safe on every MTP compose, but broader wiring is a separate maintainer decision. No-op without spec decode (the guarded event is None) and without async scheduling (the whole method is a no-op)." + drift_guard: + kind: boot + check: "Entrypoint runs install.sh before serve; boot log contains '[gdn-async-order] applied' (first boot) or '[gdn-async-order] already patched' (warm layer). Anchor drift -> '[gdn-async-order] REFUSE' + container exit 1 (never serves). Behavioral: async-ON + MTP + prefix-caching survives an agent-shaped soak past ~15k generated tokens with no CUDA illegal-access / Xid 31 (5/5 unpatched died 6-13k)." + on_fail: boot-refused + status: verified + capability: gdn-mtp-async-crash-safety + foundational: false + upstream: + ref: "OURS - not yet filed upstream. Diagnosis + reproducer posted on vllm#50021 (comment) and vllm#52873; the crash is the async half, distinct from #50021's in-kernel bounds. club-3090#1052." + status: local-vendored + drop_when: "an upstream fix for this async race lands AND the vllm-stable pin moves past it (re-run the drift_guard on the new pin, then delete the mount + entrypoint call + this entry). If we file it upstream, change upstream.status to open-rebased-local with the PR link." + - id: qwen-w4a8-int8-act model: [qwen3.6-27b, thinkingcap-27b] files: