fix(qwen3.8): patch GDN+MTP async wild-write crash on dual-fast (#1052)

models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md

@@ -0,0 +1,44 @@
+# club-3090 fix — order async input-prep after the prev-step spec-decode postprocess
+
+**Bug:** our [club-3090#1052](https://github.com/noonghunna/club-3090/issues/1052)
+· upstream [vllm#52873](https://github.com/vllm-project/vllm/issues/52873) +
+the [vllm#50021](https://github.com/vllm-project/vllm/pull/50021) thread. On a
+Qwen3-Next hybrid (GDN linear attention) with the MTP drafter **+** prefix caching
+**+** async scheduling (vLLM v0.27.1 default), the engine dies with a CUDA illegal
+memory access in `gdn_attn.py` (Xid 31 MMU fault, `VIRT_WRITE`) within 6–13k
+generated tokens of agent-shaped traffic. Drafter-off **or** `--no-async-scheduling`
+avoids it; `CUDA_LAUNCH_BLOCKING=1` suppresses it — i.e. a cross-stream race, not
+an index bug. **Distinct from vllm#50021** (whose in-kernel bounds do NOT fix this —
+built its head, crashed 2/2) and from vllm#43559/#48375 (the corruption pair).
+
+**Cause:** in `gpu_model_runner.py::synchronize_input_prep`, the next step waits
+`prepare_inputs_event` before touching reused CPU tensors — but that event is
+recorded at the END of input-prep, BEFORE the forward + spec-decode fused-align
+postprocess (`_update_states_after_model_execute`, which records
+`num_accepted_tokens_event`). So under async scheduling the next step's
+`_update_states` block-table mutation raced the still-in-flight postprocess reading
+those buffers → stale state-block index → wild GPU write.
+
+**Fix (2 lines):** also wait `num_accepted_tokens_event` (recorded AFTER the
+postprocess) at the top of `synchronize_input_prep`, closing the write-after-read
+window. It relocates a wait that align-mode already performed later inside
+`_prepare_inputs`, so there is no net GPU stall. Scoped to spec decode (the event
+is `None` otherwise) and async (the whole method is a no-op without
+`prepare_inputs_event`).
+
+**Delivery:** idempotent, anchor-checked `install.sh` → `patch_gdn_mtp_async_spec_order.py`,
+mounted at `/etc/club3090/gdn-async-order/` and invoked from the compose entrypoint
+before serve. Anchor drift → the container **refuses to boot** (exit 1) rather than
+serving unpatched.
+
+**⚠️ Attribution / status:** this is **our own fix, NOT an upstream PR** — no
+upstream commit vendored here. Validated on the reference 2×3090 (v0.27.1) only,
+not community-blessed. Evidence: async-ON survived **3/3 boots** to 33–36k generated
+tokens each (crossing ctx 32k + a compaction) where **5/5 unpatched arms died at
+6–13k**; TPS neutral (79.1 narr / 108.5 code / 1756 prefill ≥ baseline). It does
+**not** address the separate acceptance-collapse bug (vllm#52873 / Bug A).
+
+**Drop trigger:** an upstream fix for this race lands AND the `vllm-stable` pin
+moves past it → remove the mounts + entrypoint call + this dir; `patches.yml` row
+retires. If we file it upstream, update `upstream.status` from `ours` to `open`
+with the PR link.

models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+# club-3090 GDN+MTP async spec-order fix installer — runs in the container
+# entrypoint before serve. Idempotent; refuses boot (exit 1) on anchor drift so a
+# re-pinned image can't silently serve unpatched while the compose claims the fix.
+# Inert unless spec decode is active (the guarded event is None otherwise).
+set -u
+python3 /etc/club3090/gdn-async-order/patch_gdn_mtp_async_spec_order.py

models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""club-3090 fix — order async input-prep AFTER the previous step's spec-decode postprocess.
+
+Bug (our club-3090#1052, upstream vllm#52873 + the #50021 thread): on a
+Qwen3-Next hybrid (GDN linear attention) with the MTP drafter + prefix caching +
+async scheduling (vLLM v0.27.1 default), the engine dies with a CUDA illegal
+memory access in gdn_attn.py (Xid 31 MMU fault, VIRT_WRITE) within 6-13k
+generated tokens of agent-shaped traffic.
+
+Cause: in gpu_model_runner.py, `synchronize_input_prep` waits `prepare_inputs_event`
+before the next step touches reused CPU tensors -- but that event is recorded at
+the END of input-prep, BEFORE the forward + spec-decode fused-align postprocess.
+So under async scheduling the next step's `_update_states` block-table mutation
+raced the still-in-flight postprocess reading those buffers -> stale state-block
+index -> wild GPU write. This waits `num_accepted_tokens_event` (recorded AFTER
+the postprocess) as well, closing the write-after-read window.
+
+Scoped to spec decode (the event is None otherwise) and async scheduling (the
+whole method is a no-op without prepare_inputs_event). An unrecorded event is
+treated complete, so the first step is unaffected. NOT an upstream PR -- our own
+fix; validated 3/3 boots on the reference 2x3090 (async-ON survived 33-36k
+generated incl. a compaction where 5/5 unpatched died 6-13k), TPS-neutral.
+
+Idempotent; anchor-checked; exits non-zero on drift so the entrypoint can refuse
+to boot a silently-unpatched configuration.
+"""
+import io
+import sys
+
+TARGET = "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py"
+MARKER = "club-3090 GDN+MTP async spec-order fix"
+DEF = "    def synchronize_input_prep(self):"
+ANCHOR = "        self.prepare_inputs_event.synchronize()\n"
+INSERT = """        # club-3090 GDN+MTP async spec-order fix (our #1052 / upstream #52873 +
+        # PR#50021 thread). prepare_inputs_event above is recorded BEFORE the
+        # forward + spec-decode fused-align postprocess, so under async scheduling
+        # the next step's _update_states block-table mutation raced the
+        # still-in-flight postprocess -> stale state-block index -> wild GPU write
+        # (CUDA illegal access / Xid 31 VIRT_WRITE). Wait the postprocess event
+        # (recorded AFTER it) to close that write-after-read window. Scoped to spec
+        # decode (event None otherwise); an unrecorded event is complete, so the
+        # first step is unaffected.
+        if self.num_accepted_tokens_event is not None:
+            self.num_accepted_tokens_event.synchronize()
+"""
+
+def main() -> int:
+    try:
+        src = io.open(TARGET, encoding="utf-8").read()
+    except OSError as e:
+        print(f"[gdn-async-order] REFUSE: cannot read target: {e}")
+        return 1
+    if MARKER in src:
+        print("[gdn-async-order] already patched (idempotent no-op)")
+        return 0
+    d = src.find(DEF)
+    if d < 0:
+        print("[gdn-async-order] REFUSE: anchor drift - def synchronize_input_prep not found")
+        return 1
+    i = src.find(ANCHOR, d)
+    if i < 0:
+        print("[gdn-async-order] REFUSE: anchor drift - prepare_inputs_event.synchronize() not found in method")
+        return 1
+    j = i + len(ANCHOR)
+    io.open(TARGET, "w", encoding="utf-8").write(src[:j] + INSERT + src[j:])
+    print("[gdn-async-order] applied: input-prep now waits the prev-step spec-decode postprocess")
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

models/qwen3.8-27b/vllm/compose/dual/autoround-int4/mtp.yml

@@ -152,6 +152,10 @@ services:
       - ../../../../../qwen3.6-27b/vllm/patches/w4a8-int8-act:/etc/club3090/w4a8:ro
       # Lets the MTP drafter attach on this hybrid (mamba/eagle block drop).
       - ../../../../../qwen3.6-27b/vllm/patches/vllm-pr48375-mamba-drop-eagle-block:/etc/club3090/pr48375:ro
+      # club-3090 GDN+MTP async spec-order fix — closes the async wild-write race
+      # (our #1052 / upstream #52873 + #50021 thread). Ours, not an upstream PR;
+      # validated 3/3 boots on the reference 2×3090. Anchor drift → boot refused.
+      - ../../../../../qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order:/etc/club3090/gdn-async-order:ro
       - ../../../../../../scripts/detect_nvlink.sh:/etc/club3090/detect_nvlink.sh:ro
       - ../../../cache/torch_compile:/root/.cache/vllm/torch_compile_cache
       - ../../../cache/triton:/root/.triton/cache
@@ -204,6 +208,7 @@ services:
         fi
         bash /etc/club3090/w4a8/install.sh
         bash /etc/club3090/pr48375/install.sh || exit 1
+        bash /etc/club3090/gdn-async-order/install.sh || exit 1
         echo "[w4a8] VLLM_MARLIN_INPUT_DTYPE=$${VLLM_MARLIN_INPUT_DTYPE:-<unset -> W4A16>} (W4A8=$${W4A8:-1})"
         AR=""
         [ "$${_NVLINK_ENABLED:-0}" = "1" ] || AR="--disable-custom-all-reduce"

scripts/lib/profiles/patches.yml

@@ -360,6 +360,46 @@ patches:
       status: local-vendored
       drop_when: "vllm#48375 merges upstream AND the vllm-stable pin moves past it -> remove the 6 compose mounts + entrypoint calls + the patch dir; retire this row (re-run the #710 probe protocol on the new pin first)."
 
+  - id: vllm-gdn-mtp-async-spec-order
+    # club-3090's OWN fix (NOT an upstream PR) for the GDN+MTP async wild-write
+    # race: our #1052 / upstream #52873 + the #50021 thread. Shared from the
+    # qwen3.6-27b tree like pr48375. Wired on qwen3.8 dual-fast only for now
+    # (the slug it was validated on); broader MTP-compose rollout is a separate
+    # maintainer decision (the fix is engine-wide in gpu_model_runner.py).
+    model: [qwen3.8-27b]
+    files:
+      - models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh
+      - models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/patch_gdn_mtp_async_spec_order.py
+    load_bearing_when:
+      - composes:
+          - vllm/qwen38-27b-dual-fast
+        reason: "Our own 2-line fix for the async wild-write crash (club-3090#1052 / vllm#52873 + the vllm#50021 thread): on a Qwen3-Next hybrid with MTP + prefix caching + async scheduling (vLLM v0.27.1 default), gpu_model_runner.py::synchronize_input_prep waits prepare_inputs_event (recorded BEFORE the forward + spec-decode fused-align postprocess) but not num_accepted_tokens_event (recorded AFTER it), so the next step's _update_states block-table mutation raced the still-in-flight postprocess -> stale state-block index -> wild GPU write (CUDA illegal access / Xid 31 VIRT_WRITE) within 6-13k generated tokens of agent traffic. The fix also waits the postprocess event, closing the write-after-read window; it relocates a wait align-mode already did later, so TPS is neutral. Distinct from vllm#50021 (its in-kernel bounds do NOT fix this - built its head, crashed 2/2) and from vllm#43559/#48375. On-rig validation 2026-08-19 (reference 2x3090, v0.27.1): async-ON survived 3/3 boots to 33-36k generated tokens each (crossing ctx 32k + a compaction) where 5/5 unpatched arms died 6-13k; real delivery path confirmed (switch.sh -> compose mount -> entrypoint install.sh logs '[gdn-async-order] applied'); bench 79.1 narr / 108.5 code / 1756 prefill >= baseline. Does NOT address the separate acceptance-collapse bug (vllm#52873 / Bug A)."
+        evidence: "models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/README.md; club-3090#1052; vllm#52873; docs/UPSTREAM.md"
+    delivery:
+      dockerfile_bake: false
+      entrypoint_invoke: true
+      genesis: false
+    delivery_gaps: []
+    delivery_mechanism: install_script
+    delivery_spec:
+      script: models/qwen3.6-27b/vllm/patches/vllm-gdn-mtp-async-spec-order/install.sh
+      mounted_at: /etc/club3090/gdn-async-order
+      invoke: "bash /etc/club3090/gdn-async-order/install.sh"
+      invoked_before: vllm-serve
+      wired_at: [volumes, entrypoint]
+      note: "Idempotent, anchor-based (def synchronize_input_prep + the prepare_inputs_event.synchronize() line). MARKER is a unique comment string (the executable num_accepted_tokens_event.synchronize() call already exists elsewhere in the file, so it can NOT be the idempotency marker). Failure policy: anchor drift -> boot REFUSED (exit 1 in entrypoint), never a silent half-wire. Wired ONLY on vllm/qwen38-27b-dual-fast (the validated slug); the fix is engine-wide so it is safe on every MTP compose, but broader wiring is a separate maintainer decision. No-op without spec decode (the guarded event is None) and without async scheduling (the whole method is a no-op)."
+    drift_guard:
+      kind: boot
+      check: "Entrypoint runs install.sh before serve; boot log contains '[gdn-async-order] applied' (first boot) or '[gdn-async-order] already patched' (warm layer). Anchor drift -> '[gdn-async-order] REFUSE' + container exit 1 (never serves). Behavioral: async-ON + MTP + prefix-caching survives an agent-shaped soak past ~15k generated tokens with no CUDA illegal-access / Xid 31 (5/5 unpatched died 6-13k)."
+      on_fail: boot-refused
+    status: verified
+    capability: gdn-mtp-async-crash-safety
+    foundational: false
+    upstream:
+      ref: "OURS - not yet filed upstream. Diagnosis + reproducer posted on vllm#50021 (comment) and vllm#52873; the crash is the async half, distinct from #50021's in-kernel bounds. club-3090#1052."
+      status: local-vendored
+      drop_when: "an upstream fix for this async race lands AND the vllm-stable pin moves past it (re-run the drift_guard on the new pin, then delete the mount + entrypoint call + this entry). If we file it upstream, change upstream.status to open-rebased-local with the PR link."
+
   - id: qwen-w4a8-int8-act
     model: [qwen3.6-27b, thinkingcap-27b]
     files: