Conversation
… instead of baking it into the graph qwen4_exp_ple_pinned_gather took the PLE table address as a Python int, so Inductor wrote the literal address into the compiled graph and the AOT artifact. A cold boot works; the first warm boot with the compile cache on runs the artifact in a new process, where the pinned host buffer lives at a different address, and the stage holding the PLE table dies in profile_run with an illegal memory access (Qwen3.8-Flash-Next, TP2 PP2). The op now takes the table's layer name and a host/device selector, resolves the module through the forward context and reads the pointer at run time, the way qwen4_exp_compute_ple_ngram_ids already resolves its layer. The table registers itself under its prefix. Only the string and the bool reach the graph. Tests construct the table under a VllmConfig context and expose it to the op through the forward context, mirroring the existing ngram-id test. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Peuqui <peuqui@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
qwen4_exp_ple_pinned_gathertook the address of the PLE table as a Pythonint(
weight_ptr), the device table'sdata_ptr()or the UVA view of the pinned hosthalf. Inside a compiled region that integer is a constant, and Inductor writes it
verbatim into the generated code. The AOT artifact of the first pipeline stage of
Qwen3.8-Flash-Next therefore contains a literal host address:
A cold boot works, because the constant comes from the same process. The first warm
boot with the compile cache on (
Directly load AOT compilation) runs the artifact in anew process, whose pinned buffer sits at a different address. Both ranks of the stage
that holds the PLE table die in
profile_runwithCUDA error: an illegal memory access was encountered; the other stage, which has no PLE layer, loads and runs. WithVLLM_DISABLE_COMPILE_CACHE=1the same command boots. The forced compile-cache opt-outon the SM70 graph hid this since #403 introduced the pointer argument; #528 added the
device-table pointer the same way. #621 removes that opt-out, so this fix should land
first or together with it.
What changes
qwen4_exp_ple_pinned_gather(input_ids, output, weight_scale, layer_name: str, use_host_table: bool, embedding_dim: int): the op resolves the table by namethrough
get_forward_context().no_compile_layersand reads the pointer at run time,the way
qwen4_exp_compute_ple_ngram_idsalready resolves its layer. Only the stringand the bool reach the graph; both are stable across processes.
Qwen4ExpPinnedHostEmbedding.__init__registers the table under its prefix in theconfig's
static_forward_context(duplicate prefixes raise, as for the PLE layer).embedding_lookuppasses(layer_name, use_host_table)at its three call sites; thecached pointers stay where they are and keep being refreshed by
get_accelerator_weight/materialize_tables, so CUDA-graph replay after anin-place reload is unchanged (existing test).
tests/models/qwen4_exp/test_ple.py:_pinned_layerconstructs under aVllmConfigcontext; the gather test exposes the table to the op through amonkeypatched forward context, mirroring the existing ngram-id test.
Reproduction and evidence
Rig: 2x Quadro RTX 8000 (stage 0, holds the PLE table, 6 GiB of it pinned in host
memory via
VLLM_QWEN4EXP_PLE_HOST_GIB=6) + 2x Tesla V100 (stage 1), TP2 PP2,Qwen3.8-Flash-Next-180B-A4B-NVFP4 with MTP k=4, compile cache on, torch 2.10.0+cu128.
Directly load AOT→profile_run→_short_conv_fallback/qwen4_exp_ple_pinned_gather→ illegal memory access (CUDA_LAUNCH_BLOCKING=1points at the gather's Triton launch)e948a82ead51948c)e948a82ead51948c)Same command with
VLLM_DISABLE_COMPILE_CACHE=1: boots and answers before the change(350 s), which isolates the artifact reload as the trigger.
Test Plan
pre-commit run --files vllm/models/qwen4_exp/nvidia/ple_layer.py tests/models/qwen4_exp/test_ple.pyandpre-commit run mypy-3.10 --hook-stage manual --files <same>.tests/models/qwen4_exp/test_ple.pyin full on a V100 (the gather test needs pinned memory and CUDA graphs).Test Result
tests/models/qwen4_exp/test_ple.py: 60 passed (one V100 visible), on this branch and on our fork.Not a duplicate
gh pr list --state open --search "PLE pointer OR pinned_gather OR data_ptr"andgh issue list --state all --search "illegal memory access PLE OR AOT compile pointer"show nothing on this; #621 (ours) removes the opt-out that hid the bug and does not
touch the PLE layer.
AI assistance (Claude) was used to find the cause and prepare the change; I reviewed
every line and ran the boots and tests on the hardware named above.