[Bugfix] Make GPU sync checks safe under torch.compile

[khluu]

Summary

gpu_sync_debug's patched tensor-copy wrappers currently read a ContextVar
before checking whether Dynamo is tracing. A full-graph compile therefore fails
with an unsupported ContextVar.get() instead of bypassing sync checks as
intended.

This change:

• checks torch.compiler.is_compiling() before accessing either ContextVar;
• routes the original C++ Tensor.to descriptor through a weak-referenceable
  Python callable registered with torch.compiler.allow_in_graph, so Dynamo
  treats the call as opaque while AOT/Inductor can still lower it;
• adds a full-graph torch.compile regression with device and dtype derived
  dynamically from another tensor;
• runs the two new Voxtral CUDA-graph parameter cases in explicitly spawned
  processes so neither prior pytest CUDA state nor the other graph mode can
  contaminate them.

CI incident and test-selection evidence

Daily main build #88920 first exposed this in
test_voxtral_realtime_cudagraph[FULL_DECODE_ONLY] in the H200 Multimodal
Models (Extended Generation 1) job. PR #51167 added the compiled Voxtral path,
but that exact job was blocked in both its PR build #88857 and post-merge main
build #88899, so the failure was not exercised before the daily run.

I searched open vLLM issues and PRs for the exact ContextVar.get / Voxtral
full-graph failure and did not find an existing fix.

Validation

• pre-commit run --files vllm/utils/gpu_sync_debug.py tests/utils_/test_gpu_sync_debug.py tests/models/multimodal/generation/test_voxtral_realtime.py — passed all applicable hooks
• git diff --check — passed
• Local pytest was not run because this regression requires CUDA and this development host has no GPU.
• Exact H200 build #88941 passed Async/Inputs/Utils/Worker but exposed the
  saved C++ Tensor.to descriptor as Dynamo-unsupported in the Voxtral
  full-graph test.
• Replacement #88962 proved that registering the method descriptor directly
  is invalid at import time because allow_in_graph stores a weak reference.
• Replacement #88969 passed Async/Inputs/Utils/Worker and the original Voxtral
  FULL_DECODE_ONLY compile/CUDA-graph surface at literal head c2fbacb1.
  The following FULL_AND_PIECEWISE parameter then hit the normal startup
  guard with only 25.73/32.5 GiB free after the prior in-process compiled
  runner retained about 6.5 GiB; later failures were cascades from that state.
• Replacement #89000 passed Async/Inputs/Utils/Worker, but the default CUDA
  isolation helper chose fork after the parent pytest process had initialized
  CUDA. Both Voxtral cases therefore failed before engine compilation with
  Cannot re-initialize CUDA in forked subprocess.
• #89010 used an invalid copied full SHA and failed in bootstrap; it contains
  no test evidence.
• #89011 also passed Async/Inputs/Utils/Worker, then reproduced the same two
  pre-compilation CUDA-fork failures. Its literal head still contained the
  helper's default mode because the intended one-line spawn edit had not
  been staged into the prior commit.
• Current exact replacement #89026 runs only the same two H200 lanes at
  literal signed head f7f21d97caecf810db069402ed02c4624311af32, which
  explicitly passes method="spawn" to the isolation helper.

No model evaluation is required because the production change only alters
debug-wrapper control flow while Dynamo is tracing. The exact affected
generation test is included in targeted CI.

AI assistance disclosure

This change was prepared with AI assistance. Human review is required before
this draft is marked ready or merged.

[khluu]

Exact isolated validation is running in Buildkite #88941 at literal signed head b9077f56211b9255fef5e6f63a6e546bcd2dd6f5. It requests only async-engine-inputs-utils-worker (the focused CUDA regression owner) and multi-modal-models-extended-generation-1 (the escaped Voxtral end-to-end lane). Human review remains required before readiness.

[khluu]

Exact validation update:

• Buildkite #88941 passed H200 Async/Inputs/Utils/Worker, but the direct Multimodal Extended Generation 1 gate found a second full-graph integration issue: the checker now short-circuited its ContextVars, then Dynamo rejected the saved C++ TensorBase.to descriptor when the real Voxtral path supplied device and dtype from another tensor. The later memory/init failures in that job were process fallout, not independent regressions.
• I updated the same draft so the saved Tensor.to descriptor is explicitly admitted through torch.compiler.allow_in_graph, which is the public Dynamo escape hatch for this C/C++ callable. The regression now matches the real dynamic device=like.device, dtype=like.dtype call and uses the default compiler backend rather than backend="eager".
• Current signed head: 9d6c9de919e9b014f6ad580ac5441300f72fb297.
• Local validation: applicable pre-commit hooks passed (ruff, mypy, SPDX, configuration checks, and the rest of the selected suite); git diff --check passed. This host has no CUDA/PyTorch runtime, so the exact GPU test is CI-only.
• Replacement exact-head Buildkite #88962 requests the same two H200 lanes and is pending.

The PR remains draft and still requires @khluu to review and defend every line before it is made ready.

[khluu]

Replacement exact #88962 caught a genuine flaw in the previous revision before tests collected: torch.compiler.allow_in_graph(_torch_to) attempts to weak-reference the saved C++ method_descriptor, so importing gpu_sync_debug.py failed with TypeError: cannot create weak reference to 'method_descriptor' object in both selected H200 lanes.

I corrected that by registering a weak-referenceable Python wrapper instead. The wrapper calls the original descriptor, preserving the no-recursion property; Dynamo treats the wrapper as opaque and AOT/Inductor can trace/lower the underlying call. I also rebased the draft onto current main and reran all applicable pre-commit hooks plus git diff --check successfully.

Current signed head: c2fbacb13756e9b54a4ebc3902af02fe6a794c86
Replacement exact gates: https://buildkite.com/vllm/ci/builds/88969

#88969 is pinned to that literal head and only the H200 Async/Inputs/Utils/Worker and Multimodal Extended Generation 1 lanes. It is currently building the image; this draft remains held until both gates pass.

[khluu]

Exact H200 build #88969 at c2fbacb13756e9b54a4ebc3902af02fe6a794c86 validated the source fix and exposed one independent test-isolation problem:

• H200 Async Engine / Inputs / Utils / Worker: passed.
• Voxtral FULL_DECODE_ONLY: passed through Dynamo, Inductor, CUDA graph capture, and inference. This is the original failure surface (the previous head failed while Dynamo traced the saved C++ Tensor.to descriptor).
• The following FULL_AND_PIECEWISE parameter started with only 25.73/32.5 GiB free because the prior in-process compiled runner retained about 6.5 GiB, then failed the normal 0.9 startup guard; two later tests cascaded from that memory state.

Commit bee6beb40f now gives each newly added Voxtral CUDA-graph parameter a fresh process, matching the isolation pattern used by other compile-heavy tests. All applicable pre-commit hooks and git diff --check pass locally. I am launching the same two exact lanes at the literal new head; the PR remains draft for human review.

[khluu]

Current replacement exact H200 build #88988 uses literal signed head 544bf9df374ebe65a0f59defa76caf7a42400fba and only async-engine-inputs-utils-worker plus multi-modal-models-extended-generation-1. Superseded #88987 was canceled before tests when I amended the test-isolation commit to add the AI co-author trailer. (This API-triggered replacement does not publish a PR status, so the build link is the authoritative gate.)

[khluu]

Exact Buildkite #89000 update:

• H200 Async/Inputs/Utils/Worker passed at 544bf9df374e.
• Multimodal Extended Generation 1 reached both new Voxtral CUDA-graph parameters, but the default isolation helper uses fork on CUDA. Earlier tests had already initialized CUDA in the pytest parent, so both child processes failed immediately with Cannot re-initialize CUDA in forked subprocess.
• This is an isolation-mechanism failure, not a recurrence of the original Dynamo graph break: no Voxtral engine reached compilation in those forked children.

I changed the decorator to the helper's explicit spawn mode and force-pushed signed/DCO head a6b806ade02639b2412f1551898f766ff92dc5b3. All applicable local pre-commit hooks and diff checks pass. Replacement exact build #89010 uses the literal new head and only the same Async/Utils plus Multimodal Extended Generation 1 gates.

AI assistance was used; human line review remains required before readiness.

[khluu]

Correction: I copied an invalid full suffix after the displayed short SHA into #89010, so bootstrap correctly rejected a6b806ade026... as not a repository ref. That build contains no test evidence.

The verified local, fork, GitHub pull-ref, and PR head are all exactly a6b806ade04e45c10925677a14d5127deb670d38. Replacement #89011 uses that literal head and the same two H200 gates. I also corrected the PR body below; #89010 must not be cited as validation.

[khluu]

Exact #89011 closed red: Async/Inputs/Utils/Worker passed, while both new Voxtral parameters failed before engine compilation with Cannot re-initialize CUDA in forked subprocess (30 passed / 2 failed in the lane). Inspection found that its literal head still had @create_new_process_for_each_test()—the intended explicit-spawn edit was present locally but had not been staged into the prior commit. I corrected the PR body so #89011 is not misrepresented as spawn evidence.

Current signed/DCO head f7f21d97caecf810db069402ed02c4624311af32 now contains @create_new_process_for_each_test(method="spawn"). All applicable pre-commit hooks for the changed test file and git diff --check pass. Replacement exact H200 build #89026 is pinned to that literal head and only the same Async/Utils plus Multimodal Extended Generation 1 gates.

The draft remains held for terminal exact evidence and human line review. AI assistance was used.

[khluu]

Terminal exact validation at current head f7f21d97caecf810db069402ed02c4624311af32:

• Buildkite #89026 passed all four selected jobs
• H200 Async Engine / Inputs / Utils / Worker passed
• H200 Multimodal Models (Extended Generation 1) passed with 32 passed, 26 skipped, 64 deselected, then mapping 4 passed / 1 skipped
• both decisive fresh-process cases passed:
  • test_voxtral_realtime_cudagraph[FULL_DECODE_ONLY]
  • test_voxtral_realtime_cudagraph[FULL_AND_PIECEWISE]

This confirms explicit spawn isolation avoids the inherited-CUDA failure and prevents the first graph-mode run from leaking memory into the second. The PR remains draft for human line review.

AI disclosure: This validation report was prepared with AI assistance.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[njhill]

/ci test