[Bugfix] Fix XPU/ROCm compatibility in spawn_new_process_for_each_test

[dzhengAP]

Purpose

Follow-up to #41423.

mp.set_start_method("spawn") was inadvertently removed when the decorator was rewritten to use subprocess.run. While the decorator itself doesn't need it (since subprocess.run launches a fresh interpreter regardless), other parts of the test session — specifically XPU/ROCm engine workers that use mp.Process directly — depend on this global mp start method being set to spawn.

Without it, those workers default back to fork on Linux, causing:
RuntimeError: Cannot re-initialize XPU in forked subprocess

This restore has no effect on the failure-propagation fix from #41423.

cc @jikunshang @ProExpertProg @AndreasKaratzas

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dzhengAP.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request refactors the spawn_new_process_for_each_test decorator to use subprocess.run combined with cloudpickle for serializing test functions and arguments. This change ensures that exceptions occurring in the child process are captured and propagated back to the parent, preventing silent test failures. The PR also adds comprehensive unit tests for the decorator and updates existing tests in test_cudagraph_dispatch.py and test_custom_online.py to align with the new implementation. Feedback suggests that the child_script should explicitly set the multiprocessing start method to 'spawn' to avoid potential RuntimeError issues in environments where 'fork' is the default.

[luobosibing2]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1eba41a30a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chaojun-zhang]

Please check @create_new_process_for_each_test() uses spawn on XPU, which cannot pickle the server fixture (it contains a multiprocessing.Process with an unpicklable AuthenticationString).

FAILED tests/v1/logits_processors/test_custom_online.py::test_custom_logitsprocs[server1-facebook/opt-125m] - TypeError: Pickling an AuthenticationString object is disallowed for security reasons

[dzhengAP]

Please check @create_new_process_for_each_test() uses spawn on XPU, which cannot pickle the server fixture (it contains a multiprocessing.Process with an unpicklable AuthenticationString).

FAILED tests/v1/logits_processors/test_custom_online.py::test_custom_logitsprocs[server1-facebook/opt-125m] - TypeError: Pickling an AuthenticationString object is disallowed for security reasons

Yes, exactly. The RemoteOpenAIServerCustom fixture uses multiprocessing.Process internally which holds an AuthenticationString that can't be pickled across a spawn boundary. I would think of the fix to not wrap that specific test with spawn_new_process_for_each_test. Will implement this tmr. @chaojun-zhang

[AndreasKaratzas]

@dzhengAP Please fix pre-commit so we can begin evaluations of this PR.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dzhengAP.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[dzhengAP]

@dzhengAP Please fix pre-commit so we can begin evaluations of this PR.

@AndreasKaratzas Hi Andreas are you able to add a ready label?

[AndreasKaratzas]

@dzhengAP pre-commit is still failing.

[dzhengAP]

Why is GH showing 0 files changed? Has this been fixed in a different PR?
@ProExpertProg Pevious push accidentally resolved the conflicts in a way that made the branch identical to main. I've now rebased cleanly on top of the latest main — the 2 new commits are now showing correctly. The files changed should be visible now.

[dzhengAP]

@jikunshang can you help add the ready label to check CI, since focus now mainly on Intel CI tests. cc: @ProExpertProg @AndreasKaratzas

[AndreasKaratzas]

@jikunshang can you help add the ready label to check CI, since focus now mainly on Intel CI tests. cc: @ProExpertProg @AndreasKaratzas

So this does not fix rocm? So we should probably move on with reverting the other PR then.

[dzhengAP]

@jikunshang can you help add the ready label to check CI, since focus now mainly on Intel CI tests. cc: @ProExpertProg @AndreasKaratzas

So this does not fix rocm? So we should probably move on with reverting the other PR then.

@AndreasKaratzas what new ROCM failure did you see? can you help paste here? AMD CI tests were passed after the fix in this PR. — the mp.set_start_method("spawn") guard applies to both current_platform.is_rocm() or current_platform.is_xpu(). The VLLM_WORKER_MULTIPROC_METHOD=spawn is also set in the child env for both platforms. Please make sure share the specific ROCm failure you're seeing. I want to make sure we address it before any revert decision.

[dzhengAP]

Hi @ProExpertProg @jikunshang I went through the CI log and put a summary below, which seems unrelated to #41895

1. test_custom_logitsprocs — v1/logits_processors

Error: RuntimeError: Cannot re-initialize XPU in forked subprocess
Root cause: The test intentionally sets VLLM_WORKER_MULTIPROC_METHOD=fork so the entry_points monkey-patch is visible to workers. This is fundamentally incompatible with XPU which requires spawn.
Status: Pre-existing issue

2. test_qwen35_text_lora — lora/

Error: RuntimeError: Engine core initialization failed (XPU fork error in EngineCoreProc)
Root cause: The vLLM engine core is forking despite VLLM_WORKER_MULTIPROC_METHOD=spawn being set. XPU executor path is not respecting the spawn context.
Status: Pre-existing issue

---

Fix Plan

test_custom_logitsprocs may need to replace fork-based entrypoint injection with spawn-compatible approach (config file or env var instead of monkey-patching across fork boundary)
test_qwen35_text_lora need to Investigate why EngineCoreProc forks on XPU despite spawn env var; explicitly use mp.get_context("spawn") in XPU executor path

---

I can submit followup PRs for them or fix the bugs here, but it may make current PR too heavy, what do you guys think? cc @jikunshang @ProExpertProg

[jikunshang]

for intel/ci,

1. test_custom_logitsprocs — v1/logits_processors. I didn't notice it force use fork before. I will investigate more how to proper handle this case on XPU. maybe I should just skip it due to such limitation.
2. I agree test_qwen35_text_lora — lora/ is not related. we can ignore it. test_qwen35_text_lora is case with create_new_process_for_each_test decorator. it still use fork instead of spawn.

What confuse me is:
#41423 changed spawn_new_process_for_each_test method behavior, no longer use spawn, see comments #41423 (comment), this may affect some other ROCm/XPU test case with create_new_process_for_each_test decorator.

[dzhengAP]

What confuse me is:
#41423 changed spawn_new_process_for_each_test method behavior, no longer use spawn, see comments #41423 (comment), this may affect some other ROCm/XPU test case with create_new_process_for_each_test decorator.

Hi @jikunshang. The new spawn_new_process_for_each_test uses subprocess.run([sys.executable, ...]) which launches a completely fresh interpreter — this is actually more isolated than mp.spawn since there's no shared state at all. The child process sets VLLM_WORKER_MULTIPROC_METHOD=spawn and calls mp.set_start_method('spawn') before running the test, so any mp.Process calls inside the test still use spawn semantics. The behavior change is in how the subprocess is launched, not in what the subprocess does.

[dzhengAP]

@ProExpertProg @jikunshang @chaojun-zhang @AndreasKaratzas
After checking the Intel CI log for test_qwen35_text_lora — the actual error is AssertionError: lora isn't supported on XPU, not a fork/spawn issue. The test is running via subprocess.run correctly (you can see python3 -c "import sys, cloudpickle, traceback..." in the output). That means, the new decorator is working correctly and properly surfacing this pre-existing failure. This test should be skipped on XPU with pytest.mark.skip or a platform check. The fork/spawn concern doesn't apply here.

Aligned with @jikunshang

[dzhengAP]

As for the first one: Same as before — 1 failed, 33 passed — only test_custom_logitsprocs failing with the XPU fork error (Cannot re-initialize XPU in forked subprocess). This is because the test explicitly sets VLLM_WORKER_MULTIPROC_METHOD=fork internally.

In summay, both are pre-existing XPU limitations, not caused by this PR. The decorator is working correctly in both cases — it's properly surfacing failures that were previously hidden.

@ProExpertProg @jikunshang @chaojun-zhang @AndreasKaratzas

[jikunshang]

Intel CI doens't gate PR merge currently. thanks @dzhengAP for great work.
Intel folk will temperary disable these failed case in anohter PR to make ci happy and provide solid fix for these.

[dzhengAP]

Hi @jikunshang and @AndreasKaratzas, I am trying to propose a fix for in #42040
This PR fixes the LOGITPROC_SOURCE_ENTRYPOINT fork compatibility issue on XPU/ROCm that was previously failing Intel and AMD CI, as we discussed here (failure 1). The monkey-patch approach required VLLM_WORKER_MULTIPROC_METHOD=fork which is incompatible with XPU/ROCm. The fix registers a real dist-info package on disk via PYTHONPATH so any spawned subprocess can discover the entrypoint without needing fork. Please give this a look when you get a chance!
cc: @ProExpertProg