[Core] Free KV cache GPU memory on engine shutdown

[markmc]

Related to #24885

Addresses the enable_multiprocessing=False TODO in tests/v1/shutdown/test_delete.py::test_llm_delete

The trickiest part is the single-process case ("inproc" engine and "uniproc" executor") where we can't rely on shutting down child processes to release GPU memory. To address that, we:

1. Call engine_core.shutdown() from LLMEngine.__del__()
2. Free KV cache GPU memory in GPUWorker.shutdown()

Other changes include:

1. Avoid pytest timeout when waiting for GPU cleanup - use the wait_for_gpu_memory_to_clear() timeout parameter to get a nice Memory of devices not free error
2. Print memory usage at start of shutdown tests
3. assert_mp_fork_context() added to avoid We must use the 'spawn' multiprocessing start method.... silently breaking the evil_forward() monkey patch
4. Move the TP=1 enable_multiprocessing=False tests into a forked subprocess, to avoid initializing CUDA in the parent process
5. Leave test_forward_error::test_llm_model_error_without_multiprocessing disabled in the enable_multiprocessing=False case - it is still broken

[gemini-code-assist]

Code Review

This pull request effectively addresses the GPU memory leak on engine shutdown, especially for the single-process case, by introducing an explicit cleanup path. The refactoring in tests/utils.py to extract check_gpu_memory_usage is a nice improvement for test clarity.

I have two main points of feedback regarding the robustness of the shutdown mechanism:

1. The reliance on __del__ for cleanup in LLMEngine can be unreliable in complex applications with reference cycles.
2. The shutdown method in GPUWorker is now less defensive, which could lead to errors during shutdown if a worker failed to initialize completely.

Details are in the line comments.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @markmc.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[markmc]

This is turning out to be a bit of a saga:

1. The test_forward_error test was failing because the evil_forward monkey patch wasn't working, because of this: We must use the 'spawn' multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. - I've added assert_mp_fork_context() so this doesn't silently break the monkey patch in future
2. The reason this was happening is that the TP=1 enable_multiprocessing=False test was initializing CUDA in the parent process - I moved the enable_multiprocessing=False into a forked subprocess
3. This wasn't working for me locally because of [Test] Fix pytest termination with @create_new_process_for_each_test("fork") #29130
4. test_forward_error::test_llm_model_error_without_multiprocessing still isn't working - something is holding onto references - for now, I've left it disabled (as it currently is)

[github-actions]

This pull request has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this pull request should remain open. Thank you!