diff --git a/projects/rocprofiler-compute/CHANGELOG.md b/projects/rocprofiler-compute/CHANGELOG.md index b8ab0e03976d..e4137e8daed6 100644 --- a/projects/rocprofiler-compute/CHANGELOG.md +++ b/projects/rocprofiler-compute/CHANGELOG.md @@ -24,6 +24,8 @@ Full documentation for ROCm Compute Profiler is available at [https://rocm.docs. modes on partition-capable accelerators, noting that analysis derives logical XCD, L2 channel, and HBM channel counts from them. +* Added a guide for profiling vLLM workloads and its caveats. + ### Changed * Renamed the PC sampling analysis output: `pc_sampling.csv` is now `pc_sampling_summary.csv`, and the `compute_pc_sampling_view` view is now `compute_pc_sampling_summary_view`. diff --git a/projects/rocprofiler-compute/docs/how-to/profile/mode.rst b/projects/rocprofiler-compute/docs/how-to/profile/mode.rst index 5dc3885521a3..866d9115ddfa 100644 --- a/projects/rocprofiler-compute/docs/how-to/profile/mode.rst +++ b/projects/rocprofiler-compute/docs/how-to/profile/mode.rst @@ -1185,6 +1185,45 @@ The output is identical to enabling each framework's trace flag individually. Captured kernels are attributed in the ``Backend`` column and analyzed with the corresponding per-framework operator options (see :doc:`../analyze/cli`). +.. _profile-vllm-workloads: + +Profile vLLM workloads +====================== + +vLLM V1 runs GPU kernels in a worker process separate from the vLLM entry +process. ROCm Compute Profiler profiles that worker process and records its +kernel dispatches, but vLLM terminates the worker with a signal on shutdown, +and counter data is only written when a process exits normally. The profiling +run therefore reports success while leaving no GPU kernel dispatch or +performance counter data. + +To profile vLLM V1 workloads on a single GPU, set +``VLLM_ENABLE_V1_MULTIPROCESSING=0`` before launching ROCm Compute Profiler. +The vLLM workload then inherits the setting and runs the model in the profiled +process, which exits normally and writes its counter data. For example: + +.. code-block:: shell-session + + $ VLLM_ENABLE_V1_MULTIPROCESSING=0 \ + rocprof-compute profile \ + --iteration-multiplexing \ + --no-roof \ + --name vllm-offline -- \ + python offline_inference.py + +.. important:: + + This workaround applies to single-GPU runs. When the model is split across + several GPUs, vLLM starts a separate worker process for each GPU, and a + profiling run still completes without performance counter data. + +.. note:: + + Disabling vLLM V1 multiprocessing does not change the model or the kernels + it runs, so the collected counter data remains representative. It does + affect end-to-end throughput, because work that normally overlaps across two + processes is serialized into one. + .. _iteration-multiplexing: Iteration multiplexing diff --git a/projects/rocprofiler-compute/docs/reference/faq.rst b/projects/rocprofiler-compute/docs/reference/faq.rst index caa69f9cd581..e32ec9b75d0c 100644 --- a/projects/rocprofiler-compute/docs/reference/faq.rst +++ b/projects/rocprofiler-compute/docs/reference/faq.rst @@ -1,7 +1,7 @@ .. meta:: :description: ROCm Compute Profiler FAQ and troubleshooting :keywords: ROCm Compute Profiler, FAQ, troubleshooting, ROCm, profiler, tool, Instinct, - accelerator, AMD, SSH, error, version, workaround, help + accelerator, AMD, SSH, error, version, workaround, help, vLLM *** FAQ @@ -108,3 +108,10 @@ launched on separate HIP streams on the same GPU will run one after another during profiling. Kernel duration and throughput metrics reflect this serialized execution rather than the concurrent behavior that may occur during normal execution. + +Why does profiling a vLLM workload produce empty performance counter data? +========================================================================== + +vLLM V1 runs GPU kernels in a worker process that it terminates with a signal +on shutdown, and counter data is only written when a process exits normally. +See :ref:`profile-vllm-workloads` for the workaround and where it applies.