Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions projects/rocprofiler-compute/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ Full documentation for ROCm Compute Profiler is available at [https://rocm.docs.
modes on partition-capable accelerators, noting that analysis derives logical
XCD, L2 channel, and HBM channel counts from them.

* Added a guide for profiling vLLM workloads and its caveats.

### Changed

* Renamed the PC sampling analysis output: `pc_sampling.csv` is now `pc_sampling_summary.csv`, and the `compute_pc_sampling_view` view is now `compute_pc_sampling_summary_view`.
Expand Down
39 changes: 39 additions & 0 deletions projects/rocprofiler-compute/docs/how-to/profile/mode.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1185,6 +1185,45 @@ The output is identical to enabling each framework's trace flag individually.
Captured kernels are attributed in the ``Backend`` column and analyzed with the
corresponding per-framework operator options (see :doc:`../analyze/cli`).

.. _profile-vllm-workloads:

Profile vLLM workloads
======================

vLLM V1 runs GPU kernels in a worker process separate from the vLLM entry
process. ROCm Compute Profiler profiles that worker process and records its
kernel dispatches, but vLLM terminates the worker with a signal on shutdown,
and counter data is only written when a process exits normally. The profiling
run therefore reports success while leaving no GPU kernel dispatch or
performance counter data.

To profile vLLM V1 workloads on a single GPU, set
``VLLM_ENABLE_V1_MULTIPROCESSING=0`` before launching ROCm Compute Profiler.
The vLLM workload then inherits the setting and runs the model in the profiled
process, which exits normally and writes its counter data. For example:

.. code-block:: shell-session

$ VLLM_ENABLE_V1_MULTIPROCESSING=0 \
rocprof-compute profile \
--iteration-multiplexing \
--no-roof \
--name vllm-offline -- \
python offline_inference.py

.. important::

This workaround applies to single-GPU runs. When the model is split across
several GPUs, vLLM starts a separate worker process for each GPU, and a
profiling run still completes without performance counter data.

.. note::

Disabling vLLM V1 multiprocessing does not change the model or the kernels
it runs, so the collected counter data remains representative. It does
affect end-to-end throughput, because work that normally overlaps across two
processes is serialized into one.

.. _iteration-multiplexing:

Iteration multiplexing
Expand Down
9 changes: 8 additions & 1 deletion projects/rocprofiler-compute/docs/reference/faq.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. meta::
:description: ROCm Compute Profiler FAQ and troubleshooting
:keywords: ROCm Compute Profiler, FAQ, troubleshooting, ROCm, profiler, tool, Instinct,
accelerator, AMD, SSH, error, version, workaround, help
accelerator, AMD, SSH, error, version, workaround, help, vLLM

***
FAQ
Expand Down Expand Up @@ -108,3 +108,10 @@ launched on separate HIP streams on the same GPU will run one after
another during profiling. Kernel duration and throughput metrics reflect
this serialized execution rather than the concurrent behavior that may
occur during normal execution.

Why does profiling a vLLM workload produce empty performance counter data?
==========================================================================

vLLM V1 runs GPU kernels in a worker process that it terminates with a signal
on shutdown, and counter data is only written when a process exits normally.
See :ref:`profile-vllm-workloads` for the workaround and where it applies.
Loading