[Bugfix][ROCm][Build] fix profiler hang due to queue interposition bug by simondanielsson · Pull Request #54171 · vllm-project/vllm

simondanielsson · 2026-08-28T07:30:29Z

Purpose

Applies two bug fixes that are not part of the rocprofiler-sdk commit we're on (as they were not part of the therock-7.14 release):

Alternative: We could also bump to rocprofiler-sdk 1.3.5 but that will be a larger change that ver is on the therock-10.0 line.

Test Plan

Build
Ensure profiler no longer hangs. Reproducer in [Bug]: torch profiler hangs on ROCm after rocprofiler-sdk 1.3.2 bump #54087

Test Result

1. Build succeeds e2e. Here's the part where the patch is applied:

#23 [build_rocprofiler_sdk 3/4] RUN cd /src     && QI_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*'     && curl -fsSL -o /tmp/qi-7796.patch "https://github.com/ROCm/rocm-systems/commit/fae7fd57ffb8a76e811d6410b8a0cee1e94d710a.patch"     && curl -fsSL -o /tmp/qi-7924.patch "https://github.com/ROCm/rocm-systems/commit/f8db22adbe17686a9b05ae03367dd1ec73011653.patch"     && git apply -p1 --include="$QI_PATHS" --verbose /tmp/qi-7796.patch     && git apply -p1 --include="$QI_PATHS" --verbose /tmp/qi-7924.patch
#23 0.870 Checking patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp...
#23 0.870 Hunk #1 succeeded at 234 (offset 11 lines).
#23 0.870 Hunk #2 succeeded at 741 (offset 11 lines).
#23 0.870 Hunk #3 succeeded at 791 (offset 31 lines).
#23 0.870 Applied patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp cleanly.
#23 0.872 Skipped patch '.github/workflows/rocprofiler-sdk-code_coverage.yml'.
#23 0.872 Skipped patch '.github/workflows/rocprofiler-sdk-continuous_integration.yml'.
#23 0.872 Skipped patch 'projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/queue_interposition.cpp'.
#23 0.872 Checking patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp...
#23 0.872 Hunk #1 succeeded at 217 (offset 10 lines).
#23 0.872 Hunk #2 succeeded at 308 (offset 11 lines).
#23 0.872 Hunk #3 succeeded at 348 (offset 11 lines).
#23 0.872 Hunk #4 succeeded at 462 (offset 11 lines).
#23 0.872 Hunk #5 succeeded at 555 (offset 11 lines).
#23 0.872 Hunk #6 succeeded at 711 (offset 11 lines).
#23 0.872 Hunk #7 succeeded at 730 (offset 11 lines).
#23 0.872 Hunk #8 succeeded at 768 (offset 11 lines).
#23 0.872 Hunk #9 succeeded at 862 (offset 31 lines).
#23 0.872 Hunk #10 succeeded at 900 (offset 31 lines).
#23 0.872 Checking patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.hpp...
#23 0.872 Hunk #2 succeeded at 163 (offset 4 lines).
#23 0.872 Applied patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp cleanly.
#23 0.872 Applied patch projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.hpp cleanly.
#23 DONE 0.9s

#24 [build_rocprofiler_sdk 4/4] RUN cd /src/projects/rocprofiler-sdk     && if [ "$USE_SCCACHE" = "1" ]; then            export CMAKE_C_COMPILER_LAUNCHER=sccache            && export CMAKE_CXX_COMPILER_LAUNCHER=sccache;        fi     && cmake -B build -G Ninja         -DCMAKE_BUILD_TYPE=Release         -DCMAKE_PREFIX_PATH=/opt/rocm         -DCMAKE_INSTALL_PREFIX=/opt/rocm         -DROCPROFILER_BUILD_TESTS=OFF         -DROCPROFILER_BUILD_SAMPLES=OFF         -DROCPROFILER_BUILD_DOCS=OFF     && cmake --build build --parallel "$(nproc)"     && cmake --install build --prefix /staging --strip
#24 0.165 -- The C compiler identification is GNU 11.4.0
#24 0.196 -- The CXX compiler identification is GNU 11.4.0
...
#24 118.1 -- Installing: /staging/share/modulefiles/rocprofiler-sdk-rocattach/1.3.2
#24 118.1 -- Installing: /staging/lib/cmake/rocprofiler-sdk-rocattach/rocprofiler-sdk-rocattach-config.cmake
#24 118.1 -- Installing: /staging/lib/cmake/rocprofiler-sdk-rocattach/rocprofiler-sdk-rocattach-config-version.cmake
#24 118.1 -- Installing: /staging/share/doc/rocprofiler-sdk-rocattach/LICENSE.md
#24 DONE 118.2s
...

2. Profiling no longer hangs

(built base and final image as vllm/vllm-openai-rocm:qi-fix):

docker run --name vllm --rm --entrypoint "" \
  --device /dev/kfd --device /dev/dri -v /sys:/sys \
  -v ~/.cache/huggingface/hub:/root/.cache/huggingface/hub \
  --group-add video --group-add render --network host --ipc host \
  --privileged --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  --shm-size 256G --ulimit memlock=-1:-1 --ulimit stack=67108864:67108864 -e HIP_VISIBLE_DEVICES=4 \
  vllm/vllm-openai-rocm:qi-fix \
  vllm serve RedHatAI/gemma-4-31B-it-FP8-block \
    --load_format dummy --tensor_parallel_size 1 \
    --port 8200 --profiler-config.profiler torch \
    --profiler-config.torch_profiler_dir /tmp/traces/1318414/ \
    --profiler-config.torch_profiler_with_stack false

(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]        █     █     █▄   ▄█
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.28.1rc1.dev53+gab1ba6c31
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]   █▄█▀ █     █     █     █  model   RedHatAI/gemma-4-31B-it-FP8-block
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:347]
(APIServer pid=1) INFO 08-28 10:54:43 [api_utils.py:286] non-default args: {'model_tag': 'RedHatAI/gemma-4-31B-it-FP8-block', 'port': 8200, 'model': 'RedHatAI/gemma-4-31B-it-FP8-block', 'load_format': 'dummy', 'profiler_config': ProfilerConfig(profiler='torch', torch_profiler_dir='/tmp/traces/1318414', proton_profiler_dir='', proton_context='shadow', proton_data='tree', proton_backend=None, proton_mode=None, proton_hook=None, proton_output_format=None, torch_profiler_with_stack=False, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, capture_torch_profiler=False, detailed_trace_annotation=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)}
...
USDT:2026-08-28 11:00:04 1:1404 ActivityProfilerController.cpp:415] profiler_start
(APIServer pid=1) INFO 08-28 11:00:04 [api_router.py:25] Profiler started.
(APIServer pid=1) INFO:     127.0.0.1:46248 - "POST /start_profile HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46386 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46488 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46274 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46398 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46466 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO:     127.0.0.1:46484 - "POST /v1/completions HTTP/1.1" 200 OK
...
(APIServer pid=1) INFO 08-28 11:02:24 [api_router.py:31] Stopping profiler...
USDT:2026-08-28 11:02:24 1:1404 ActivityProfilerController.cpp:455] profiler_stop
USDT:2026-08-28 11:02:24 603:603 ActivityProfilerController.cpp:455] profiler_stop
[rank0]:[W828 11:02:26.720282776 collection.cpp:1182] Warning: ROCTracer produced duplicate flow start: 3 (function operator())
(APIServer pid=1) INFO 08-28 11:02:33 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 125.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.0%
(APIServer pid=1) INFO 08-28 11:02:43 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 11.0%
(EngineCore pid=603) -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
(EngineCore pid=603)                                                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls
(EngineCore pid=603) -------------------------------------------------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------  ------------
(EngineCore pid=603)                         _w8a8_triton_block_scaled_mm.kd         0.00%       0.000us         0.00%       0.000us       0.000us       89.466s        64.09%       89.466s

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

…qi-hang-fix

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

…-hang Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

github-actions · 2026-08-28T19:41:37Z

✅ @simondanielsson, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

Rohan138

LGTM, thanks!

dllehr-amd

Whelp...we carrying this one around a little longer. I'm good merging.

dllehr-amd · 2026-08-29T02:59:51Z

/ci run

github-actions · 2026-08-29T03:00:09Z

✅ Triggered Buildkite CI #86091 for commit ab1ba6c312b8.

simondanielsson · 2026-08-29T08:09:49Z

/ci retry

github-actions · 2026-08-29T08:10:06Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #86091.

simondanielsson · 2026-08-29T09:52:33Z

/ci retry

github-actions · 2026-08-29T09:52:50Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #86091.

simondanielsson · 2026-08-29T12:24:02Z

/ci retry

github-actions · 2026-08-29T12:24:18Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #86091.

AndreasKaratzas · 2026-08-29T19:07:11Z

/ci run

github-actions · 2026-08-29T19:07:29Z

✅ Triggered Buildkite CI #86165 for commit 43833c16e83d.

AndreasKaratzas · 2026-08-29T19:08:02Z

@simondanielsson Please don't use excessive retries without any particular evidence that the failed test group is fixed

dllehr-amd · 2026-08-31T15:34:00Z

/ci retry

github-actions · 2026-08-31T15:34:19Z

✅ Queued 1 failed job(s) for retry in Buildkite CI #86165.

Rohan138 · 2026-08-31T18:54:54Z

Now that the 7.14 docker image is merged in #49925, we should probably copy this over in a follow-up to that dockerfile as well

AndreasKaratzas · 2026-09-01T19:00:10Z

/ci run

github-actions · 2026-09-01T19:00:34Z

✅ Triggered Buildkite CI #86680 for commit 47cc444a336d.

AndreasKaratzas · 2026-09-02T03:39:44Z

/ci run

github-actions · 2026-09-02T03:41:31Z

✅ Triggered Buildkite CI #86767 for commit da4d8a374c57.

shen-shanshan · 2026-09-02T07:52:27Z

/ci retry

github-actions · 2026-09-02T07:52:48Z

✅ Queued 2 failed job(s) for retry in Buildkite CI #86767.

vllm-project#54171) Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com> Co-authored-by: Andreas Karatzas <akaratza@amd.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

fix: patch rocprofiler-sdk with two prs

f350f2c

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

mergify Bot added ci/build rocm Related to AMD ROCm labels Aug 28, 2026

github-project-automation Bot added this to AMD Aug 28, 2026

github-project-automation Bot moved this to Todo in AMD Aug 28, 2026

simondanielsson changed the title ~~fix: patch rocprofiler-sdk with two prs~~ [Bugfix][ROCm][Build]: fix profiler hang Aug 28, 2026

mergify Bot added the bug Something isn't working label Aug 28, 2026

simondanielsson changed the title ~~[Bugfix][ROCm][Build]: fix profiler hang~~ [Bugfix][ROCm][Build]: fix profiler hang due to queue interposition bug Aug 28, 2026

simondanielsson added 3 commits August 28, 2026 09:53

fix: simplify comment

8e76595

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

Merge branch 'main' of github.com:vllm-project/vllm into rocprofiler-…

a032205

…qi-hang-fix

chore: rename to hsa paths

03f7e2d

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

simondanielsson mentioned this pull request Aug 28, 2026

[Bug]: torch profiler hangs on ROCm after rocprofiler-sdk 1.3.2 bump #54087

Closed

1 task

Merge remote-tracking branch 'upstream/main' into fix/rocprofiler-sdk…

ab1ba6c

…-hang Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>

simondanielsson marked this pull request as ready for review August 28, 2026 11:02

simondanielsson requested review from AndreasKaratzas, dllehr-amd and tjtanaa as code owners August 28, 2026 11:02

claude Bot reviewed Aug 28, 2026

View reviewed changes

Rohan138 added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 28, 2026

Rohan138 approved these changes Aug 28, 2026

View reviewed changes

dllehr-amd approved these changes Aug 29, 2026

View reviewed changes

Merge branch 'main' into fix/rocprofiler-sdk-hang

43833c1

simondanielsson changed the title ~~[Bugfix][ROCm][Build]: fix profiler hang due to queue interposition bug~~ [Bugfix][ROCm][Build] fix profiler hang due to queue interposition bug Aug 29, 2026

Merge branch 'main' into fix/rocprofiler-sdk-hang

737afa3

Merge branch 'main' into fix/rocprofiler-sdk-hang

47cc444

Merge branch 'main' into fix/rocprofiler-sdk-hang

da4d8a3

AndreasKaratzas enabled auto-merge (squash) September 2, 2026 03:57

AndreasKaratzas merged commit b205750 into vllm-project:main Sep 2, 2026
127 of 128 checks passed

github-project-automation Bot moved this from Todo to Done in AMD Sep 2, 2026

simondanielsson deleted the fix/rocprofiler-sdk-hang branch September 2, 2026 09:04

mawong-amd mentioned this pull request Sep 3, 2026

[ROCm][Perf][Bugfix] Multi-stream perf improvements; rocprofiler fixes #55099

Merged

4 tasks

JohnQinAMD mentioned this pull request Sep 11, 2026

[Bug][ROCm]: Intermittent worker segfault in libhsa-runtime64 — ROCPROFILER_QUEUE_INTERPOSITION=0 forces an unfixed ROCr heap overflow #56521

Open

Uh oh!

Uh oh!

Conversation

simondanielsson commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

1. Build succeeds e2e. Here's the part where the patch is applied:

2. Profiling no longer hangs

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

github-actions Bot commented Aug 28, 2026

Uh oh!

Rohan138 left a comment

Choose a reason for hiding this comment

Uh oh!

dllehr-amd left a comment

Choose a reason for hiding this comment

Uh oh!

dllehr-amd commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

simondanielsson commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

simondanielsson commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

simondanielsson commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

AndreasKaratzas commented Aug 29, 2026

Uh oh!

github-actions Bot commented Aug 29, 2026

Uh oh!

AndreasKaratzas commented Aug 29, 2026

Uh oh!

dllehr-amd commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

Rohan138 commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndreasKaratzas commented Sep 1, 2026

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

AndreasKaratzas commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

shen-shanshan commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

simondanielsson commented Aug 28, 2026 •

edited

Loading

Rohan138 commented Aug 31, 2026 •

edited

Loading