Skip to content

test(rocprofiler-sdk): range replay week of 2026-10-12: C8 (stacked) - #13272

Draft
amd-vkale wants to merge 2 commits into
users/vkale/range-replay-tests-2026-10-05-with-cursor-fixes-helpfrom
users/vkale/range-replay-tests-2026-10-12-with-cursor-fixes-help
Draft

amd-vkale wants to merge 2 commits into
users/vkale/range-replay-tests-2026-10-05-with-cursor-fixes-helpfrom
users/vkale/range-replay-tests-2026-10-12-with-cursor-fixes-help

Conversation

@amd-vkale

Copy link
Copy Markdown
Contributor

Motivation

Week two of the range replay test roadmap (source/docs/conceptual/range_replay/range_replay_test_plan.md): C8. A replay window restores the agent's whole tracked inventory before every pass and at its end, so another thread's work on the same agent must wait for the window instead of running inside it. Otherwise its writes are restored away. #10864 fixed a case where a graph launched from another thread took the queue fast path and could run inside the window; this is its regression test. With it, all three bugs fixed in #10864 without a test (C1, C2 and C8) have one.

Stacked on #12815 (week one: C1, C2).

Technical Details

New tests/range-replay-concurrency/, in the style of tests/range-replay-local-context. Its handshake follows tests/kernel-replay-concurrency, a block of atomics the tool exports and the application finds with dlsym:

  • Application (main.cpp). The main thread holds 64 MiB of ballast, so each restore takes time. It records a range of three dependent kernels and replays it in rocprofiler_range_replay_end(). A second thread captures a one-kernel graph that increments its own counter, launches it once before the range opens, then waits. Every allocation and the graph setup happen before the range opens, because an allocation would decline the range, and so would a dispatch from the second thread while the range is recording.
  • Tool (client.cpp). It asks for six passes and drives the handshake:
    • at the first replayed pass, it releases the graph thread (the range has left its thread by then, so a launch cannot decline it as a concurrent dispatch);
    • at the last pass, it waits until the graph thread has begun its 64 launches and gives them time to reach the queue while the window still holds the agent;
    • at CLOSE, it signals that the range is over, so the graph thread never waits on a window that did not open.
  • Pass condition. The range is REPLAYED with the expected pass and dispatch counts and divergence_count == 0 (ROCPROF_RANGE_REPLAY_VERIFY=1). The handshake overlapped the window. The application's result is intact, and the counter holds all 64 increments plus the warm-up. A launch that ran inside the window would have its increment restored away, and would also appear as a divergent region.
  • No GPU. The test skips, the same way the decline tests do.

The roadmap's coverage table gains the test, and its list of untested #10864 bugs now names the test that guards each one.

Issue Tracking

Related: #10864 (range replay, whose replay window this test exercises against another thread).

Test Plan

  • test-range-replay-concurrent-graph-launch on the mi325 CI jobs.
  • Locally, with no GPU or ROCm on the build machine:
    • clang-format 11 and cmake-format are clean;
    • the tool compiles with -W -Wall -Wextra -Wshadow -Werror against the in-tree headers;
    • the application's host code compiles with the same flags against the in-tree HIP and HSA host headers;
    • a CMake harness with stand-in imported targets configures the directory and registers the test with the intended environment, pass, fail and skip expressions.

Test Result

CI for this PR is pending. Nothing could be run on a GPU locally, and the device code was not compiled here, because the in-tree HIP headers need a newer clang than the one on the build machine.

Week one (#12815, 1417207891): C1 and C2 passed on every mi325 job (rhel-8.8, rhel-9.5, sles-15.6, ubuntu-22.04 twice, and code coverage). Its red jobs came from other tests:

  • rocprofv3-test-kernel-replay-validate (ubuntu-22.04): pass 0 counted 81911 SQ_WAVES against 16384 in each replayed pass, well beyond the 8-wave tolerance. That is not the few-wave spread develop's runs showed; develop's own CI has not reached tests since 2026-10-09 (see below), so newer develop runs could not be compared.
  • rocprofv3-test-conversion-script-validate pmc1, both prefixes (sles-15.6): a counter value of 0.0 where every value must be positive.
  • Code coverage hit the 30-minute action limit after the range replay tests had passed.

#10864 is unchanged at 0844152fe6; its CI was summarized in #12815.

Separately, develop's own SDK build has failed since 2026-10-09: ROCr #13129 added HSA_ACCESS_PERMISSION_EX, which hsa/utils.hpp does not handle under -Werror=switch. This stack has not merged develop since then, so its CI is unaffected.

Submission Checklist

Open in Web Open in Cursor 

cursoragent and others added 2 commits October 12, 2026 15:07
…nge replay window (C8)

Add tests/range-replay-concurrency. The replay window restores the
agent's whole tracked inventory before every pass and at its end, so a
second thread's work on the same agent has to wait for the window rather
than run inside it, or its writes are restored away. A graph launch from
another thread used to take the queue fast path and could do exactly that.

The application holds 64 MiB of ballast so each restore takes time,
records a three-kernel range on its main thread, and replays it six
times. A second thread warms up a one-kernel graph before the range opens
and waits. The tool releases it at the first replayed pass, after the
range has left its thread, so the launches cannot decline it as a
concurrent dispatch. At the last pass the tool waits until the thread has
begun launching, so the launches are issued while the window still holds
the agent.

The test passes only if the range was replayed with divergence checking
reporting no changed region, the range's result is intact, and the
graph thread's counter holds all 64 increments plus the warm-up. A
launch that ran inside the window would have its increment restored away
and show up as a divergent region. A run without a GPU skips.

Co-authored-by: Vivek Kale <vivek.kale@amd.com>
The coverage table gains the concurrency test, and every bug fixed in
#10864 that had no regression test (copies unseen without copy tracing,
hsa_amd_memory_async_batch_copy bypassing the decline, and another
thread's graph launch inside a replay window) now names the test that
guards it.

Co-authored-by: Vivek Kale <vivek.kale@amd.com>
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

  • Add new commits
  • Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

@github-actions github-actions Bot added the Stale PR has no activity for 25+ days label Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation organization: ROCm project: rocprofiler-sdk Stale PR has no activity for 25+ days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants