feat(rocprofv3): job_replay_mode lets input-file jobs describe kernel replay passes by amd-vkale · Pull Request #12729 · ROCm/rocm-systems

amd-vkale · 2026-10-02T23:17:13Z

Motivation

An input file's jobs already say which services each job uses, but every job is a separate application run (application replay). This lets a file opt in to running its jobs as the passes of one kernel replay run instead: each job picks what its pass collects. The opt-in is a new top-level field, so existing JSON and YAML files keep working unchanged.

Stacked on #12776 (the --att replay pass, which builds on #11335). Review the commits after that branch; the merge commit only brings in that branch's test fix.

Technical Details

Schema and opt-in. A new top-level job_replay_mode is either application (the default) or kernel:

{
  "job_replay_mode": "kernel",
  "jobs": [
    {"pmc": ["SQ_WAVES", "GRBM_COUNT"], "kernel_include_regex": "gemm", "kernel_replay_beta_enabled": true},
    {"advanced_thread_trace": true, "att_target_cu": 1},
    {"pmc": ["GRBM_GUI_ACTIVE"]}
  ]
}

This runs each gemm dispatch three times: counters, then the thread trace (with counters locally stopped), then counters (with the trace stopped). It produces one run and one output directory.

Which options are per pass. A job selects one service: pmc (a counter group) or advanced_thread_trace with its att_* settings. att_library_path may sit on any job. Every other option applies to the whole run, and jobs that set it must agree.
What is rejected:
- a job with both services, or neither;
- more than one thread-trace job, pmc_groups, or att_* settings on a counter job;
- a file without any pmc job, or replay_mode: application;
- --pmc or --att on the command line, attach mode, and --collection-period.
The existing replay checks (beta acknowledgement, PC sampling, SPM, device-mode ATT) then apply to the folded run.
Backward compatibility. Without the field, or with application, main() takes exactly the old path, with or without --replay-mode kernel. CLI tests pin this, and an older rocprofv3 simply ignores the field.
rocprofv3_input_schema.json now documents job_replay_mode, replay_mode, kernel_replay_beta_enabled, advanced_thread_trace and the ATT job options. The schema is documentation only, so a CLI test checks that its enum matches what rocprofv3 accepts. That check skips where the schema isn't installed.

Tool. The thread-trace pass was hard-wired to pass 0. A new ROCPROF_KERNEL_REPLAY_ATT_AFTER_GROUPS setting places it after N configured groups; 0 keeps the CLI's ATT-first order.

The layout logic moves to kernel_replay_plan.{hpp,cpp} and is unit-tested.
Agents skip groups they can't collect, so each per-agent profile records its configured group. That way the ATT pass follows exactly the groups configured ahead of it on every GPU.
Every pass now sets both contexts explicitly, rather than toggling only at passes 0 and 1.

Found, not fixed here (they change existing behavior and need their own PRs):

In application mode, a multi-job file silently drops every job without pmc. Two trace-only jobs produce zero runs, and an ATT job next to a pmc job never runs.
A job setting att_serialize_all: true fails with a RuntimeError, because the argparse default False counts as set in get_args. I left that key out of the schema.

Test Plan

GPU-free:
- tool-tests gains 7 kernel_replay_plan cases.
- test_kernel_replay_cli.py gains 22 job-replay cases: folding, ordering, YAML, run-option merging, every rejection, defaults, legacy files, and schema sync.
GPU (mi325 CI):
- rocprofv3-test-kernel-replay-jobs-{generate,validate}: two counter jobs, checked with the full replay validator.
- ...-jobs-att-{generate,validate}: the ATT job between the counter jobs (the trace runs on pass 1), checked with --thread-trace.
- The ...-att-* tests from the base branch also run here.

Test Result

Second CI run (head a7272b3: Core, Coverage, TheRock). Every test this branch adds or changes passed in every job that ran it:

the 8 new replay tests (att-* and jobs-*, generate and validate);
the plan and CLI unit tests.

Formatting passes, and Core rhel-8.8, rhel-9.5 and sles-15.6 pass outright. The other jobs failed only on tests outside this change, and each of those tests passed in the other jobs built from this same commit.

First CI run (head d7c6f27). All generate steps passed, and the behavioral checks passed on hardware:

each replayed dispatch was traced exactly once, and the traced dispatches matched the ones with counter records, both with ATT first and with ATT on pass 1;
replay_pass stayed 0..N-1 per dispatch;
sanity counters were constant across passes.

The validate steps failed on three test-expectation bugs, fixed for the second run:

test_expected_counters_present hardcodes the five-group counter list (log). A new --expected-counters option takes the two-group runs' list.
The first-launch test expected 3 profiled dispatches; the right number is 4 (log). The HIP runtime's copy kernel for hipMemcpyToSymbol also gets its first launch profiled.
The schema-sync check read a source-tree path that TheRock's installed tests don't have (log). It now skips there.

Separately, black-formatted validate.py, which the formatting job flagged once it was touched (log).

Remaining failures are pre-existing or infrastructure. None of them runs code this branch changes. The tool changes only take effect with kernel replay plus counters, and the rocprofv3 changes only with kernel replay. I checked each one against 330 rocprofiler-sdk Core, Coverage and TheRock jobs on 52 branches (2026-09-30 to 2026-10-05). The counts below exclude this branch.

WSL Build & Test (job, second run): the runner lost contact with GitHub during "Set up WSL", before the repository was copied in. The job has no log.
rocprofv3-test-rocshmem-tracing timeout (second run: Core ubuntu-22.04 log; first run: TheRock gfx94X log): timed out in 21 jobs on 17 other branches, including develop (Core ubuntu, TheRock gfx94X).
thread-trace-api-agent-start-stop-before-hsa-init-test timeout (second run, Coverage log): failed in 7 jobs on 7 other branches, including develop (log).
thread-trace-api-triple-buffer-slow-test abort (second run, Coverage log): the same abort hit [CLR] Add hipDeviceGetP2PAtomicCapabilities #10635 (log) and fix(rocprofiler-sdk): split requirements file and install the runtime one into share/rocprofiler-sdk #12304 (log).
app-without-signal-handling-fork-exec, "tool finalization" not printed (second run, TheRock gfx94X log):
- the same message hit the spawn variant on [hip] feat: Add hipLibraryGetModule API (AIRUNTIME-2608) #12625 (log) and the single-process variant on perf(rocprofiler-sdk): kernel replay phase 2: performance optimizations #10916 (log);
- this scenario's test_flushed_markers check failed on [hip] feat: Add hipLibraryGetModule API (AIRUNTIME-2608) #12625 (log), perf(rocprofiler-sdk): kernel replay phase 2: performance optimizations #10916 (log), fix(rocprofiler-sdk): split requirements file and install the runtime one into share/rocprofiler-sdk #12304 (log) and refactor(rocprofiler-sdk): migrate SPM off the per-queue callback registry (moved PR 8887 to here) #11968 (log).
attach-once-att and att-no-intercept validate (first run, Core rhel-9.5: attach-once-att, att-no-intercept):
- the same checks failed together on Release imported VMM handles and mempool pointers #12760 (attach-once-att, att-no-intercept);
- attach-once-att failed on 14 other branches, including develop (log), and att-no-intercept on 8.
kernel_replay_snapshot.restore_reverts_multiple_buffers abort (first run, Coverage log):
- this is its only failure in the window;
- its binary is built only from source/lib/rocprofiler-sdk/kernel_replay/, which this branch doesn't touch;
- it passed in that run's four Core jobs and in Coverage on the second run.

Locally (no GPU):

Built against ROCm 7.2.4 with no warnings. clang-tidy with CI's settings exits 0, and clang-format 11, cmake-format, black 26 and flake8 are clean.
tool-tests 24/24; CLI unit tests 54/54, both with the schema present and without it.
rocprofv3 --echo: the opt-in file gives one run with two groups and the ATT position at 1. The same jobs without the field give two runs in pass_1/ and pass_2/.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

…CONFIG The kernel filters (--kernel-include-regex, --kernel-exclude-regex, --kernel-iteration-range) were only consulted from the counter dispatch callback, after the SDK had already been told to replay the dispatch once per counter group. A dispatch no filter selects was still drained, snapshotted and re-executed N times, collecting nothing on any pass. Consult the filter once per logical dispatch in the CONFIG callback instead, when the SDK asks for the pass count. A dispatch that is not selected reports one pass and runs once on the ordinary path. The decision is kept per thread, keyed by the dispatch id the SDK reserves before CONFIG, and every later callback for that id -- each replay pass, or the single run of a dispatch that is not replayed -- reuses it, so the stateful iteration filter still advances exactly once per application launch. This replaces the pass-0 memo in is_targeted_kernel, which could only take effect once a pass had started. Counter collection's iteration counts move to namespace scope so CONFIG and the counter callback number launches the same way. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

Co-authored-by: Vivek Kale <vivek.kale@amd.com>

rocprofv3 rejected --att with --replay-mode kernel because only counter collection was pass-aware: the thread trace would stay on for every pass and report each dispatch once per pass. The SDK has let a tool switch services per pass since kernel replay landed (replay_start_context/replay_stop_context, and the att_client and local-context samples); rocprofv3 simply never used it. With --att, each targeted dispatch now gets one extra pass ahead of its counter passes. Pass 0 runs the dispatch thread trace with counter collection locally stopped, pass 1 locally stops the trace and restarts counter collection, and counter group g runs on pass g + 1. The toggles are sticky for the rest of the loop and the SDK restores both contexts afterwards. The trace never shares a pass with counter collection: SQTT traffic and serialization perturb the counters, and both services write the one per-dispatch user_data slot the SDK hands out, so whichever dispatch callback runs last overwrites the other's state. Under replay the dispatch thread trace moves to its own context so a pass can switch it without dropping that pass's kernel dispatch, memory copy and API records from the client context. It only traces a dispatch the CONFIG plan selected, on the thread trace pass, or on the single run of a dispatch with no counter group to collect; a replay that falls back to a single run, and a dispatch replay never planned (HIP graph, multi-packet submission), keep counter collection instead. Counter records still number their groups 0..N-1 in replay_pass. Device-mode thread trace (--att-consecutive-kernels, --att-no-intercept, --selected-regions) and --collection-period cannot be confined to one pass, so rocprofv3 rejects them together with --replay-mode kernel and --att. PC sampling and --spm stay rejected. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

Two GPU integration tests, disabled when rocprof-trace-decoder is not found, like the other rocprofv3 ATT tests: - --att with two counter groups over the kernel-replay app. validate.py checks the usual replay invariants (counter records carry replay_pass 0..1 once per dispatch, sanity counters constant across passes, one distinct batch per pass), and a new test_one_thread_trace_per_dispatch requires each dispatch to be traced exactly once and the traced dispatches to match the ones with counter records. - The same run with three launches per kernel. With --att and no iteration range only each kernel's first launch is profiled, so exactly three logical dispatches must remain, each traced once and carrying both counter passes. The new check is opt-in through --thread-trace so the counters-only runs are unaffected. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

Describe how --att combines with --replay-mode kernel, how passes map to counter groups, which thread trace options are rejected and why, that replay_pass still numbers counter groups, and that dispatches the kernel filters do not select are no longer replayed. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

mi325 CI ran the new thread trace replay tests for the first time. The behavior they exist for held -- every replayed dispatch was traced exactly once, the traced dispatches matched the ones with counter records, and replay_pass still ran 0..1 -- but two expectations were wrong: - test_expected_counters_present checks a fixed counter list written for the five-group run. A --expected-counters option now names the union of a run's groups, and the thread trace runs pass their two-group list. - With first-launch-only targeting the app yields four profiled dispatches, not three: besides its three kernels, the HIP runtime dispatches a copy kernel for hipMemcpyToSymbol, and that kernel's first launch is profiled too. Also black-format validate.py; it predates the formatter's current output, and the formatting job checks every touched Python file. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

…ially rocprofv3-test-kernel-replay-att-validate checks that SQ_WAVES and SQ_INSTS_VALU stay constant across a dispatch's replay passes. Those SQ counters count every wave on the device, and the generate test only holds the rocprofiler_kernel_replay lock, so GPU tests that run alongside it add waves and instructions to some of the passes and the check fails intermittently. Run rocprofv3-test-kernel-replay-att-generate with RUN_SERIAL, the same fix #12888 makes for the other kernel replay generate tests. The first-iteration variant keeps running in parallel: its validate step selects only checks that do not compare counter values.

…the counter passes The thread trace pass was hard-wired to pass 0. A new ROCPROF_KERNEL_REPLAY_ATT_AFTER_GROUPS setting says how many of the configured counter groups run ahead of it (0, the default, keeps it first), so a front end can put it anywhere in the pass order. The pass layout moves into kernel_replay_plan.{hpp,cpp} as plain logic: counter_group(pass) maps a pass to its counter group, or to the thread trace, and thread_trace_pass_index() places the trace on an agent. An agent skips the configured groups it cannot collect, so its profiles now record which configured group each was built from, and its thread trace pass follows exactly the groups configured ahead of it rather than a fixed count of its own groups. Each pass now sets both contexts explicitly -- counters on and the trace off, or the reverse -- instead of only toggling at passes 0 and 1, which only held while the trace came first. GPU-free unit tests in tool-tests cover the layout with the trace first, between groups, last, and on an agent missing a group. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

A JSON or YAML input file can now opt in with a top-level job_replay_mode: kernel. Its jobs then become the passes of a single kernel replay run, in job order, instead of separate runs of the application, and each job selects what its pass collects: - pmc: one counter group, or - advanced_thread_trace: the dispatch thread trace, configured by the att_* options on the same job; at most one job may trace. Every other option a job sets applies to the whole run, so it may sit on any job and jobs that set it must agree. att_library_path only locates the decoder and may also sit on any job. The planner rejects a job with both or neither service, a second thread trace job, pmc_groups, att_* settings on a counter job, a file without counters, replay_mode: application, --pmc or --att on the command line, attach mode and --collection-period; the existing kernel replay checks (beta acknowledgement, PC sampling, SPM, device-mode thread trace) then apply to the folded run. The run writes one output directory, and the thread trace pass goes where its job sits through ROCPROF_KERNEL_REPLAY_ATT_AFTER_GROUPS. job_replay_mode defaults to application. A file without it, or with application, takes exactly the path it took before -- each job is a run of its own, with or without --replay-mode kernel -- so existing input files are unaffected; an older rocprofv3 ignores the field. rocprofv3_input_schema.json documents job_replay_mode, replay_mode, kernel_replay_beta_enabled, advanced_thread_trace and the thread trace job options. The schema is documentation only, so a CLI unit test now checks that its job_replay_mode values match what rocprofv3 accepts. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

Two input files with job_replay_mode: kernel over the kernel-replay app: - two counter jobs, validated like two --pmc groups (one dispatch id per logical dispatch, replay_pass 0..1, constant sanity counters, a distinct batch per pass); - the same counter jobs with a thread trace job between them, so the trace runs on pass 1. Counter records must still number their groups 0..1 and each dispatch must be traced exactly once. Disabled without rocprof-trace-decoder, like the other thread trace tests. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

Describe how an input file opts in to running its jobs as kernel replay passes, which options select a pass and which apply to the run, what is rejected, how output and replay_pass look, and where the thread trace pass falls. List job_replay_mode in the input schema summaries and the CHANGELOG. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

The job_replay_mode GPU runs collect two counter groups, so pass their counter list to validate.py through the new --expected-counters option instead of letting it look for the five-group set. The schema check in the CLI unit test reads rocprofv3_input_schema.json from the source tree. TheRock runs the installed tests, which have no source tree, so the check now skips when the schema is not present rather than failing; it still runs in-tree, where the Core jobs passed it. Co-authored-by: Vivek Kale <vivek.kale@amd.com>

rocprofv3-test-kernel-replay-jobs-validate and -jobs-att-validate run the full validate.py, which checks that SQ_WAVES and SQ_INSTS_VALU stay constant across a dispatch's replay passes. The SQ counters count every wave on the device, and the jobs generate tests only hold the rocprofiler_kernel_replay lock, so GPU tests that run alongside them make the check fail intermittently. Run rocprofv3-test-kernel-replay-jobs-generate and -jobs-att-generate with RUN_SERIAL, as the thread trace generate test does.

github-actions · 2026-11-04T00:15:26Z

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

Add new commits
Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

github-actions Bot added documentation Improvements or additions to documentation project: rocprofiler-sdk labels Oct 2, 2026

systems-assistant Bot added the organization: ROCm label Oct 2, 2026

amd-vkale mentioned this pull request Oct 5, 2026

feat(rocprofv3): run dispatch thread trace as its own kernel replay pass #12776

Open

1 task

cursoragent and others added 13 commits October 9, 2026 00:44

style(rocprofv3): clang-format the kernel replay plan comments

40df217

Co-authored-by: Vivek Kale <vivek.kale@amd.com>

amd-vkale force-pushed the cursor/kernel-replay-att-pass-d7b8 branch from 101e8a3 to 101b092 Compare October 9, 2026 07:21

amd-vkale force-pushed the cursor/kernel-replay-json-passes-d7b8 branch from cecd78c to d6d85d4 Compare October 9, 2026 07:21

cursor Bot mentioned this pull request Oct 16, 2026

[DO NOT MERGE] Integration preview: range replay + kernel replay perf stack on develop #10193

Open

github-actions Bot added the Stale PR has no activity for 25+ days label Nov 4, 2026

cursor Bot force-pushed the cursor/kernel-replay-att-pass-d7b8 branch from a85bc14 to 84ed986 Compare November 5, 2026 08:21

github-actions Bot removed the Stale PR has no activity for 25+ days label Nov 6, 2026

cursor Bot mentioned this pull request Nov 6, 2026

[Tracking] rocprofiler-sdk: kernel replay and range replay #15178

Closed

Navigation Menu

Repository navigation

feat(rocprofv3): job_replay_mode lets input-file jobs describe kernel replay passes - #12729

feat(rocprofv3): job_replay_mode lets input-file jobs describe kernel replay passes#12729
amd-vkale wants to merge 13 commits into
cursor/kernel-replay-att-pass-d7b8from
cursor/kernel-replay-json-passes-d7b8

amd-vkale commented Oct 2, 2026 •

edited by cursor Bot

Loading

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

amd-vkale commented Oct 2, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

github-actions Bot commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

amd-vkale commented Oct 2, 2026 •

edited by cursor Bot

Loading