test(rocprofiler-sdk): add a kernel replay benchmark harness and benchmarking guide by amd-vkale · Pull Request #12604 · ROCm/rocm-systems

amd-vkale · 2026-10-01T04:20:04Z

Motivation

Underlying Problem

The kernel replay performance work needs numbers that are comparable from one change to the next, and today nothing in the tree produces them:

The CI perf tests only bound wall time. tests/kernel-replay-perf/ and validate_perf.py on develop assert that end-to-end time stays under a threshold. They catch a catastrophic slowdown, but they record no value an operator can compare across builds, so they cannot say whether an optimization helped.
The benchmark database merged replay and non-replay runs. rocprofv3 emits the same config JSON for a kernel replay run and a non-replay run that request the same counters. The benchmark harness hashes that JSON to pick a benchmark_config row, so the two runs shared one row and their measurements were averaged together. The database could not express the comparison the replay work needs.
The benchmark configuration compared against the wrong thing. Its app_replay job passed four --pmc flags without enabling replay, assuming rocprofv3 then runs the application once per group. It does not: without replay the groups rotate across dispatches within one run, so no dispatch carries more than one group. Comparing kernel replay against that job compared "every group on every dispatch" with "one group on any dispatch", and kernel replay was always going to look expensive.
The docs said the same wrong thing. The kernel replay index listed several --pmc groups as application replay, which contradicts the cost table on the same page.

Notes on Bigger Picture

This PR is the measurement layer for the perf stack. #12608 adds per-phase timings inside one replay window; this harness measures whole runs across pass counts and memory footprints. Together they let each optimization PR say which phase moved and by how much end to end. The 9.4× BabelStream figure in #10916 is the first number to re-measure with it.

Technical Details

There are three ways to collect several counter groups, and only two of them are comparable.

flowchart LR
  G["G counter groups"] --> AR["Application replay: G full runs, one group each"]
  G --> MX["Multiplexed (several --pmc, no replay): one run, groups rotate across dispatches"]
  G --> KR["Kernel replay: one run, every dispatch replayed once per group"]
  AR -. "same data on every dispatch" .- KR

Application replay and kernel replay both put every group on every dispatch, so they are the pair to compare. Multiplexing collects less data and is not a substitute for either.

Operator-run harness

benchmark/scripts/replay_perf.py runs a kernel replay application under rocprofv3 and writes one CSV row per measurement, each carrying the full command that produced it. Every measurement is the median of --repeats timed runs taken after --warmup untimed runs, the same convention the CI perf tests use, because the first run pays for page-cache misses and code-object loading. It is operator-run, not part of CI.

P1 collection modes: application replay vs. multiplexed vs. kernel replay, for the same groups.
P2 pass-count scaling.
P3/P4 footprint scaling: the tracked device footprint, and the host peak RSS that comes with it.
P6 the non-replayed dispatch tax: what dispatches that are not replayed pay while replay is merely active.
P5 (drain cost as a fraction of a window) needs internal timers rather than wall clock, so it is left to the INFO phase timings in feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608.

Benchmark database bookkeeping

derive_replay_config reads the collection mode off the rocprofv3 command line: --kernel-replay-beta-enabled in all its value forms, the number of --pmc groups, and input files. It records kernel_replay, counter_group_count and counter_collection_mode on benchmark_config, so runs that differ only in how their groups are collected land in separate rows.
CONST_ADDED_COLUMNS: databases created before these columns existed are migrated on connect, because CREATE TABLE IF NOT EXISTS leaves an existing table as it was and the insert would otherwise fail on the missing columns.
benchmark_replay_{{metric}}: a view that puts kernel replay and application replay on one axis, the cost of collecting every requested group. Application replay pays one full run per group; kernel replay pays one run plus a pass per dispatch.

Benchmark configuration and docs

benchmark/kernel-replay.yaml, benchmark/README.md: the app_replay job is renamed to what it measures, and the single-group run becomes the reference for application replay, since collecting G groups that way is that run repeated G times. A sweep over how many dispatches the application enqueues between synchronizations is added: kernel replay drains the agent for every replayed dispatch, so how much a workload loses to that drain depends on how deep its queue usually is, and nothing measured that before.
kernel_replay_benchmarking.md covers the three collection modes, which two are comparable and why, what the suite now records about a run, and what it still cannot express: the bytes the snapshot moves, wall time against GPU time, and how much of a run was declined rather than replayed. The kernel replay index table is corrected, and the performance assessment links to the new page.

The four commits are cherry-picked unchanged from users/vkale/kernel-replay-bench-improvements-with-cursor-help onto current develop.

Reviewer Guide

derive_replay_config accepts every value form of --kernel-replay-beta-enabled (bare, separate value, inline value, falsy value) and never consumes a counter whose name looks like a truth value (TestDeriveReplayConfig).
Runs that differ only in collection mode or group count get different benchmark_config rows, and identical runs reuse one row (TestConfigRows).
The migration adds the missing columns, preserves existing rows, and is idempotent (TestMigration).
The replay view's projection and speedup are right with and without a single-group reference row, and repeated runs do not duplicate the reference (TestReplayView).
In replay_perf.py, warm-up runs are excluded from the median, a run that always fails yields no number, and a failed measurement is recorded rather than dropped (TestRepeat, TestRecorder).

Out of scope: registering the harness in CI (it needs a GPU and minutes per data point), and the snapshot-bytes and declined-dispatch metrics the new docs list as not yet expressible.

Issue Tracking

JIRA ID: AIPROFSDK-68

Test Plan

test_replay_perf.py, 28 tests:
- command construction: one --pmc per group, the replay flag only when requested and before the application, and a per-measurement output directory;
- process measurement: wall time, return codes, signalled children, and peak RSS that is not carried over from an earlier child;
- repeat, median and warm-up logic;
- the CSV recorder;
- argument validation;
- the collection-mode vocabulary shared with the database.
test_benchmark_replay.py, 29 tests:
- deriving the replay configuration from rocprofv3 arguments;
- separating benchmark_config rows;
- the schema migration;
- the replay view;
- that the schema and views execute and are idempotent.
Both files are pure Python and sqlite3, with no GPU and no built rocprofiler. They are registered as the test-benchmark-replay-bookkeeping ctest (CMakeLists.txt L15-L20) when ROCPROFILER_BUILD_BENCHMARK=ON. That option is off by default, so CI does not run them.
Run locally with python3 -m unittest discover -s projects/rocprofiler-sdk/benchmark/tests -p "test_*.py" -v.

Test Result

All 57 tests pass locally (Python 3.12, 0.7 s).
black 26 and cmake-format 0.6.13 are clean on the changed files.
rocprofv3-benchmark.py needs Python 3.7 (subprocess.run(text=...)); that requirement predates this PR.
Treat the Checks tab as the source of truth for CI.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

Related PRs

Kernel replay host-side perf stack: docs(rocprofiler-sdk): note why kernel replay keeps restoring constant symbols #12605 ← perf(rocprofiler-sdk): take the kernel replay drain signal from the signal pool #12607 ← feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608 ← perf(rocprofiler-sdk): skip HIP's compilation-unit markers in the kernel replay snapshot #12609 ← perf(rocprofiler-sdk): keep kernel replay snapshot staging across dispatches #12610.
perf(rocprofiler-sdk): kernel replay phase 2: performance optimizations #10916: device-side snapshot and restore optimizations (Mythreya Kuricheti). Its 9.4× BabelStream figure is the first number to re-measure with this harness.
feat(rocprofiler-sdk): kernel replay service (callback tracing) #7960: the kernel replay service this measures.

…harness The benchmark suite measures a single rocprofv3 invocation per job, which is enough for kernel replay and for multiplexed collection but cannot express application replay: that means running the whole application once per counter group and adding the runs up. Without it there is nothing to compare replay against, since application replay is the thing replay is meant to replace. Add a harness that drives rocprofv3 directly and measures all three ways of collecting the same groups, alongside pass-count scaling, footprint scaling and the cost replay imposes on dispatches that opt out. Every row carries the command that produced it so a number can be traced back. Measurements are a median over repeated runs taken after untimed warmup runs, matching the convention the CI performance tests already use, and peak RSS comes from os.wait4 so it describes the run being measured rather than the largest child reaped so far. Co-authored-by: Vivek Kale <vivek.kale@amd.com> (cherry picked from commit c8dfafb)

… benchmark database A kernel replay run and a non-replay run that request the same counters emit the same rocprofv3 config JSON, because rocprofv3 does not report replay there. The benchmark harness hashes that JSON to pick a benchmark_config row, so the two runs shared one row and their measurements were averaged together: the database could not express the comparison the replay work needs. Read the collection mode off the rocprofv3 command line instead and record it alongside the group count, so runs that differ only in how the groups are collected land in separate rows. Add a view that puts kernel replay and application replay on one axis, the cost of collecting every requested group, since application replay pays for a full application run per group while kernel replay pays for one run plus a pass per dispatch. Databases created before these columns existed are migrated on connect, because CREATE TABLE IF NOT EXISTS leaves them as they were and the insert would otherwise fail on the missing columns. Co-authored-by: Vivek Kale <vivek.kale@amd.com> (cherry picked from commit 12d3516)

… configuration The job named app_replay passed four --pmc flags without enabling replay, on the assumption that this makes rocprofv3 run the application once per group. It does not: the tool rotates the counter group every few dispatches within a single run, so no dispatch carries more than one group. Comparing replay against it compared a run that collects four groups everywhere against a run that collects one group anywhere, and replay was always going to look expensive. Rename it to what it measures and make the single-group run the reference for application replay, since collecting G groups that way is that run repeated G times. Add a sweep over how many dispatches the application enqueues between synchronizations. Kernel replay drains the agent per replayed dispatch, so how much a workload loses to that drain depends on how deep its queue usually is, and nothing measured that. It is also the axis range replay changes, so having kernel replay numbers along it now means the comparison later does not need a fresh baseline. Co-authored-by: Vivek Kale <vivek.kale@amd.com> (cherry picked from commit be06c4a)

… the alternatives The conceptual index listed several --pmc groups as application replay, which is what it looks like from the command line and not what the tool does with it. It rotates the group across dispatches instead, so it is the group rotation row of the same table, and the two rows contradicted each other on cost. Anyone benchmarking against that row was comparing a run that collects every group everywhere against a run that collects one group anywhere. Correct the table and add a page covering the three ways to collect several groups, which two of them are comparable and why, what the benchmark suite now records about a run, and the three things it still cannot express: the bytes the snapshot moves, wall time against GPU time, and how much of a run was declined rather than replayed. Co-authored-by: Vivek Kale <vivek.kale@amd.com> (cherry picked from commit e99a02f)

therock-pr-bot · 2026-10-01T04:20:41Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-01T04:20:43Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

Copilot review overview

🟡 Changes recommended

The harness and database currently misclassify rocprofv3 application replay as multiplexing and fail to enable kernel replay correctly.

Review effort: Balanced
Findings: 2 · 4 · 4

Open (10)

Application replay is mislabeled as multiplexing · New
CLI --pmc groups are application replay, not multiplexing · New
Schema omits default application-replay mode · New
Projection incorrectly multiplies non-additive metrics · New
Reference grouping omits benchmark mode · New
Test expects incorrect multiplexed classification · New
CLI groups misidentified as cheap multiplexed measurement · New
CLI guidance reverses application replay and multiplexing behavior · New
Documentation incorrectly describes CLI groups as multiplexing · New
Benchmark unnecessarily requires projection for application replay · New

What changed in this PR

Adds kernel-replay performance measurement, database bookkeeping, tests, and operator documentation.

Changes:

Adds an operator-run replay benchmark harness and tests.
Extends benchmark schemas, migrations, and comparison views.
Updates benchmark configurations and kernel-replay guidance.

File	Description
`benchmark/CMakeLists.txt`	Registers benchmark tests.
`benchmark/README.md`	Documents replay benchmarking.
`benchmark/kernel-replay.yaml`	Updates replay workloads and sweeps.
`benchmark/scripts/replay_perf.py`	Implements the measurement harness.
`benchmark/source/bin/rocprofv3-benchmark.py`	Derives and stores replay metadata.
`benchmark/source/share/rocprofiler-sdk/benchmark_tables.sql`	Adds replay columns.
`benchmark/source/share/rocprofiler-sdk/benchmark_views.sql`	Adds comparison views.
`benchmark/tests/CMakeLists.txt`	Registers Python tests.
`benchmark/tests/test_benchmark_replay.py`	Tests database bookkeeping.
`benchmark/tests/test_replay_perf.py`	Tests the harness.
`source/docs/_toc.yml.in`	Adds the guide to navigation.
`source/docs/conceptual/kernel_replay/index.md`	Updates the conceptual overview.
`source/docs/conceptual/kernel_replay/kernel_replay_benchmarking.md`	Adds benchmarking guidance.
`source/docs/conceptual/kernel_replay/kernel_replay_performance.md`	Links the new guide.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    # The same four groups without replay, which rotates them across dispatches instead of
+    # collecting them all everywhere. Not comparable to replay_4_groups on data collected, only on
+    # what the group machinery costs when nothing is replayed.
+    multiplex_4_groups:
      [--pmc, SQ_WAVES, --pmc, GRBM_COUNT, --pmc, SQ_INSTS_VALU, --pmc, SQ_BUSY_CYCLES]


+def _profile_cmd(args, groups, outdir, replay=False, extra=None):
+    return (
+        [args.rocprofv3]
+        + _pmc_args(groups)


+    SG.mean * BC.counter_group_count AS application_replay_projected,
+    CASE
+        WHEN ST.mean > 0 THEN (SG.mean * BC.counter_group_count) / ST.mean
+    END AS `speedup vs application replay`


+    LEFT JOIN single_group SG ON (
+        SG.app_id = ST.app_id
+        AND SG.sdk_id = ST.sdk_id
+    )


+    def test_several_pmc_groups_without_replay_are_multiplexed(self):
+        data = benchmark.derive_replay_config(
+            ["--pmc", "SQ_WAVES", "--pmc", "GRBM_COUNT", "--pmc", "FETCH_SIZE"]
+        )
+        self.assertEqual(data["counter_group_count"], 3)
+        self.assertEqual(data["counter_collection_mode"], "multiplexed")


+Passing several `--pmc` flags without `--kernel-replay-beta-enabled` is a third thing again: the
+group is rotated every few dispatches, so one run collects several groups but no dispatch carries
+all of them. It is measured here as a floor, not as a comparison, because it does not collect the
+same data.


+| Counter group rotation (multiple `--pmc` groups) | amortized across dispatches | none; different dispatches sample different groups | `O(1 ×` app runtime`)` |
+
+Only the first two collect every group on every dispatch, so only those two are comparable on the
+data they produce. Group rotation is cheaper because it collects less: passing several `--pmc` flags
+without enabling replay rotates the group every few dispatches rather than running the application
+again. See [Benchmarking](kernel_replay_benchmarking.md).


+**Multiplexed collection** is what `rocprofv3` does when it is given several `--pmc` flags and
+`--kernel-replay-beta-enabled` is not set. The tool counts dispatches and rotates the counter group
+every `ROCPROF_COUNTER_GROUPS_INTERVAL` of them, so one run yields all the groups but no single
+dispatch carries more than one. Where dispatches are homogeneous and numerous, that is a reasonable
+statistical trade; where the interesting kernel runs twice, it is not a measurement of that kernel


+A benchmark job is a single `rocprofv3` invocation, so application replay cannot be written as a
+job: it is G runs whose costs add up. The suite handles this by measuring one of those runs and
+projecting the rest.


amd-vkale · 2026-10-06T07:01:31Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

Co-authored-by: amd-vkale <291155109+amd-vkale@users.noreply.github.com>

Copilot · 2026-10-06T07:08:31Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply ...

Addressed the review feedback in commit 06ec0022.

cursoragent and others added 4 commits October 1, 2026 00:41

github-actions Bot added documentation Improvements or additions to documentation project: rocprofiler-sdk labels Oct 1, 2026

therock-pr-bot Bot added the Not ready to Review label Oct 1, 2026

systems-assistant Bot added the organization: ROCm label Oct 1, 2026

amd-vkale marked this pull request as ready for review October 1, 2026 05:25

amd-vkale requested review from a team as code owners October 1, 2026 05:25

Copilot AI balanced review requested due to automatic review settings October 1, 2026 05:25

Copilot started reviewing on behalf of amd-vkale October 1, 2026 05:26

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Copilot started work on behalf of amd-vkale October 6, 2026 07:02

View session

fix(rocprofiler-sdk): correct replay benchmark modes and projections

06ec002

Co-authored-by: amd-vkale <291155109+amd-vkale@users.noreply.github.com>

Copilot finished work on behalf of amd-vkale October 6, 2026 07:09

amd-vkale closed this Oct 6, 2026

amd-vkale mentioned this pull request Oct 6, 2026

test(rocprofiler-sdk): add a kernel replay benchmark harness and benchmarking guide #12885

Open

1 task done

Navigation Menu

Repository navigation

test(rocprofiler-sdk): add a kernel replay benchmark harness and benchmarking guide - #12604

test(rocprofiler-sdk): add a kernel replay benchmark harness and benchmarking guide#12604
amd-vkale wants to merge 5 commits into
developfrom
users/vkale/kr-perf-0-replay-benchmark-harness--with-cursor-help

amd-vkale commented Oct 1, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

amd-vkale commented Oct 6, 2026

Uh oh!

Copilot AI commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

amd-vkale commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Underlying Problem

Notes on Bigger Picture

Technical Details

Operator-run harness

Benchmark database bookkeeping

Benchmark configuration and docs

Reviewer Guide

Issue Tracking

Test Plan

Test Result

Submission Checklist

Related PRs

Uh oh!

therock-pr-bot Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

amd-vkale commented Oct 6, 2026

Uh oh!

Copilot AI commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amd-vkale commented Oct 1, 2026 •

edited

Loading

therock-pr-bot Bot commented Oct 1, 2026 •

edited

Loading