feat(rocdbgapi): introduce clusters of workgroups by aktemur · Pull Request #7992 · ROCm/rocm-systems

aktemur · 2026-06-30T05:37:55Z

Motivation

Add a new level of hierarchy, "clusters", to the layering of wave -> workgroup -> cluster -> dispatch.

Technical Details

Define clusters as a new layer, similar to how workgroups are defined.

JIRA ID: AIROCGDB-77

Test Plan

To be done covered by GDB tests.

Test Result

Not yet available.

therock-pr-bot · 2026-06-30T05:38:44Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	⚠️ Warning	Error: Source/code files changed without an accompanying unit test. Expected: add at least one test file named like `test_<name>.py` / `test_<name>.cpp` (or `<name>_test.`). Current:* code file(s) changed: `projects/rocdbgapi/src/architecture.cpp`, `projects/rocdbgapi/src/architecture.h`, `projects/rocdbgapi/src/cluster.cpp`, `projects/rocdbgapi/src/cluster.h`, `projects/rocdbgapi/src/dispatch.cpp` (+11 more); no test file found
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-06-30T05:38:45Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

lancesix

A first quick pass, this looks promising already, thanks!

In the commit msg for "rocdbgapi: create clusters of workgroups"

"We have to consider take the cluster id" either "consider" or "take" is too much.

aktemur · 2026-07-03T10:06:54Z

The update above has two major differences wrt to the previous revision:

Use first-of-cluster/last-of-cluster to determine if a wave is in a cluster.
Define an API to query cluster sizes (similar to workgroup sizes and grid sizes).

aktemur · 2026-07-07T13:37:31Z

I addressed some of Lancelot's comments. There are still a few opens that need to be clarified.

lancesix

(review for patch 3)

lancesix

(patch 4)

aktemur · 2026-07-16T12:23:17Z

The latest push is only a rebase.

aktemur · 2026-07-16T13:37:33Z

The update above addresses the comments received so far.

lancesix

I think we are almost there.

aktemur · 2026-08-07T20:48:32Z

@lancesix, the last force-push updates the CLUSTER_SIZES query. Please check.

aktemur · 2026-08-12T14:10:43Z

The force-push above introduces queries to get workgroup ids within the cluster.

In two places, we get a workgroup's ids by calling `group_ids ()`, which returns an optional tuple, and if the optional is non-empty, we call `group_ids ()` again. Avoid the duplicated call.

A workgroup is alive as long as it has a wave. Set the epoch mark unconditionally, whenever the mark of any wave of the workgroup is being set. In practice, this should not change the epoch value of a workgroup. This change is a refactoring to obtain clearer and simpler code.

Starting with gfx12-5, workgroups may belong to clusters. Add methods to read cluster ids from a CWSR record. The cluster and group ids are read from TTMP registers. The meaning of the registers depend on whether the wave is in cluster mode. If not in a cluster, group ids are obtained from TTMP7 and TTMP9 as in previous gfx12 architecture. If in a cluster, what we read from TTMP7 and TTMP9 are cluster ids. The group ids in this case are read from TTMP6. To figure out if the wave is in cluster mode, we keep track of the "first_of_cluster" and "last_of_cluster" bits in the compute relaunch wave register. All CWSR records created for waves between the first and the last of cluster (inclusive) are marked to be in a cluster. Bug: AIROCGDB-77

Add a first definition for clusters, which are groups of workgroups. Bug: AIROCGDB-77

Starting with gfx12-5, workgroups may be grouped into clusters. When creating workgroups, take this into account. Note that being part of a cluster is not mandatory. In that case, the cluster id would be std::nullopt. We nevertheless create clusters and associate workgroups to them, to avoid having to handle non-cluster mode separately. In cluster mode, a cluster id would be a valid 3D coordinate. In this case, a workgroup's group id is not necessarily unique. We have to take also the cluster id into account to identify workgroups uniquely. This patch adds clusters into the wave -> workgroup -> cluster -> dispatch hierarchy. API queries are added in the next patch. Bug: AIROCGDB-77

Define new enums in the API to query the cluster coordinate of a wave and a workgroup, and other cluster-related information. Bug: AIROCGDB-77

Add API query to obtain cluster sizes. To be in consistence with the existing queries that return workgroup sizes and grid sizes (i.e. AMD_DBGAPI_DISPATCH_INFO_WORKGROUP_SIZES and AMD_DBGAPI_DISPATCH_INFO_GRID_SIZES), the new query (i.e. AMD_DBGAPI_DISPATCH_INFO_CLUSTER_SIZES) returns the sizes in units of work-items. Bug: AIROCGDB-77

aktemur · 2026-08-17T11:51:01Z

The latest push above is only a rebase.

…uster Add a workgroup query and a wave query to get the workgroup coordinates within the cluster the workgroup belongs to.

aktemur · 2026-08-17T11:54:35Z

The push above removes two more "... in units of ..." sentences.

github-actions · 2026-09-12T00:21:34Z

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

Add new commits
Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

aktemur · 2026-09-14T10:22:08Z

This PR should remain open as a reference to #10241.

aktemur · 2026-09-29T15:22:00Z

The PR that replaces this PR has been merged: #10241
Closing.

aktemur requested a review from a team as a code owner June 30, 2026 05:37

github-actions Bot added the project: rocdbgapi label Jun 30, 2026

therock-pr-bot Bot added the Not ready to Review label Jun 30, 2026

aktemur commented Jun 30, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.cpp Resolved

systems-assistant Bot added the organization: ROCm label Jun 30, 2026

aktemur changed the title ~~rocdbgapi: introduce clusters of workgroups~~ feat(rocdbgapi): introduce clusters of workgroups Jun 30, 2026

lancesix reviewed Jun 30, 2026

View reviewed changes

aktemur removed the Not ready to Review label Jun 30, 2026

aktemur force-pushed the users/aktemur/cluster branch from c6dff6d to 5c4c116 Compare July 3, 2026 10:02

therock-pr-bot Bot added the Not ready to Review label Jul 3, 2026

aktemur assigned lancesix Jul 3, 2026

lancesix reviewed Jul 7, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 5c4c116 to 6c615bc Compare July 7, 2026 13:29

aktemur force-pushed the users/aktemur/cluster branch 2 times, most recently from 8fd0911 to e9ea232 Compare July 16, 2026 07:14

aktemur requested a review from lancesix July 16, 2026 07:17

lancesix reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/architecture.cpp Resolved

Comment thread projects/rocdbgapi/src/architecture.cpp Resolved

Comment thread projects/rocdbgapi/src/architecture.cpp Resolved

lancesix reviewed Jul 16, 2026

View reviewed changes

Comment thread projects/rocdbgapi/src/cluster.h Resolved

lumachad reviewed Jul 16, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from e9ea232 to 5ccf7d9 Compare July 16, 2026 12:16

aktemur assigned aktemur and unassigned lancesix Jul 16, 2026

aktemur force-pushed the users/aktemur/cluster branch from 5ccf7d9 to d171615 Compare July 16, 2026 13:36

aktemur assigned lancesix Jul 16, 2026

aktemur force-pushed the users/aktemur/cluster branch from 972adda to 62cfcaf Compare August 3, 2026 11:08

aktemur assigned lancesix and unassigned aktemur Aug 3, 2026

lancesix reviewed Aug 5, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 62cfcaf to 60892d9 Compare August 7, 2026 20:46

aktemur force-pushed the users/aktemur/cluster branch from 60892d9 to 42d3a6e Compare August 12, 2026 14:09

lancesix reviewed Aug 17, 2026

View reviewed changes

aktemur force-pushed the users/aktemur/cluster branch from 42d3a6e to 18c37dd Compare August 17, 2026 11:19

aktemur added 8 commits August 17, 2026 06:50

doc(rocdbgapi): fix a documentation typo

08ede9b

refactor(rocdbgapi): avoid calling workgroup::group_ids () twice

d2df2a8

In two places, we get a workgroup's ids by calling `group_ids ()`, which returns an optional tuple, and if the optional is non-empty, we call `group_ids ()` again. Avoid the duplicated call.

feature(rocdbgapi): add cluster.h and cluster.cpp

3e5e389

Add a first definition for clusters, which are groups of workgroups. Bug: AIROCGDB-77

feature(rocdbgapi): add API definitions to query cluster information

88e1e25

Define new enums in the API to query the cluster coordinate of a wave and a workgroup, and other cluster-related information. Bug: AIROCGDB-77

aktemur force-pushed the users/aktemur/cluster branch from 18c37dd to 95fde49 Compare August 17, 2026 11:50

feature(rocdbgapi): add API definition to query workgroup coord in cl…

5a105b3

…uster Add a workgroup query and a wave query to get the workgroup coordinates within the cluster the workgroup belongs to.

aktemur force-pushed the users/aktemur/cluster branch from 95fde49 to 5a105b3 Compare August 17, 2026 11:54

aktemur mentioned this pull request Aug 17, 2026

feat(rocdbgapi): introduce clusters of workgroups (alternative) #10241

Merged

github-actions Bot added the Stale PR has no activity for 25+ days label Sep 12, 2026

aktemur removed the Stale PR has no activity for 25+ days label Sep 14, 2026

aktemur closed this Sep 29, 2026

Conversation

aktemur commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Uh oh!

therock-pr-bot Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ All Policy Checks Passed

Uh oh!

therock-pr-bot Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 7, 2026

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jul 16, 2026

Uh oh!

aktemur commented Jul 16, 2026

Uh oh!

lancesix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Aug 7, 2026

Uh oh!

aktemur commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aktemur commented Jun 30, 2026 •

edited

Loading

therock-pr-bot Bot commented Jun 30, 2026 •

edited

Loading

therock-pr-bot Bot commented Jun 30, 2026 •

edited

Loading