Skip to content

[rocjitsu] add per-XCD workgroup dispatch observability - #10283

Merged
atgutier merged 1 commit into
developfrom
users/agutierr/xcd-dispatch-observability
Aug 19, 2026
Merged

atgutier merged 1 commit into
developfrom
users/agutierr/xcd-dispatch-observability

Conversation

@atgutier

Copy link
Copy Markdown
Contributor

A HW queue is bound to a single XCD by SoC::assign_queue_cp(), so every workgroup of every dispatch on that queue is placed by that one XCD's command processor on that one XCD's compute units. On the 8-XCD CDNA4 configs a single-queue application reaches 32 of the 256 CUs the topology advertises, but nothing in the tree measured this.

Count workgroups placed per command processor and expose the per-XCD histogram from the SoC, then pin current behavior in xcd_distribution_test: one queue confines the whole grid to one XCD, and only opening multiple queues reaches multiple XCDs. The follow-on fan-out work flips the first expectation.

Copilot AI lite review requested due to automatic review settings August 18, 2026 02:51
@atgutier
atgutier requested a review from a team as a code owner August 18, 2026 02:51
@therock-pr-bot

therock-pr-bot Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
📝 PR Description ❌ Fail Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword.
Expected: include a JIRA ID / ISSUE ID line (separator : or -, or omitted; value may be a JIRA key, a number with/without #, or a link), OR a closing keyword + issue reference. Accepted examples:
• JIRA ID : TESTAUTO-6039
• JIRA ID - #330
• JIRA ID #330
• JIRA ID (on separate line)
ROCM-25757
• ISSUE ID : TESTUTO-3334
• ISSUE ID #3334
• ISSUE ID - TESTAUTO-3433
• ISSUE ID (on separate line)
AIRUNTIME-2352
• ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234
• Closes #10
• Fixes octo-org/octo-repo#100
• Resolves: #123
• #123
• https://github.com/<org_name>/<repo_name>/issues/123
Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds observability for how workgroups are distributed across XCDs in rocjitsu’s multi-XCD SoC emulation, and introduces a regression test that pins current single-queue behavior (all workgroups land on one XCD) versus multi-queue behavior (queues rotate across XCDs).

Changes:

  • Track a per-command-processor cumulative count of dispatched workgroups and expose it via a new SoC::dispatched_workgroups_per_xcd() histogram helper.
  • Add xcd_distribution_test.cpp to validate current XCD placement behavior for one queue vs. multiple queues.
  • Wire the new test into the rocjitsu_tests CMake target.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
emulation/rocjitsu/tests/xcd_distribution_test.cpp New tests validating XCD workgroup distribution behavior.
emulation/rocjitsu/tests/CMakeLists.txt Adds the new test source to the test executable.
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/vm/soc.h Adds SoC-level per-XCD workgroup histogram accessor.
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/vm/amdgpu/command_processor.h Adds CP accessor and storage for cumulative workgroup dispatch count.
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/vm/amdgpu/command_processor.cpp Increments the new cumulative workgroup counter on each workgroup placement.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@atgutier
atgutier force-pushed the users/agutierr/xcd-dispatch-observability branch from f0b185a to f2c198a Compare August 18, 2026 03:56

@kuhar kuhar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to apply the documented RocJITsu style rules consistently in xcd_distribution_test.cpp: rename CONFIG_PATH to kConfigPath (line 37), make XcdDistributionFixture a class with an explicit public: section (line 46), and spell out std::vector<uint64_t> for both counts declarations (lines 96 and 126). These are direct requirements in emulation/rocjitsu/docs/style.md.

@atgutier
atgutier force-pushed the users/agutierr/xcd-dispatch-observability branch from f2c198a to e8683ef Compare August 18, 2026 23:34
A HW queue is bound to a single XCD by SoC::assign_queue_cp(), so every
workgroup of every dispatch on that queue is placed by that one XCD's command
processor on that one XCD's compute units. On the 8-XCD CDNA4 configs a
single-queue application reaches 32 of the 256 CUs the topology advertises,
but nothing in the tree measured this.

Count workgroups placed per command processor and expose the per-XCD histogram
from the SoC, then pin current behavior in xcd_distribution_test: one queue
confines the whole grid to one XCD, and only opening multiple queues reaches
multiple XCDs. The follow-on fan-out work flips the first expectation.
@atgutier
atgutier force-pushed the users/agutierr/xcd-dispatch-observability branch from e8683ef to 9f40910 Compare August 19, 2026 00:21
@atgutier
atgutier requested a review from kuhar August 19, 2026 00:39

@kuhar kuhar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@atgutier
atgutier merged commit 53a939b into develop Aug 19, 2026
235 of 247 checks passed
@atgutier
atgutier deleted the users/agutierr/xcd-dispatch-observability branch August 19, 2026 19:09
atgutier added a commit that referenced this pull request Aug 19, 2026
A HW queue is bound to a single XCD by SoC::assign_queue_cp(), so every
workgroup of every dispatch on that queue is placed by that one XCD's
command processor on that one XCD's compute units. On the 8-XCD CDNA4
configs a single-queue application reaches 32 of the 256 CUs the
topology advertises, but nothing in the tree measured this.

Count workgroups placed per command processor and expose the per-XCD
histogram from the SoC, then pin current behavior in
xcd_distribution_test: one queue confines the whole grid to one XCD, and
only opening multiple queues reaches multiple XCDs. The follow-on
fan-out work flips the first expectation.

(cherry picked from commit 53a939b)
atgutier added a commit that referenced this pull request Aug 20, 2026
…t calls (#10444)

## Motivation

hipTensor and rocThrust gfx1250 objects are refused at code-object load
time with "indirect branch or call target recovery is not implemented
for relocated kernel text" on `s_swap_pc_i64`. A refused object leaves
HIP holding an invalid kernel
handle, so the failure surfaces as a segfault rather than a diagnostic.
The fix is upstream as #10376; this backports it to
`release/bkc/therock-10.1-20260811`.

## Commits Cherry-Picked

  | # | Upstream | PR | Backport | Subject |
  |---|----------|-----|----------|---------|
| 1 | `b6d838028e` | #10197 | `846fb16d12` | [rocjitsu][docs] Refresh
architecture support documentation |
| 2 | `5c8e33e391` | #10200 | `db5c58b1be` | [rocjitsu] Define
standalone decoder ownership |
| 3 | `be1f91998f` | #10199 | `ed102baf1c` | [rocjitsu] Apply async
IOFFSET to LDS addresses |
| 4 | `fda635e5b4` | — | `ad32ff7c18` | [rocjitsu] Extract per-scope
rollback and object-level checks from translate() |
| 5 | `8f75eb781e` | — | `4c0cb05207` | [rocjitsu] Emit leading V_NOPs
ahead of a gfx1250 MODE write |
| 6 | `00aa1e260f` | #9941 | `ded43b1787` | fix(rocjitsu): disable
zero-count TDM descriptors |
| 7 | `c810647609` | #10192 | `50ca7f1732` | [rocjitsu] Apply MUBUF
LDS-load instruction offset to the LDS address |
| 8 | `1d7866269d` | #9560 | `2e6bf26943` | [rocjitsu] code rewrite
condition validator |
| 9 | `95b6c410bb` | #10175 | `8418901076` | [rocjitsu] Fix
ds_read_b64_tr_b16 lane layout in gfx950 simulator |
| 10 | `bfaebcfb5c` | #10240 | `c4eac1e1f7` | [rocjitsu][util] Add
result and diagnostic utilities |
| 11 | `dbb7221fa7` | #9775 | `cdd0993a5f` | fix(rocjitsu): route tensor
DMA through wave VMID |
| 12 | `2779247222` | #9779 | `7bfeba6f5d` | feat(rocjitsu): Lazy vgpr
allocation system |
| 13 | `a659172a31` | #9936 | `180cf00325` | fix(rocjitsu): honor TDM
tensor bounds across iterations |
| 14 | `60c29efb39` | #10204 | `370bfb9796` | [rocjitsu] Fix gfx950
decoder extension handling |
| 15 | `c1dba564e7` | #10345 | `536110a083` | [rocjitsu] Model gfx1250
wave ABI state |
| 16 | `7f71be9f32` | #10346 | `3f922ad5f2` | [rocjitsu] Model gfx1250
barrier synchronization |
| 17 | `bea7ac8937` | #10368 | `7b5ecf6c7b` | [rocjitsu] Replace decoder
exceptions with results |
| 18 | `f2e878694e` | #10371 | `8cf750d23f` | [rocjitsu] Dispatch
operand validation by type |
| 19 | `dd619bed77` | #9735 | `1a612c47c0` | fix(rocjitsu): respect
same-wave LDS ordering |
| 20 | `c5fae8d2b2` | #10137 | `4360c4b978` | [rocjitsu] Rename gfx1250
to cdna5 for enums, configs, amdisa codegen helpers and generated
symbols. (NFC) |
| 21 | `b05a587f89` | #10274 | `bfb5f9fab0` | [rocjitsu] Align CDNA4
MI350X configs with captured topology |
| 22 | `f81f9d40db` | #10362 | `8177b5725c` | feat(rocjitsu): cdna5 OOB
behavior implementation |
| 23 | `53a939bd00` | #10283 | `141b109432` | [rocjitsu] add per-XCD
workgroup dispatch observability |
| 24 | `a0fb11bc17` | #10414 | `6bb4c5a959` | [rocjitsu] Refactor test
encoding handling in execution tests (NFC) |
| 25 | `e5109e315e` | #9844 | `322ac26b44` | feat(emulation): debug
emulated GPU kernels with real ROCgdb |
| 26 | `efa24f23c9` | — | `c3c9641073` | rocjitsu: Point the XCD
distribution test at the config that exists |
| 27 | `8b7763ef5e` | #10376 | `c3f419555a` | [rocjitsu] Translate
gfx1250 lane-banked and kernarg-supplied indirect calls |

## Issue Tracking

[ROCM-29436]

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.


[ROCM-29346]:
https://amd-hub.atlassian.net/browse/ROCM-29346?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[ROCM-29436]:
https://amd-hub.atlassian.net/browse/ROCM-29436?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Signed-off-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Co-authored-by: Jakub Kuderski <jakub@nod-labs.com>
Co-authored-by: James Newling <james.newling@gmail.com>
Co-authored-by: Fei Kou <feikou@meta.com>
Co-authored-by: Austin Lu <austin.lu@amd.com>
Co-authored-by: Fei Kou <feikou@anthropic.com>
Co-authored-by: Erick Ochoa Lopez <eochoalo@amd.com>
Co-authored-by: Han-Chung Wang <hanhan0912@gmail.com>
Co-authored-by: Rosa, Ada <arosa@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
cmcknigh pushed a commit that referenced this pull request Aug 25, 2026
A HW queue is bound to a single XCD by SoC::assign_queue_cp(), so every
workgroup of every dispatch on that queue is placed by that one XCD's
command processor on that one XCD's compute units. On the 8-XCD CDNA4
configs a single-queue application reaches 32 of the 256 CUs the
topology advertises, but nothing in the tree measured this.

Count workgroups placed per command processor and expose the per-XCD
histogram from the SoC, then pin current behavior in
xcd_distribution_test: one queue confines the whole grid to one XCD, and
only opening multiple queues reaches multiple XCDs. The follow-on
fan-out work flips the first expectation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants