Skip to content

[rocjitsu] Align CDNA4 MI350X configs with captured topology - #10274

Merged
kuhar merged 4 commits into
ROCm:developfrom
hanhanW:codex/cdna4-config-fidelity
Aug 19, 2026
Merged

kuhar merged 4 commits into
ROCm:developfrom
hanhanW:codex/cdna4-config-fidelity

Conversation

@hanhanW

@hanhanW hanhanW commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Take the updated topology and device properties from the checked-in MI350X KFD captures:

  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
    simd_count 1024
    array_count 32
    simd_arrays_per_engine 1
    cu_per_simd_array 9
    simd_per_cu 4
    device_id 30112
    num_sdma_engines 2
    num_sdma_xgmi_engines 14
    num_cp_queues 24
    max_engine_clk_fcompute 2200
    capability 2889327232
    capability2 1
    debug_prop 1494
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
    size_in_bytes 309220868096
    width 8192
    mem_clk_max 1900

The topology values describe 288 physical CU slots but only 256 active CUs:

  physical CUs = 32 arrays * 9 CUs/array = 288
  active CUs   = 1024 SIMDs / 4 SIMDs/CU = 256

With eight XCDs, the physical geometry becomes four shader engines and 36 CU slots per XCD:

  shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
  physical CUs/XCD   = 4 engines * 1 array/engine * 9 CUs/array = 36

The config's local_mem_size field represents advertised VRAM capacity in bytes, so set it from the memory-bank size_in_bytes value. rocminfo displays the same capacity in KB:

  309220868096 bytes / 1024 = 301973504 KB

Adjust topology stress workloads to cover the 288 instantiated CU slots and bound matrix workgroups beyond their 256 useful output tiles. Add sysfs and rocminfo regressions to prevent the physical topology and externally reported active-CU count from drifting apart.

rocJITsu does not currently model a harvested-CU mask, so all 288 physical slots remain schedulable even though rocminfo reports 256 active CUs.

Issue Tracking

N/A

Submission Checklist

@hanhanW
hanhanW requested review from IanWood1 and kuhar August 18, 2026 00:19
@hanhanW
hanhanW requested a review from atgutier as a code owner August 18, 2026 00:19
Copilot AI lite review requested due to automatic review settings August 18, 2026 00:19
@hanhanW
hanhanW requested a review from a team as a code owner August 18, 2026 00:19
@github-actions github-actions Bot added documentation Improvements or additions to documentation project: rocjitsu labels Aug 18, 2026
@therock-pr-bot

therock-pr-bot Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
📝 PR Description ❌ Fail Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword.
Expected: include a JIRA ID / ISSUE ID line (separator : or -, or omitted; value may be a JIRA key, a number with/without #, or a link), OR a closing keyword + issue reference. Accepted examples:
• JIRA ID : TESTAUTO-6039
• JIRA ID - #330
• JIRA ID #330
• JIRA ID (on separate line)
ROCM-25757
• ISSUE ID : TESTUTO-3334
• ISSUE ID #3334
• ISSUE ID - TESTAUTO-3433
• ISSUE ID (on separate line)
AIRUNTIME-2352
• ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234
• Closes #10
• Fixes octo-org/octo-repo#100
• Resolves: #123
• #123
• https://github.com/<org_name>/<repo_name>/issues/123
Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass —
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the MI350X (gfx950/CDNA4) emulation configs and related stress/regression tests to match the checked-in KFD capture: 288 physical CU slots (8 XCDs × 36) while reporting 256 active CUs via simd_count, and updates advertised VRAM size and key device properties accordingly.

Changes:

  • Update gfx950 CDNA4 JSON configs to the captured MI350X topology/device properties (36 CUs/XCD, memory size/clock, SDMA/CP queue counts, device_id, clocks).
  • Adjust stress tests (vector add, matmul, scaling) to dispatch across all 288 physical CU slots, adding bounds checks where needed.
  • Add sysfs + rocminfo regression tests to pin both physical topology geometry and externally reported active CU/clock values.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
emulation/rocjitsu/tests/vector_add_test.cpp Update dispatch sizing to 288 workgroups (36 CUs/XCD).
emulation/rocjitsu/tests/sysfs_test.cpp Add MI350X physical-vs-active CU regression + device property pinning.
emulation/rocjitsu/tests/scaling_test.cpp Update CU geometry constants to 36 CUs/XCD.
emulation/rocjitsu/tests/rocminfo_test.cpp Add rocminfo output regressions for active CU count and max clock.
emulation/rocjitsu/tests/race-detector/CMakeLists.txt Update documentation comment for 288 physical / 256 active CU context.
emulation/rocjitsu/tests/matmul_test.cpp Dispatch matmul stress across 288 physical CUs; clarify behavior for bounded extra groups.
emulation/rocjitsu/tests/kernels/vector_add.hip Update kernel docstring formatting; reflect 288-workgroup stress run.
emulation/rocjitsu/tests/kernels/matmul_tiled.hip Document 288-workgroup dispatch and bounded exits for extra groups.
emulation/rocjitsu/tests/kernels/matmul_mfma.hip Add bounds check for extra workgroups in 288-WG stress dispatch.
emulation/rocjitsu/tests/config_test.cpp Update expected MI350X physical geometry (9 CUs/SE) and note harvesting.
emulation/rocjitsu/tests/CMakeLists.txt Register new rocminfo regression tests.
emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/sysfs.h Update comment clarifying array_count derivation consistency.
emulation/rocjitsu/docs/configuration.md Update config example link indexing for 9 CUs/SE.
emulation/rocjitsu/configs/gfx950_cdna4.json Align standalone MI350X config to captured topology/device properties.
emulation/rocjitsu/configs/gfx950_cdna4_kmd.json Align KMD-mode MI350X config to captured topology/device properties.
emulation/rocjitsu/configs/gfx950_cdna4_kmd_2gpu.json Align multi-GPU KMD config to captured topology/device properties.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread emulation/rocjitsu/tests/sysfs_test.cpp Outdated
EXPECT_EQ(active_cus, 256u);

EXPECT_EQ(loaded.device.device_id, 30112u);
EXPECT_EQ(loaded.device.local_mem_size, 309220868096u);
@kuhar kuhar changed the title Align CDNA4 MI350X configs with captured topology. [rocjitsu] Align CDNA4 MI350X configs with captured topology Aug 18, 2026

@kuhar kuhar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some comments

"num_sdma_xgmi_engines": 12,
"num_cp_queues": 128,
"max_engine_clk_fcompute": 2400
"num_sdma_engines": 2,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should model and test num_sdma_queues_per_engine with the other captured SDMA properties. The checked-in node capture at projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties says 8, but launching this config currently emits 2, so the synthetic topology exposes 4 regular queues instead of 16. Please add the field through the device/config-to-sysfs path, set the MI350X configs to 8, and assert the generated property.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made num_sdma_queues_per_engine, and the current default is 2 because of legacy. I'll send a follow-up to fix all the configs for this field and we can remove the default.

I'm not doing it in this PR because the scope is mainly for mi350x.

"num_shader_arrays_per_engine": 2,
"num_cu_per_sh": 4,
"num_shader_arrays_per_engine": 1,
"num_cu_per_sh": 9,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we pin the DRM-visible value here and make cu_active_number follow the intended active-CU semantics? This geometry change makes array_count_per_xcc() 4 and the current helper returns 36, while KFD and rocminfo report 256 active CUs for the device. No CDNA config test covers that divergence. If 36 is intentionally a per-XCC contract, please document and test it; otherwise derive the value from simd_count / simd_per_cu.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated cu_active_number to use simd_count / simd_per_cu, so MI350X now reports 1024 / 4 = 256 active CUs through AMDGPU_INFO_DEV_INFO.

@hanhanW
hanhanW requested a review from kuhar August 18, 2026 20:58
@hanhanW
hanhanW force-pushed the codex/cdna4-config-fidelity branch from f5e659c to b4eecdb Compare August 18, 2026 21:09

@kuhar kuhar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'd just rename this config json to gfx950_mi355x.json, similar to the gfx1100/gfx1201 targets. @amd-arosa does this also need changes on the mirage side?

@hanhanW

hanhanW commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

LGTM, I'd just rename this config json to gfx950_mi355x.json, similar to the gfx1100/gfx1201 targets. @amd-arosa does this also need changes on the mirage side?

let me give it a shot and see if there are complains from CI or not.

@hanhanW
hanhanW requested a review from amd-arosa as a code owner August 19, 2026 04:46
Take the updated topology and device properties from the checked-in
MI350X KFD captures:

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
    simd_count 1024
    array_count 32
    simd_arrays_per_engine 1
    cu_per_simd_array 9
    simd_per_cu 4
    device_id 30112
    num_sdma_engines 2
    num_sdma_xgmi_engines 14
    num_cp_queues 24
    max_engine_clk_fcompute 2200
    capability 2889327232
    capability2 1
    debug_prop 1494
```

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
    size_in_bytes 309220868096
    width 8192
    mem_clk_max 1900
```

The topology values describe 288 physical CU slots but only 256 active
CUs:

```
  physical CUs = 32 arrays * 9 CUs/array = 288
  active CUs   = 1024 SIMDs / 4 SIMDs/CU = 256
```

With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:

```
  shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
  physical CUs/XCD   = 4 engines * 1 array/engine * 9 CUs/array = 36
```

The config's local_mem_size field represents advertised VRAM capacity
in bytes, so set it from the memory-bank size_in_bytes value.
rocminfo displays the same capacity in KB:

```
  309220868096 bytes / 1024 = 301973504 KB
```

Adjust topology stress workloads to cover the 288 instantiated CU
slots and bound matrix workgroups beyond their 256 useful output
tiles. Add sysfs and rocminfo regressions to prevent the physical
topology and externally reported active-CU count from drifting apart.

rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.

Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
@hanhanW
hanhanW force-pushed the codex/cdna4-config-fidelity branch from 71cdaca to 5cd42eb Compare August 19, 2026 17:38
@kuhar
kuhar merged commit b05a587 into ROCm:develop Aug 19, 2026
27 of 28 checks passed
@hanhanW
hanhanW deleted the codex/cdna4-config-fidelity branch August 19, 2026 18:24
atgutier pushed a commit that referenced this pull request Aug 19, 2026
Take the updated topology and device properties from the checked-in
MI350X KFD captures:

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
    simd_count 1024
    array_count 32
    simd_arrays_per_engine 1
    cu_per_simd_array 9
    simd_per_cu 4
    device_id 30112
    num_sdma_engines 2
    num_sdma_xgmi_engines 14
    num_cp_queues 24
    max_engine_clk_fcompute 2200
    capability 2889327232
    capability2 1
    debug_prop 1494
```

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
    size_in_bytes 309220868096
    width 8192
    mem_clk_max 1900
```

The topology values describe 288 physical CU slots but only 256 active
CUs:

```
  physical CUs = 32 arrays * 9 CUs/array = 288
  active CUs   = 1024 SIMDs / 4 SIMDs/CU = 256
```

With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:

```
  shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
  physical CUs/XCD   = 4 engines * 1 array/engine * 9 CUs/array = 36
```

The config's local_mem_size field represents advertised VRAM capacity in
bytes, so set it from the memory-bank size_in_bytes value. rocminfo
displays the same capacity in KB:

```
  309220868096 bytes / 1024 = 301973504 KB
```

Adjust topology stress workloads to cover the 288 instantiated CU slots
and bound matrix workgroups beyond their 256 useful output tiles. Add
sysfs and rocminfo regressions to prevent the physical topology and
externally reported active-CU count from drifting apart.

rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.

## Issue Tracking

N/A

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

---------

Signed-off-by: hanhanW <hanhan0912@gmail.com>
(cherry picked from commit b05a587)
atgutier added a commit that referenced this pull request Aug 20, 2026
…t calls (#10444)

## Motivation

hipTensor and rocThrust gfx1250 objects are refused at code-object load
time with "indirect branch or call target recovery is not implemented
for relocated kernel text" on `s_swap_pc_i64`. A refused object leaves
HIP holding an invalid kernel
handle, so the failure surfaces as a segfault rather than a diagnostic.
The fix is upstream as #10376; this backports it to
`release/bkc/therock-10.1-20260811`.

## Commits Cherry-Picked

  | # | Upstream | PR | Backport | Subject |
  |---|----------|-----|----------|---------|
| 1 | `b6d838028e` | #10197 | `846fb16d12` | [rocjitsu][docs] Refresh
architecture support documentation |
| 2 | `5c8e33e391` | #10200 | `db5c58b1be` | [rocjitsu] Define
standalone decoder ownership |
| 3 | `be1f91998f` | #10199 | `ed102baf1c` | [rocjitsu] Apply async
IOFFSET to LDS addresses |
| 4 | `fda635e5b4` | — | `ad32ff7c18` | [rocjitsu] Extract per-scope
rollback and object-level checks from translate() |
| 5 | `8f75eb781e` | — | `4c0cb05207` | [rocjitsu] Emit leading V_NOPs
ahead of a gfx1250 MODE write |
| 6 | `00aa1e260f` | #9941 | `ded43b1787` | fix(rocjitsu): disable
zero-count TDM descriptors |
| 7 | `c810647609` | #10192 | `50ca7f1732` | [rocjitsu] Apply MUBUF
LDS-load instruction offset to the LDS address |
| 8 | `1d7866269d` | #9560 | `2e6bf26943` | [rocjitsu] code rewrite
condition validator |
| 9 | `95b6c410bb` | #10175 | `8418901076` | [rocjitsu] Fix
ds_read_b64_tr_b16 lane layout in gfx950 simulator |
| 10 | `bfaebcfb5c` | #10240 | `c4eac1e1f7` | [rocjitsu][util] Add
result and diagnostic utilities |
| 11 | `dbb7221fa7` | #9775 | `cdd0993a5f` | fix(rocjitsu): route tensor
DMA through wave VMID |
| 12 | `2779247222` | #9779 | `7bfeba6f5d` | feat(rocjitsu): Lazy vgpr
allocation system |
| 13 | `a659172a31` | #9936 | `180cf00325` | fix(rocjitsu): honor TDM
tensor bounds across iterations |
| 14 | `60c29efb39` | #10204 | `370bfb9796` | [rocjitsu] Fix gfx950
decoder extension handling |
| 15 | `c1dba564e7` | #10345 | `536110a083` | [rocjitsu] Model gfx1250
wave ABI state |
| 16 | `7f71be9f32` | #10346 | `3f922ad5f2` | [rocjitsu] Model gfx1250
barrier synchronization |
| 17 | `bea7ac8937` | #10368 | `7b5ecf6c7b` | [rocjitsu] Replace decoder
exceptions with results |
| 18 | `f2e878694e` | #10371 | `8cf750d23f` | [rocjitsu] Dispatch
operand validation by type |
| 19 | `dd619bed77` | #9735 | `1a612c47c0` | fix(rocjitsu): respect
same-wave LDS ordering |
| 20 | `c5fae8d2b2` | #10137 | `4360c4b978` | [rocjitsu] Rename gfx1250
to cdna5 for enums, configs, amdisa codegen helpers and generated
symbols. (NFC) |
| 21 | `b05a587f89` | #10274 | `bfb5f9fab0` | [rocjitsu] Align CDNA4
MI350X configs with captured topology |
| 22 | `f81f9d40db` | #10362 | `8177b5725c` | feat(rocjitsu): cdna5 OOB
behavior implementation |
| 23 | `53a939bd00` | #10283 | `141b109432` | [rocjitsu] add per-XCD
workgroup dispatch observability |
| 24 | `a0fb11bc17` | #10414 | `6bb4c5a959` | [rocjitsu] Refactor test
encoding handling in execution tests (NFC) |
| 25 | `e5109e315e` | #9844 | `322ac26b44` | feat(emulation): debug
emulated GPU kernels with real ROCgdb |
| 26 | `efa24f23c9` | — | `c3c9641073` | rocjitsu: Point the XCD
distribution test at the config that exists |
| 27 | `8b7763ef5e` | #10376 | `c3f419555a` | [rocjitsu] Translate
gfx1250 lane-banked and kernarg-supplied indirect calls |

## Issue Tracking

[ROCM-29436]

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.


[ROCM-29346]:
https://amd-hub.atlassian.net/browse/ROCM-29346?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[ROCM-29436]:
https://amd-hub.atlassian.net/browse/ROCM-29436?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

---------

Signed-off-by: Tony Gutierrez <anthony.gutierrez@amd.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Co-authored-by: Jakub Kuderski <jakub@nod-labs.com>
Co-authored-by: James Newling <james.newling@gmail.com>
Co-authored-by: Fei Kou <feikou@meta.com>
Co-authored-by: Austin Lu <austin.lu@amd.com>
Co-authored-by: Fei Kou <feikou@anthropic.com>
Co-authored-by: Erick Ochoa Lopez <eochoalo@amd.com>
Co-authored-by: Han-Chung Wang <hanhan0912@gmail.com>
Co-authored-by: Rosa, Ada <arosa@amd.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
cmcknigh pushed a commit that referenced this pull request Aug 25, 2026
Take the updated topology and device properties from the checked-in
MI350X KFD captures:

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
    simd_count 1024
    array_count 32
    simd_arrays_per_engine 1
    cu_per_simd_array 9
    simd_per_cu 4
    device_id 30112
    num_sdma_engines 2
    num_sdma_xgmi_engines 14
    num_cp_queues 24
    max_engine_clk_fcompute 2200
    capability 2889327232
    capability2 1
    debug_prop 1494
```

```
  projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
    size_in_bytes 309220868096
    width 8192
    mem_clk_max 1900
```

The topology values describe 288 physical CU slots but only 256 active
CUs:

```
  physical CUs = 32 arrays * 9 CUs/array = 288
  active CUs   = 1024 SIMDs / 4 SIMDs/CU = 256
```

With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:

```
  shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
  physical CUs/XCD   = 4 engines * 1 array/engine * 9 CUs/array = 36
```

The config's local_mem_size field represents advertised VRAM capacity in
bytes, so set it from the memory-bank size_in_bytes value. rocminfo
displays the same capacity in KB:

```
  309220868096 bytes / 1024 = 301973504 KB
```

Adjust topology stress workloads to cover the 288 instantiated CU slots
and bound matrix workgroups beyond their 256 useful output tiles. Add
sysfs and rocminfo regressions to prevent the physical topology and
externally reported active-CU count from drifting apart.

rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.

## Issue Tracking

N/A

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

---------

Signed-off-by: hanhanW <hanhan0912@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants