Repository navigation
[rocjitsu] Align CDNA4 MI350X configs with captured topology - #10274
Conversation
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
There was a problem hiding this comment.
Pull request overview
This PR updates the MI350X (gfx950/CDNA4) emulation configs and related stress/regression tests to match the checked-in KFD capture: 288 physical CU slots (8 XCDs × 36) while reporting 256 active CUs via simd_count, and updates advertised VRAM size and key device properties accordingly.
Changes:
- Update gfx950 CDNA4 JSON configs to the captured MI350X topology/device properties (36 CUs/XCD, memory size/clock, SDMA/CP queue counts, device_id, clocks).
- Adjust stress tests (vector add, matmul, scaling) to dispatch across all 288 physical CU slots, adding bounds checks where needed.
- Add sysfs + rocminfo regression tests to pin both physical topology geometry and externally reported active CU/clock values.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| emulation/rocjitsu/tests/vector_add_test.cpp | Update dispatch sizing to 288 workgroups (36 CUs/XCD). |
| emulation/rocjitsu/tests/sysfs_test.cpp | Add MI350X physical-vs-active CU regression + device property pinning. |
| emulation/rocjitsu/tests/scaling_test.cpp | Update CU geometry constants to 36 CUs/XCD. |
| emulation/rocjitsu/tests/rocminfo_test.cpp | Add rocminfo output regressions for active CU count and max clock. |
| emulation/rocjitsu/tests/race-detector/CMakeLists.txt | Update documentation comment for 288 physical / 256 active CU context. |
| emulation/rocjitsu/tests/matmul_test.cpp | Dispatch matmul stress across 288 physical CUs; clarify behavior for bounded extra groups. |
| emulation/rocjitsu/tests/kernels/vector_add.hip | Update kernel docstring formatting; reflect 288-workgroup stress run. |
| emulation/rocjitsu/tests/kernels/matmul_tiled.hip | Document 288-workgroup dispatch and bounded exits for extra groups. |
| emulation/rocjitsu/tests/kernels/matmul_mfma.hip | Add bounds check for extra workgroups in 288-WG stress dispatch. |
| emulation/rocjitsu/tests/config_test.cpp | Update expected MI350X physical geometry (9 CUs/SE) and note harvesting. |
| emulation/rocjitsu/tests/CMakeLists.txt | Register new rocminfo regression tests. |
| emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/sysfs.h | Update comment clarifying array_count derivation consistency. |
| emulation/rocjitsu/docs/configuration.md | Update config example link indexing for 9 CUs/SE. |
| emulation/rocjitsu/configs/gfx950_cdna4.json | Align standalone MI350X config to captured topology/device properties. |
| emulation/rocjitsu/configs/gfx950_cdna4_kmd.json | Align KMD-mode MI350X config to captured topology/device properties. |
| emulation/rocjitsu/configs/gfx950_cdna4_kmd_2gpu.json | Align multi-GPU KMD config to captured topology/device properties. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| EXPECT_EQ(active_cus, 256u); | ||
|
|
||
| EXPECT_EQ(loaded.device.device_id, 30112u); | ||
| EXPECT_EQ(loaded.device.local_mem_size, 309220868096u); |
| "num_sdma_xgmi_engines": 12, | ||
| "num_cp_queues": 128, | ||
| "max_engine_clk_fcompute": 2400 | ||
| "num_sdma_engines": 2, |
There was a problem hiding this comment.
We should model and test num_sdma_queues_per_engine with the other captured SDMA properties. The checked-in node capture at projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties says 8, but launching this config currently emits 2, so the synthetic topology exposes 4 regular queues instead of 16. Please add the field through the device/config-to-sysfs path, set the MI350X configs to 8, and assert the generated property.
There was a problem hiding this comment.
I made num_sdma_queues_per_engine, and the current default is 2 because of legacy. I'll send a follow-up to fix all the configs for this field and we can remove the default.
I'm not doing it in this PR because the scope is mainly for mi350x.
| "num_shader_arrays_per_engine": 2, | ||
| "num_cu_per_sh": 4, | ||
| "num_shader_arrays_per_engine": 1, | ||
| "num_cu_per_sh": 9, |
There was a problem hiding this comment.
Could we pin the DRM-visible value here and make cu_active_number follow the intended active-CU semantics? This geometry change makes array_count_per_xcc() 4 and the current helper returns 36, while KFD and rocminfo report 256 active CUs for the device. No CDNA config test covers that divergence. If 36 is intentionally a per-XCC contract, please document and test it; otherwise derive the value from simd_count / simd_per_cu.
There was a problem hiding this comment.
I updated cu_active_number to use simd_count / simd_per_cu, so MI350X now reports 1024 / 4 = 256 active CUs through AMDGPU_INFO_DEV_INFO.
f5e659c to
b4eecdb
Compare
kuhar
left a comment
There was a problem hiding this comment.
LGTM, I'd just rename this config json to gfx950_mi355x.json, similar to the gfx1100/gfx1201 targets. @amd-arosa does this also need changes on the mirage side?
let me give it a shot and see if there are complains from CI or not. |
Take the updated topology and device properties from the checked-in
MI350X KFD captures:
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
simd_count 1024
array_count 32
simd_arrays_per_engine 1
cu_per_simd_array 9
simd_per_cu 4
device_id 30112
num_sdma_engines 2
num_sdma_xgmi_engines 14
num_cp_queues 24
max_engine_clk_fcompute 2200
capability 2889327232
capability2 1
debug_prop 1494
```
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
size_in_bytes 309220868096
width 8192
mem_clk_max 1900
```
The topology values describe 288 physical CU slots but only 256 active
CUs:
```
physical CUs = 32 arrays * 9 CUs/array = 288
active CUs = 1024 SIMDs / 4 SIMDs/CU = 256
```
With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:
```
shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
physical CUs/XCD = 4 engines * 1 array/engine * 9 CUs/array = 36
```
The config's local_mem_size field represents advertised VRAM capacity
in bytes, so set it from the memory-bank size_in_bytes value.
rocminfo displays the same capacity in KB:
```
309220868096 bytes / 1024 = 301973504 KB
```
Adjust topology stress workloads to cover the 288 instantiated CU
slots and bound matrix workgroups beyond their 256 useful output
tiles. Add sysfs and rocminfo regressions to prevent the physical
topology and externally reported active-CU count from drifting apart.
rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Signed-off-by: hanhanW <hanhan0912@gmail.com>
71cdaca to
5cd42eb
Compare
Take the updated topology and device properties from the checked-in
MI350X KFD captures:
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
simd_count 1024
array_count 32
simd_arrays_per_engine 1
cu_per_simd_array 9
simd_per_cu 4
device_id 30112
num_sdma_engines 2
num_sdma_xgmi_engines 14
num_cp_queues 24
max_engine_clk_fcompute 2200
capability 2889327232
capability2 1
debug_prop 1494
```
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
size_in_bytes 309220868096
width 8192
mem_clk_max 1900
```
The topology values describe 288 physical CU slots but only 256 active
CUs:
```
physical CUs = 32 arrays * 9 CUs/array = 288
active CUs = 1024 SIMDs / 4 SIMDs/CU = 256
```
With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:
```
shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
physical CUs/XCD = 4 engines * 1 array/engine * 9 CUs/array = 36
```
The config's local_mem_size field represents advertised VRAM capacity in
bytes, so set it from the memory-bank size_in_bytes value. rocminfo
displays the same capacity in KB:
```
309220868096 bytes / 1024 = 301973504 KB
```
Adjust topology stress workloads to cover the 288 instantiated CU slots
and bound matrix workgroups beyond their 256 useful output tiles. Add
sysfs and rocminfo regressions to prevent the physical topology and
externally reported active-CU count from drifting apart.
rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.
## Issue Tracking
N/A
## Submission Checklist
- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.
---------
Signed-off-by: hanhanW <hanhan0912@gmail.com>
(cherry picked from commit b05a587)
…t calls (#10444) ## Motivation hipTensor and rocThrust gfx1250 objects are refused at code-object load time with "indirect branch or call target recovery is not implemented for relocated kernel text" on `s_swap_pc_i64`. A refused object leaves HIP holding an invalid kernel handle, so the failure surfaces as a segfault rather than a diagnostic. The fix is upstream as #10376; this backports it to `release/bkc/therock-10.1-20260811`. ## Commits Cherry-Picked | # | Upstream | PR | Backport | Subject | |---|----------|-----|----------|---------| | 1 | `b6d838028e` | #10197 | `846fb16d12` | [rocjitsu][docs] Refresh architecture support documentation | | 2 | `5c8e33e391` | #10200 | `db5c58b1be` | [rocjitsu] Define standalone decoder ownership | | 3 | `be1f91998f` | #10199 | `ed102baf1c` | [rocjitsu] Apply async IOFFSET to LDS addresses | | 4 | `fda635e5b4` | — | `ad32ff7c18` | [rocjitsu] Extract per-scope rollback and object-level checks from translate() | | 5 | `8f75eb781e` | — | `4c0cb05207` | [rocjitsu] Emit leading V_NOPs ahead of a gfx1250 MODE write | | 6 | `00aa1e260f` | #9941 | `ded43b1787` | fix(rocjitsu): disable zero-count TDM descriptors | | 7 | `c810647609` | #10192 | `50ca7f1732` | [rocjitsu] Apply MUBUF LDS-load instruction offset to the LDS address | | 8 | `1d7866269d` | #9560 | `2e6bf26943` | [rocjitsu] code rewrite condition validator | | 9 | `95b6c410bb` | #10175 | `8418901076` | [rocjitsu] Fix ds_read_b64_tr_b16 lane layout in gfx950 simulator | | 10 | `bfaebcfb5c` | #10240 | `c4eac1e1f7` | [rocjitsu][util] Add result and diagnostic utilities | | 11 | `dbb7221fa7` | #9775 | `cdd0993a5f` | fix(rocjitsu): route tensor DMA through wave VMID | | 12 | `2779247222` | #9779 | `7bfeba6f5d` | feat(rocjitsu): Lazy vgpr allocation system | | 13 | `a659172a31` | #9936 | `180cf00325` | fix(rocjitsu): honor TDM tensor bounds across iterations | | 14 | `60c29efb39` | #10204 | `370bfb9796` | [rocjitsu] Fix gfx950 decoder extension handling | | 15 | `c1dba564e7` | #10345 | `536110a083` | [rocjitsu] Model gfx1250 wave ABI state | | 16 | `7f71be9f32` | #10346 | `3f922ad5f2` | [rocjitsu] Model gfx1250 barrier synchronization | | 17 | `bea7ac8937` | #10368 | `7b5ecf6c7b` | [rocjitsu] Replace decoder exceptions with results | | 18 | `f2e878694e` | #10371 | `8cf750d23f` | [rocjitsu] Dispatch operand validation by type | | 19 | `dd619bed77` | #9735 | `1a612c47c0` | fix(rocjitsu): respect same-wave LDS ordering | | 20 | `c5fae8d2b2` | #10137 | `4360c4b978` | [rocjitsu] Rename gfx1250 to cdna5 for enums, configs, amdisa codegen helpers and generated symbols. (NFC) | | 21 | `b05a587f89` | #10274 | `bfb5f9fab0` | [rocjitsu] Align CDNA4 MI350X configs with captured topology | | 22 | `f81f9d40db` | #10362 | `8177b5725c` | feat(rocjitsu): cdna5 OOB behavior implementation | | 23 | `53a939bd00` | #10283 | `141b109432` | [rocjitsu] add per-XCD workgroup dispatch observability | | 24 | `a0fb11bc17` | #10414 | `6bb4c5a959` | [rocjitsu] Refactor test encoding handling in execution tests (NFC) | | 25 | `e5109e315e` | #9844 | `322ac26b44` | feat(emulation): debug emulated GPU kernels with real ROCgdb | | 26 | `efa24f23c9` | — | `c3c9641073` | rocjitsu: Point the XCD distribution test at the config that exists | | 27 | `8b7763ef5e` | #10376 | `c3f419555a` | [rocjitsu] Translate gfx1250 lane-banked and kernarg-supplied indirect calls | ## Issue Tracking [ROCM-29436] ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. [ROCM-29346]: https://amd-hub.atlassian.net/browse/ROCM-29346?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ [ROCM-29436]: https://amd-hub.atlassian.net/browse/ROCM-29436?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Signed-off-by: Tony Gutierrez <anthony.gutierrez@amd.com> Signed-off-by: hanhanW <hanhan0912@gmail.com> Co-authored-by: Jakub Kuderski <jakub@nod-labs.com> Co-authored-by: James Newling <james.newling@gmail.com> Co-authored-by: Fei Kou <feikou@meta.com> Co-authored-by: Austin Lu <austin.lu@amd.com> Co-authored-by: Fei Kou <feikou@anthropic.com> Co-authored-by: Erick Ochoa Lopez <eochoalo@amd.com> Co-authored-by: Han-Chung Wang <hanhan0912@gmail.com> Co-authored-by: Rosa, Ada <arosa@amd.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Take the updated topology and device properties from the checked-in
MI350X KFD captures:
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/properties:
simd_count 1024
array_count 32
simd_arrays_per_engine 1
cu_per_simd_array 9
simd_per_cu 4
device_id 30112
num_sdma_engines 2
num_sdma_xgmi_engines 14
num_cp_queues 24
max_engine_clk_fcompute 2200
capability 2889327232
capability2 1
debug_prop 1494
```
```
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/data/topology/nodes/5/mem_banks/0/properties:
size_in_bytes 309220868096
width 8192
mem_clk_max 1900
```
The topology values describe 288 physical CU slots but only 256 active
CUs:
```
physical CUs = 32 arrays * 9 CUs/array = 288
active CUs = 1024 SIMDs / 4 SIMDs/CU = 256
```
With eight XCDs, the physical geometry becomes four shader engines and
36 CU slots per XCD:
```
shader engines/XCD = (32 arrays / 1 array/engine) / 8 XCDs = 4
physical CUs/XCD = 4 engines * 1 array/engine * 9 CUs/array = 36
```
The config's local_mem_size field represents advertised VRAM capacity in
bytes, so set it from the memory-bank size_in_bytes value. rocminfo
displays the same capacity in KB:
```
309220868096 bytes / 1024 = 301973504 KB
```
Adjust topology stress workloads to cover the 288 instantiated CU slots
and bound matrix workgroups beyond their 256 useful output tiles. Add
sysfs and rocminfo regressions to prevent the physical topology and
externally reported active-CU count from drifting apart.
rocJITsu does not currently model a harvested-CU mask, so all 288
physical slots remain schedulable even though rocminfo reports 256
active CUs.
## Issue Tracking
N/A
## Submission Checklist
- [x] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.
---------
Signed-off-by: hanhanW <hanhan0912@gmail.com>
Take the updated topology and device properties from the checked-in MI350X KFD captures:
The topology values describe 288 physical CU slots but only 256 active CUs:
With eight XCDs, the physical geometry becomes four shader engines and 36 CU slots per XCD:
The config's local_mem_size field represents advertised VRAM capacity in bytes, so set it from the memory-bank size_in_bytes value. rocminfo displays the same capacity in KB:
Adjust topology stress workloads to cover the 288 instantiated CU slots and bound matrix workgroups beyond their 256 useful output tiles. Add sysfs and rocminfo regressions to prevent the physical topology and externally reported active-CU count from drifting apart.
rocJITsu does not currently model a harvested-CU mask, so all 288 physical slots remain schedulable even though rocminfo reports 256 active CUs.
Issue Tracking
N/A
Submission Checklist