Repository navigation
feat(rocprofiler-sdk): WSL2 compute and profiling support for RDNA 3.5 - #7016
Conversation
rocprofiler_sdk_get_gfx_architectures() shells out to rocminfo at configure time to populate the per-GPU architecture list. In environments where rocminfo enumerates no GPU agent at configure time (WSL, where the GPU agent only becomes visible after the runtime loads; containers without /dev/kfd; cross-compile), the regex returns an empty list, which breaks ~8 downstream list(GET rocprofiler-sdk-tests-gfx-info 0 ...) call sites with "list GET given empty list". When rocminfo returns no gfx info and GPU_TARGETS is explicitly set, fall back to that. Behavior unchanged for native Linux + KFD setups where rocminfo correctly enumerates the GPU.
Add empty <gfx1150 base="gfx11"></gfx1150> entries to basic_counters.xml and derived_counters.xml. gfx1150 (RDNA 3.5; Strix Point integrated GPU) shares the gfx11 counter set, so inheriting from gfx11 makes SQ_WAVES / GRBM_COUNT / GL2C_HIT etc. resolvable via rocprofv3-avail without duplicating definitions. No new counters introduced; this is purely arch registration so the existing gfx11 counter set becomes available on gfx1150.
On the WSL synth-agent path the agent is populated from DXCore and
does not expose KFD topology attributes (array_count, simd_count,
wave_front_size, etc.) through the standard counter registry. The
AST loader fails to resolve these names when an AQL agent isn't
available, and downstream AQL queries trip CHECK_EQ / SIGABRT on
metrics that have no hardware block.
Make the three counter paths tolerate that gap:
evaluate_ast.cpp
At AST load time inject seven topology Metric stubs (array_count,
simd_count, wave_front_size, simd_arrays_per_engine,
cu_per_simd_array, simd_per_cu, max_waves_per_simd) with
synthetic IDs in the 0xFFFF0000ULL+ range. These resolve common
topology attribute names without an AQL agent.
dimensions.cpp
getBlockDimensions(): for synthetic IDs (or any metric with no
block / expression / constant) return a single-instance
dimension, avoiding CHECK_EQ in the AQL query path.
aql/helpers.cpp
get_query_info(): for synthetic IDs (or metrics with no block)
return an empty query result; aqlprofile_get_pmc_info would
otherwise SIGABRT dereferencing the null block name.
Transitional; should be revisited once libhsakmt-windows is
available to populate the real topology attributes on WSL.
The three hw-counter execute tests (rocprofv3-avail-test-hw-counters, -test-check-hw-counters, -test-check-hw-counters_2) were gated on IS_DISABLED, which disables ALL gfx10/11/12xx targets. That gate was originally for PC-sampling-disabled cases, but hw-counter enumeration and pmc-check do not depend on PC sampling support; they only require that the counters exist on the arch. On gfx1150 (RDNA 3.5) under WSL2, basic_counters.xml inherits from gfx11 (via the "counters: register gfx1150 ..." commit in this series) so SQ_WAVES / GRBM_COUNT / GL2C_HIT enumerate cleanly via rocprofv3-avail info --pmc; pmc-check on the same counters succeeds. Split out IS_HW_COUNTERS_DISABLED, currently only set for gfx906 and ROCPROFILER_MEMCHECK. Wire it into the three hw-counter execute tests; leave the pytest validate test on the original IS_DISABLED for safety. After this change on RDNA 3.5 / WSL2 / ROCm 6.4.2: rocprofv3-avail-test-hw-counters PASS rocprofv3-avail-test-check-hw-counters PASS rocprofv3-avail-test-check-hw-counters_2 PASS rocprofv3-avail-test-pc-sample-config Disabled (IS_PC_SAMPLING_DISABLED)
Add an opt-in path that populates the synthesized WSL agent's compute topology from the libhsakmt-windows (librocdxg) shim instead of the hardcoded per-arch fallback. fetch_libhsakmt_topology() opens KFD, acquires system properties, and matches the DXCore adapter to its KFD node by Windows LUID, then mirrors NumFComputeCores / NumArrays / NumShaderBanks / MaxEngineClockMhzFCompute / EngineId into the agent. Gated at build time by ROCPROFILER_USE_LIBHSAKMT_WINDOWS (requires the hsakmt headers and a built librocdxg) and at runtime by the ROCPROFILER_USE_LIBROCDXG env var; otherwise the hardcoded fallback is used and behavior is unchanged. Verified in-process on WSL2 (Radeon 890M / gfx1150): the matched KFD node yields gfx11.5.0, 32 FCompute, 1 shader bank, 2 arrays, wave32, 2900 MHz.
On platforms without a usable KFD device node (e.g. WSL2, which exposes /dev/dxg but not /dev/kfd), opening the KFD handle fails and the previous code aborted via ROCP_FATAL_IF in kfd_device_fd(), bringing down the whole rocprofv3 run. Add a kfd_device_available() probe and gate kfd::init() on it: when the device cannot be opened, log a warning and return ROCPROFILER_STATUS_SUCCESS instead of constructing kfd_device_fd. get_version() and poll_kfd_t / config::init are only reached after the probe, so the fatal path is fully gated. On hardware with a real /dev/kfd the probe succeeds and the original code path is taken unchanged (no behavior change on real GPUs).
1f466e4 to
a73ea21
Compare
a73ea21 to
444fe6d
Compare
4e6fe3f to
b95d690
Compare
|
question, just curious: will we be getting wsl2 runners to test this change? |
4da66b7 to
afdf5ac
Compare
afdf5ac to
fcee884
Compare
…rocprofv3-compute # Conflicts: # projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/platform/wsl/agent.cpp # projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/agent.cpp
72e44ae to
9873a7a
Compare
…rocprofv3-compute Two conflicts, both resolved semantically rather than textually. device_counting.cpp: kept the branch's SetUp() WSL/no-KFD skip and took develop's test_run(const device_counting_run_options&), dropping the branch's 4-argument test_run declaration. Develop re-supplies that signature as a wrapper below it, so keeping both would have declared the same signature twice. validate.py: took develop's body and dropped the branch's pandas and skip_if_hw_counter_values_unavailable imports. This is behavior-neutral: that helper calls pytest.fail in every branch rather than skipping -- commit c4ef285 on this branch deliberately converted it from skip to fail -- so develop's strict assertions fail in exactly the cases the helper did. pandas was already unused before the merge. The helper is still used by the extra_counters, input1, input2, input3 and kernel_filtering validate scripts, so nothing is orphaned.
rocr::atomic::Load() routes through __atomic_load(), whose _WIN32 implementation was InterlockedOr(object, 0). That reads the value back unchanged, but it is still a locked read-modify-write, so it requires the page to be writable. Memory the runtime only ever reads is not always writable: the WDDM monitored-fence value is a read-only CPU mapping, documented as such in d3dukmdt.h:1805 for D3DDDI_SYNCHRONIZATIONOBJECTINFO2::MonitoredFence and in d3dkmthk.h:5138 for D3DKMT_CREATEHWQUEUE. Querying a live adapter confirms it: the fence VA it returns reports PAGE_READONLY, a locked read-modify-write against that address raises 0xC0000005, and a plain volatile read of the same address succeeds. Load the value instead. A naturally aligned integer load is already atomic on the architectures built here, and volatile is what keeps the compiler from reusing a stale value. Keep the barrier the interlocked form supplied incidentally, so that no existing caller sees weaker ordering than before: relaxed loads have been receiving a full barrier whether or not they asked for one, and this is not the change in which to take that away. MemoryBarrier() goes before the load because the only reordering a lock prefix suppresses that x86-TSO still permits is an older store passing this load. Dropping the size dispatch also removes a narrower hazard: loads of 16-bit AQL packet headers were going through the 32-bit InterlockedOr, touching two bytes past the object.
LoadSyncValue() is correct as written now that the Windows atomic load no longer performs a read-modify-write, so the call site needs no code change. What it does need is the constraint written down: nothing near these spins says that sync_addr is the KMD's read-only CPU mapping of the monitored fence, or that an atomic read-modify-write against it raises 0xC0000005. Note it next to the load so that a later change reaching for a fetch-or or a compare-exchange here does not have to rediscover it from a crash.
The CI is now passing, and the MemoryTest1.exe issue has been resolved. |
|
I investigated the rocprofiler-sdk continuous integration CI failures that happens on MI325.The root cause seems to be that the SQ_WAVES values collected can have small couple of number variation at least on MI325 and values should be tested with small tolerance. I have patch ready for fixing this test, it's now in separate PR. |
…rocprofv3-compute Resolve the import conflict in counter-collection/input3/validate.py: take develop's removal of the unused pandas/os imports and keep this branch's hardware_counters skip helper.
Jonathan has approved the PR, and the team confirmed that his approval is sufficient to move forward with the merge.
|
Merging this now, existing tests that still fails seems to be flaky. rocprofiler-sdk related tests are failing for SQ_WAVES check which value can have little variation that test does not take into account and PR #12085 should fix that later. SQ_WAVES measures:
Variance is normal because:
|
TheRock Submodule Bump ActivityNewest first
|
…7016) ## Motivation Enable `rocprofv3` counter collection and profiling on WSL2 for gfx11-class GPUs. On WSL, the complete KMT topology is read up front through `librocdxg`, using the same thunk interface as the HSA runtime. An agent is published only after all required topology information has been collected, so the published record remains stable for its lifetime. As a result, `rocprofiler_query_available_agents()` returns consistent agent information regardless of when it is called, including before the HSA runtime has initialized. This is required by `rocprofv3-avail` and counter metric lookup. Native Linux continues to use the existing `gnulinux` path. `select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()` first, which succeeds whenever the KFD sysfs nodes directory is present, so the WSL path is not selected on a normal Linux system. There are two ungated changes that also affect native Linux. Both are fixes and are called out under **Native-Linux impact** below. This work builds on [[#6399](#6399)], which introduced the platform dispatch structure used here. `platform_dispatch.cpp` and its six tests came from that PR (`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**. This PR adds WSL as another implementation behind the existing `select_platform()` interface. ## Dependency on [#10034](#10034) [#10034](#10034) contains the libhsakmt and ROCr prerequisites for this work. There is no CI gate, submodule pin, or build-time dependency that enforces merge order, so the two PRs can technically merge in either order. Their changes to `core/runtime/amd_topology.cpp` also do not overlap. However, two parts of this PR depend on #10034 before they are fully functional. ### 1. WSL enumeration is disabled in a stock build until the required hsakmt headers are installed WSL enumeration is controlled by the CMake probe `rocprofiler_sdk_dxg_topology_supported()` (`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`). The probe requires the hsakmt headers used by the build to provide: * `HsaStructureSizes` * `HsaNodeProperties::LuidLowPart` * `HsaNodeProperties::LuidHighPart` * `OverrideEngineId` Released ROCm headers, including 7.2.4, do not provide all of these. When the probe fails, `agent_unsupported.cpp` is built instead of the WSL enumerator, `platform::wsl::is_available()` returns false, and `enumerate()` returns an empty result. The copies of `hsakmttypes.h` in this repository already contain the required definitions (lines 359-360 and 1578), but they are not currently installed. #10034 adds the WSL header installation rules in `libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`. Until that lands, a stock build of this PR will not enable WSL enumeration. ### 2. The DXG topology snapshot is not reference-counted on current `develop` Current `develop` unconditionally calls `topology_drop_snapshot()` from `hsaKmtReleaseSystemProperties()` (`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology snapshot and destroys its `WDDMDevice` objects even if another consumer still has the topology acquired. #10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot destruction conditional on the reference count. The normal `rocprofv3` path is not affected because the SDK topology read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before `main()`, and therefore before the application's `hsa_init()`. `rocprof-attach` is different because it injects into an already-running process. In that case there is no ordering guarantee between the SDK and the existing HSA runtime consumer, so late attach can fail until the reference-counting change from #10034 is present. ## Technical Details ### Agent enumeration (WSL) * `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology through `librocdxg` using: * `hsaKmtOpenKFD()` * `hsaKmtAcquireSystemProperties()` * `hsaKmtGetNodeProperties()` * `hsaKmtReleaseSystemProperties()` * `hsaKmtCloseKFD()` The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies that all five are available before any are called. * `HsaNodeProperties` has no size parameter, so the destination buffer includes `kNodePropsSlack = 128` bytes of additional space. * `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID, with device ID as a fallback. Enumeration is allowed to be partial: an unmatched adapter is skipped with a diagnostic while successfully matched GPUs are still published. * Memory banks, caches, and IO links are not currently published by the WSL path. * If the thunk reports no GPU node, no GPU agent is published. A warning is emitted and the application continues. * `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL logs mismatches, keeps successfully paired GPUs, and continues. ### Counters * `counters/controller.cpp` emits a one-time warning that WSL counters are collected through the DXG vendor-packet path rather than the KFD profiler ioctl. * `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of `is_available()`. This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old placement was bypassed. Without the environment variable, the DXG path drops aqlprofile's vendor-specific PM4 IB and counter values are returned as zero. * `basic_counters.xml` and `derived_counters.xml` each gain an empty `<gfx1150 base="gfx11">` section to keep the installed metric files consistent. These XML files are not used by the runtime metric loader. Runtime metrics come from `config.yaml` (`counters/metrics.cpp:348`), which already supports gfx1150 and is unchanged by this PR. ### KFD * `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is unavailable or cannot be opened, initialization continues with a warning instead of aborting in `kfd_device_fd()`. ## Native-Linux impact Two changes are not WSL-gated and also apply to native Linux: * `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for metrics that do not have a hardware block. Constant and topology metrics such as `simd_count` have an empty block name. Passing that empty name to `aqlprofile_get_pmc_info()` can dereference a null pointer and abort. * `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A system without a usable `/dev/kfd`, for example because the driver is not loaded or access is denied, now receives a warning and continues instead of aborting. Other rocprofiler services are unaffected. All other WSL-specific behavior remains isolated from native Linux. The DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4 command-buffer bounds change is limited to libhsakmt's DXG/WDDM submission path. ## Tests * **44 new tests** * 14 `wsl_dxg_topology` tests * 16 `wsl_dxg_thunk` tests * 10 `agent_mapping` tests * 4 `parse_gfx_target_version` tests The first two groups are in new executables. The mapping and target-version tests are added to the existing `rocprofiler-sdk-lib-tests`. None of these tests requires a GPU, HSA runtime, or real DXG thunk. The DXG tests use a fake thunk and fake dynamic loader. The 30 DXG-specific tests use the same CMake capability probe as the WSL enumerator, so they are only built when the required hsakmt interface is available. In a stock configuration, 14 of the 44 new tests are built. * `platform-dispatch-tests` and its six tests are pre-existing from #6399. * `rocprofv3-avail` * Three tests are moved out of the PC-sampling gate and are now enabled on Navi. * The L2 hit counter is selected by GPU family instead of being hardcoded. These tests have not previously run in Navi CI, so they may expose existing Navi-specific issues. * `hsa_barrier` * Skip when no GPU agent is available instead of dereferencing an empty agent map. * `agent` * `agent_visibility_multigpu` performs its `substr(1)` calls after the no-GPU skip, avoiding `std::out_of_range` when no agents are present. * `grid_max_size` tolerates the difference between rocprofiler's hardcoded `INT32_MAX` value and the value reported by the installed ROCr runtime. ## JIRA ID JIRA ID: ROCM-23912 ## Test Plan ## Test Result ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. --------- Co-authored-by: Mika Laitio <mika.laitio@amd.com>
…7016) ## Motivation Enable `rocprofv3` counter collection and profiling on WSL2 for gfx11-class GPUs. On WSL, the complete KMT topology is read up front through `librocdxg`, using the same thunk interface as the HSA runtime. An agent is published only after all required topology information has been collected, so the published record remains stable for its lifetime. As a result, `rocprofiler_query_available_agents()` returns consistent agent information regardless of when it is called, including before the HSA runtime has initialized. This is required by `rocprofv3-avail` and counter metric lookup. Native Linux continues to use the existing `gnulinux` path. `select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()` first, which succeeds whenever the KFD sysfs nodes directory is present, so the WSL path is not selected on a normal Linux system. There are two ungated changes that also affect native Linux. Both are fixes and are called out under **Native-Linux impact** below. This work builds on [[#6399](#6399)], which introduced the platform dispatch structure used here. `platform_dispatch.cpp` and its six tests came from that PR (`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**. This PR adds WSL as another implementation behind the existing `select_platform()` interface. ## Dependency on [#10034](#10034) [#10034](#10034) contains the libhsakmt and ROCr prerequisites for this work. There is no CI gate, submodule pin, or build-time dependency that enforces merge order, so the two PRs can technically merge in either order. Their changes to `core/runtime/amd_topology.cpp` also do not overlap. However, two parts of this PR depend on #10034 before they are fully functional. ### 1. WSL enumeration is disabled in a stock build until the required hsakmt headers are installed WSL enumeration is controlled by the CMake probe `rocprofiler_sdk_dxg_topology_supported()` (`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`). The probe requires the hsakmt headers used by the build to provide: * `HsaStructureSizes` * `HsaNodeProperties::LuidLowPart` * `HsaNodeProperties::LuidHighPart` * `OverrideEngineId` Released ROCm headers, including 7.2.4, do not provide all of these. When the probe fails, `agent_unsupported.cpp` is built instead of the WSL enumerator, `platform::wsl::is_available()` returns false, and `enumerate()` returns an empty result. The copies of `hsakmttypes.h` in this repository already contain the required definitions (lines 359-360 and 1578), but they are not currently installed. #10034 adds the WSL header installation rules in `libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`. Until that lands, a stock build of this PR will not enable WSL enumeration. ### 2. The DXG topology snapshot is not reference-counted on current `develop` Current `develop` unconditionally calls `topology_drop_snapshot()` from `hsaKmtReleaseSystemProperties()` (`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology snapshot and destroys its `WDDMDevice` objects even if another consumer still has the topology acquired. #10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot destruction conditional on the reference count. The normal `rocprofv3` path is not affected because the SDK topology read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before `main()`, and therefore before the application's `hsa_init()`. `rocprof-attach` is different because it injects into an already-running process. In that case there is no ordering guarantee between the SDK and the existing HSA runtime consumer, so late attach can fail until the reference-counting change from #10034 is present. ## Technical Details ### Agent enumeration (WSL) * `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology through `librocdxg` using: * `hsaKmtOpenKFD()` * `hsaKmtAcquireSystemProperties()` * `hsaKmtGetNodeProperties()` * `hsaKmtReleaseSystemProperties()` * `hsaKmtCloseKFD()` The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies that all five are available before any are called. * `HsaNodeProperties` has no size parameter, so the destination buffer includes `kNodePropsSlack = 128` bytes of additional space. * `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID, with device ID as a fallback. Enumeration is allowed to be partial: an unmatched adapter is skipped with a diagnostic while successfully matched GPUs are still published. * Memory banks, caches, and IO links are not currently published by the WSL path. * If the thunk reports no GPU node, no GPU agent is published. A warning is emitted and the application continues. * `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL logs mismatches, keeps successfully paired GPUs, and continues. ### Counters * `counters/controller.cpp` emits a one-time warning that WSL counters are collected through the DXG vendor-packet path rather than the KFD profiler ioctl. * `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of `is_available()`. This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old placement was bypassed. Without the environment variable, the DXG path drops aqlprofile's vendor-specific PM4 IB and counter values are returned as zero. * `basic_counters.xml` and `derived_counters.xml` each gain an empty `<gfx1150 base="gfx11">` section to keep the installed metric files consistent. These XML files are not used by the runtime metric loader. Runtime metrics come from `config.yaml` (`counters/metrics.cpp:348`), which already supports gfx1150 and is unchanged by this PR. ### KFD * `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is unavailable or cannot be opened, initialization continues with a warning instead of aborting in `kfd_device_fd()`. ## Native-Linux impact Two changes are not WSL-gated and also apply to native Linux: * `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for metrics that do not have a hardware block. Constant and topology metrics such as `simd_count` have an empty block name. Passing that empty name to `aqlprofile_get_pmc_info()` can dereference a null pointer and abort. * `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A system without a usable `/dev/kfd`, for example because the driver is not loaded or access is denied, now receives a warning and continues instead of aborting. Other rocprofiler services are unaffected. All other WSL-specific behavior remains isolated from native Linux. The DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4 command-buffer bounds change is limited to libhsakmt's DXG/WDDM submission path. ## Tests * **44 new tests** * 14 `wsl_dxg_topology` tests * 16 `wsl_dxg_thunk` tests * 10 `agent_mapping` tests * 4 `parse_gfx_target_version` tests The first two groups are in new executables. The mapping and target-version tests are added to the existing `rocprofiler-sdk-lib-tests`. None of these tests requires a GPU, HSA runtime, or real DXG thunk. The DXG tests use a fake thunk and fake dynamic loader. The 30 DXG-specific tests use the same CMake capability probe as the WSL enumerator, so they are only built when the required hsakmt interface is available. In a stock configuration, 14 of the 44 new tests are built. * `platform-dispatch-tests` and its six tests are pre-existing from #6399. * `rocprofv3-avail` * Three tests are moved out of the PC-sampling gate and are now enabled on Navi. * The L2 hit counter is selected by GPU family instead of being hardcoded. These tests have not previously run in Navi CI, so they may expose existing Navi-specific issues. * `hsa_barrier` * Skip when no GPU agent is available instead of dereferencing an empty agent map. * `agent` * `agent_visibility_multigpu` performs its `substr(1)` calls after the no-GPU skip, avoiding `std::out_of_range` when no agents are present. * `grid_max_size` tolerates the difference between rocprofiler's hardcoded `INT32_MAX` value and the value reported by the installed ROCr runtime. ## JIRA ID JIRA ID: ROCM-23912 ## Test Plan ## Test Result ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests. --------- Co-authored-by: Mika Laitio <mika.laitio@amd.com>
) ## Motivation Enable rocprofv3 thread trace (`--att`) support on WSL2, completing the ATT portion of the WSL2 enablement introduced in #7016. Previously, the ATT samples and tests only searched `${ROCM_PATH}` for `rocprof-trace-decoder`. As a result, separately built decoders could not be detected, causing the corresponding ATT tests to be skipped. ## Technical Details - **Shared decoder lookup**: Added a new `rocprofiler_sdk_find_trace_decoder()` macro to `rocprofiler-sdk-utilities.cmake`, replacing five duplicated `find_library` blocks across the ATT samples and tests. The lookup now searches `ROCPROFILER_SDK_TRACE_DECODER_ROOT` before `${ROCM_PATH}`. If `ROCPROFILER_SDK_TRACE_DECODER_ROOT` is not set, the existing behavior remains unchanged. - **`att-no-intercept` test gating**: This test requires multi-buffer thread trace support, which is currently available only on MI3xx and gfx12. On unsupported platforms, the test could time out. It now uses the existing `rocprofiler_sdk_sqtt_triple_buffer_disabled()` check to skip the test where multi-buffer support is unavailable. - **WSL2 documentation**: Added a **Thread trace (ATT)** section to the WSL2 how-to covering: - ATT decoder setup - HSA runtime path configuration when `/dev/kfd` is unavailable - The current `--selected-regions` limitation on WSL2 ATT has also been removed from the **Not covered on WSL2** section. No C++ changes are included in this PR. ## Issue Tracking JIRA ID: ROCM-23912 ## Test Plan WSL2, gfx1150 (no `/dev/kfd`), ROCr/librocdxg/SDK built from this branch, rocprof-trace-decoder 0.2.2 found via `ROCPROFILER_SDK_TRACE_DECODER_ROOT`. - `ctest -R 'att|thread-trace|thread_trace'` - `rocprofv3 --att --kernel-trace -- simple-transpose` ## Test Result - ctest: 165/171 passed; `thread-trace-sample` and rocattach 12/12 pass; `att-no-intercept` is now Disabled on gfx1150. - `rocprofv3 --att`: both dispatches captured and decoded. - 6 failures, unrelated to this change: 5 × `unit.thread_trace.*` caused by the ctest-baked `LD_LIBRARY_PATH` (they pass when run directly), and the `att-marker-trace` validate (`--selected-regions` gives an empty trace on WSL2, documented). ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.
Motivation
Enable rocprofv3 counter collection and profiling on WSL2, with support validated on Strix Point (gfx1150).
On WSL, the complete KMT topology is read up front through
librocdxg, using the same thunk interface as the HSA runtime. An agent is published only after all required topology information has been collected, so the published record remains stable for its lifetime.As a result,
rocprofiler_query_available_agents()returns consistent agent information regardless of when it is called, including before the HSA runtime has initialized. This is required byrocprofv3-availand counter metric lookup.Native Linux continues to use the existing
gnulinuxpath.select_platform()(agent.cpp:355) checksgnulinux::is_available()first, which succeeds whenever the KFD sysfs nodes directory is present, so the WSL path is not selected on a normal Linux system.There are two ungated changes that also affect native Linux. Both are fixes and are called out under Native-Linux impact below.
This work builds on [#6399], which introduced the platform dispatch structure used here.
platform_dispatch.cppand its six tests came from that PR (d2f0b27ad5) and are therefore listed as pre-existing under Tests. This PR adds WSL as another implementation behind the existingselect_platform()interface.Dependency on #10034
#10034 contains the libhsakmt and ROCr prerequisites for this work.
There is no CI gate, submodule pin, or build-time dependency that enforces merge order, so the two PRs can technically merge in either order. Their changes to
core/runtime/amd_topology.cppalso do not overlap. However, two parts of this PR depend on #10034 before they are fully functional.1. WSL enumeration is disabled in a stock build until the required hsakmt headers are installed
WSL enumeration is controlled by the CMake probe
rocprofiler_sdk_dxg_topology_supported()(cmake/Modules/rocprofiler-sdk-utilities.cmake:120).The probe requires the hsakmt headers used by the build to provide:
HsaStructureSizesHsaNodeProperties::LuidLowPartHsaNodeProperties::LuidHighPartOverrideEngineIdReleased ROCm headers, including 7.2.4, do not provide all of these. When the probe fails,
agent_unsupported.cppis built instead of the WSL enumerator,platform::wsl::is_available()returns false, andenumerate()returns an empty result.The copies of
hsakmttypes.hin this repository already contain the required definitions (lines 359-360 and 1578), but they are not currently installed. #10034 adds the WSL header installation rules inlibhsakmt/CMakeLists_wsl.txtthroughROCDXG_PUBLIC_HEADERS.Until that lands, a stock build of this PR will not enable WSL enumeration.
2. The DXG topology snapshot is not reference-counted on current
developCurrent
developunconditionally callstopology_drop_snapshot()fromhsaKmtReleaseSystemProperties()(libhsakmt/src/dxg/topology.cpp:417). This frees the global topology snapshot and destroys itsWDDMDeviceobjects even if another consumer still has the topology acquired.#10034 adds
src/dxg/snapshot_refcount.hppand makes snapshot destruction conditional on the reference count.The normal
rocprofv3path is not affected because the SDK topology read happens fromROCPROFILER_LIBRARY_CTORand completes beforemain(), and therefore before the application'shsa_init().rocprof-attachis different because it injects into an already-running process. In that case there is no ordering guarantee between the SDK and the existing HSA runtime consumer, so late attach can fail until the reference-counting change from #10034 is present.Technical Details
Agent enumeration (WSL)
platform/wsl/dxg_topology.cppanddxg_thunk.hppread KMT topology throughlibrocdxgusing:hsaKmtOpenKFD()hsaKmtAcquireSystemProperties()hsaKmtGetNodeProperties()hsaKmtReleaseSystemProperties()hsaKmtCloseKFD()The symbols are resolved with
dlsym.DxgThunk::complete()verifies that all five are available before any are called.HsaNodePropertieshas no size parameter, so the destination buffer includeskNodePropsSlack = 128bytes of additional space.platform/wsl/agent.cppmatches DXG adapters to HSA nodes by LUID, with device ID as a fallback. Enumeration is allowed to be partial: an unmatched adapter is skipped with a diagnostic while successfully matched GPUs are still published.Memory banks, caches, and IO links are not currently published by the WSL path.
If the thunk reports no GPU node, no GPU agent is published. A warning is emitted and the application continues.
agent_mapping.{hpp,cpp}adds a WSL-specific mapping policy. Native Linux keeps the existing strict behavior (agent.cpp:755), while WSL logs mismatches, keeps successfully paired GPUs, and continues.Counters
counters/controller.cppemits a one-time warning that WSL counters are collected through the DXG vendor-packet path rather than the KFD profiler ioctl.WSLKMT_VENDOR_PACKET=1is now set fromwsl::enumerate()instead ofis_available().This matters when using
ROCPROFILER_FORCE_PLATFORM=wsl, where the old placement was bypassed. Without the environment variable, the DXG path drops aqlprofile's vendor-specific PM4 IB and counter values are returned as zero.basic_counters.xmlandderived_counters.xmleach gain an empty<gfx1150 base="gfx11">section to keep the installed metric files consistent.These XML files are not used by the runtime metric loader. Runtime metrics come from
config.yaml(counters/metrics.cpp:348), which already supports gfx1150 and is unchanged by this PR.KFD
kfd::init()now probes for a usable KFD device. If/dev/kfdis unavailable or cannot be opened, initialization continues with a warning instead of aborting inkfd_device_fd().Native-Linux impact
Two changes are not WSL-gated and also apply to native Linux:
aql/helpers.cpp:51:get_query_info()returns an empty ID query for metrics that do not have a hardware block.Constant and topology metrics such as
simd_counthave an empty block name. Passing that empty name toaqlprofile_get_pmc_info()can dereference a null pointer and abort.kfd/kfd.cpp:1586: the KFD availability check is unconditional. A system without a usable/dev/kfd, for example because the driver is not loaded or access is denied, now receives a warning and continues instead of aborting.Other rocprofiler services are unaffected.
All other WSL-specific behavior remains isolated from native Linux. The DXG implementation requires
/dev/dxgandlibdxcore.so, and the PM4 command-buffer bounds change is limited to libhsakmt's DXG/WDDM submission path.Tests
44 new tests
wsl_dxg_topologytestswsl_dxg_thunktestsagent_mappingtestsparse_gfx_target_versiontestsThe first two groups are in new executables. The mapping and target-version tests are added to the existing
rocprofiler-sdk-lib-tests.None of these tests requires a GPU, HSA runtime, or real DXG thunk. The DXG tests use a fake thunk and fake dynamic loader.
The 30 DXG-specific tests use the same CMake capability probe as the WSL enumerator, so they are only built when the required hsakmt interface is available. In a stock configuration, 14 of the 44 new tests are built.
platform-dispatch-testsand its six tests are pre-existing from [rocprofiler-sdk] Platform specific topology information #6399.rocprofv3-availThese tests have not previously run in Navi CI, so they may expose existing Navi-specific issues.
hsa_barrieragentagent_visibility_multigpuperforms itssubstr(1)calls after the no-GPU skip, avoidingstd::out_of_rangewhen no agents are present.grid_max_sizetolerates the difference between rocprofiler's hardcodedINT32_MAXvalue and the value reported by the installed ROCr runtime.JIRA ID
JIRA ID: ROCM-23912
Test Plan
Test Result
Submission Checklist