Skip to content

feat(rocprofiler-sdk): WSL2 compute and profiling support for RDNA 3.5 - #7016

Merged
lamikr merged 100 commits into
developfrom
users/ihhuang/wsl-rocprofv3-compute
Sep 24, 2026
Merged

lamikr merged 100 commits into
developfrom
users/ihhuang/wsl-rocprofv3-compute

Conversation

@ihhethan

@ihhethan ihhethan commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Enable rocprofv3 counter collection and profiling on WSL2, with support validated on Strix Point (gfx1150).

On WSL, the complete KMT topology is read up front through librocdxg, using the same thunk interface as the HSA runtime. An agent is published only after all required topology information has been collected, so the published record remains stable for its lifetime.

As a result, rocprofiler_query_available_agents() returns consistent agent information regardless of when it is called, including before the HSA runtime has initialized. This is required by rocprofv3-avail and counter metric lookup.

Native Linux continues to use the existing gnulinux path. select_platform() (agent.cpp:355) checks gnulinux::is_available() first, which succeeds whenever the KFD sysfs nodes directory is present, so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are fixes and are called out under Native-Linux impact below.

This work builds on [#6399], which introduced the platform dispatch structure used here. platform_dispatch.cpp and its six tests came from that PR (d2f0b27ad5) and are therefore listed as pre-existing under Tests. This PR adds WSL as another implementation behind the existing select_platform() interface.

Dependency on #10034

#10034 contains the libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that enforces merge order, so the two PRs can technically merge in either order. Their changes to core/runtime/amd_topology.cpp also do not overlap. However, two parts of this PR depend on #10034 before they are fully functional.

1. WSL enumeration is disabled in a stock build until the required hsakmt headers are installed

WSL enumeration is controlled by the CMake probe rocprofiler_sdk_dxg_topology_supported() (cmake/Modules/rocprofiler-sdk-utilities.cmake:120).

The probe requires the hsakmt headers used by the build to provide:

  • HsaStructureSizes
  • HsaNodeProperties::LuidLowPart
  • HsaNodeProperties::LuidHighPart
  • OverrideEngineId

Released ROCm headers, including 7.2.4, do not provide all of these. When the probe fails, agent_unsupported.cpp is built instead of the WSL enumerator, platform::wsl::is_available() returns false, and enumerate() returns an empty result.

The copies of hsakmttypes.h in this repository already contain the required definitions (lines 359-360 and 1578), but they are not currently installed. #10034 adds the WSL header installation rules in libhsakmt/CMakeLists_wsl.txt through ROCDXG_PUBLIC_HEADERS.

Until that lands, a stock build of this PR will not enable WSL enumeration.

2. The DXG topology snapshot is not reference-counted on current develop

Current develop unconditionally calls topology_drop_snapshot() from hsaKmtReleaseSystemProperties() (libhsakmt/src/dxg/topology.cpp:417). This frees the global topology snapshot and destroys its WDDMDevice objects even if another consumer still has the topology acquired.

#10034 adds src/dxg/snapshot_refcount.hpp and makes snapshot destruction conditional on the reference count.

The normal rocprofv3 path is not affected because the SDK topology read happens from ROCPROFILER_LIBRARY_CTOR and completes before main(), and therefore before the application's hsa_init().

rocprof-attach is different because it injects into an already-running process. In that case there is no ordering guarantee between the SDK and the existing HSA runtime consumer, so late attach can fail until the reference-counting change from #10034 is present.

Technical Details

Agent enumeration (WSL)

  • platform/wsl/dxg_topology.cpp and dxg_thunk.hpp read KMT topology through librocdxg using:

    • hsaKmtOpenKFD()
    • hsaKmtAcquireSystemProperties()
    • hsaKmtGetNodeProperties()
    • hsaKmtReleaseSystemProperties()
    • hsaKmtCloseKFD()

    The symbols are resolved with dlsym. DxgThunk::complete() verifies that all five are available before any are called.

  • HsaNodeProperties has no size parameter, so the destination buffer includes kNodePropsSlack = 128 bytes of additional space.

  • platform/wsl/agent.cpp matches DXG adapters to HSA nodes by LUID, with device ID as a fallback. Enumeration is allowed to be partial: an unmatched adapter is skipped with a diagnostic while successfully matched GPUs are still published.

  • Memory banks, caches, and IO links are not currently published by the WSL path.

  • If the thunk reports no GPU node, no GPU agent is published. A warning is emitted and the application continues.

  • agent_mapping.{hpp,cpp} adds a WSL-specific mapping policy. Native Linux keeps the existing strict behavior (agent.cpp:755), while WSL logs mismatches, keeps successfully paired GPUs, and continues.

Counters

  • counters/controller.cpp emits a one-time warning that WSL counters are collected through the DXG vendor-packet path rather than the KFD profiler ioctl.

  • WSLKMT_VENDOR_PACKET=1 is now set from wsl::enumerate() instead of is_available().

    This matters when using ROCPROFILER_FORCE_PLATFORM=wsl, where the old placement was bypassed. Without the environment variable, the DXG path drops aqlprofile's vendor-specific PM4 IB and counter values are returned as zero.

  • basic_counters.xml and derived_counters.xml each gain an empty <gfx1150 base="gfx11"> section to keep the installed metric files consistent.

    These XML files are not used by the runtime metric loader. Runtime metrics come from config.yaml (counters/metrics.cpp:348), which already supports gfx1150 and is unchanged by this PR.

KFD

  • kfd::init() now probes for a usable KFD device. If /dev/kfd is unavailable or cannot be opened, initialization continues with a warning instead of aborting in kfd_device_fd().

Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

  • aql/helpers.cpp:51: get_query_info() returns an empty ID query for metrics that do not have a hardware block.

    Constant and topology metrics such as simd_count have an empty block name. Passing that empty name to aqlprofile_get_pmc_info() can dereference a null pointer and abort.

  • kfd/kfd.cpp:1586: the KFD availability check is unconditional. A system without a usable /dev/kfd, for example because the driver is not loaded or access is denied, now receives a warning and continues instead of aborting.

    Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The DXG implementation requires /dev/dxg and libdxcore.so, and the PM4 command-buffer bounds change is limited to libhsakmt's DXG/WDDM submission path.

Tests

  • 44 new tests

    • 14 wsl_dxg_topology tests
    • 16 wsl_dxg_thunk tests
    • 10 agent_mapping tests
    • 4 parse_gfx_target_version tests

    The first two groups are in new executables. The mapping and target-version tests are added to the existing rocprofiler-sdk-lib-tests.

    None of these tests requires a GPU, HSA runtime, or real DXG thunk. The DXG tests use a fake thunk and fake dynamic loader.

    The 30 DXG-specific tests use the same CMake capability probe as the WSL enumerator, so they are only built when the required hsakmt interface is available. In a stock configuration, 14 of the 44 new tests are built.

  • platform-dispatch-tests and its six tests are pre-existing from [rocprofiler-sdk] Platform specific topology information #6399.

  • rocprofv3-avail

    • Three tests are moved out of the PC-sampling gate and are now enabled on Navi.
    • The L2 hit counter is selected by GPU family instead of being hardcoded.

    These tests have not previously run in Navi CI, so they may expose existing Navi-specific issues.

  • hsa_barrier

    • Skip when no GPU agent is available instead of dereferencing an empty agent map.
  • agent

    • agent_visibility_multigpu performs its substr(1) calls after the no-GPU skip, avoiding std::out_of_range when no agents are present.
    • grid_max_size tolerates the difference between rocprofiler's hardcoded INT32_MAX value and the value reported by the installed ROCr runtime.

JIRA ID

JIRA ID: ROCM-23912

Test Plan

Test Result

Submission Checklist

ihhethan added 6 commits June 9, 2026 23:09
rocprofiler_sdk_get_gfx_architectures() shells out to rocminfo at
configure time to populate the per-GPU architecture list. In
environments where rocminfo enumerates no GPU agent at configure
time (WSL, where the GPU agent only becomes visible after the
runtime loads; containers without /dev/kfd; cross-compile), the
regex returns an empty list, which breaks ~8 downstream
list(GET rocprofiler-sdk-tests-gfx-info 0 ...) call sites with
"list GET given empty list".

When rocminfo returns no gfx info and GPU_TARGETS is explicitly
set, fall back to that. Behavior unchanged for native Linux + KFD
setups where rocminfo correctly enumerates the GPU.
Add empty <gfx1150 base="gfx11"></gfx1150> entries to
basic_counters.xml and derived_counters.xml. gfx1150 (RDNA 3.5;
Strix Point integrated GPU) shares the gfx11 counter set, so
inheriting from gfx11 makes SQ_WAVES / GRBM_COUNT / GL2C_HIT etc.
resolvable via rocprofv3-avail without duplicating definitions.

No new counters introduced; this is purely arch registration so
the existing gfx11 counter set becomes available on gfx1150.
On the WSL synth-agent path the agent is populated from DXCore and
does not expose KFD topology attributes (array_count, simd_count,
wave_front_size, etc.) through the standard counter registry. The
AST loader fails to resolve these names when an AQL agent isn't
available, and downstream AQL queries trip CHECK_EQ / SIGABRT on
metrics that have no hardware block.

Make the three counter paths tolerate that gap:

  evaluate_ast.cpp
    At AST load time inject seven topology Metric stubs (array_count,
    simd_count, wave_front_size, simd_arrays_per_engine,
    cu_per_simd_array, simd_per_cu, max_waves_per_simd) with
    synthetic IDs in the 0xFFFF0000ULL+ range. These resolve common
    topology attribute names without an AQL agent.

  dimensions.cpp
    getBlockDimensions(): for synthetic IDs (or any metric with no
    block / expression / constant) return a single-instance
    dimension, avoiding CHECK_EQ in the AQL query path.

  aql/helpers.cpp
    get_query_info(): for synthetic IDs (or metrics with no block)
    return an empty query result; aqlprofile_get_pmc_info would
    otherwise SIGABRT dereferencing the null block name.

Transitional; should be revisited once libhsakmt-windows is
available to populate the real topology attributes on WSL.
The three hw-counter execute tests (rocprofv3-avail-test-hw-counters,
-test-check-hw-counters, -test-check-hw-counters_2) were gated on
IS_DISABLED, which disables ALL gfx10/11/12xx targets. That gate was
originally for PC-sampling-disabled cases, but hw-counter
enumeration and pmc-check do not depend on PC sampling support;
they only require that the counters exist on the arch.

On gfx1150 (RDNA 3.5) under WSL2, basic_counters.xml inherits from
gfx11 (via the "counters: register gfx1150 ..." commit in this
series) so SQ_WAVES / GRBM_COUNT / GL2C_HIT enumerate cleanly via
rocprofv3-avail info --pmc; pmc-check on the same counters succeeds.

Split out IS_HW_COUNTERS_DISABLED, currently only set for gfx906
and ROCPROFILER_MEMCHECK. Wire it into the three hw-counter execute
tests; leave the pytest validate test on the original IS_DISABLED
for safety.

After this change on RDNA 3.5 / WSL2 / ROCm 6.4.2:
  rocprofv3-avail-test-hw-counters         PASS
  rocprofv3-avail-test-check-hw-counters   PASS
  rocprofv3-avail-test-check-hw-counters_2 PASS
  rocprofv3-avail-test-pc-sample-config    Disabled (IS_PC_SAMPLING_DISABLED)
Add an opt-in path that populates the synthesized WSL agent's compute
topology from the libhsakmt-windows (librocdxg) shim instead of the
hardcoded per-arch fallback. fetch_libhsakmt_topology() opens KFD,
acquires system properties, and matches the DXCore adapter to its KFD
node by Windows LUID, then mirrors NumFComputeCores / NumArrays /
NumShaderBanks / MaxEngineClockMhzFCompute / EngineId into the agent.

Gated at build time by ROCPROFILER_USE_LIBHSAKMT_WINDOWS (requires the
hsakmt headers and a built librocdxg) and at runtime by the
ROCPROFILER_USE_LIBROCDXG env var; otherwise the hardcoded fallback is
used and behavior is unchanged.

Verified in-process on WSL2 (Radeon 890M / gfx1150): the matched KFD
node yields gfx11.5.0, 32 FCompute, 1 shader bank, 2 arrays, wave32,
2900 MHz.
On platforms without a usable KFD device node (e.g. WSL2, which exposes
/dev/dxg but not /dev/kfd), opening the KFD handle fails and the previous
code aborted via ROCP_FATAL_IF in kfd_device_fd(), bringing down the whole
rocprofv3 run.

Add a kfd_device_available() probe and gate kfd::init() on it: when the
device cannot be opened, log a warning and return ROCPROFILER_STATUS_SUCCESS
instead of constructing kfd_device_fd. get_version() and poll_kfd_t /
config::init are only reached after the probe, so the fatal path is fully
gated. On hardware with a real /dev/kfd the probe succeeds and the original
code path is taken unchanged (no behavior change on real GPUs).
@ihhethan
ihhethan requested review from a team as code owners June 10, 2026 04:20
@ihhethan ihhethan changed the title Users/ihhuang/wsl rocprofv3 compute [rocprofiler-sdk] WSL2 compute and profiling support for RDNA 3.5 Jun 10, 2026
@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch from 1f466e4 to a73ea21 Compare June 10, 2026 16:15
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jun 10, 2026
@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch from a73ea21 to 444fe6d Compare June 10, 2026 17:02
@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch 2 times, most recently from 4e6fe3f to b95d690 Compare June 11, 2026 23:42
@MythreyaK

Copy link
Copy Markdown
Member

question, just curious: will we be getting wsl2 runners to test this change?

@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch 2 times, most recently from 4da66b7 to afdf5ac Compare June 15, 2026 16:58
@ihhethan ihhethan closed this Jun 15, 2026
@ihhethan ihhethan reopened this Jun 15, 2026
@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch from afdf5ac to fcee884 Compare June 15, 2026 22:29
ihhethan and others added 4 commits September 3, 2026 05:02
@ihhethan
ihhethan force-pushed the users/ihhuang/wsl-rocprofv3-compute branch from 72e44ae to 9873a7a Compare September 15, 2026 21:50
ihhethan and others added 4 commits September 15, 2026 17:29
…rocprofv3-compute

Two conflicts, both resolved semantically rather than textually.

device_counting.cpp: kept the branch's SetUp() WSL/no-KFD skip and
took develop's test_run(const device_counting_run_options&), dropping
the branch's 4-argument test_run declaration. Develop re-supplies that
signature as a wrapper below it, so keeping both would have declared
the same signature twice.

validate.py: took develop's body and dropped the branch's pandas and
skip_if_hw_counter_values_unavailable imports. This is
behavior-neutral: that helper calls pytest.fail in every branch rather
than skipping -- commit c4ef285 on this branch deliberately
converted it from skip to fail -- so develop's strict assertions fail
in exactly the cases the helper did. pandas was already unused before
the merge. The helper is still used by the extra_counters, input1,
input2, input3 and kernel_filtering validate scripts, so nothing is
orphaned.
rocr::atomic::Load() routes through __atomic_load(), whose _WIN32
implementation was InterlockedOr(object, 0). That reads the value back
unchanged, but it is still a locked read-modify-write, so it requires the
page to be writable. Memory the runtime only ever reads is not always
writable: the WDDM monitored-fence value is a read-only CPU mapping,
documented as such in d3dukmdt.h:1805 for
D3DDDI_SYNCHRONIZATIONOBJECTINFO2::MonitoredFence and in d3dkmthk.h:5138
for D3DKMT_CREATEHWQUEUE. Querying a live adapter confirms it: the fence
VA it returns reports PAGE_READONLY, a locked read-modify-write against
that address raises 0xC0000005, and a plain volatile read of the same
address succeeds.

Load the value instead. A naturally aligned integer load is already
atomic on the architectures built here, and volatile is what keeps the
compiler from reusing a stale value.

Keep the barrier the interlocked form supplied incidentally, so that no
existing caller sees weaker ordering than before: relaxed loads have been
receiving a full barrier whether or not they asked for one, and this is
not the change in which to take that away. MemoryBarrier() goes before
the load because the only reordering a lock prefix suppresses that
x86-TSO still permits is an older store passing this load.

Dropping the size dispatch also removes a narrower hazard: loads of
16-bit AQL packet headers were going through the 32-bit InterlockedOr,
touching two bytes past the object.
LoadSyncValue() is correct as written now that the Windows atomic load
no longer performs a read-modify-write, so the call site needs no code
change. What it does need is the constraint written down: nothing near
these spins says that sync_addr is the KMD's read-only CPU mapping of
the monitored fence, or that an atomic read-modify-write against it
raises 0xC0000005.

Note it next to the load so that a later change reaching for a fetch-or
or a compare-exchange here does not have to rediscover it from a crash.
@ihhethan
ihhethan requested a review from jrmadsen September 21, 2026 14:52
@ihhethan

Copy link
Copy Markdown
Contributor Author

Rescinding approval due to all the failing tests.

The CI is now passing, and the MemoryTest1.exe issue has been resolved.

@lamikr

lamikr commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

I investigated the rocprofiler-sdk continuous integration CI failures that happens on MI325.The root cause seems to be that the SQ_WAVES values collected can have small couple of number variation at least on MI325 and values should be tested with small tolerance.

I have patch ready for fixing this test, it's now in separate PR.
#12085

…rocprofv3-compute

Resolve the import conflict in counter-collection/input3/validate.py:
take develop's removal of the unused pandas/os imports and keep this
branch's hardware_counters skip helper.
@ihhethan
ihhethan dismissed ApoKalipse-V’s stale review September 23, 2026 14:52

Jonathan has approved the PR, and the team confirmed that his approval is sufficient to move forward with the merge.

@lamikr

lamikr commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

Merging this now, existing tests that still fails seems to be flaky. rocprofiler-sdk related tests are failing for SQ_WAVES check which value can have little variation that test does not take into account and PR #12085 should fix that later.

SQ_WAVES measures:

  • Total wavefronts dispatched to shader queues
  • Fundamental unit of GPU parallelism
  • Proxy for work distribution across compute units

Variance is normal because:

  • Wave scheduling is non-deterministic
  • Partial waves may be handled differently
  • System overhead varies slightly
  • MI325 has more aggressive scheduler than MI300X

@lamikr
lamikr merged commit b0aa8f2 into develop Sep 24, 2026
232 of 242 checks passed
@lamikr
lamikr deleted the users/ihhuang/wsl-rocprofv3-compute branch September 24, 2026 12:38
@systems-assistant

Copy link
Copy Markdown
Contributor

TheRock Submodule Bump Activity

Newest first

anujshuk-amd pushed a commit that referenced this pull request Sep 25, 2026
…7016)

## Motivation

Enable `rocprofv3` counter collection and profiling on WSL2 for
gfx11-class GPUs.

On WSL, the complete KMT topology is read up front through `librocdxg`,
using the same thunk interface as the HSA runtime. An agent is published
only after all required topology information has been collected, so the
published record remains stable for its lifetime.

As a result, `rocprofiler_query_available_agents()` returns consistent
agent information regardless of when it is called, including before the
HSA runtime has initialized. This is required by `rocprofv3-avail` and
counter metric lookup.

Native Linux continues to use the existing `gnulinux` path.
`select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()`
first, which succeeds whenever the KFD sysfs nodes directory is present,
so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are
fixes and are called out under **Native-Linux impact** below.

This work builds on
[[#6399](#6399)], which
introduced the platform dispatch structure used here.
`platform_dispatch.cpp` and its six tests came from that PR
(`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**.
This PR adds WSL as another implementation behind the existing
`select_platform()` interface.

## Dependency on
[#10034](#10034)

[#10034](#10034) contains the
libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that
enforces merge order, so the two PRs can technically merge in either
order. Their changes to `core/runtime/amd_topology.cpp` also do not
overlap. However, two parts of this PR depend on #10034 before they are
fully functional.

### 1. WSL enumeration is disabled in a stock build until the required
hsakmt headers are installed

WSL enumeration is controlled by the CMake probe
`rocprofiler_sdk_dxg_topology_supported()`
(`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`).

The probe requires the hsakmt headers used by the build to provide:

* `HsaStructureSizes`
* `HsaNodeProperties::LuidLowPart`
* `HsaNodeProperties::LuidHighPart`
* `OverrideEngineId`

Released ROCm headers, including 7.2.4, do not provide all of these.
When the probe fails, `agent_unsupported.cpp` is built instead of the
WSL enumerator, `platform::wsl::is_available()` returns false, and
`enumerate()` returns an empty result.

The copies of `hsakmttypes.h` in this repository already contain the
required definitions (lines 359-360 and 1578), but they are not
currently installed. #10034 adds the WSL header installation rules in
`libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`.

Until that lands, a stock build of this PR will not enable WSL
enumeration.

### 2. The DXG topology snapshot is not reference-counted on current
`develop`

Current `develop` unconditionally calls `topology_drop_snapshot()` from
`hsaKmtReleaseSystemProperties()`
(`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology
snapshot and destroys its `WDDMDevice` objects even if another consumer
still has the topology acquired.

#10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot
destruction conditional on the reference count.

The normal `rocprofv3` path is not affected because the SDK topology
read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before
`main()`, and therefore before the application's `hsa_init()`.

`rocprof-attach` is different because it injects into an already-running
process. In that case there is no ordering guarantee between the SDK and
the existing HSA runtime consumer, so late attach can fail until the
reference-counting change from #10034 is present.

## Technical Details

### Agent enumeration (WSL)

* `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology
through `librocdxg` using:

  * `hsaKmtOpenKFD()`
  * `hsaKmtAcquireSystemProperties()`
  * `hsaKmtGetNodeProperties()`
  * `hsaKmtReleaseSystemProperties()`
  * `hsaKmtCloseKFD()`

The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies
that all five are available before any are called.

* `HsaNodeProperties` has no size parameter, so the destination buffer
includes `kNodePropsSlack = 128` bytes of additional space.

* `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID,
with device ID as a fallback. Enumeration is allowed to be partial: an
unmatched adapter is skipped with a diagnostic while successfully
matched GPUs are still published.

* Memory banks, caches, and IO links are not currently published by the
WSL path.

* If the thunk reports no GPU node, no GPU agent is published. A warning
is emitted and the application continues.

* `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native
Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL
logs mismatches, keeps successfully paired GPUs, and continues.

### Counters

* `counters/controller.cpp` emits a one-time warning that WSL counters
are collected through the DXG vendor-packet path rather than the KFD
profiler ioctl.

* `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of
`is_available()`.

This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old
placement was bypassed. Without the environment variable, the DXG path
drops aqlprofile's vendor-specific PM4 IB and counter values are
returned as zero.

* `basic_counters.xml` and `derived_counters.xml` each gain an empty
`<gfx1150 base="gfx11">` section to keep the installed metric files
consistent.

These XML files are not used by the runtime metric loader. Runtime
metrics come from `config.yaml` (`counters/metrics.cpp:348`), which
already supports gfx1150 and is unchanged by this PR.

### KFD

* `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is
unavailable or cannot be opened, initialization continues with a warning
instead of aborting in `kfd_device_fd()`.

## Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

* `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for
metrics that do not have a hardware block.

Constant and topology metrics such as `simd_count` have an empty block
name. Passing that empty name to `aqlprofile_get_pmc_info()` can
dereference a null pointer and abort.

* `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A
system without a usable `/dev/kfd`, for example because the driver is
not loaded or access is denied, now receives a warning and continues
instead of aborting.

  Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The
DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4
command-buffer bounds change is limited to libhsakmt's DXG/WDDM
submission path.

## Tests

* **44 new tests**

  * 14 `wsl_dxg_topology` tests
  * 16 `wsl_dxg_thunk` tests
  * 10 `agent_mapping` tests
  * 4 `parse_gfx_target_version` tests

The first two groups are in new executables. The mapping and
target-version tests are added to the existing
`rocprofiler-sdk-lib-tests`.

None of these tests requires a GPU, HSA runtime, or real DXG thunk. The
DXG tests use a fake thunk and fake dynamic loader.

The 30 DXG-specific tests use the same CMake capability probe as the WSL
enumerator, so they are only built when the required hsakmt interface is
available. In a stock configuration, 14 of the 44 new tests are built.

* `platform-dispatch-tests` and its six tests are pre-existing from
#6399.

* `rocprofv3-avail`

* Three tests are moved out of the PC-sampling gate and are now enabled
on Navi.
* The L2 hit counter is selected by GPU family instead of being
hardcoded.

These tests have not previously run in Navi CI, so they may expose
existing Navi-specific issues.

* `hsa_barrier`

* Skip when no GPU agent is available instead of dereferencing an empty
agent map.

* `agent`

* `agent_visibility_multigpu` performs its `substr(1)` calls after the
no-GPU skip, avoiding `std::out_of_range` when no agents are present.
* `grid_max_size` tolerates the difference between rocprofiler's
hardcoded `INT32_MAX` value and the value reported by the installed ROCr
runtime.

## JIRA ID

JIRA ID: ROCM-23912

## Test Plan


## Test Result



## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Mika Laitio <mika.laitio@amd.com>
dgaliffiAMD pushed a commit that referenced this pull request Sep 25, 2026
…7016)

## Motivation

Enable `rocprofv3` counter collection and profiling on WSL2 for
gfx11-class GPUs.

On WSL, the complete KMT topology is read up front through `librocdxg`,
using the same thunk interface as the HSA runtime. An agent is published
only after all required topology information has been collected, so the
published record remains stable for its lifetime.

As a result, `rocprofiler_query_available_agents()` returns consistent
agent information regardless of when it is called, including before the
HSA runtime has initialized. This is required by `rocprofv3-avail` and
counter metric lookup.

Native Linux continues to use the existing `gnulinux` path.
`select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()`
first, which succeeds whenever the KFD sysfs nodes directory is present,
so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are
fixes and are called out under **Native-Linux impact** below.

This work builds on
[[#6399](#6399)], which
introduced the platform dispatch structure used here.
`platform_dispatch.cpp` and its six tests came from that PR
(`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**.
This PR adds WSL as another implementation behind the existing
`select_platform()` interface.

## Dependency on
[#10034](#10034)

[#10034](#10034) contains the
libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that
enforces merge order, so the two PRs can technically merge in either
order. Their changes to `core/runtime/amd_topology.cpp` also do not
overlap. However, two parts of this PR depend on #10034 before they are
fully functional.

### 1. WSL enumeration is disabled in a stock build until the required
hsakmt headers are installed

WSL enumeration is controlled by the CMake probe
`rocprofiler_sdk_dxg_topology_supported()`
(`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`).

The probe requires the hsakmt headers used by the build to provide:

* `HsaStructureSizes`
* `HsaNodeProperties::LuidLowPart`
* `HsaNodeProperties::LuidHighPart`
* `OverrideEngineId`

Released ROCm headers, including 7.2.4, do not provide all of these.
When the probe fails, `agent_unsupported.cpp` is built instead of the
WSL enumerator, `platform::wsl::is_available()` returns false, and
`enumerate()` returns an empty result.

The copies of `hsakmttypes.h` in this repository already contain the
required definitions (lines 359-360 and 1578), but they are not
currently installed. #10034 adds the WSL header installation rules in
`libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`.

Until that lands, a stock build of this PR will not enable WSL
enumeration.

### 2. The DXG topology snapshot is not reference-counted on current
`develop`

Current `develop` unconditionally calls `topology_drop_snapshot()` from
`hsaKmtReleaseSystemProperties()`
(`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology
snapshot and destroys its `WDDMDevice` objects even if another consumer
still has the topology acquired.

#10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot
destruction conditional on the reference count.

The normal `rocprofv3` path is not affected because the SDK topology
read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before
`main()`, and therefore before the application's `hsa_init()`.

`rocprof-attach` is different because it injects into an already-running
process. In that case there is no ordering guarantee between the SDK and
the existing HSA runtime consumer, so late attach can fail until the
reference-counting change from #10034 is present.

## Technical Details

### Agent enumeration (WSL)

* `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology
through `librocdxg` using:

  * `hsaKmtOpenKFD()`
  * `hsaKmtAcquireSystemProperties()`
  * `hsaKmtGetNodeProperties()`
  * `hsaKmtReleaseSystemProperties()`
  * `hsaKmtCloseKFD()`

The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies
that all five are available before any are called.

* `HsaNodeProperties` has no size parameter, so the destination buffer
includes `kNodePropsSlack = 128` bytes of additional space.

* `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID,
with device ID as a fallback. Enumeration is allowed to be partial: an
unmatched adapter is skipped with a diagnostic while successfully
matched GPUs are still published.

* Memory banks, caches, and IO links are not currently published by the
WSL path.

* If the thunk reports no GPU node, no GPU agent is published. A warning
is emitted and the application continues.

* `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native
Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL
logs mismatches, keeps successfully paired GPUs, and continues.

### Counters

* `counters/controller.cpp` emits a one-time warning that WSL counters
are collected through the DXG vendor-packet path rather than the KFD
profiler ioctl.

* `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of
`is_available()`.

This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old
placement was bypassed. Without the environment variable, the DXG path
drops aqlprofile's vendor-specific PM4 IB and counter values are
returned as zero.

* `basic_counters.xml` and `derived_counters.xml` each gain an empty
`<gfx1150 base="gfx11">` section to keep the installed metric files
consistent.

These XML files are not used by the runtime metric loader. Runtime
metrics come from `config.yaml` (`counters/metrics.cpp:348`), which
already supports gfx1150 and is unchanged by this PR.

### KFD

* `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is
unavailable or cannot be opened, initialization continues with a warning
instead of aborting in `kfd_device_fd()`.

## Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

* `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for
metrics that do not have a hardware block.

Constant and topology metrics such as `simd_count` have an empty block
name. Passing that empty name to `aqlprofile_get_pmc_info()` can
dereference a null pointer and abort.

* `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A
system without a usable `/dev/kfd`, for example because the driver is
not loaded or access is denied, now receives a warning and continues
instead of aborting.

  Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The
DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4
command-buffer bounds change is limited to libhsakmt's DXG/WDDM
submission path.

## Tests

* **44 new tests**

  * 14 `wsl_dxg_topology` tests
  * 16 `wsl_dxg_thunk` tests
  * 10 `agent_mapping` tests
  * 4 `parse_gfx_target_version` tests

The first two groups are in new executables. The mapping and
target-version tests are added to the existing
`rocprofiler-sdk-lib-tests`.

None of these tests requires a GPU, HSA runtime, or real DXG thunk. The
DXG tests use a fake thunk and fake dynamic loader.

The 30 DXG-specific tests use the same CMake capability probe as the WSL
enumerator, so they are only built when the required hsakmt interface is
available. In a stock configuration, 14 of the 44 new tests are built.

* `platform-dispatch-tests` and its six tests are pre-existing from
#6399.

* `rocprofv3-avail`

* Three tests are moved out of the PC-sampling gate and are now enabled
on Navi.
* The L2 hit counter is selected by GPU family instead of being
hardcoded.

These tests have not previously run in Navi CI, so they may expose
existing Navi-specific issues.

* `hsa_barrier`

* Skip when no GPU agent is available instead of dereferencing an empty
agent map.

* `agent`

* `agent_visibility_multigpu` performs its `substr(1)` calls after the
no-GPU skip, avoiding `std::out_of_range` when no agents are present.
* `grid_max_size` tolerates the difference between rocprofiler's
hardcoded `INT32_MAX` value and the value reported by the installed ROCr
runtime.

## JIRA ID

JIRA ID: ROCM-23912

## Test Plan


## Test Result



## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Mika Laitio <mika.laitio@amd.com>
ihhethan added a commit that referenced this pull request Oct 5, 2026
)

## Motivation

Enable rocprofv3 thread trace (`--att`) support on WSL2, completing the
ATT portion of the WSL2 enablement introduced in #7016.

Previously, the ATT samples and tests only searched `${ROCM_PATH}` for
`rocprof-trace-decoder`. As a result, separately built decoders could
not be detected, causing the corresponding ATT tests to be skipped.

## Technical Details

- **Shared decoder lookup**: Added a new
`rocprofiler_sdk_find_trace_decoder()` macro to
`rocprofiler-sdk-utilities.cmake`, replacing five duplicated
`find_library` blocks across the ATT samples and tests. The lookup now
searches `ROCPROFILER_SDK_TRACE_DECODER_ROOT` before `${ROCM_PATH}`. If
`ROCPROFILER_SDK_TRACE_DECODER_ROOT` is not set, the existing behavior
remains unchanged.

- **`att-no-intercept` test gating**: This test requires multi-buffer
thread trace support, which is currently available only on MI3xx and
gfx12. On unsupported platforms, the test could time out. It now uses
the existing `rocprofiler_sdk_sqtt_triple_buffer_disabled()` check to
skip the test where multi-buffer support is unavailable.

- **WSL2 documentation**: Added a **Thread trace (ATT)** section to the
WSL2 how-to covering:
  - ATT decoder setup
  - HSA runtime path configuration when `/dev/kfd` is unavailable
  - The current `--selected-regions` limitation on WSL2

  ATT has also been removed from the **Not covered on WSL2** section.

No C++ changes are included in this PR.

## Issue Tracking

JIRA ID: ROCM-23912

## Test Plan

WSL2, gfx1150 (no `/dev/kfd`), ROCr/librocdxg/SDK built from this
branch, rocprof-trace-decoder 0.2.2 found via
`ROCPROFILER_SDK_TRACE_DECODER_ROOT`.

- `ctest -R 'att|thread-trace|thread_trace'`
- `rocprofv3 --att --kernel-trace -- simple-transpose`

## Test Result

- ctest: 165/171 passed; `thread-trace-sample` and rocattach 12/12 pass;
`att-no-intercept` is now Disabled on gfx1150.
- `rocprofv3 --att`: both dispatches captured and decoded.
- 6 failures, unrelated to this change: 5 × `unit.thread_trace.*` caused
by the ctest-baked `LD_LIBRARY_PATH` (they pass when run directly), and
the `att-marker-trace` validate (`--selected-regions` gives an empty
trace on WSL2, documented).

## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.
@ihhethan ihhethan changed the title feat(rocprofiler-sdk): WSL2 compute and profiling support for RDNA 3 feat(rocprofiler-sdk): WSL2 compute and profiling support for RDNA 3.5 Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants