Skip to content

[rocprofiler-sdk] Platform specific topology information - #6399

Merged
bgopesh merged 9 commits into
developfrom
users/bgopesh/topology-provider
Jun 2, 2026
Merged

bgopesh merged 9 commits into
developfrom
users/bgopesh/topology-provider

Conversation

@bgopesh

@bgopesh bgopesh commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

This PR decouples agent (CPU/GPU) discovery from `agent.cpp` by introducing per-platform free functions under `rocprofiler::platform::{gnulinux,wsl,windows}`, preserving the existing Linux/KFD enumeration verbatim while adding WSL (DXCore/D3DKMT) and native Windows (D3DKMT) backends.

Changes:

Directory structure

```
source/lib/rocprofiler-sdk/
├── platform/
│ ├── CMakeLists.txt
│ ├── agent.hpp # dispatch header: is_available / enumerate / name
│ ├── gnulinux/
│ │ ├── CMakeLists.txt
│ │ ├── agent.hpp
│ │ └── agent.cpp # byte-identical move of Linux/KFD sysfs logic from agent.cpp
│ ├── wsl/
│ │ ├── CMakeLists.txt
│ │ ├── agent.hpp
│ │ └── agent.cpp # DXCore enumeration via D3DKMTEnumAdapters3 + RAII DxcoreHandle
│ └── windows/
│ ├── CMakeLists.txt
│ ├── agent.hpp
│ └── agent.cpp # native Windows D3DKMT skeleton (stub, follow-up)
└── agent.cpp # runtime dispatch: gnulinux -> wsl -> windows
```

The earlier `topology/` subdirectory (with abstract `TopologyProvider`, `LinuxKfdProvider`, `DxgProvider`, `D3dkmtProvider` classes) has been removed in favour of this flat, namespace-per-platform layout per CONTRIBUTING.md §8.

Key design points

  • No abstract base class. Each platform exposes three free functions in its own namespace: `is_available()`, `enumerate()`, `name()`. `agent.cpp` dispatches at runtime in precedence order: `gnulinux → wsl → windows`.
  • `ROCPROFILER_FORCE_PLATFORM={gnulinux|wsl|windows}` env var overrides the runtime selection (replaces the earlier `ROCPROFILER_FORCE_DXG` / `ROCPROFILER_FORCE_D3DKMT` vars).
  • Linux behaviour unchanged. The gnulinux body is a verbatim move of `read_topology()` and helpers from `agent.cpp`.
  • WSL body moved from the old `dxg_provider`; uses `D3DKMTEnumAdapters3` + `D3DKMTQueryAdapterInfo` (PHYSICALADAPTERDEVICEIDS, ADAPTERADDRESS, ADAPTERREGISTRYINFO, GETSEGMENTSIZE); RAII `DxcoreHandle` wraps the dlopen'd library; filters AMD adapters, packs PCI BDF into `location_id`, converts wide names to UTF-8. Fields DXCore cannot supply (gfx_target_version, CU/SDMA counts, mem_banks/caches/io_links) are left zero/empty.
  • Windows body is a stub skeleton; full D3DKMT enumeration is a follow-up.
  • `agent_internal.hpp` removed. `update_agent_runtime_visibility` is now declared in `agent.hpp` directly.

Technical Details

  • `platform/gnulinux/agent.cpp`: byte-identical move of `read_topology()` and helpers (`parse_cpu_info`, `read_map`, `read_property`, etc.) out of `agent.cpp`. Linux behaviour is unchanged.
  • `platform/wsl/agent.cpp`: WSL backend using D3DKMT/DXCore. Probes `D3DKMTEnumAdapters3` symbol; also checks `/dev/dxg` presence for `is_available()` so a box with `libdxcore.so` but no `/dev/dxg` does not silently return an empty topology.
  • `platform/windows/agent.cpp`: native Windows backend skeleton. All D3DKMT calls are `#ifdef _WIN32` guarded so the class still links on Linux (stubs return empty + warning). Full enumeration body is a follow-up.
  • `agent.cpp`: `read_topology()` and its helpers deleted. `internal_refresh_topology()` dispatches through the platform functions. No other public function changed. Unused `fs` alias and `filesystem.hpp` include removed (was flagged by clang-tidy `misc-unused-alias-decls`).

JIRA ID

Test Plan

Test Result

Submission Checklist

Copilot AI review requested due to automatic review settings May 25, 2026 18:10
@bgopesh
bgopesh requested review from a team as code owners May 25, 2026 18:10
@bgopesh bgopesh changed the title Users/bgopesh/topology provider [rocprofiler-sdk] Introduce TopologyProvider abstraction May 25, 2026

This comment was marked as outdated.

@github-actions

github-actions Bot commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

Code Coverage Report

Code Coverage Report

Tests Only

code coverage tests.png

Samples Only

code coverage samples.png

Tests + Samples

code coverage all.png

@jrmadsen

Copy link
Copy Markdown
Contributor

Please avoid polymorphic abstractions. If we need to have a design discussion, please let me know. I'd prefer an approach/organization like:

  • agent.hpp (platform neutral interface)
  • agent.cpp (handles calling platform-specific implementation)
  • linux/
    • agent.hpp (Linux interface in linux namespace)
    • agent.cpp (existing agent.cpp in linux namespace)
  • windows/
    • agent.hpp (Windows interface in windows namespace)
    • agent.cpp
  • wsl/
    • agent.hpp (WSL interface in wsl namespace)
    • agent.cpp

This pattern will keep Linux/Windows/WSL code separate, work well with our current source code organization (for example, we may need a linux/registration.cpp and a windows/registration.cpp), and the namespacing will make calls explicit, e.g. rocprofiler::linux::<some-agent-function>(...).

@bgopesh

bgopesh commented May 26, 2026

Copy link
Copy Markdown
Contributor Author

Please avoid polymorphic abstractions. If we need to have a design discussion, please let me know. I'd prefer an approach/organization like:

  • agent.hpp (platform neutral interface)

  • agent.cpp (handles calling platform-specific implementation)

  • linux/

    • agent.hpp (Linux interface in linux namespace)
    • agent.cpp (existing agent.cpp in linux namespace)
  • windows/

    • agent.hpp (Windows interface in windows namespace)
    • agent.cpp
  • wsl/

    • agent.hpp (WSL interface in wsl namespace)
    • agent.cpp

This pattern will keep Linux/Windows/WSL code separate, work well with our current source code organization (for example, we may need a linux/registration.cpp and a windows/registration.cpp), and the namespacing will make calls explicit, e.g. rocprofiler::linux::<some-agent-function>(...).

Thanks for looking into it.
ok. makes sense. Let me redesign this with the suggested approach. I think the suggested approach would be good in long term for all platform specific differences.

@bgopesh

bgopesh commented May 26, 2026 •

Copy link
Copy Markdown
Contributor Author

@jrmadsen : Before I refactor this, can you please approve this structure with namespace:

Tentative layout I'd implement once these are settled (using kfd as a placeholder instead of linux since linux (and unix) are predefined as 1 by GCC unless we compile in strict mode):

source/lib/rocprofiler-sdk/
agent.hpp
agent.cpp # dispatcher: kfd:: vs wsl:: at runtime, plus update_agent_runtime_visibility
kfd/
agent.{hpp,cpp} # namespace rocprofiler::kfd (old read_topology() body, verbatim move)
CMakeLists.txt # always built on UNIX
wsl/
agent.{hpp,cpp} # namespace rocprofiler::wsl (old DxgProvider body, verbatim move)
CMakeLists.txt # always built on UNIX (WSL is Linux)
windows/
agent.{hpp,cpp} # namespace rocprofiler::windows (D3DKMT skeleton)

@meserve-amd meserve-amd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not reviewed dxg_provider yet, but have some concerns about the architecture so far that can be discussed.

@bgopesh

bgopesh commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

@meserve-amd : sure. Most of your concerns will go away once I remove the polymorphic topology and us ethe platform specific directory structure as Jonathan mentioned above.

@bgopesh

bgopesh commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

@jrmadsen : Before I refactor this, can you please approve this structure with namespace:

Tentative layout I'd implement once these are settled (using kfd as a placeholder instead of linux since linux (and unix) are predefined as 1 by GCC unless we compile in strict mode):

source/lib/rocprofiler-sdk/ agent.hpp agent.cpp # dispatcher: kfd:: vs wsl:: at runtime, plus update_agent_runtime_visibility kfd/ agent.{hpp,cpp} # namespace rocprofiler::kfd (old read_topology() body, verbatim move) CMakeLists.txt # always built on UNIX wsl/ agent.{hpp,cpp} # namespace rocprofiler::wsl (old DxgProvider body, verbatim move) CMakeLists.txt # always built on UNIX (WSL is Linux) windows/ agent.{hpp,cpp} # namespace rocprofiler::windows (D3DKMT skeleton)

@jrmadsen This part is done.

@bgopesh bgopesh changed the title [rocprofiler-sdk] Introduce TopologyProvider abstraction [rocprofiler-sdk] Platform specific topology information May 27, 2026
@bgopesh

bgopesh commented May 28, 2026

Copy link
Copy Markdown
Contributor Author

@meserve-amd and @ApoKalipse-V : reminder for re-review

@meserve-amd meserve-amd left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conditional approval on addressing the one Major comment.

Thanks for reworking the architecture here; this looks much better now.

I understand gnulinux was copied code, but there are a few nits Claude found in there when I let it loose. I think it's a good time to fix up things, but I'll leave it up to you.

@bgopesh

bgopesh commented May 29, 2026

Copy link
Copy Markdown
Contributor Author

Conditional approval on addressing the one Major comment.

Thanks for reworking the architecture here; this looks much better now.

I understand gnulinux was copied code, but there are a few nits Claude found in there when I let it loose. I think it's a good time to fix up things, but I'll leave it up to you.

Thanks for the review. Addressed the Major comment. I want to keep Linux implementation verbatim as I want to limit the scope of this PR to just focus on getting the structure right. I know changes you requested are small but still would like to address them after this is merged.

@bgopesh
bgopesh force-pushed the users/bgopesh/topology-provider branch 3 times, most recently from 0b4794f to 8796818 Compare June 2, 2026 05:04
@bgopesh
bgopesh requested a review from ApoKalipse-V June 2, 2026 05:28

@ApoKalipse-V ApoKalipse-V left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks okay, so pre-approving, but we are missing tests.
Just make sure to add unit tests ensuring the different options return or fail accordingly to get some coverage.
e.g. WSL is compiled in linux but not covered by our CI because it doesn't run standard tests in WSL.

@bgopesh
bgopesh force-pushed the users/bgopesh/topology-provider branch 3 times, most recently from 7aab19d to 7342b2f Compare June 2, 2026 11:12
@bgopesh

bgopesh commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Code looks okay, so pre-approving, but we are missing tests. Just make sure to add unit tests ensuring the different options return or fail accordingly to get some coverage. e.g. WSL is compiled in linux but not covered by our CI because it doesn't run standard tests in WSL.

Added tests

bgopesh and others added 9 commits June 2, 2026 20:06
Factor GPU/CPU agent discovery behind a polymorphic TopologyProvider
interface so the existing Linux/KFD path is preserved verbatim while
WSL and native Windows gain dedicated backends.

  - topology/linux_kfd_provider: byte-identical move of read_topology()
    and its helpers (parse_cpu_info, read_map, read_property, etc.) out
    of agent.cpp. Linux behaviour is unchanged.
  - topology/dxg_provider: WSL backend skeleton. Dlopens libdxcore.so
    and probes CreateDXCoreAdapterFactory; full DXCore-to-agent
    enumeration is left as a follow-up.
  - topology/d3dkmt_provider: native Windows backend skeleton. All
    D3DKMT calls are #ifdef _WIN32 guarded so the class still links on
    Linux (stubs return empty + warning). The full enumeration body
    is left as a follow-up.
  - topology/topology_provider: factory make_default_provider() with
    runtime selection precedence:
      ROCPROFILER_FORCE_DXG -> ROCPROFILER_FORCE_D3DKMT -> sysfs ->
      libdxcore.so loadable -> _WIN32 -> LinuxKfd fallback.
    The ROCPROFILER_FORCE_* env vars are new and intended for triage.
  - agent.cpp: read_topology() and its helpers deleted. The singleton
    accessor and internal_refresh_topology() now go through the
    provider. No other public function changed.
  - agent_internal.hpp: minimal forward declaration of
    update_agent_runtime_visibility so providers can call it without
    touching the public header.
Fixes probe_libdxcore() to look up D3DKMTEnumAdapters3 (the symbol
actually called) instead of CreateDXCoreAdapterFactory. Fills in
DxgProvider::enumerate() using D3DKMTEnumAdapters3 +
D3DKMTQueryAdapterInfo (PHYSICALADAPTERDEVICEIDS, ADAPTERADDRESS,
ADAPTERREGISTRYINFO, GETSEGMENTSIZE). Filters AMD adapters only,
packs PCI BDF into location_id, converts wide adapter names to UTF-8,
and closes every adapter handle. Fields DXCore cannot supply
(gfx_target_version, CU/SDMA counts, mem_banks/caches/io_links) are
left zero/empty rather than fabricated.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
After the topology refactor moved all sysfs walking into
LinuxKfdProvider, agent.cpp no longer references the fs namespace
alias or lib/common/filesystem.hpp. clang-tidy on ubuntu 22.04
flagged this as misc-unused-alias-decls (warnings-as-errors).

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
- dxg_provider.cpp: drop redundant 'static' from constexpr defs inside
  anonymous namespace (readability-static-definition-in-anonymous-namespace).
- d3dkmt_provider.hpp: mark stub helpers static; they don't touch
  instance state (readability-convert-member-functions-to-static).
- topology/CMakeLists.txt: cmake-format normalization.
- agent.cpp: drop stray blank line per clang-format.

Verified locally with clang-format 11.1.0, clang-tidy 19.1.0, and
cmake-format 0.6.13 from requirements.txt.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
- DxgProvider::is_available() now also checks /dev/dxg presence so
  selection matches the documented "libdxcore.so + /dev/dxg" intent.
  Without this, a box with libdxcore.so but no /dev/dxg would pick
  DxgProvider and then return an empty topology silently.
- ROCPROFILER_FORCE_D3DKMT is now gated on _WIN32. On non-Windows
  builds the override is ignored with a warning instead of selecting
  the stub provider that always returns empty.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Drops the abstract TopologyProvider base in favor of per-platform free
functions under rocprofiler::platform::{gnulinux,wsl,windows}, per
CONTRIBUTING.md item 8. Each platform owns is_available()/enumerate()/name;
agent.cpp dispatches at runtime in precedence gnulinux -> wsl -> windows,
with ROCPROFILER_FORCE_PLATFORM={gnulinux|wsl|windows} as override.

The gnulinux body is moved verbatim from linux_kfd_provider.cpp so existing
KFD behavior stays byte-identical. The WSL body is moved from dxg_provider
and gains an RAII DxcoreHandle wrapper for the dlopen'd library; review
comments on get_agent_offset rationale and the WSL num_xcc=1 limitation
are addressed inline. update_agent_runtime_visibility moves to agent.hpp
so the now-redundant agent_internal.hpp is removed.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Three D3DKMTQueryAdapterInfo calls (ADAPTERADDRESS, ADAPTERREGISTRYINFO,
GETSEGMENTSIZE) previously dropped their status and continued, so a failure
would publish a rocprofiler_agent_t with a zeroed BDF, missing adapter name,
or zero VRAM size. Check each status; on failure log a WARNING with the
adapter index and vendor/device IDs and skip the adapter.

Adapter cleanup now uses common::scope_destructor instead of duplicating
DxcCloseAdapter at every exit path, matching how the rest of the codebase
handles scope-bound resources and also closing the leak window if the
rocprofiler_agent_t allocation throws.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Adds tests/platform_dispatch.cpp covering the runtime selection in
agent.cpp and the per-platform entry points:
  - wsl::is_available() / wsl::enumerate() degrade safely off-WSL
  - gnulinux::is_available() honors the local-topology fixture
  - ROCPROFILER_FORCE_PLATFORM={gnulinux,wsl,unknown} dispatch behavior
Tests with /dev/dxg dependencies self-skip when not in WSL so they are
useful on both bare-metal Linux CI and on a WSL workstation.

wsl/agent.cpp: cache the random agent offset as const and drop the
redundant gpu_count counter (was always equal to the logical id since
every DXCore-enumerated adapter is a GPU).

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@bgopesh
bgopesh force-pushed the users/bgopesh/topology-provider branch from 7342b2f to 4882a01 Compare June 2, 2026 14:36
@bgopesh
bgopesh merged commit d2f0b27 into develop Jun 2, 2026
40 checks passed
@bgopesh
bgopesh deleted the users/bgopesh/topology-provider branch June 2, 2026 18:47
bgopesh added a commit that referenced this pull request Jun 3, 2026
…ws/ subdirs

Relocate the platform-abstraction implementations into per-OS
subdirectories under details/platform/, matching the layout introduced in
rocprofiler-sdk PR #6399:

  loader_posix.cpp     -> gnulinux/loader.cpp
  pe_parser_posix.cpp  -> gnulinux/pe_parser.cpp
  loader_win.cpp       -> windows/loader.cpp
  pe_parser_win.cpp    -> windows/pe_parser.cpp
  encoding_win.hpp     -> windows/encoding.hpp

loader.hpp and pe_parser.hpp stay in details/platform/ since they are the
platform-neutral interface. Includes in the moved .cpp files now reach
those headers via the rooted "details/platform/..." form (resolved against
the rocprofiler-register target's include path on
source/lib/rocprofiler-register), and the Windows .cpp files include the
relocated encoding helper via "details/platform/windows/encoding.hpp".

Layout-only change: no behavioral edits to the .cpp content.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
bgopesh added a commit that referenced this pull request Jun 3, 2026
…yout

Replace the suffix-based file selection in details/platform/CMakeLists.txt
with the add_subdirectory pattern used by rocprofiler-sdk PR #6399. Each
per-OS subdirectory now owns its own sources via a self-contained
CMakeLists.txt:

  * details/platform/CMakeLists.txt: dispatches to windows/ on WIN32 and
    gnulinux/ otherwise.
  * details/platform/gnulinux/CMakeLists.txt: adds loader.cpp and
    pe_parser.cpp to rocprofiler-register.
  * details/platform/windows/CMakeLists.txt: adds loader.cpp, pe_parser.cpp,
    and encoding.hpp, and links psapi (used by EnumProcessModulesEx /
    GetModuleInformation in loader.cpp and pe_parser.cpp).

CONTRIBUTING.md compliance: only target-scoped commands, and each
subdirectory's CMakeLists.txt owns its own sources.

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
lamikr added a commit that referenced this pull request Sep 24, 2026
…7016)

## Motivation

Enable `rocprofv3` counter collection and profiling on WSL2 for
gfx11-class GPUs.

On WSL, the complete KMT topology is read up front through `librocdxg`,
using the same thunk interface as the HSA runtime. An agent is published
only after all required topology information has been collected, so the
published record remains stable for its lifetime.

As a result, `rocprofiler_query_available_agents()` returns consistent
agent information regardless of when it is called, including before the
HSA runtime has initialized. This is required by `rocprofv3-avail` and
counter metric lookup.

Native Linux continues to use the existing `gnulinux` path.
`select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()`
first, which succeeds whenever the KFD sysfs nodes directory is present,
so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are
fixes and are called out under **Native-Linux impact** below.

This work builds on
[[#6399](#6399)], which
introduced the platform dispatch structure used here.
`platform_dispatch.cpp` and its six tests came from that PR
(`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**.
This PR adds WSL as another implementation behind the existing
`select_platform()` interface.

## Dependency on
[#10034](#10034)

[#10034](#10034) contains the
libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that
enforces merge order, so the two PRs can technically merge in either
order. Their changes to `core/runtime/amd_topology.cpp` also do not
overlap. However, two parts of this PR depend on #10034 before they are
fully functional.

### 1. WSL enumeration is disabled in a stock build until the required
hsakmt headers are installed

WSL enumeration is controlled by the CMake probe
`rocprofiler_sdk_dxg_topology_supported()`
(`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`).

The probe requires the hsakmt headers used by the build to provide:

* `HsaStructureSizes`
* `HsaNodeProperties::LuidLowPart`
* `HsaNodeProperties::LuidHighPart`
* `OverrideEngineId`

Released ROCm headers, including 7.2.4, do not provide all of these.
When the probe fails, `agent_unsupported.cpp` is built instead of the
WSL enumerator, `platform::wsl::is_available()` returns false, and
`enumerate()` returns an empty result.

The copies of `hsakmttypes.h` in this repository already contain the
required definitions (lines 359-360 and 1578), but they are not
currently installed. #10034 adds the WSL header installation rules in
`libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`.

Until that lands, a stock build of this PR will not enable WSL
enumeration.

### 2. The DXG topology snapshot is not reference-counted on current
`develop`

Current `develop` unconditionally calls `topology_drop_snapshot()` from
`hsaKmtReleaseSystemProperties()`
(`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology
snapshot and destroys its `WDDMDevice` objects even if another consumer
still has the topology acquired.

#10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot
destruction conditional on the reference count.

The normal `rocprofv3` path is not affected because the SDK topology
read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before
`main()`, and therefore before the application's `hsa_init()`.

`rocprof-attach` is different because it injects into an already-running
process. In that case there is no ordering guarantee between the SDK and
the existing HSA runtime consumer, so late attach can fail until the
reference-counting change from #10034 is present.

## Technical Details

### Agent enumeration (WSL)

* `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology
through `librocdxg` using:

  * `hsaKmtOpenKFD()`
  * `hsaKmtAcquireSystemProperties()`
  * `hsaKmtGetNodeProperties()`
  * `hsaKmtReleaseSystemProperties()`
  * `hsaKmtCloseKFD()`

The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies
that all five are available before any are called.

* `HsaNodeProperties` has no size parameter, so the destination buffer
includes `kNodePropsSlack = 128` bytes of additional space.

* `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID,
with device ID as a fallback. Enumeration is allowed to be partial: an
unmatched adapter is skipped with a diagnostic while successfully
matched GPUs are still published.

* Memory banks, caches, and IO links are not currently published by the
WSL path.

* If the thunk reports no GPU node, no GPU agent is published. A warning
is emitted and the application continues.

* `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native
Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL
logs mismatches, keeps successfully paired GPUs, and continues.

### Counters

* `counters/controller.cpp` emits a one-time warning that WSL counters
are collected through the DXG vendor-packet path rather than the KFD
profiler ioctl.

* `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of
`is_available()`.

This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old
placement was bypassed. Without the environment variable, the DXG path
drops aqlprofile's vendor-specific PM4 IB and counter values are
returned as zero.

* `basic_counters.xml` and `derived_counters.xml` each gain an empty
`<gfx1150 base="gfx11">` section to keep the installed metric files
consistent.

These XML files are not used by the runtime metric loader. Runtime
metrics come from `config.yaml` (`counters/metrics.cpp:348`), which
already supports gfx1150 and is unchanged by this PR.

### KFD

* `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is
unavailable or cannot be opened, initialization continues with a warning
instead of aborting in `kfd_device_fd()`.

## Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

* `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for
metrics that do not have a hardware block.

Constant and topology metrics such as `simd_count` have an empty block
name. Passing that empty name to `aqlprofile_get_pmc_info()` can
dereference a null pointer and abort.

* `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A
system without a usable `/dev/kfd`, for example because the driver is
not loaded or access is denied, now receives a warning and continues
instead of aborting.

  Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The
DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4
command-buffer bounds change is limited to libhsakmt's DXG/WDDM
submission path.

## Tests

* **44 new tests**

  * 14 `wsl_dxg_topology` tests
  * 16 `wsl_dxg_thunk` tests
  * 10 `agent_mapping` tests
  * 4 `parse_gfx_target_version` tests

The first two groups are in new executables. The mapping and
target-version tests are added to the existing
`rocprofiler-sdk-lib-tests`.

None of these tests requires a GPU, HSA runtime, or real DXG thunk. The
DXG tests use a fake thunk and fake dynamic loader.

The 30 DXG-specific tests use the same CMake capability probe as the WSL
enumerator, so they are only built when the required hsakmt interface is
available. In a stock configuration, 14 of the 44 new tests are built.

* `platform-dispatch-tests` and its six tests are pre-existing from
#6399.

* `rocprofv3-avail`

* Three tests are moved out of the PC-sampling gate and are now enabled
on Navi.
* The L2 hit counter is selected by GPU family instead of being
hardcoded.

These tests have not previously run in Navi CI, so they may expose
existing Navi-specific issues.

* `hsa_barrier`

* Skip when no GPU agent is available instead of dereferencing an empty
agent map.

* `agent`

* `agent_visibility_multigpu` performs its `substr(1)` calls after the
no-GPU skip, avoiding `std::out_of_range` when no agents are present.
* `grid_max_size` tolerates the difference between rocprofiler's
hardcoded `INT32_MAX` value and the value reported by the installed ROCr
runtime.

## JIRA ID

JIRA ID: ROCM-23912

## Test Plan


## Test Result



## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Mika Laitio <mika.laitio@amd.com>
anujshuk-amd pushed a commit that referenced this pull request Sep 25, 2026
…7016)

## Motivation

Enable `rocprofv3` counter collection and profiling on WSL2 for
gfx11-class GPUs.

On WSL, the complete KMT topology is read up front through `librocdxg`,
using the same thunk interface as the HSA runtime. An agent is published
only after all required topology information has been collected, so the
published record remains stable for its lifetime.

As a result, `rocprofiler_query_available_agents()` returns consistent
agent information regardless of when it is called, including before the
HSA runtime has initialized. This is required by `rocprofv3-avail` and
counter metric lookup.

Native Linux continues to use the existing `gnulinux` path.
`select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()`
first, which succeeds whenever the KFD sysfs nodes directory is present,
so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are
fixes and are called out under **Native-Linux impact** below.

This work builds on
[[#6399](#6399)], which
introduced the platform dispatch structure used here.
`platform_dispatch.cpp` and its six tests came from that PR
(`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**.
This PR adds WSL as another implementation behind the existing
`select_platform()` interface.

## Dependency on
[#10034](#10034)

[#10034](#10034) contains the
libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that
enforces merge order, so the two PRs can technically merge in either
order. Their changes to `core/runtime/amd_topology.cpp` also do not
overlap. However, two parts of this PR depend on #10034 before they are
fully functional.

### 1. WSL enumeration is disabled in a stock build until the required
hsakmt headers are installed

WSL enumeration is controlled by the CMake probe
`rocprofiler_sdk_dxg_topology_supported()`
(`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`).

The probe requires the hsakmt headers used by the build to provide:

* `HsaStructureSizes`
* `HsaNodeProperties::LuidLowPart`
* `HsaNodeProperties::LuidHighPart`
* `OverrideEngineId`

Released ROCm headers, including 7.2.4, do not provide all of these.
When the probe fails, `agent_unsupported.cpp` is built instead of the
WSL enumerator, `platform::wsl::is_available()` returns false, and
`enumerate()` returns an empty result.

The copies of `hsakmttypes.h` in this repository already contain the
required definitions (lines 359-360 and 1578), but they are not
currently installed. #10034 adds the WSL header installation rules in
`libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`.

Until that lands, a stock build of this PR will not enable WSL
enumeration.

### 2. The DXG topology snapshot is not reference-counted on current
`develop`

Current `develop` unconditionally calls `topology_drop_snapshot()` from
`hsaKmtReleaseSystemProperties()`
(`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology
snapshot and destroys its `WDDMDevice` objects even if another consumer
still has the topology acquired.

#10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot
destruction conditional on the reference count.

The normal `rocprofv3` path is not affected because the SDK topology
read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before
`main()`, and therefore before the application's `hsa_init()`.

`rocprof-attach` is different because it injects into an already-running
process. In that case there is no ordering guarantee between the SDK and
the existing HSA runtime consumer, so late attach can fail until the
reference-counting change from #10034 is present.

## Technical Details

### Agent enumeration (WSL)

* `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology
through `librocdxg` using:

  * `hsaKmtOpenKFD()`
  * `hsaKmtAcquireSystemProperties()`
  * `hsaKmtGetNodeProperties()`
  * `hsaKmtReleaseSystemProperties()`
  * `hsaKmtCloseKFD()`

The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies
that all five are available before any are called.

* `HsaNodeProperties` has no size parameter, so the destination buffer
includes `kNodePropsSlack = 128` bytes of additional space.

* `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID,
with device ID as a fallback. Enumeration is allowed to be partial: an
unmatched adapter is skipped with a diagnostic while successfully
matched GPUs are still published.

* Memory banks, caches, and IO links are not currently published by the
WSL path.

* If the thunk reports no GPU node, no GPU agent is published. A warning
is emitted and the application continues.

* `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native
Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL
logs mismatches, keeps successfully paired GPUs, and continues.

### Counters

* `counters/controller.cpp` emits a one-time warning that WSL counters
are collected through the DXG vendor-packet path rather than the KFD
profiler ioctl.

* `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of
`is_available()`.

This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old
placement was bypassed. Without the environment variable, the DXG path
drops aqlprofile's vendor-specific PM4 IB and counter values are
returned as zero.

* `basic_counters.xml` and `derived_counters.xml` each gain an empty
`<gfx1150 base="gfx11">` section to keep the installed metric files
consistent.

These XML files are not used by the runtime metric loader. Runtime
metrics come from `config.yaml` (`counters/metrics.cpp:348`), which
already supports gfx1150 and is unchanged by this PR.

### KFD

* `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is
unavailable or cannot be opened, initialization continues with a warning
instead of aborting in `kfd_device_fd()`.

## Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

* `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for
metrics that do not have a hardware block.

Constant and topology metrics such as `simd_count` have an empty block
name. Passing that empty name to `aqlprofile_get_pmc_info()` can
dereference a null pointer and abort.

* `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A
system without a usable `/dev/kfd`, for example because the driver is
not loaded or access is denied, now receives a warning and continues
instead of aborting.

  Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The
DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4
command-buffer bounds change is limited to libhsakmt's DXG/WDDM
submission path.

## Tests

* **44 new tests**

  * 14 `wsl_dxg_topology` tests
  * 16 `wsl_dxg_thunk` tests
  * 10 `agent_mapping` tests
  * 4 `parse_gfx_target_version` tests

The first two groups are in new executables. The mapping and
target-version tests are added to the existing
`rocprofiler-sdk-lib-tests`.

None of these tests requires a GPU, HSA runtime, or real DXG thunk. The
DXG tests use a fake thunk and fake dynamic loader.

The 30 DXG-specific tests use the same CMake capability probe as the WSL
enumerator, so they are only built when the required hsakmt interface is
available. In a stock configuration, 14 of the 44 new tests are built.

* `platform-dispatch-tests` and its six tests are pre-existing from
#6399.

* `rocprofv3-avail`

* Three tests are moved out of the PC-sampling gate and are now enabled
on Navi.
* The L2 hit counter is selected by GPU family instead of being
hardcoded.

These tests have not previously run in Navi CI, so they may expose
existing Navi-specific issues.

* `hsa_barrier`

* Skip when no GPU agent is available instead of dereferencing an empty
agent map.

* `agent`

* `agent_visibility_multigpu` performs its `substr(1)` calls after the
no-GPU skip, avoiding `std::out_of_range` when no agents are present.
* `grid_max_size` tolerates the difference between rocprofiler's
hardcoded `INT32_MAX` value and the value reported by the installed ROCr
runtime.

## JIRA ID

JIRA ID: ROCM-23912

## Test Plan


## Test Result



## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Mika Laitio <mika.laitio@amd.com>
dgaliffiAMD pushed a commit that referenced this pull request Sep 25, 2026
…7016)

## Motivation

Enable `rocprofv3` counter collection and profiling on WSL2 for
gfx11-class GPUs.

On WSL, the complete KMT topology is read up front through `librocdxg`,
using the same thunk interface as the HSA runtime. An agent is published
only after all required topology information has been collected, so the
published record remains stable for its lifetime.

As a result, `rocprofiler_query_available_agents()` returns consistent
agent information regardless of when it is called, including before the
HSA runtime has initialized. This is required by `rocprofv3-avail` and
counter metric lookup.

Native Linux continues to use the existing `gnulinux` path.
`select_platform()` (`agent.cpp:355`) checks `gnulinux::is_available()`
first, which succeeds whenever the KFD sysfs nodes directory is present,
so the WSL path is not selected on a normal Linux system.

There are two ungated changes that also affect native Linux. Both are
fixes and are called out under **Native-Linux impact** below.

This work builds on
[[#6399](#6399)], which
introduced the platform dispatch structure used here.
`platform_dispatch.cpp` and its six tests came from that PR
(`d2f0b27ad5`) and are therefore listed as pre-existing under **Tests**.
This PR adds WSL as another implementation behind the existing
`select_platform()` interface.

## Dependency on
[#10034](#10034)

[#10034](#10034) contains the
libhsakmt and ROCr prerequisites for this work.

There is no CI gate, submodule pin, or build-time dependency that
enforces merge order, so the two PRs can technically merge in either
order. Their changes to `core/runtime/amd_topology.cpp` also do not
overlap. However, two parts of this PR depend on #10034 before they are
fully functional.

### 1. WSL enumeration is disabled in a stock build until the required
hsakmt headers are installed

WSL enumeration is controlled by the CMake probe
`rocprofiler_sdk_dxg_topology_supported()`
(`cmake/Modules/rocprofiler-sdk-utilities.cmake:120`).

The probe requires the hsakmt headers used by the build to provide:

* `HsaStructureSizes`
* `HsaNodeProperties::LuidLowPart`
* `HsaNodeProperties::LuidHighPart`
* `OverrideEngineId`

Released ROCm headers, including 7.2.4, do not provide all of these.
When the probe fails, `agent_unsupported.cpp` is built instead of the
WSL enumerator, `platform::wsl::is_available()` returns false, and
`enumerate()` returns an empty result.

The copies of `hsakmttypes.h` in this repository already contain the
required definitions (lines 359-360 and 1578), but they are not
currently installed. #10034 adds the WSL header installation rules in
`libhsakmt/CMakeLists_wsl.txt` through `ROCDXG_PUBLIC_HEADERS`.

Until that lands, a stock build of this PR will not enable WSL
enumeration.

### 2. The DXG topology snapshot is not reference-counted on current
`develop`

Current `develop` unconditionally calls `topology_drop_snapshot()` from
`hsaKmtReleaseSystemProperties()`
(`libhsakmt/src/dxg/topology.cpp:417`). This frees the global topology
snapshot and destroys its `WDDMDevice` objects even if another consumer
still has the topology acquired.

#10034 adds `src/dxg/snapshot_refcount.hpp` and makes snapshot
destruction conditional on the reference count.

The normal `rocprofv3` path is not affected because the SDK topology
read happens from `ROCPROFILER_LIBRARY_CTOR` and completes before
`main()`, and therefore before the application's `hsa_init()`.

`rocprof-attach` is different because it injects into an already-running
process. In that case there is no ordering guarantee between the SDK and
the existing HSA runtime consumer, so late attach can fail until the
reference-counting change from #10034 is present.

## Technical Details

### Agent enumeration (WSL)

* `platform/wsl/dxg_topology.cpp` and `dxg_thunk.hpp` read KMT topology
through `librocdxg` using:

  * `hsaKmtOpenKFD()`
  * `hsaKmtAcquireSystemProperties()`
  * `hsaKmtGetNodeProperties()`
  * `hsaKmtReleaseSystemProperties()`
  * `hsaKmtCloseKFD()`

The symbols are resolved with `dlsym`. `DxgThunk::complete()` verifies
that all five are available before any are called.

* `HsaNodeProperties` has no size parameter, so the destination buffer
includes `kNodePropsSlack = 128` bytes of additional space.

* `platform/wsl/agent.cpp` matches DXG adapters to HSA nodes by LUID,
with device ID as a fallback. Enumeration is allowed to be partial: an
unmatched adapter is skipped with a diagnostic while successfully
matched GPUs are still published.

* Memory banks, caches, and IO links are not currently published by the
WSL path.

* If the thunk reports no GPU node, no GPU agent is published. A warning
is emitted and the application continues.

* `agent_mapping.{hpp,cpp}` adds a WSL-specific mapping policy. Native
Linux keeps the existing strict behavior (`agent.cpp:755`), while WSL
logs mismatches, keeps successfully paired GPUs, and continues.

### Counters

* `counters/controller.cpp` emits a one-time warning that WSL counters
are collected through the DXG vendor-packet path rather than the KFD
profiler ioctl.

* `WSLKMT_VENDOR_PACKET=1` is now set from `wsl::enumerate()` instead of
`is_available()`.

This matters when using `ROCPROFILER_FORCE_PLATFORM=wsl`, where the old
placement was bypassed. Without the environment variable, the DXG path
drops aqlprofile's vendor-specific PM4 IB and counter values are
returned as zero.

* `basic_counters.xml` and `derived_counters.xml` each gain an empty
`<gfx1150 base="gfx11">` section to keep the installed metric files
consistent.

These XML files are not used by the runtime metric loader. Runtime
metrics come from `config.yaml` (`counters/metrics.cpp:348`), which
already supports gfx1150 and is unchanged by this PR.

### KFD

* `kfd::init()` now probes for a usable KFD device. If `/dev/kfd` is
unavailable or cannot be opened, initialization continues with a warning
instead of aborting in `kfd_device_fd()`.

## Native-Linux impact

Two changes are not WSL-gated and also apply to native Linux:

* `aql/helpers.cpp:51`: `get_query_info()` returns an empty ID query for
metrics that do not have a hardware block.

Constant and topology metrics such as `simd_count` have an empty block
name. Passing that empty name to `aqlprofile_get_pmc_info()` can
dereference a null pointer and abort.

* `kfd/kfd.cpp:1586`: the KFD availability check is unconditional. A
system without a usable `/dev/kfd`, for example because the driver is
not loaded or access is denied, now receives a warning and continues
instead of aborting.

  Other rocprofiler services are unaffected.

All other WSL-specific behavior remains isolated from native Linux. The
DXG implementation requires `/dev/dxg` and `libdxcore.so`, and the PM4
command-buffer bounds change is limited to libhsakmt's DXG/WDDM
submission path.

## Tests

* **44 new tests**

  * 14 `wsl_dxg_topology` tests
  * 16 `wsl_dxg_thunk` tests
  * 10 `agent_mapping` tests
  * 4 `parse_gfx_target_version` tests

The first two groups are in new executables. The mapping and
target-version tests are added to the existing
`rocprofiler-sdk-lib-tests`.

None of these tests requires a GPU, HSA runtime, or real DXG thunk. The
DXG tests use a fake thunk and fake dynamic loader.

The 30 DXG-specific tests use the same CMake capability probe as the WSL
enumerator, so they are only built when the required hsakmt interface is
available. In a stock configuration, 14 of the 44 new tests are built.

* `platform-dispatch-tests` and its six tests are pre-existing from
#6399.

* `rocprofv3-avail`

* Three tests are moved out of the PC-sampling gate and are now enabled
on Navi.
* The L2 hit counter is selected by GPU family instead of being
hardcoded.

These tests have not previously run in Navi CI, so they may expose
existing Navi-specific issues.

* `hsa_barrier`

* Skip when no GPU agent is available instead of dereferencing an empty
agent map.

* `agent`

* `agent_visibility_multigpu` performs its `substr(1)` calls after the
no-GPU skip, avoiding `std::out_of_range` when no agents are present.
* `grid_max_size` tolerates the difference between rocprofiler's
hardcoded `INT32_MAX` value and the value reported by the installed ROCr
runtime.

## JIRA ID

JIRA ID: ROCM-23912

## Test Plan


## Test Result



## Submission Checklist

- [ ] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

---------

Co-authored-by: Mika Laitio <mika.laitio@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants