rocprofv3: add rocprofv3 --doctor self-diagnostic option by bgopesh · Pull Request #12889 · ROCm/rocm-systems

bgopesh · 2026-10-06T09:04:46Z

Motivation

When rocprofv3 fails on first use, the cause is usually the environment rather than the profiler: a missing or mismatched library, /dev/kfd permissions, GPU devices not passed into a container, a conflicting LD_PRELOAD, or a broken Python environment. Today users have to work through each of these by hand.

rocprofv3 --doctor checks all of them in one command, explains what it found, and prints a copy-pasteable fix for each problem. It is an option of rocprofv3, so there is nothing separate to install or find, and it works with every ROCm installation layout, not only /opt/rocm.

--doctor is a placeholder name while naming is discussed; renaming it is a three-constant change plus docs.

Technical Details

How it works

flowchart TD
    A["rocprofv3 --doctor [options]<br/>(only rocprofv3 options before -- count;<br/>rocprofv3 execs libexec/rocprofiler-sdk/rocprofv3-doctor)"] --> B

    subgraph LOCATE["1. Locate the ROCm installation"]
        B["--rocm-root, if given"] -.->|else first hit| B2["install prefix of this script"]
        B2 -.-> B3["ROCM_PATH / ROCM_HOME / ROCM_DIR"]
        B3 -.-> B4["rocprofv3 on PATH"]
        B4 -.-> B5["TheRock Python package _rocm_sdk_core"]
        B5 -.-> B6["/opt/rocm, then newest /opt/rocm/core-X.Y"]
        B6 --> K["classify install kind<br/>system-package | therock-package |<br/>python-wheel | distro-package | standalone"]
    end

    K --> S

    subgraph SELECT["2. Select and schedule checks"]
        S["--only / --skip globs<br/>opt-in smoke checks via --run-smoke-test or --only"] --> S2{"every --only pattern<br/>matches a check?"}
        S2 -->|no| X2["exit 2: selection error"]
        S2 -->|yes| S3["add prerequisites of selected checks"]
        S3 --> S4["order so prerequisites run first"]
    end

    S4 --> R

    subgraph RUN["3. Run each check through SystemAccessor"]
        R{"prerequisites<br/>all passed?"} -->|no| SK["SKIP: names the prerequisite"]
        R -->|yes| P{"probe category"}
        P -->|passive| P1["read files, /sys, /proc, env;<br/>ldconfig -p; exclusive temp-file probe"]
        P -->|process| P2["child process with timeout:<br/>load libraries RTLD_NOW, import packages,<br/>run rocprofv3-avail --help"]
        P -->|gpu| P3["child process with timeout:<br/>rocprofv3-avail info --pmc,<br/>opt-in smoke checks"]
        P2 --> D{"command failed<br/>or crashed?"}
        P3 --> D
        D -->|yes| DG["diagnose the output<br/>(see next diagram)"]
        P1 --> RES
        D -->|no| RES
        DG --> RES
        P -.->|check raised an exception| ER["ERROR: doctor bug,<br/>not a system finding"]
    end

    RES["PASS / WARN / FAIL<br/>+ detail, remediation, data"] --> OUT
    SK --> OUT
    ER --> OUT

    subgraph REPORT["4. Report"]
        OUT["grouped text report + How to Fix<br/>or --format json (schema_version 1)"] --> EC{"exit code"}
        EC -->|any FAIL| E1["1"]
        EC -->|no FAIL, any ERROR| E2["2"]
        EC -->|otherwise| E0["0"]
    end

When a command the doctor runs fails, its output is turned into a diagnosis. A matched error message is treated as evidence, not as a conclusion:

flowchart LR
    O["exit status + stdout/stderr<br/>of the failed command"] --> M{"matches a known<br/>message?"}
    M -->|no| U["keep last output line as evidence;<br/>suggest ROCPROFILER_LOG_LEVEL=info"]
    M -->|yes| OB["observation<br/>what the output literally says"]
    OB --> C["corroborate with machine facts<br/>/dev/kfd access, LD_LIBRARY_PATH / LD_PRELOAD,<br/>*_VISIBLE_DEVICES, injected tools"]
    C --> H{"facts support<br/>one cause?"}
    H -->|yes| L["likely cause<br/>confidence: high"]
    H -->|no| PO["possible cause<br/>confidence: possible"]
    L --> F["remediation for that mechanism only<br/>+ related checks"]
    PO --> N["next check that tells<br/>the causes apart"]

For example, ROCr reports HSA_STATUS_ERROR_OUT_OF_RESOURCES both when /dev/kfd cannot be opened and for genuine allocation failures. The doctor checks /dev/kfd before naming a cause, and suggests group-membership fixes only when the device really is inaccessible.

Usage

rocprofv3 --doctor                       # full report
rocprofv3 --doctor --format json         # same report, as JSON
rocprofv3 --doctor --only runtime        # one group (plus its prerequisites)
rocprofv3 --doctor --rocm-root /opt/rocm/core-10.0
rocprofv3 --doctor --run-smoke-test      # also run rocprofv3 itself
rocprofv3 --doctor --list-checks         # every check, with its probe category

Checks

54 checks in 10 groups. 52 run by default; the 2 smoke checks are opt-in.

Group	Checks	What it covers
`install`	10	ROCm root and install kind, SDK/tool/roctx/avail/kokkosp/rocattach libraries (exact soversion), tool-vs-library version skew, conflicting ROCm installs, counter definitions
`driver`	7	`/dev/kfd`, DRI render nodes, `render`/`video` membership (account vs. current session), amdgpu driver, KFD topology
`runtime`	6	HSA, HIP, AQLProfile, ATT decoder, rocprofiler-register, and a load test of each ROCm library
`counters`	3	CAP_PERFMON for device locking, another profiler holding the GPU, counter enumeration
`environ`	7	running as root on behalf of another user (`sudo`, `su`), `LD_PRELOAD` / `HSA_TOOLS_LIB` / `ROCP_TOOL_LIBRARIES` conflicts, GPU visibility masks, beta gates, `ROCPROF_*` dump
`container`	3	container detection, device passthrough, virtualized GPU
`python`	8	interpreter, `sqlite3`/`ctypes`/`importlib`, optional `pyyaml`/`pandas`/`otf2`, the `rocprofv3` package
`filesystem`	3	output directory writable, free space, `/tmp`
`tools`	5	`rocprofv3`, `rocprofv3-avail`, `rocprof-attach`, `rocpd`, and `rocprofv3-avail --help` runs
`smoke` (opt-in)	2	`rocprofv3-avail info` finds a GPU; `rocprofv3` starts, injects its tool, and runs a program

What a run does to the machine

Every check declares a probe category, shown in --list-checks and in the JSON output.

Probe	What it does	When
`passive`	Reads files, `/sys`, `/proc`, and the environment; `ldconfig -p`. The output-directory checks create and delete one exclusively-created temporary file.	every run
`process`	Starts ROCm code in a time-limited child process: loads the ROCm libraries with `RTLD_NOW`, imports Python packages, runs `rocprofv3-avail --help`.	every run
`gpu`	Initializes the GPU runtime: `rocprofv3-avail info --pmc`.	every run; `--skip counters.avail-enumeration` to avoid it
opt-in	The `smoke` checks run `rocprofv3-avail info` and `rocprofv3` itself.	`--run-smoke-test`, or naming them with `--only`

There is no --fix mode and nothing is changed persistently. On timeout, a child's whole process tree is killed. Run it as the user who runs rocprofv3, without sudo; it warns when run as root on behalf of another account, and inside a root container the group-membership checks pass because they do not apply.

Finding the ROCm installation

ROCm is not assumed to live in /opt/rocm. The doctor recognizes the following layouts and tailors its fix commands to each:

ROCm system packages (/opt/rocm linked to /opt/rocm-X.Y.Z)
TheRock system packages (/opt/rocm/core-X.Y, with /opt/rocm/{bin,lib} as compatibility links)
TheRock Python wheels (_rocm_sdk_core in a virtualenv)
distribution packages (/usr)
standalone trees: tarballs, source builds and custom prefixes

For example, a missing library in a pip install suggests pip, not apt. Installations are compared by the files their paths resolve to, so /opt/rocm and /opt/rocm/core-X.Y count as one installation.

Results and exit codes

Status	Meaning
`PASS` / `WARN` / `FAIL`	finding about the system
`SKIP`	a prerequisite did not pass, or was excluded with `--skip`
`ERROR`	the check itself broke, which is a doctor bug and not a system finding

Exit codes:

0: no failures
1: at least one FAIL
2: the doctor could not run, an --only pattern matched no check, or a check broke and nothing failed

JSON output carries schema_version: 1.

Changes to existing files

source/bin/rocprofv3.py: --doctor execs the doctor script (no lingering rocprofv3 process; signals and exit status pass through) and is recognized only among rocprofv3's own options, so in rocprofv3 -- app --doctor the flag is still passed to app. The diff also contains black-only formatting changes: removed blank lines and one reflowed string.
CMake: installs the doctor script to <libexecdir>/rocprofiler-sdk/rocprofv3-doctor (off PATH, like rocprofiler-sdk-launch-compiler) and its Python modules, and exports the rocprofiler-sdk::rocprofv3-doctor target through config.cmake.in for the tests.
Docs: new how-to page, docs/how-to/using-rocprofv3-doctor.rst, added to the TOC and index.
CHANGELOG.md: new entry under rocprofv3.

New Python modules are prefixed doctor_, because rocprofiler_rocprofv3_python() installs into one flat directory. All system access goes through SystemAccessor (doctor_env.py), which is what makes every check testable without a GPU.

Issue Tracking

Test Plan

Tests live in tests/rocprofv3-doctor/. None require a GPU.

Suite	Kind	Registered as
`test_doctor_framework.py`	registry, selection, prerequisite scheduling, error status, rendering, JSON, `--doctor` dispatch	`rocprofv3-doctor-unit-tests` (label `unittests`)
`test_doctor_checks.py`	each check against synthetic machines through a fake accessor	same
`test_doctor_layout.py`	root detection, install kinds, and same-installation comparison for each layout; soversion resolution	same
`test_doctor_diagnose.py`	every error signature, corroboration and confidence, the load test	same
`test_doctor_system.py`	real processes and files: the write probe never follows a symlink, a timeout kills the whole process tree, signal vs. timeout codes, an import that aborts is contained, imports ignore modules in the current directory, output is capped	same
`validate.py`	the installed tool: text and JSON output, filters, exit codes, `--rocm-root`, `rocprofv3 --doctor` help and passthrough match the script	`rocprofv3-doctor` (label `integration-tests`)

In addition, these were run by hand:

The real CLI on a ROCm 6.2.0 machine.
The load test against shared libraries that are genuinely broken, built with gcc: a missing dependency, an undefined symbol, and a corrupt file.
Synthetic TheRock-package and tarball layouts on disk.

Test Result

261 tests passed, run with pytest directly against the source tree. The CMake/CTest registration has not been run from a build yet.
black --check is clean.
Doctor run on this machine: about 0.7 s, with no leftover temporary files.
The broken-library cases were each diagnosed correctly:
- the missing dependency was named
- the unresolved symbol was named
- the corrupt file was reported as an invalid shared object

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

🤖 Generated with Claude Code

therock-pr-bot · 2026-10-06T09:05:50Z

❌ PR Check — Action Required

Check	Status	Details
📝 PR Description	❌ Fail	Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword. Expected: include a `JIRA ID` / `ISSUE ID` line (separator `:` or `-`, or omitted; value may be a JIRA key, a number with/without `#`, or a link), OR a closing keyword + issue reference. Accepted examples: • `JIRA ID : TESTAUTO-6039` • `JIRA ID - #330` • `JIRA ID #330` • `JIRA ID` (on separate line) `ROCM-25757` • `ISSUE ID : TESTUTO-3334` • `ISSUE ID #3334` • `ISSUE ID - TESTAUTO-3433` • `ISSUE ID` (on separate line) `AIRUNTIME-2352` • `ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234` • `Closes #10` • `Fixes octo-org/octo-repo#100` • `Resolves: #123` • `#123` • `https://github.com/<org_name>/<repo_name>/issues/123` Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-06T09:05:53Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved issues affect package selection, counter database validation, process isolation, and diagnostic accuracy.

Review effort: Balanced
Findings: 1 · 10 · 1

Open (12)

Run sqlite3 probes in an isolated subprocess · New
Report hypervisor detection without asserting GPU virtualization · New
Exclude the dispatching parent from profiler process detection · New
Include primary GID when checking group membership · New
Recognize HIP_VISIBLE_DEVICES=-1 as hiding all GPUs · New
Clear all inspected visibility variables in remediation · New
Check versioned SDK libraries in the loader cache · New
Mirror SDK search paths and fallback for metrics configuration · New
Validate the counter configuration file actually consumed by the SDK · New
Mark checks invoking imports or SQLite as process probes · New
Limit high confidence to masks that hide all devices · New
Format entry and entries correctly in user messages · New

What changed in this PR

Adds the rocprofv3-doctor diagnostic CLI, including installation discovery, system checks, actionable reporting, integration with rocprofv3, documentation, and extensive tests.

Changes:

Adds 53 extensible diagnostic checks with text/JSON reporting.
Integrates rocprofv3 --doctor and installation targets.
Adds unit, system-boundary, integration, and layout tests.

File	Description
`projects/rocprofiler-sdk/CHANGELOG.md`	Documents the diagnostic tool.
`projects/rocprofiler-sdk/cmake/Templates/rocprofiler-sdk/config.cmake.in`	Exports the doctor target.
`projects/rocprofiler-sdk/source/bin/CMakeLists.txt`	Configures and installs the CLI.
`projects/rocprofiler-sdk/source/bin/rocprofv3-doctor.py`	Implements the standalone entry point.
`projects/rocprofiler-sdk/source/bin/rocprofv3.py`	Adds `--doctor` dispatch.
`projects/rocprofiler-sdk/source/docs/_toc.yml.in`	Adds documentation navigation.
`projects/rocprofiler-sdk/source/docs/index.rst`	Links the how-to page.
`projects/rocprofiler-sdk/source/docs/how-to/using-rocprofv3-doctor.rst`	Documents usage and diagnostics.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/CMakeLists.txt`	Installs doctor modules.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor.py`	Runs checks and renders reports.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_diagnose.py`	Diagnoses known failures.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_env.py`	Provides isolated system access.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_layout.py`	Detects installation layouts.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_registry.py`	Implements check registration and execution.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_result.py`	Defines result and probe types.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_container.py`	Adds container checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_counters.py`	Adds counter checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_driver.py`	Adds driver/device checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_environ.py`	Adds environment checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_filesystem.py`	Adds filesystem checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_install.py`	Adds installation checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_python_env.py`	Adds Python checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_runtime.py`	Adds runtime loading checks.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_smoke.py`	Adds opt-in smoke tests.
`projects/rocprofiler-sdk/source/lib/python/rocprofv3/doctor_checks_tools.py`	Adds companion-tool checks.
`projects/rocprofiler-sdk/tests/CMakeLists.txt`	Enables the test directory.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/CMakeLists.txt`	Registers doctor tests.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/conftest.py`	Provides synthetic-system fixtures.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/pytest.ini`	Configures test discovery.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/test_doctor_checks.py`	Tests individual checks.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/test_doctor_diagnose.py`	Tests failure diagnosis.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/test_doctor_framework.py`	Tests framework behavior.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/test_doctor_layout.py`	Tests installation layouts.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/test_doctor_system.py`	Tests real system boundaries.
`projects/rocprofiler-sdk/tests/rocprofv3-doctor/validate.py`	Tests CLI contracts.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The tool no longer installs a separate `rocprofv3-doctor` command (#12889): `rocprofv3 --doctor` is the only entry point. - SKILL.md finds and runs `rocprofv3 --doctor`, from PATH, a ROCm prefix, or a source checkout's rocprofv3.py for ROCm releases whose rocprofv3 rejects the option, and uses that command throughout. - It tells the agent to run the check as the user who runs rocprofv3, never with sudo, matching the tool's new run-as-user warning. - summarize_report.py, its tests, the skill card, and the evals use the new command; the behavior-graded evals also require `--doctor` in the logs and treat running the check as root as a mistake. The skill keeps its name, rocprofv3-doctor, until the flag's name is settled. Co-Authored-By: Claude <noreply@anthropic.com>

Reviewer (venkat1361): - WSL2: accept /dev/dxg when /dev/kfd is absent (KFD still wins when both exist). kfd-readable checks /dev/dxg; dri-devices, amdgpu-module and the render/video group checks SKIP; gpu-topology verifies libdxcore.so and librocdxg.so and HSA_ENABLE_DXG_DETECTION. The container passthrough check and the OUT_OF_RESOURCES diagnosis are WSL-aware. - Remove the standalone aqlprofile check: it is linked statically by default, and a separate one is covered by the SDK load test. - install.kokkosp-library is an error: rocprofv3.py resolves it with required=True on every launch. - The bootstrap honors CMAKE_INSTALL_LIBDIR, then lib64 and lib. Copilot: - Probe sqlite3 in an isolated child interpreter so a broken native extension fails one check instead of the doctor. - Python import checks declare probe=PROBE_PROCESS. - Resolve versioned SONAMEs through LD_LIBRARY_PATH and the ldconfig cache too; SDK libraries keep the exact-version rule. - ROCPROFILER_METRICS_PATH is a ':'-separated list with a fallback to the install path, and the metrics file is config.yaml. - Count the primary GID in the group checks. - HIP_VISIBLE_DEVICES=-1 hides every GPU; the remediation unsets every visibility variable that is set; hipErrorNoDevice blames a partial mask only as a possible cause. - The hypervisor message no longer claims GPU passthrough. - Proper singular/plural text instead of "entr(y|ies)". Unit tests cover each change, and the doctor docs are updated. Co-Authored-By: Claude <noreply@anthropic.com>

- GPU visibility: one helper, gpu_visibility_masks() in doctor_env, now feeds both environ.gpu-visibility and the hipErrorNoDevice diagnosis. It follows CLR and ROCr: an empty ROCR_VISIBLE_DEVICES hides every GPU, but an empty HIP/CUDA_VISIBLE_DEVICES restricts nothing; -1 hides every GPU in all three; a non-empty HIP_VISIBLE_DEVICES overrides CUDA_VISIBLE_DEVICES. - install.rocm-root fails when detection found no ROCm (SOURCE_NOT_FOUND) or the root has no ROCm libraries, so the library checks SKIP behind one root cause instead of each failing. driver.kfd-device no longer depends on it: the driver checks do not use the ROCm root and still run when none is found. - python.sqlite3 is a warning: only the rocpd CLI needs Python sqlite3; rocprofv3 writes rocpd output from C++. Tests cover each change, and the doctor docs are updated. Co-Authored-By: Claude <noreply@anthropic.com>

bgopesh · 2026-10-13T07:22:57Z

@bwelton and @venkat1361 Thanks for the review. I have addressed the feedback. Please re-review.

Adds `rocprofv3-doctor`, a self-diagnostic CLI that inspects the machine and the rocprofiler-sdk installation, reports what is healthy versus broken, and tells the user how to fix each problem. Usable standalone or as `rocprofv3 --doctor`. 53 checks across 10 groups (51 run by default; 2 smoke checks that run rocprofv3 itself are opt-in via --run-smoke-test or --only): - installation: SDK/tool/roctx/kokkosp/avail/rocattach libraries with exact-soversion resolution, tool-vs-library version skew, conflicting ROCm installs on the search path, counter metrics database - driver/device: /dev/kfd, DRI render nodes, render/video group membership, amdgpu driver, KFD topology GPU enumeration - runtime: ROCr, HIP, aqlprofile (required for --pmc and ATT), ATT trace decoder, rocprofiler-register, and a load test that loads each ROCm library in a child process with RTLD_NOW - counters: CAP_PERFMON/CAP_SYS_ADMIN (device locking), competing profiler holding the device, counter enumeration - environment: LD_PRELOAD conflicts, HSA_TOOLS_LIB, GPU visibility masking, beta gates, ROCPROF_*/ROCPROFILER_* dump - container: containerization detection, device passthrough, virt GPU - python: interpreter version, sqlite3/ctypes/importlib, optional pyyaml/pandas/otf2, rocprofv3 package importability - filesystem: output directory writability, disk space, /tmp - tools: rocprofv3, rocprofv3-avail, rocprof-attach, rocpd Design notes: - All system access is funnelled through a single injectable SystemAccessor, so every check is unit-testable without a GPU and the suite runs green in CI. - Checks declare prerequisites, which always run first and are pulled into --only selections; when one does not pass, its dependents report `skip` naming it, so a single root cause produces a single red line. An --only pattern matching no check is an error. - Each check is isolated: an exception becomes an `error` (a doctor bug) carrying the traceback, never a `fail` about the system. Every child process has a timeout that kills its whole process tree; native library loads and Python package imports run in children. - Every check declares its probe category (passive, process, gpu), published in --list-checks and JSON. The only file written is an exclusively-created temporary probe; smoke output goes to a private temporary directory. - Severity is deliberately conservative: ambiguous conditions report `warn` rather than `fail` to avoid false alarms. - Install-layout aware: ROCm is not assumed to live under /opt/rocm. The root is located from the script's install prefix, ROCM_PATH/ROCM_HOME/ROCM_DIR, rocprofv3 on PATH, an importable TheRock Python package (_rocm_sdk_core), then /opt/rocm and the newest /opt/rocm/core-X.Y. Legacy system packages, TheRock system packages, TheRock Python wheels, distro packages, and standalone trees (tarballs, source builds, custom prefixes) are classified, installations are compared by the files they resolve to rather than by path text, and remediation commands match the install type. - Diagnoses rather than only inventories: libraries are actually loaded, so missing dependencies (all of them, via ldd), mixed ROCm versions, old libstdc++, and corrupt files are caught. When a command the doctor runs fails or crashes, its output is matched against known loader, ROCr, HIP, SDK, and Python error messages and the report names the cause and fix instead of the exit code. A match is evidence, not a conclusion: each diagnosis separates the observation from the candidate cause, rates the cause `likely` only when facts on the machine corroborate it, and remediates only the mechanism actually implicated. Output is grouped human-readable text with a copy-pasteable remediation section, or versioned-schema JSON (--format json) for CI and bug reports. Supports --only/--skip with glob matching, --list-checks, --verbose/--quiet, --no-color (honouring NO_COLOR and non-TTY stdout), --rocm-root and --output. Exit codes: 0 pass/warn-only, 1 at least one failure, 2 tool error, unmatched --only, or a broken check with no failure. Includes 243 tests (unit, system-boundary, and integration) under tests/rocprofv3-doctor, a how-to doc page, and CHANGELOG entry. Co-Authored-By: Claude <noreply@anthropic.com>

Optional Python packages are imported in a child interpreter so that a native extension that aborts or hangs costs one check, not the report. That child ran as `python -c`, which puts the current directory first on sys.path: a stray rocprofv3.py, pandas.py, or yaml.py in the directory the doctor was run from shadowed the real module and was executed. Running from /tmp with a leftover /tmp/rocprofv3.py made python.rocprofv3-package report that file as the loaded package. Run the child isolated (-I) and hand it the doctor's exact sys.path instead of passing it through PYTHONPATH, so module resolution matches an in-process import and nothing in the working directory is imported. Co-Authored-By: Claude <noreply@anthropic.com>

The doctor is meant to run as the user who runs rocprofv3, but nothing noticed when it did not. Under sudo, root opens /dev/kfd regardless of group membership (hiding the real user's access problem), the group checks told the user to add root to render/video, the capability check described root, and sudo's reset environment changed what the environment and layout checks saw. Root inside a container -- the default for ROCm images, and legitimate there -- drew warnings that the render/video groups do not exist. - New check environ.run-as-user warns when the process is root on behalf of another account: SUDO_USER for sudo, and the audit login uid in /proc/self/loginuid for su, doas, and pkexec. A direct root login (loginuid 0) and containers and services (loginuid unset) are genuinely root and pass. - The render/video group checks pass for root, which does not need them; driver.kfd-readable still verifies real access. - A root process that cannot open /dev/kfd is pointed at user namespaces / rootless containers instead of usermod. - The how-to states who should run the tool. Co-Authored-By: Claude <noreply@anthropic.com>

The checks were reachable two ways: a separate `rocprofv3-doctor` command in bin/ and `rocprofv3 --doctor`, which spawned that command. Keep one: `rocprofv3 --doctor`. Users already have rocprofv3 on PATH, there is no second top-level command to name and package, and it matches how other tools are expected to expose the shared checks (`<tool> --doctor`). rocprofv3 imports only the standard library and dispatches before parsing arguments, so `--doctor` still starts in the broken setups it exists to diagnose. - The script moves to source/libexec/rocprofiler-sdk/rocprofv3-doctor/ and installs to <libexecdir>/rocprofiler-sdk/rocprofv3-doctor, off PATH, following rocprofiler-sdk-launch-compiler. The exported rocprofiler-sdk::rocprofv3-doctor target points there. - rocprofv3 finds it through the configured libexec dir (falling back to the source-checkout path) and execs it instead of running it as a child, so signals and the exit status pass straight through and no rocprofv3 process lingers to be mistaken for a running profiler. - counters.no-profiler-lock recognizes doctor runs exactly -- the script, or rocprofv3 with --doctor among its own options -- replacing a substring match that missed `rocprofv3 --doctor` and the shell that launched it. `rocprofv3 ... -- app --doctor` is still a profiler. - The command name lives in one place per component (DOCTOR_COMMAND / DOCTOR_FLAG, COMMAND in the script, DOCTOR_FLAG in rocprofv3.py) and every message that tells the user what to run uses it. `--doctor` is a placeholder pending naming; renaming it is a three-constant change plus docs. - Root detection derives the install prefix from the libexec location. - The CTest execute tests run through `rocprofv3 --doctor`; the docs, help text, and CHANGELOG describe a single option. Co-Authored-By: Claude <noreply@anthropic.com>

Three unit tests that load the rocprofv3 launcher failed in CI with FileNotFoundError on <build>/source/bin/rocprofv3.py: - CTest runs the tests from their copy in the build tree, but the unit-test registration passed only --rocprofv3-package-dir. The rocprofv3_path fixture then fell back to a source-tree path computed relative to the copy, which does not exist there. - Given the real path, loading still failed: the configured launcher is plain `rocprofv3`, and spec_from_file_location returns None for a file without a .py suffix. Pass --rocprofv3-path and --doctor-path (the imported targets' locations) to the unit tests as the validate test already does, load the launcher with an explicit SourceFileLoader, and make the path fixtures fail with a clear message instead of guessing a nonexistent path. Verified in a reproduction of the CI layout (tests copied into the build tree, extensionless configured rocprofv3): the same three tests failed before and pass after, with the full unit and validate suites green. Co-Authored-By: Claude <noreply@anthropic.com>

Reviewer (venkat1361): - WSL2: accept /dev/dxg when /dev/kfd is absent (KFD still wins when both exist). kfd-readable checks /dev/dxg; dri-devices, amdgpu-module and the render/video group checks SKIP; gpu-topology verifies libdxcore.so and librocdxg.so and HSA_ENABLE_DXG_DETECTION. The container passthrough check and the OUT_OF_RESOURCES diagnosis are WSL-aware. - Remove the standalone aqlprofile check: it is linked statically by default, and a separate one is covered by the SDK load test. - install.kokkosp-library is an error: rocprofv3.py resolves it with required=True on every launch. - The bootstrap honors CMAKE_INSTALL_LIBDIR, then lib64 and lib. Copilot: - Probe sqlite3 in an isolated child interpreter so a broken native extension fails one check instead of the doctor. - Python import checks declare probe=PROBE_PROCESS. - Resolve versioned SONAMEs through LD_LIBRARY_PATH and the ldconfig cache too; SDK libraries keep the exact-version rule. - ROCPROFILER_METRICS_PATH is a ':'-separated list with a fallback to the install path, and the metrics file is config.yaml. - Count the primary GID in the group checks. - HIP_VISIBLE_DEVICES=-1 hides every GPU; the remediation unsets every visibility variable that is set; hipErrorNoDevice blames a partial mask only as a possible cause. - The hypervisor message no longer claims GPU passthrough. - Proper singular/plural text instead of "entr(y|ies)". Unit tests cover each change, and the doctor docs are updated. Co-Authored-By: Claude <noreply@anthropic.com>

- GPU visibility: one helper, gpu_visibility_masks() in doctor_env, now feeds both environ.gpu-visibility and the hipErrorNoDevice diagnosis. It follows CLR and ROCr: an empty ROCR_VISIBLE_DEVICES hides every GPU, but an empty HIP/CUDA_VISIBLE_DEVICES restricts nothing; -1 hides every GPU in all three; a non-empty HIP_VISIBLE_DEVICES overrides CUDA_VISIBLE_DEVICES. - install.rocm-root fails when detection found no ROCm (SOURCE_NOT_FOUND) or the root has no ROCm libraries, so the library checks SKIP behind one root cause instead of each failing. driver.kfd-device no longer depends on it: the driver checks do not use the ROCm root and still run when none is found. - python.sqlite3 is a warning: only the rocpd CLI needs Python sqlite3; rocprofv3 writes rocpd output from C++. Tests cover each change, and the doctor docs are updated. Co-Authored-By: Claude <noreply@anthropic.com>

CI formats with black 26 (--required-version 26); these files had been formatted with black 23: - rocprofv3.py: hug the multi-line warning() string in parse_input, matching develop. - test_doctor_framework.py: drop the parentheses around a tuple unpacking target. Also remove the unused `import pytest` from validate.py (flake8 F401). Co-Authored-By: Claude <noreply@anthropic.com>

The tool no longer installs a separate `rocprofv3-doctor` command (#12889): `rocprofv3 --doctor` is the only entry point. - SKILL.md finds and runs `rocprofv3 --doctor`, from PATH, a ROCm prefix, or a source checkout's rocprofv3.py for ROCm releases whose rocprofv3 rejects the option, and uses that command throughout. - It tells the agent to run the check as the user who runs rocprofv3, never with sudo, matching the tool's new run-as-user warning. - summarize_report.py, its tests, the skill card, and the evals use the new command; the behavior-graded evals also require `--doctor` in the logs and treat running the check as root as a mistake. The skill keeps its name, rocprofv3-doctor, until the flag's name is settled. Co-Authored-By: Claude <noreply@anthropic.com>

Follow the project's tier + feature labeling (as rocpd, pc-sampling and kfd do): every doctor test now carries the rocprofv3-doctor label next to unittests or integration-tests, so `ctest -L rocprofv3-doctor` selects them all while CI's existing -L unittests / integration-tests selections are unchanged. Name the discovered unit tests rocprofv3-doctor.<function> and trim the repeated "test_doctor_" prefix, so rocprofv3-doctor-unit-tests.test_doctor_checks_kfd_device_pass becomes rocprofv3-doctor.checks_kfd_device_pass The integration tests keep the tests.integration.{execute,validate}.* names their macros generate. Verified with the real pytest discovery module: 294 tests discovered, all labeled unittests and rocprofv3-doctor, all passing. Co-Authored-By: Claude <noreply@anthropic.com>

Conflict in projects/rocprofiler-sdk/CHANGELOG.md: develop added an "HSA API tracing support for hsa_amd_enable_high_precision_timestamps" entry under Unreleased -> Added, where this branch adds the rocprofv3 --doctor entry. Keep both, develop's first. Co-Authored-By: Claude <noreply@anthropic.com>

The tool no longer installs a separate `rocprofv3-doctor` command (#12889): `rocprofv3 --doctor` is the only entry point. - SKILL.md finds and runs `rocprofv3 --doctor`, from PATH, a ROCm prefix, or a source checkout's rocprofv3.py for ROCm releases whose rocprofv3 rejects the option, and uses that command throughout. - It tells the agent to run the check as the user who runs rocprofv3, never with sudo, matching the tool's new run-as-user warning. - summarize_report.py, its tests, the skill card, and the evals use the new command; the behavior-graded evals also require `--doctor` in the logs and treat running the check as root as a mistake. The skill keeps its name, rocprofv3-doctor, until the flag's name is settled. Co-Authored-By: Claude <noreply@anthropic.com>

## Motivation Adds a `rocprofv3-doctor` agent skill that teaches AI coding agents to work out why rocprofv3 does not work on a machine. The agent runs `rocprofv3 --doctor`, finds the root causes in its report, and gives the user an ordered list of fixes. Like the `rocprofv3` skill in #12814, it is meant to be pulled into the amd/skills catalog from this repo. The two skills split the work. `rocprofv3` covers profiling once the setup works. `rocprofv3-doctor` covers what to do when rocprofv3 fails, crashes, finds no GPU, or will not start. > **Stacked on #12889**, which adds the `rocprofv3 --doctor` option this skill runs. This PR's base is that branch; it will be retargeted to `develop` once #12889 merges, and should not merge before it. ## Technical Details New folder `projects/rocprofiler-sdk/skills/rocprofv3-doctor/`, in the same layout as `skills/rocprofv3/` from #12814: - `SKILL.md`: - finds `rocprofv3 --doctor`: on `PATH`, under ROCm and TheRock prefixes, or from a source checkout's `rocprofv3.py` for ROCm releases whose `rocprofv3` rejects the option - runs it as the user who runs `rocprofv3`, never with `sudo` - runs it with JSON output and reads the exit codes - presents ordered fixes, with these rules: - act on `likely` causes; for `possible` ones, propose the step that tells causes apart - never run `sudo`, `usermod`, `sysctl`, package managers, or `pip` on the user's behalf - never add remediations the report does not give; in particular, no `perf_event_paranoid` changes and no `setcap` on the `rocprofv3` Python script - Its description routes plain profiling questions to the `rocprofv3` skill. - `scripts/summarize_report.py`: turns the doctor's JSON report (`schema_version` 1) into a markdown triage. It lists: - root causes, failures first, each with the checks it blocked - diagnoses, with their confidence and evidence - doctor errors, kept apart from findings about the system - what the run did not verify It uses only the Python standard library, and on input it cannot read it exits 2 with a message instead of a traceback. - `tests/test_summarize_report.py`: unit tests that need no GPU. - `evals/evals.json`: - 5 should-trigger cases, 2 of them graded on behavior: a readiness check, and a pasted `HSA_STATUS_ERROR_OUT_OF_RESOURCES` error that must be corroborated before a cause is named - 3 should-not cases: profiling hot kernels and choosing counters (the `rocprofv3` skill's job), and a PyTorch profiler trace - `evals/machine.yml`: `os: [Linux]`, `labels: [gpu, rocm]`. - `skill-card.md`. ## Issue Tracking JIRA ID: <TODO> ## Test Plan - Ran the skill's unit tests with Python 3.8 and 3.10. - Ran the skill's steps against #12889 on a ROCm 6.2.0 machine: the installed `rocprofv3` rejects `--doctor` as the skill expects, the source-checkout fallback runs, and the script summarizes the real report. - Ran amd/skills structural validation: `skillscope structural` v0.1.2, the version and arguments its `check.sh` uses, with the skill copied into a catalog checkout. - Confirmed that the same validation rejects a deliberately broken dataset. - Ran black. ## Test Result - 11 unit tests pass on both Python versions, and black is clean. - The script folds skipped checks into their root causes and separates informational findings. - Catalog structural checks pass: 11 skill folders and 85 cases in shape. - Routing and behavioral evals (`skillscope routing` / `behavioral`) have not been run yet; they need the `claude` CLI and a ROCm machine. ## Submission Checklist - [ ] Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

systems-assistant · 2026-10-19T11:48:40Z

TheRock Submodule Bump Activity

Newest first

2026-10-19 — Included in TheRock via ROCm/TheRock#9353.

Resolve the CHANGELOG conflict by keeping the rocprofv3 --doctor (#12889) and hsa_amd_queue_create (#13720) entries from develop alongside this PR's Unreleased entries. Replace the queue-interposition, SPM agent-scoping, and kernel replay iteration-range entries with more detailed versions (#11967-#11970, #11335). Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI balanced review requested due to automatic review settings October 6, 2026 09:04

bgopesh requested review from a team as code owners October 6, 2026 09:04

bgopesh marked this pull request as draft October 6, 2026 09:04

github-actions Bot added documentation Improvements or additions to documentation project: rocprofiler-sdk labels Oct 6, 2026

Copilot started reviewing on behalf of bgopesh October 6, 2026 09:05

View session

therock-pr-bot Bot added the Not ready to Review label Oct 6, 2026

Copilot AI reviewed Oct 6, 2026

View reviewed changes

systems-assistant Bot added the organization: ROCm label Oct 6, 2026

bgopesh mentioned this pull request Oct 6, 2026

feat(rocprofiler-sdk): add rocprofv3-doctor agent skill #12895

Merged

1 task

bgopesh added this pull request to stack #12896 October 6, 2026 10:45

bgopesh changed the title ~~rocprofv3: add rocprofv3-doctor self-diagnostic tool~~ rocprofv3: add rocprofv3 --doctor self-diagnostic tool Oct 7, 2026

bgopesh changed the title ~~rocprofv3: add rocprofv3 --doctor self-diagnostic tool~~ rocprofv3: add rocprofv3 --doctor self-diagnostic option Oct 7, 2026

bgopesh marked this pull request as ready for review October 7, 2026 08:24

venkat1361 requested changes Oct 12, 2026

View reviewed changes

bgopesh requested a review from venkat1361 October 12, 2026 14:02

bwelton reviewed Oct 12, 2026

View reviewed changes

bgopesh requested a review from bwelton October 13, 2026 07:23

bgopesh and others added 5 commits October 13, 2026 13:08

bgopesh and others added 3 commits October 13, 2026 13:09

bgopesh force-pushed the users/bgopesh/rocprofv3-doctor branch from b151c1f to 150b7aa Compare October 13, 2026 07:41

venkat1361 approved these changes Oct 13, 2026

View reviewed changes

bgopesh merged commit e7b4327 into develop Oct 17, 2026
136 of 139 checks passed

bgopesh deleted the users/bgopesh/rocprofv3-doctor branch October 17, 2026 13:59

This was referenced Oct 21, 2026

[rocjitsu] Correct legacy floating min/max and median selection #13898

Merged

[rocjitsu] Expose backed CDNA3 and CDNA4 hardware identity fields #13899

Merged

[rocjitsu] Implement scratch LDS transfers and scalar cache discard #13900

Merged

pmoutsias-amd mentioned this pull request Nov 3, 2026

Move ROCprofiler-SDK changelog entries for code merged after the ROCm 10.2.0 branch to the Unreleased section #14943

Open

Conversation

bgopesh commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

How it works

Usage

Checks

What a run does to the machine

Finding the ROCm installation

Results and exit codes

Changes to existing files

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 6, 2026

❌ PR Check — Action Required

Uh oh!

therock-pr-bot Bot commented Oct 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bgopesh commented Oct 13, 2026

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 19, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bgopesh commented Oct 6, 2026 •

edited

Loading