Repository navigation
fix(rocprofiler-sdk): skip HIP visibility matching for ROCR-hidden agents - #12951
Merged
venkat1361 merged 1 commit intoOct 27, 2026
Merged
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
itrowbri
approved these changes
Oct 26, 2026
bgopesh
approved these changes
Oct 27, 2026
…ents HIP_VISIBLE_DEVICES indexes the ROCR-visible device list, but agents hidden by ROCR_VISIBLE_DEVICES were matched against HIP tokens using their absolute GPU ordinal. Any process setting both variables (e.g. ROCR=4 HIP=0) warned 'Attempt to enable hip visibility for agent-N which is not visible to HSA' for an unrelated agent. Mark ROCR-hidden agents invisible to HSA and HIP without consulting the HIP-family variables. Fixes AIPROFSDK-1169
venkat1361
force-pushed
the
users/bewelton/aiprofsdk-1169-visibility-warning
branch
from
October 27, 2026 16:04
3afd834 to
99adfe7
Compare
venkat1361
deleted the
users/bewelton/aiprofsdk-1169-visibility-warning
branch
October 27, 2026 22:50
ywang103-amd
pushed a commit
that referenced
this pull request
Oct 28, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HIP_VISIBLE_DEVICESindexes the list of devices ROCR exposes, but rocprofiler-sdk matched HIP tokens against the absolute GPU ordinal for agents hidden byROCR_VISIBLE_DEVICES. Any process setting both variables (e.g.ROCR_VISIBLE_DEVICES=4 HIP_VISIBLE_DEVICES=0) logged a spurious warning naming an unrelated agent:This breaks PyTorch's
dynamo/test_logging.py::LoggingTests::test_logs_out(AIPYTORCH-1297) because of the unexpected stderr output.Changes
agent.cpp: whenROCR_VISIBLE_DEVICESis set and does not include the agent, mark it invisible to HSA and HIP and return before theHIP_VISIBLE_DEVICES/CUDA_VISIBLE_DEVICES/GPU_DEVICE_ORDINALmatching. HIP (rocclr/device/rocm/rocdevice.cpp) only enumerates agents returned byhsa_iterate_agents, so a ROCR-hidden agent can never be HIP-visible. The resulting visibility values are unchanged; only the false warning is removed.tests/agent.cpp: newagent_visibility_rocr_hip_stackedtest covering the ticket's table (ROCRunset /HIP=4,ROCR=3,4 HIP=1,ROCR=4,5 HIP=0,ROCR=4 HIP=4). It checks both the per-agent HSA/HIP/RCCL/rocDecode visibility and that the warning is not emitted. It uses the warning thatget_gpu_agents()always logs as a positive control, so the test cannot pass just because stderr capture failed.Testing
On an 8x MI325X (gfx942) host, standalone rocprofiler-sdk build against TheRock 10.1.0a20260910:
rocprofiler-sdk-lib-tests(RelWithDebInfo): clean, 0 warnings.agent.cppchange reverted,agent_visibility_rocr_hip_stackedfails on exactly theROCR=3,4 HIP=1andROCR=4,5 HIP=0cases, with thenot visible to HSAwarning for agents 4/5/6. With the fix it passes.ctest -R '^unit\.(rocprofiler_lib|agent_mapping|parse_gfx_target_version|contexts|enum_string|hsa|LateStartTest|queue_controller|queue_interposition|hsa_barrier)\.': every test whose binary was built passes, exceptunit.rocprofiler_lib.agent. It fails identically without this change (product_name"unknown" vs emptydevice_mkt_nameon this host's runtime).rocprofiler_lib.agent_visibility*+agent_abipass with zero sanitizer reports.Tracking
JIRA ID : AIPROFSDK-1169