Add automatic PyTorch library discovery for Python applications by mradosav-amd · Pull Request #2623 · ROCm/rocm-systems

mradosav-amd · 2026-01-15T11:57:27Z

Motivation

When profiling PyTorch applications with rocprof-sys-run or rocprof-sys-sample, the PyTorch libraries may not be in LD_LIBRARY_PATH, causing runtime linking issues. This change automatically discovers and adds the PyTorch library path when profiling Python applications that use PyTorch.

Technical Details

Added is_python_interpreter() function to detect if the target executable is a Python interpreter (python, python3, python3.X)
Added discover_torch_libpath() function that runs Python to query torch installation path via torch.__path__
Added add_torch_library_path() template function to prepend discovered torch lib directory to LD_LIBRARY_PATH
Added consolidate_env_entries() function to deduplicate environment variable entries (removes duplicate paths in colon-separated values)
Integrated torch library path discovery into both rocprof-sys-run and rocprof-sys-sample tools

Known issue:

Unable to detect when to add LD_LIBRARY_PATH if Python is run via a shell script, for example

JIRA ID

AIPROFSYST-107

Test Plan

Unit tests for is_python_interpreter() covering various Python interpreter names and paths
Unit tests for consolidate_env_entries() verifying deduplication of environment variable entries
Manual testing: Profile a PyTorch Python application with rocprof-sys-run and verify torch libraries are loaded correctly

Test Result

is_python_interpreter() should correctly identify python, python3, python3.X executables (including full paths) and reject non-Python executables
consolidate_env_entries() should merge duplicate environment variable keys and deduplicate path entries
PyTorch applications should run successfully under profiling without manual LD_LIBRARY_PATH configuration

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Copilot

Pull request overview

This PR adds automatic PyTorch library path discovery and configuration for Python applications being profiled with rocprof-sys tools, eliminating the need for manual LD_LIBRARY_PATH setup.

Changes:

Added Python interpreter detection via is_python_interpreter() to identify python/python3/python3.X executables
Added PyTorch library path discovery via discover_torch_libpath() that queries torch installation location
Added environment variable consolidation via consolidate_env_entries() to deduplicate colon-separated path entries
Integrated torch library path discovery into rocprof-sys-run and rocprof-sys-sample tools

Reviewed changes

Copilot reviewed 5 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
projects/rocprofiler-systems/source/lib/core/argparse.hpp	Declares `add_torch_library_path()` function for torch library path integration
projects/rocprofiler-systems/source/lib/core/argparse.cpp	Implements wrapper for `add_torch_library_path()` that delegates to common library
projects/rocprofiler-systems/source/lib/common/tests/test_environment.cpp	Adds unit tests for Python interpreter detection and environment variable deduplication
projects/rocprofiler-systems/source/lib/common/tests/CMakeLists.txt	Registers new test file in build system
projects/rocprofiler-systems/source/lib/common/environment.hpp	Implements core logic for Python detection, torch path discovery, path consolidation, and LD_LIBRARY_PATH manipulation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

applications (#2623) * Add automatic PyTorch library discovery for Python applications (#2623) [rocm-systems] ROCm/rocm-systems#2623 (commit b533f56)

…nal-merge-new-729 (#459) * SWDEV-549518 - Enable logging dynamically through HIP APIS. (#1079) * SWDEV-549518 - Enable logging dynamically through HIP APIS. * SWDEV-549518 - Adding ROCProfiler related new API changes. * rocprofiler-sdk changes for hip api additions. --------- Co-authored-by: Venkateshwar Reddy Kandula <venkateshwar.kandula1306@gmail.com> Co-authored-by: jainprad <92369414+jainprad@users.noreply.github.com> * [rocprof-compute] Pin ruff version for consistent formatting (#2680) * pin ruff versions each to current latest * Update rocprofiler-compute-formatting.yml * Downgrade .pre-commit-config.yaml to match develop * Fix Python Formatting (#2679) Updated version of black to 26.1.0 updated some formatting rules Signed-off-by: David Galiffi <David.Galiffi@amd.com> * Add automatic PyTorch library discovery for Python applications (#2623) * Add automatic PyTorch library discovery for Python applications (#2623) * Add notice for the newly deprecated env variables (#2690) * [rocprofiler-compute] Use TheRock nightly builds in testing container (#2661) * Use TheRock nightly builds in testing container * Add HIP_DEVICE_LIB_PATH env var for hipcc to work * Add HIP_PLATFORM env var for cmake hip package * Add tarball placeholder * Add -f to curl command to fail on HTTP error * [rocprofiler-compute] Fix kernel/dispatch filtering (#2479) * Fix kernel/dispatch fitlering in GUI * Disallow --kernel and --dispatch filtering in analyze --gui mode since GUI frontend offers dropdown menu for kernel and dispatch filtering * Update CHANGELOG and documentation * Gracefully handle N/A values * Ensure workload path is valid before using it in GUI * Ignore kernel filters if dispatch filters provided * Add documentation for dispatch filtering overriding kernel filtering * Fix typo * Fix documentation * remove unnecessary whitespace * Address review comments * Allow kernel/dispatch filtering with --gui * Address review comments * Address review comments * Update CHANGELOG * Fix formatting * [rocprofiler-systems] Add build option for "examples" to specify gfx-arch (#2626) ## Motivation - Added `check_rocminfo` function that returns true if the provided regex was found, false otherwise. Can also use `GET_OUTPUT` to get the raw output filtered with or without a regex. - Moved `rocprofiler_systems_get_gfx_archs()` to `MacroUtilities.cmake` - Added `rocprofiler_systems_lookup_gfx()`, which detects whether a given `gfx` is from the `instinct`, `radeon` or `apu` family. - Added `ROCPROFSYS_GFX_TARGETS` as a build argument. Used to specify the offloading architectures that GPU examples should compile for. If empty, defaults to whatever your system has. - GPU examples now check if the given `gfx` targets (from `ROCPROFSYS_GFX_TARGETS`) are supported. - OMPVV offload tests now only compile if `amdflang` version is `>= 20` - Improve link time by reducing the number of GFX targets that binaries need to support. - RCCL is now passed a `GPU_TARGETS` var specifying the architectures to build/link against. * Reset HIP_VERSION_PATCH to 0 (#2590) * [SWDEV-559965] Update Changelog for power cap type (#2647) * [SWDEV-559965] Update Changelog for amd-smi set --power-cap Updated Changelog to mention flexible argument ordering for power cap type in amdsmi power cap set. Corrected Changelog documentation on PPT1 reset power_cap command. Signed-off-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> Co-authored-by: Maisam Arif <Maisam.Arif@amd.com> * [rocprofiler-system]: Enable UCX Communication API tracing (#2306) ## Motivation Enable UCX communication tracing and communication metadata ## Technical Details Implement UCX API wrappers to trace transport-layer communication. This adds communication data tracking and exposes “UCX Comm Send/Recv” timelines, enabling detailed analysis of MPI, OpenSHMEM, and other UCX-based runtime communication patterns. - Implements function interception for UCX functions across multiple categories using gotcha component. - Extended comm_data component to track UCX send/recv operations - Added ucx_send and ucx_recv labels for Perfetto counter tracks. Integrated UCX data tracking with existing MPI/RCCL tracking infrastructure. - Added ROCPROFSYS_USE_UCX configuration option (enabled by default). - Created FindUCX.cmake module for UCX header detection. Falls back to internal UCX headers if system headers not found. - Updated all Dockerfiles to include UCX dependencies. * hip-issue-3876 : Take into account thread-local capture mode in checks for valid capture (#2177) * Revert "rocr: Switch back to legacy IPC (#1744)" (#2676) This reverts commit 7e4b622. * SWDEV-558849 - Add support for static linking with ROCR (#2659) * [rocprofiler-systems] Fix MPI recv_data calculation (#2694) Fix incorrect `mpi_recv` calculation. It was using `_send_size` instead of `_recv_size` for `mpi_recv`. * [ROCmInfo] docs: mono-repo changes and style edits (#2584) * initial edits * mono repo related updates * standardize component name * style edits * more edits * Update CODEOWNERS (#2705): add /project/amdsmi owner * Use size_t datatype for global dimensions. (#2604) * rocrtst: set HSA_ENABLE_INTERRUPT after TestExample creation (#2687) Signed-off-by: Horatio Zhang <Hongkun.Zhang@amd.com> Co-authored-by: cfreeamd <166262151+cfreeamd@users.noreply.github.com> * Rework clock based unit tests (#2646) * Update hip_api_trace.cpp for HIP_RUNTIME_API_TABLE_STEP_VERSION 22 * Update hip_api_trace.hpp for HIP_RUNTIME_API_TABLE_STEP_VERSION 22 --------- Signed-off-by: David Galiffi <David.Galiffi@amd.com> Signed-off-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> Signed-off-by: Horatio Zhang <Hongkun.Zhang@amd.com> Co-authored-by: Karthik Jayaprakash <54370791+kjayapra-amd@users.noreply.github.com> Co-authored-by: Venkateshwar Reddy Kandula <venkateshwar.kandula1306@gmail.com> Co-authored-by: jainprad <92369414+jainprad@users.noreply.github.com> Co-authored-by: jamessiddeley-amd <James.Siddeley@amd.com> Co-authored-by: David Galiffi <David.Galiffi@amd.com> Co-authored-by: Milan Radosavljevic <milan.radosavljevic@amd.com> Co-authored-by: marantic-amd <marantic@amd.com> Co-authored-by: vedithal-amd <Vignesh.Edithal@amd.com> Co-authored-by: Kian Cossettini <Kian.Cossettini@amd.com> Co-authored-by: Rakesh Roy <137397847+rakesroy@users.noreply.github.com> Co-authored-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Co-authored-by: Maisam Arif <Maisam.Arif@amd.com> Co-authored-by: Sajina PK <sputhala@amd.com> Co-authored-by: Ioannis Assiouras <38722728+iassiour@users.noreply.github.com> Co-authored-by: Alysa Liu <Alysa.Liu@amd.com> Co-authored-by: German Andryeyev <56892148+gandryey@users.noreply.github.com> Co-authored-by: yugang-amd <yugang.wang@amd.com> Co-authored-by: JeniferC99 <150404595+JeniferC99@users.noreply.github.com> Co-authored-by: hongkzha-amd <Hongkun.Zhang@amd.com> Co-authored-by: cfreeamd <166262151+cfreeamd@users.noreply.github.com> Co-authored-by: Jatin Chaudhary <51944368+cjatin@users.noreply.github.com> Co-authored-by: kjayapra-amd <karthik.jayaprakash@amd.com> Co-authored-by: hsivasun_amdeng <haresh.sivasuntharampillai@amd.com> Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>

mradosav-amd requested review from a team and jrmadsen as code owners January 15, 2026 11:57

github-actions Bot added the project: rocprofiler-systems label Jan 15, 2026

systems-assistant Bot added the organization: ROCm label Jan 15, 2026

dgaliffiAMD requested a review from Copilot January 15, 2026 14:41

Copilot AI reviewed Jan 15, 2026

View reviewed changes

dgaliffiAMD approved these changes Jan 16, 2026

View reviewed changes

Comment thread projects/rocprofiler-systems/source/lib/common/tests/test_environment.cpp Outdated

Comment thread projects/rocprofiler-systems/source/lib/common/tests/test_environment.cpp

mradosav-amd force-pushed the users/mradosav-amd/libcaffe-torch-discovery branch 2 times, most recently from 529f486 to 72c9a3f Compare January 19, 2026 14:23

habajpai-amd reviewed Jan 19, 2026

View reviewed changes

Comment thread projects/rocprofiler-systems/source/lib/common/environment.hpp

mradosav-amd added 6 commits January 19, 2026 23:59

Fix torch lib

320d3e8

cleanup

04431a8

cleanup2

7ec8c01

Applied suggestions from code review

d215fe2

Applied suggestions from code review

90d8afb

Applied suggestions from code review

b3b4c6f

dgaliffiAMD force-pushed the users/mradosav-amd/libcaffe-torch-discovery branch from ef0d27f to b3b4c6f Compare January 20, 2026 04:59

mradosav-amd merged commit b533f56 into develop Jan 20, 2026
60 of 61 checks passed

mradosav-amd deleted the users/mradosav-amd/libcaffe-torch-discovery branch January 20, 2026 07:42

charis-poag-amd mentioned this pull request Jan 20, 2026

[SWDEV-552020/SWDEV-563971] Fail memory partition tests if ASIC supports memory partitions & kmod not installed #2695

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add automatic PyTorch library discovery for Python applications#2623

Add automatic PyTorch library discovery for Python applications#2623
mradosav-amd merged 6 commits into
developfrom
users/mradosav-amd/libcaffe-torch-discovery

mradosav-amd commented Jan 15, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mradosav-amd commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mradosav-amd commented Jan 15, 2026 •

edited

Loading