[Rocprofiler-system]: Enable UCX Communication tracing by sputhala-amd · Pull Request #2306 · ROCm/rocm-systems

sputhala-amd · 2025-12-13T03:38:59Z

Motivation

Enable UCX communication tracing and communication metadata

Technical Details

Implement UCX API wrappers to trace transport-layer communication. This adds communication data tracking and exposes “UCX Comm Send/Recv” timelines, enabling detailed analysis of MPI, OpenSHMEM, and other UCX-based runtime communication patterns.

Implements function interception for UCX functions across multiple categories using gotcha component.
Extended comm_data component to track UCX send/recv operations - Added ucx_send and ucx_recv labels for Perfetto counter tracks. Integrated UCX data tracking with existing MPI/RCCL tracking infrastructure.
Added ROCPROFSYS_USE_UCX configuration option (enabled by default).
Created FindUCX.cmake module for UCX header detection. Falls back to internal UCX headers if system headers not found.
Updated all Dockerfiles to include UCX dependencies:

For documentation created a separate PR- #2478.

JIRA ID

AIPROFSYST-36

Test Plan

Adds automation ctests for UCX trace validation using existing MPI application with UCX transport layer.
Adds unit tests for UCX gotcha component as well as communication data tracking.

Test Result

All tests pass successfully.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

(#2306) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Motivation Enable UCX communication tracing and communication metadata ## Technical Details Implement UCX API wrappers to trace transport-layer communication. This adds communication data tracking and exposes “UCX Comm Send/Recv” timelines, enabling detailed analysis of MPI, OpenSHMEM, and other UCX-based runtime communication patterns. - Implements function interception for UCX functions across multiple categories using gotcha component. - Extended comm_data component to track UCX send/recv operations - Added ucx_send and ucx_recv labels for Perfetto counter tracks. Integrated UCX data tracking with existing MPI/RCCL tracking infrastructure. - Added ROCPROFSYS_USE_UCX configuration option (enabled by default). - Created FindUCX.cmake module for UCX header detection. Falls back to internal UCX headers if system headers not found. - Updated all Dockerfiles to include UCX dependencies. [rocm-systems] ROCm/rocm-systems#2306 (commit 15c82d6)

…nal-merge-new-729 (#459) * SWDEV-549518 - Enable logging dynamically through HIP APIS. (#1079) * SWDEV-549518 - Enable logging dynamically through HIP APIS. * SWDEV-549518 - Adding ROCProfiler related new API changes. * rocprofiler-sdk changes for hip api additions. --------- Co-authored-by: Venkateshwar Reddy Kandula <venkateshwar.kandula1306@gmail.com> Co-authored-by: jainprad <92369414+jainprad@users.noreply.github.com> * [rocprof-compute] Pin ruff version for consistent formatting (#2680) * pin ruff versions each to current latest * Update rocprofiler-compute-formatting.yml * Downgrade .pre-commit-config.yaml to match develop * Fix Python Formatting (#2679) Updated version of black to 26.1.0 updated some formatting rules Signed-off-by: David Galiffi <David.Galiffi@amd.com> * Add automatic PyTorch library discovery for Python applications (#2623) * Add automatic PyTorch library discovery for Python applications (#2623) * Add notice for the newly deprecated env variables (#2690) * [rocprofiler-compute] Use TheRock nightly builds in testing container (#2661) * Use TheRock nightly builds in testing container * Add HIP_DEVICE_LIB_PATH env var for hipcc to work * Add HIP_PLATFORM env var for cmake hip package * Add tarball placeholder * Add -f to curl command to fail on HTTP error * [rocprofiler-compute] Fix kernel/dispatch filtering (#2479) * Fix kernel/dispatch fitlering in GUI * Disallow --kernel and --dispatch filtering in analyze --gui mode since GUI frontend offers dropdown menu for kernel and dispatch filtering * Update CHANGELOG and documentation * Gracefully handle N/A values * Ensure workload path is valid before using it in GUI * Ignore kernel filters if dispatch filters provided * Add documentation for dispatch filtering overriding kernel filtering * Fix typo * Fix documentation * remove unnecessary whitespace * Address review comments * Allow kernel/dispatch filtering with --gui * Address review comments * Address review comments * Update CHANGELOG * Fix formatting * [rocprofiler-systems] Add build option for "examples" to specify gfx-arch (#2626) ## Motivation - Added `check_rocminfo` function that returns true if the provided regex was found, false otherwise. Can also use `GET_OUTPUT` to get the raw output filtered with or without a regex. - Moved `rocprofiler_systems_get_gfx_archs()` to `MacroUtilities.cmake` - Added `rocprofiler_systems_lookup_gfx()`, which detects whether a given `gfx` is from the `instinct`, `radeon` or `apu` family. - Added `ROCPROFSYS_GFX_TARGETS` as a build argument. Used to specify the offloading architectures that GPU examples should compile for. If empty, defaults to whatever your system has. - GPU examples now check if the given `gfx` targets (from `ROCPROFSYS_GFX_TARGETS`) are supported. - OMPVV offload tests now only compile if `amdflang` version is `>= 20` - Improve link time by reducing the number of GFX targets that binaries need to support. - RCCL is now passed a `GPU_TARGETS` var specifying the architectures to build/link against. * Reset HIP_VERSION_PATCH to 0 (#2590) * [SWDEV-559965] Update Changelog for power cap type (#2647) * [SWDEV-559965] Update Changelog for amd-smi set --power-cap Updated Changelog to mention flexible argument ordering for power cap type in amdsmi power cap set. Corrected Changelog documentation on PPT1 reset power_cap command. Signed-off-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> Co-authored-by: Maisam Arif <Maisam.Arif@amd.com> * [rocprofiler-system]: Enable UCX Communication API tracing (#2306) ## Motivation Enable UCX communication tracing and communication metadata ## Technical Details Implement UCX API wrappers to trace transport-layer communication. This adds communication data tracking and exposes “UCX Comm Send/Recv” timelines, enabling detailed analysis of MPI, OpenSHMEM, and other UCX-based runtime communication patterns. - Implements function interception for UCX functions across multiple categories using gotcha component. - Extended comm_data component to track UCX send/recv operations - Added ucx_send and ucx_recv labels for Perfetto counter tracks. Integrated UCX data tracking with existing MPI/RCCL tracking infrastructure. - Added ROCPROFSYS_USE_UCX configuration option (enabled by default). - Created FindUCX.cmake module for UCX header detection. Falls back to internal UCX headers if system headers not found. - Updated all Dockerfiles to include UCX dependencies. * hip-issue-3876 : Take into account thread-local capture mode in checks for valid capture (#2177) * Revert "rocr: Switch back to legacy IPC (#1744)" (#2676) This reverts commit 7e4b622. * SWDEV-558849 - Add support for static linking with ROCR (#2659) * [rocprofiler-systems] Fix MPI recv_data calculation (#2694) Fix incorrect `mpi_recv` calculation. It was using `_send_size` instead of `_recv_size` for `mpi_recv`. * [ROCmInfo] docs: mono-repo changes and style edits (#2584) * initial edits * mono repo related updates * standardize component name * style edits * more edits * Update CODEOWNERS (#2705): add /project/amdsmi owner * Use size_t datatype for global dimensions. (#2604) * rocrtst: set HSA_ENABLE_INTERRUPT after TestExample creation (#2687) Signed-off-by: Horatio Zhang <Hongkun.Zhang@amd.com> Co-authored-by: cfreeamd <166262151+cfreeamd@users.noreply.github.com> * Rework clock based unit tests (#2646) * Update hip_api_trace.cpp for HIP_RUNTIME_API_TABLE_STEP_VERSION 22 * Update hip_api_trace.hpp for HIP_RUNTIME_API_TABLE_STEP_VERSION 22 --------- Signed-off-by: David Galiffi <David.Galiffi@amd.com> Signed-off-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Signed-off-by: Maisam Arif <Maisam.Arif@amd.com> Signed-off-by: Horatio Zhang <Hongkun.Zhang@amd.com> Co-authored-by: Karthik Jayaprakash <54370791+kjayapra-amd@users.noreply.github.com> Co-authored-by: Venkateshwar Reddy Kandula <venkateshwar.kandula1306@gmail.com> Co-authored-by: jainprad <92369414+jainprad@users.noreply.github.com> Co-authored-by: jamessiddeley-amd <James.Siddeley@amd.com> Co-authored-by: David Galiffi <David.Galiffi@amd.com> Co-authored-by: Milan Radosavljevic <milan.radosavljevic@amd.com> Co-authored-by: marantic-amd <marantic@amd.com> Co-authored-by: vedithal-amd <Vignesh.Edithal@amd.com> Co-authored-by: Kian Cossettini <Kian.Cossettini@amd.com> Co-authored-by: Rakesh Roy <137397847+rakesroy@users.noreply.github.com> Co-authored-by: Bindhiya Kanangot Balakrishnan <Bindhiya.KanangotBalakrishnan@amd.com> Co-authored-by: Maisam Arif <Maisam.Arif@amd.com> Co-authored-by: Sajina PK <sputhala@amd.com> Co-authored-by: Ioannis Assiouras <38722728+iassiour@users.noreply.github.com> Co-authored-by: Alysa Liu <Alysa.Liu@amd.com> Co-authored-by: German Andryeyev <56892148+gandryey@users.noreply.github.com> Co-authored-by: yugang-amd <yugang.wang@amd.com> Co-authored-by: JeniferC99 <150404595+JeniferC99@users.noreply.github.com> Co-authored-by: hongkzha-amd <Hongkun.Zhang@amd.com> Co-authored-by: cfreeamd <166262151+cfreeamd@users.noreply.github.com> Co-authored-by: Jatin Chaudhary <51944368+cjatin@users.noreply.github.com> Co-authored-by: kjayapra-amd <karthik.jayaprakash@amd.com> Co-authored-by: hsivasun_amdeng <haresh.sivasuntharampillai@amd.com> Co-authored-by: Kandula, Venkateshwar reddy <Venkateshwarreddy.Kandula@amd.com>

github-actions Bot added the project: rocprofiler-systems label Dec 13, 2025

systems-assistant Bot added the organization: ROCm label Dec 13, 2025

mradosav-amd reviewed Dec 15, 2025

View reviewed changes

adjordje-amd reviewed Dec 15, 2025

View reviewed changes

Comment thread projects/rocprofiler-systems/source/lib/rocprof-sys/library/components/ucx_gotcha.hpp Outdated

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch from 442957e to 67396e6 Compare December 17, 2025 02:03

marantic-amd reviewed Dec 17, 2025

View reviewed changes

Comment thread ...ects/rocprofiler-systems/source/lib/rocprof-sys/library/components/tests/test_ucx_gotcha.cpp Outdated

marantic-amd reviewed Dec 17, 2025

View reviewed changes

Comment thread ...ects/rocprofiler-systems/source/lib/rocprof-sys/library/components/tests/test_ucx_gotcha.cpp Outdated

marantic-amd reviewed Dec 17, 2025

View reviewed changes

Comment thread ...ects/rocprofiler-systems/source/lib/rocprof-sys/library/components/tests/test_ucx_gotcha.cpp Outdated

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch from 67396e6 to 022376d Compare December 22, 2025 17:33

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch 6 times, most recently from a30a8e9 to 8b9e7bf Compare January 5, 2026 19:56

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch 6 times, most recently from 106d83d to 2fd870b Compare January 12, 2026 02:14

sputhala-amd marked this pull request as ready for review January 12, 2026 02:25

sputhala-amd requested review from a team and jrmadsen as code owners January 12, 2026 02:25

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch 5 times, most recently from 70ac03c to c345f00 Compare January 16, 2026 15:28

mradosav-amd requested changes Jan 19, 2026

View reviewed changes

Comment thread ...ects/rocprofiler-systems/source/lib/rocprof-sys/library/components/tests/test_ucx_gotcha.cpp Outdated

Comment thread ...ects/rocprofiler-systems/source/lib/rocprof-sys/library/components/tests/test_ucx_gotcha.cpp Outdated

dgaliffiAMD reviewed Jan 19, 2026

View reviewed changes

Comment thread projects/rocprofiler-systems/source/lib/rocprof-sys/CMakeLists.txt Outdated

sputhala-amd added 6 commits January 19, 2026 19:15

Enable UCX Communication tracing

1140d44

remove debug print and fix copyright

73daacf

Rebase with develop

6a56465

Docker ucx package name fix

8088807

Fix build errors and unit test logic

53402e5

Unit test segfault temp fix

6f35221

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch from 5778b21 to 7e41ff5 Compare January 20, 2026 01:50

Remove UCX gtest unit tests and associated CMake configuration

22498b8

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch from 2619c4f to 22498b8 Compare January 20, 2026 01:56

Merge branch 'develop' into users/sputhala/ucx_tracing

de3a287

sputhala-amd force-pushed the users/sputhala/ucx_tracing branch from a4060a6 to de3a287 Compare January 20, 2026 02:17

dgaliffiAMD approved these changes Jan 20, 2026

View reviewed changes

Merge branch 'develop' into users/sputhala/ucx_tracing

1584473

mradosav-amd self-requested a review January 20, 2026 11:49

sputhala-amd added 2 commits January 20, 2026 07:34

Exclude UCX tests for MPICH

4b0436a

Changelog modification

f71dcda

sputhala-amd requested a review from a team as a code owner January 20, 2026 18:04

Merge branch 'develop' into users/sputhala/ucx_tracing

0c03bb6

dgaliffiAMD approved these changes Jan 20, 2026

View reviewed changes

dgaliffiAMD merged commit 15c82d6 into develop Jan 20, 2026
125 of 127 checks passed

dgaliffiAMD deleted the users/sputhala/ucx_tracing branch January 20, 2026 18:16

sputhala-amd mentioned this pull request Jan 21, 2026

[Rocprofiler-system]: Add Documentation for communication API tracing #2478

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Rocprofiler-system]: Enable UCX Communication tracing#2306

[Rocprofiler-system]: Enable UCX Communication tracing#2306
dgaliffiAMD merged 12 commits into
developfrom
users/sputhala/ucx_tracing

sputhala-amd commented Dec 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

sputhala-amd commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

JIRA ID

Test Plan

Test Result

Submission Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sputhala-amd commented Dec 13, 2025 •

edited

Loading