[abouteiller]

Motivation

On certain test systems we do not have a shared filesystem, the deadlock_analysis script harness would depend on a shared filesystem to collect traces and artifacts. Remove that limitation.

Technical Details

Use MPI stdout redirection to coalesce the trace artifacts to the headnode in a headnode local output directory, perform analysis on the headnode.

Prior filesystem based mode remains available and default.

JIRA ID

Test Plan

Intentionally added a deadlock to a functional test, compiled with Debug.
Run mpiexec_deadlock_attach script and observe artifacts.

Test Result

 ROCSHMEM_DEBUG_LEVEL=trace:stats ~/ompi/bin/mpirun -n 2 tests/functional_tests/rocshmem_functional_tests -n 1 -a 26  & (sleep 2; /home/abouteil/rocm-systems/projects/rocshmem/scripts/debug/mpiexec_deadlock_analysis.sh  rocshmem_functional_tests --stats --color --no-shared-fs )
### Creating Test:      Blocking WG level Puts  B=ipc PE=2 W=1 Z=64 ###
Deploying deadlock analysis...
Executable:       rocshmem_functional_tests
Output directory: ./rocshmem_deadlock_analysis_20260530_013437
Filesystem mode:  no shared fs (--output-filename)
Stats:            yes
Color:            --color

Found 4 process(es) matching 'rocshmem_functional_tests': 717472 717473 751554 751555
Output directory: /tmp/rocshmem_deadlock_20260530_013437
Per-process timeout: 120s

Analyzing PID 717472 (PE 0) ... OK
Analyzing PID 717473 (PE 1) ... OK (no GPU threads)
Analyzing PID 751554 (PE 0) ... OK
Analyzing PID 751555 (PE 1) ... OK (no GPU threads)

=== Attach and Analyze Summary ===
Processes analyzed: 4 / 4

  PE 0     PID 751554    groups=1    \033[1;31m1\033[0m wf(s) in rocSHMEM; \033[1;36mWaiting for remote PE memory update (barrier/sync or user rocshmem_wait_until). Check if the remote PE is alive and making progress. (at: #0  0x0000792efa5af3c8 in rocshmem::Context::wait_until<long> (this=0x792eed407000, ivars=0x792efac04008, cmp=0, val=1) at src/context_tmpl_device.hpp:224)\033[0m
  PE 1     PID 751555    groups=?    (no GPU threads found)
=== BEGIN FILE: pe0_pid_717472.txt ===
[New LWP 717493]
[New LWP 717490]
[New LWP 717488]
[New LWP 717485]
[New LWP 717481]
[New LWP 717480]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x000077a09e06695f in ?? () from /opt/rocm-7.2.1/lib/libhsa-runtime64.so.1
=== rocSHMEM Deadlock Analysis ===
Process(es): 717472
Total GPU wavefronts analyzed: 1
Unique backtrace groups: 1

--- Group 1 (1 wavefront(s)) ---
  WFs: (0,0,0)/0
  Backtrace:
    #0  0x0000776005fafce0 in rocshmem::uncached_load<long volatile> (src=0x776006604008) at src/assembly.hpp:125
    #1  rocshmem::Context::test<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:432
    #2  rocshmem::Context::wait_until<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:224
    #3  rocshmem::IPCContext::internal_direct_barrier (this=0x775ff8e07000, pe=0, PE_start=0, stride=1, n_pes=2, pSync=0x776006604000) at build/src/ipc/context_ipc_device_coll.cpp:44
    #4  0x0000776005fb7c24 in rocshmem::IPCContext::internal_sync_wg (this=0x775ff8e07000, pe=0, PE_start=0, stride=1, PE_size=2, pSync=0x776006604000) at build/src/ipc/context_ipc_device_coll.cpp:115
    #5  0x000077600603d018 in rocshmem::IPCContext::sync_all_wg (this=0x775ff8e07000) at build/src/ipc/context_ipc_device_coll.cpp:168
    #6  0x000077600603d558 in rocshmem::IPCContext::barrier_all_wg (this=0x775ff8e07000) at build/src/ipc/context_ipc_device_coll.cpp:187
    #7  0x000077600603dedc in rocshmem::Context::barrier_all_wg (this=0x775ff8e07000) at build/src/context_device.cpp:151
    #8  0x000077600603e33c in rocshmem::rocshmem_barrier_all_wg () at build/src/rocshmem_gpu.cpp:885
    #9  0x00007760060437cc in WorkGroupPrimitiveTest (loop=1, skip=10, start_time=0x775ff8e31000, end_time=0x775ff8e32000, source=0x776013c00480 "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqr"..., dest=0x776013d00500 "1", size=1, type=WGPutTestType, ctx_type=4) at /home/abouteil/rocm-systems/projects/rocshmem/tests/functional_tests/workgroup_primitives.cpp:72
  [HINT] Waiting for remote PE memory update (barrier/sync or user rocshmem_wait_until). Check if the remote PE is alive and making progress. (at: #2  rocshmem::Context::wait_until<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:224)
...
=== END FILE: pe1_pid_751555.txt ===

Reconstructing per-PE output files from mpiexec output...
Reconstructed output from 1 rank file(s).

Running cross-rank coalescing pass...

=== Cross-Rank Deadlock Analysis ===
Directory: ./rocshmem_deadlock_analysis_20260530_013437
Ranks with analysis files: 2  (0, 1)
Unique backtrace patterns: 2

--- Pattern 1 [UNIQUE: rank 0] ---
  Ranks: 0
  Total wavefronts: 1
  Backtrace:
    #0  0x0000776005fafce0 in rocshmem::uncached_load<long volatile> (src=0x776006604008) at src/assembly.hpp:125
    #1  rocshmem::Context::test<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:432
    #2  rocshmem::Context::wait_until<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:224
    #3  rocshmem::IPCContext::internal_direct_barrier (this=0x775ff8e07000, pe=0, PE_start=0, stride=1, n_pes=2, pSync=0x776006604000) at build/src/ipc/context_ipc_device_coll.cpp:44
    #4  0x0000776005fb7c24 in rocshmem::IPCContext::internal_sync_wg (this=0x775ff8e07000, pe=0, PE_start=0, stride=1, PE_size=2, pSync=0x776006604000) at build/src/ipc/context_ipc_device_coll.cpp:115
    #5  0x000077600603d018 in rocshmem::IPCContext::sync_all_wg (this=0x775ff8e07000) at build/src/ipc/context_ipc_device_coll.cpp:168
    #6  0x000077600603d558 in rocshmem::IPCContext::barrier_all_wg (this=0x775ff8e07000) at build/src/ipc/context_ipc_device_coll.cpp:187
    #7  0x000077600603dedc in rocshmem::Context::barrier_all_wg (this=0x775ff8e07000) at build/src/context_device.cpp:151
    #8  0x000077600603e33c in rocshmem::rocshmem_barrier_all_wg () at build/src/rocshmem_gpu.cpp:885
    #9  0x00007760060437cc in WorkGroupPrimitiveTest (loop=1, skip=10, start_time=0x775ff8e31000, end_time=0x775ff8e32000, source=0x776013c00480 "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqr"..., dest=0x776013d00500 "1", size=1, type=WGPutTestType, ctx_type=4) at /home/abouteil/rocm-systems/projects/rocshmem/tests/functional_tests/workgroup_primitives.cpp:72
  [HINT] Waiting for remote PE memory update (barrier/sync or user rocshmem_wait_until). Check if the remote PE is alive and making progress. (at: #2  rocshmem::Context::wait_until<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:224)

=== Per-Rank rocSHMEM Wavefront Count ===
  Rank 0: 2 wf(s) inside rocSHMEM
  Rank 1: 0 wf(s) inside rocSHMEM

=== Summary ===
  0 pattern(s) common to all 2 rank(s)
  1 pattern(s) present in exactly one rank
  Most common hint: Waiting for remote PE memory update (barrier/sync or user rocshmem_wait_until). Check if the remote PE is alive and making progress. (at: #2  rocshmem::Context::wait_until<long> (this=0x775ff8e07000, ivars=0x776006604008, cmp=0, val=1) at src/context_tmpl_device.hpp:224)

[github-actions]

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

• Add new commits
• Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

[kesag]

LGTM