[amd-vkale]

Note

Experiment, stacked on #13083. The base is #13083's branch, so this diff is only 398062b4fc. Opened as a draft to get a GPU run.

Motivation

Kernel replay keeps its per-pass toggles in thread-local state, and the replay loop holds the replayed agent's replay mutex exclusively (queue.cpp#L1136-L1137). A tool callback that runs on the replaying thread (CONFIG/PASS phase callbacks, dispatch callbacks) can still submit GPU work while the loop runs:

• To the replayed agent: the dispatch takes the same mutex shared (queue.cpp#L1288-L1290) while the thread holds it exclusively. With libstdc++/glibc this throws std::system_error ("Resource deadlock avoided") out of WriteInterceptor, and the process aborts with no explanation. That includes runtime copies and fills implemented as kernels. Nothing in the docs warns tools about it.
• To another agent: the dispatch runs, but its dispatch hooks see this loop's toggles. If that dispatch is itself replayed, the nested loop clears the thread's toggle routing when it exits, so the outer loop's remaining passes silently lose their toggles.

Technical Details

• local_context_control_t records the replayed agent. kernel_replay::replaying_agent() reports it for the loop's lifetime.
• The counter collection, SPM and thread trace enter hooks, and the kernel dispatch tracing filter, pass the dispatch's agent to is_locally_enabled(). Overrides apply only on the replayed agent.
• WriteInterceptor never replays a dispatch issued while its thread is replaying. It stops with a clear message when a replay callback submits work to the replayed agent.
• is_locally_enabled() reads the thread-local state directly instead of calling local_context_override(). Benchmarked, that saves the extra out-of-line call: about 17.5 to 12.8 ns per dispatch with 8 contexts outside a loop.
• kernel_replay.h documents the constraint and the per-agent scope of the toggles.

Issue Tracking

None.

Test Plan

• New GPU-free tests: kernel_replay_local_context.overrides_apply_only_on_the_replay_agent and .replaying_agent_tracks_the_loop.
• Randomized property test against the real local_context.cpp (out of tree): random replay agents, dispatch agents and toggles, with a reference model.
• CBMC model of agent scoping: the current design yields a counterexample; the agent-aware version verifies.
• CI: the mi325 kernel replay suites exercise the changed paths on every replay pass (a dispatch on the replayed agent behaves exactly as before). No CI test drives a tool callback that submits GPU work.

Test Result

• Local build against ROCm 6.4.1: clean except the 4 test apps that need a newer ROCm. -Werror and clang-format are clean on all touched files.
• Unit tests (no GPU): 644/745, up from 642 with the 2 new tests passing; the same 101 environment failures as refactor(rocprofiler-sdk): resolve kernel-replay pass toggles in the dispatch hooks #13083, and no test changed status.
• Property test: about 130M checks across 8 threads and 4 seeds, 0 failures.
• GPU: pending this PR's CI.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

  

[github-actions]

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

• Add new commits
• Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.