[amd-vkale]

Motivation

The purpose of this PR is to isolate the counter-collection slice of the callback-removal effort (reference PRs #8730 / #8586) into a small, single-service change, per review guidance to land callback removal one service at a time.

Underlying Problem

Dispatch counter collection currently registers a queue_callbacks_t on every HSA queue. That hides “is this service active?” inside a per-queue map, makes enqueue and completion the same walk, and serializes every GPU while any counter context is active.

This PR:

1. Stops registering counter collection with that map. The write interceptor and async signal handler call explicit free functions instead.
2. Scopes collection to GPU agents via rocprofiler_dispatch_counting_service_set_agents, so disjoint agent sets can be active together.

Public dispatch callback/record APIs are unchanged except for set_agents. Device counting is untouched. The per-queue callback map remains for unmigrated services.

Notes on Bigger Picture

Note the associated focused effort of the PR #8790 for migrating thread trace off the per-queue callback registry.

The kernel replay feature PR #7960 is greatly enhanced by this PR. However, note this is independent of the kernel replay feature and can be used for other features in ROCprofiler-SDK.

Technical Details

Enter vs exit use different context sets. Enter iterates active contexts; exit iterates registered contexts so in-flight dispatches still complete after stop.

flowchart LR
  WriteInterceptor --> EnterHook["kernel_dispatch_phase_enter_hook: active"]
  AsyncSignalHandler --> ExitHook["kernel_dispatch_phase_exit_hook: registered"]
  ContextStop --> Drain["enabled=false, drain, unserialize, callback_thread_stop, clear slot"]

Loading

Architecture: queue_callback_removal_architecture.md. Permalinks pin to 70bef64.

Hooks and call sites

• Declarations: queue_hooks.hpp
• Enter (collects_on, COUNTERS_CLIENT_ID): queue_hooks.cpp#L45-L84
• Exit (provenance via packet_return_map): queue_hooks.cpp#L86-L107
• is_any_active: queue_hooks.cpp#L109-L113
• no_real_consumers gate: queue.cpp#L318-L328
• Enter call site: queue.cpp#L633-L644
• Exit call site: queue.cpp#L172-L179
• Batching disabled while active: queue.cpp#L768-L781
• start_context (no add_callback): core.cpp#L148-L172
• Stop order (enabled → drain → unserialize → callback thread): core.cpp#L174-L209
• Stop services before active-slot CAS: context.cpp#L377-L405
• Producer tags: client_ids.hpp#L33-L56

Per-agent scoping

• API: dispatch_counting_service.h#L148-L185
• collects_on / intersects: context.hpp#L82-L110
• Conflict uses intersects: context.cpp#L305-L314
• set_agents locked while active: core.cpp#L211-L248
• Serialization refcount: queue_controller.hpp#L97-L175

Reviewer Guide

1. Confirm enter = get_active_contexts, exit = get_registered_contexts (queue_hooks.cpp).
2. Counters-only runs still enter WriteInterceptor (queue.cpp#L318-L328).
3. start_context does not call add_callback (core.cpp#L148-L172).
4. context::stop_context stops counters before the active-slot CAS (context.cpp#L377-L405).
5. counters::stop_context drains before teardown (core.cpp#L174-L209).
6. collects_on filters before queue_cb (queue_hooks.cpp#L62-L66).
7. In-flight regression: queue_hooks_test.cpp#L194-L326 — map drops by one per completion and every dispatch reaches the record callback after stop.
8. Per-agent: per_agent_scoping_test.cpp, integration per_agent_scoping_app.cpp.

Answer: In-flight record after stop? GPU-1-only context leave GPU-0 alone? packet_return_map empty with every record delivered?

Out of scope: deleting Queue::_callbacks; migrating other services; device-counting.

Issue Tracking

JIRA ticket:
AIPROFSDK-1017

Test Plan

• is_any_active; stop-while-in-flight
• Per-agent unit + multi-GPU integration (links above)
• Start/stop tests assert enabled, not iterate_callbacks (core.cpp)

Test Result

Intended coverage listed above. Treat the Checks tab as source of truth for CI.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[therock-pr-bot]

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

• ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

[vlaindic]

Overall, I'm fine with the PR, added couple optional comments.

Also, please check the AI generated comments.

PR #8891 Review — refactor(rocprofiler-sdk): migrate counter collection off the per-queue callback registry

Status: "Not ready to Review" label is set — this review is early feedback.

---

Findings

---

⚠️ IMPORTANT [hygiene]: Test Plan and Test Result sections are empty

Issues Detected

• Both ## Test Plan and ## Test Result contain only the template placeholder comments — no actual content.

Suggested Actions

• Fill in the Test Plan: at minimum, state which tests were run (counter-tests, rocprofv3 counter integration tests) and on which hardware.
• Fill in the Test Result: paste pass/fail summary. Even "counter-tests: all pass on MI300X" is sufficient.

Risk/Impact Assessment

• Medium — reviewers cannot verify that the change was exercised before merging.

---

⚠️ IMPORTANT [hygiene]: JIRA ID is indirect and exempt condition requires validation

Issues Detected

• The PR says "This PR does not have a JIRA ticket but is motivated by a ticket like that of kernel replay: AIPROFSDK-68." The JIRA ID is a reference to a related ticket, not the ticket for this work.
• The description does provide a clear, standalone justification (problem, change, validation path via cross-linked PRs), so the exemption partially applies — but the "motivated by a ticket like" phrasing is weaker than needed.

Suggested Actions

• Either create a JIRA ticket for this refactor and link it directly, or explicitly state "No JIRA ticket; standalone PR — see Technical Details for full justification." Don't use "like AIPROFSDK-68" phrasing, which implies an indirect relationship.

Risk/Impact Assessment

• Low — traceability concern only.

---

⚠️ IMPORTANT [clarity]: Copyright year is 2025 on all newly added files — should be 2026

Issues Detected
All five newly added files carry Copyright (c) 2025:

• counters/queue_hooks.cpp
• counters/queue_hooks.hpp
• counters/tests/queue_hooks_test.cpp
• hsa/queue_hooks/client_ids.hpp
• (CMakeLists.txt has no copyright header — that's fine)

Suggested Actions

• Update copyright header in counters/queue_hooks.cpp from 2025 to 2026 — file is newly added in this PR.
• Update copyright header in counters/queue_hooks.hpp from 2025 to 2026 — file is newly added in this PR.
• Update copyright header in counters/tests/queue_hooks_test.cpp from 2025 to 2026 — file is newly added in this PR.
• Update copyright header in hsa/queue_hooks/client_ids.hpp from 2025 to 2026 — file is newly added in this PR.

Risk/Impact Assessment

• Low — policy/compliance issue only.

---

⚠️ IMPORTANT [generic+clarity]: Attach-mode in-flight dispatch race is deleted without explanation

Issues Detected
The old stop_context contained an explicit guard: when registration::is_attached(), it removed the per-queue callback so new dispatches during the detach drain could pass through without incrementing _active_kernels. That block is deleted in this PR with only the comment:

"No per-queue callback to remove; counters::write_hook no-ops once dispatch_counter_collection is disabled above."

The comment asserts the new path is safe, but the assertion is not obviously correct:

• stop_context sets enabled = false under a write lock.
• write_hook calls queue_cb, which reads enabled under a read lock — so the no-op is real for new dispatches.
• But get_active_contexts() is a separate snapshot: a context can still appear in get_active_contexts() for some window after enabled is set to false (the context is only removed from the active set when rocprofiler_stop_context fully deactivates it).
• During that window, is_any_active() returns true, should_batch_packets stays false, and write_hook is called — but queue_cb correctly short-circuits. This is safe.

However: the PR description does not explain this reasoning, and the inline comment does not mention the window. A reviewer reading this will correctly ask "what about in-flight dispatches during attach-mode detach?" and have no answer in the code.

Suggested Actions

• Add a comment at the deletion site in stop_context explaining the two-phase safety: enabled is set to false first (making queue_cb no-op for new dispatches), and the context will be removed from get_active_contexts() when the stop completes, at which point is_any_active() returns false and should_batch_packets can return to true.
• Add a sentence to the PR description's "Technical Details" section covering the attach-mode case and why the old is_attached() block is no longer needed.

Risk/Impact Assessment

• Medium — correctness argument is sound based on the code, but the missing explanation creates review friction and risks a future author misunderstanding the invariant.

---

⚠️ IMPORTANT [generic]: write_hook and signal_completion_hook in queue_hooks.cpp lack function docstrings

Issues Detected
The header queue_hooks.hpp has terse one-line comments above write_hook and signal_completion_hook, but the .cpp implementations have no docstrings at all. The functions carry non-trivial contracts:

• write_hook mutates inst_pkt (appends) and OR-folds is_serialized — callers must understand these are out-params, not inputs.
• signal_completion_hook iterates all active contexts (not the one that produced the packet) — the reason for this is subtle.

Suggested Actions

• Add a brief comment to write_hook in queue_hooks.hpp noting that inst_pkt and is_serialized are output params (appended/OR-folded respectively), and that the function is called once per kernel dispatch from WriteInterceptor, not once per active context.
• Add a comment to signal_completion_hook explaining that it re-queries active contexts rather than taking a captured context pointer because the completion fires asynchronously and contexts may have started/stopped since the dispatch.

Risk/Impact Assessment

• Low — maintainability concern, not correctness.

---

💡 SUGGESTION [generic]: queue_hooks_test.cpp — single negative-path test is thin for a new public hook

Issues Detected
queue_hooks_test.cpp tests only is_any_active() returning false with no active context. The two primary hooks (write_hook, signal_completion_hook) have no unit-level tests beyond what the existing core.cpp integration tests indirectly exercise.

Suggested Actions

• This is acknowledged as a host-only, GPU-free unit test file, which limits what can be tested without standing up HSA. At minimum, extend queue_hooks_test.cpp with a test that activates a mock context (the existing core.cpp test infrastructure already does this) and asserts is_any_active() returns true. This validates the full active/inactive cycle without GPU hardware.
• Consider extending counters/tests/core.cpp:start_stop_buffered_ctx to assert that counters::is_any_active() is true after start and false after stop — symmetrical to the old iterate_callbacks check that was removed.

Risk/Impact Assessment

• Low — the existing integration tests cover real behavior; this is about defense-in-depth at the unit level.

---

💡 SUGGESTION [hygiene]: client_ids.hpp "unused ids reserved" comment could name the follow-up PRs

Issues Detected
client_ids.hpp says:

"Services are migrated off the per-queue callback registry one at a time; the unused ids are reserved so the tag values stay stable across those PRs."

The constants THREAD_TRACE_CLIENT_ID, PC_SAMPLING_CLIENT_ID, SPM_CLIENT_ID are not used by any code in this PR.

Suggested Actions

• Add references to the follow-up PRs (PR #8790 for thread trace is already mentioned in the description) as comments next to each unused constant, or at least note the planned migration order. This prevents a future developer from wondering if these are dead code.

Risk/Impact Assessment

• Low — cosmetic/maintainability only.

---

Summary

Severity	Count	Tags
⚠️ IMPORTANT	4	hygiene (2), clarity (1), generic+clarity (1)
💡 SUGGESTION	2	generic (1), hygiene (1)

Overall Risk: Medium — The refactor logic is sound, the primary behaviors (enabled guard, no_real_consumers gate, should_batch disable) are correctly wired. The main concern is the missing test plan, the unexplained attach-mode safety argument, and stale copyright years.

[vlaindic]

Suggested change

	// Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved.
	// Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.

[cursor]

Commit 6f56146 on the stacked #9586 moves the header on this file, and on the other three files this PR adds, from 2025 to 2026.

One caveat worth stating: the fix is not in #8891's own diff. This branch's head is still ecf8d24cad, where the line reads 2025. It rides in on #9586, which bases on this branch, so it lands when that merges.

[cursor]

Posting note: drafted with Cursor, but I (@amd-vkale) own this post.

Ack — applied. The header on this file and on the other three files this PR adds now reads 2026. Sorry for the slow acknowledgement from me directly; the fix landed a while back but I never closed the loop with you on it.

[vlaindic]

Could we think of a better name? kernel_dispatch_hook or something similar. Or kernel_dispatch_phase_enter_hook or something. Write might be a bit misleading as it's not clear that this is about write interceptor hook.

[amd-vkale]

I'd agree the name isn't great - I actually think kernel_dispatch_phase_enter_hook would be reasonably appropriate here. I will look into changing it to that.

[cursor]

Renamed to kernel_dispatch_phase_enter_hook, the name agreed in the reply above — commit 6f56146 on the stacked #9586.

signal_completion_hook became kernel_dispatch_phase_exit_hook in the same commit, for symmetry. Only the enter hook was flagged, but leaving one half of the pair renamed reads worse than either alternative, and the paired names make the enqueue-versus-completion asymmetry this branch introduces easier to follow. is_any_active kept its name, being neither phase. SPM mirrors these names on its own branch and was deliberately left alone until the review of #8887 settles, so those two are briefly inconsistent by choice rather than oversight.

Same caveat as the other threads: the rename is not in #8891's diff, it is upstack on #9586.

[amd-vkale]

See above for reference for where this fix is in the stacked PR and the specifics of the fix.

The main point is that the new names I chose were kernel_dispatch_phase_enter_hook and kernel_dispatch_phase_exit_hook.

[vlaindic]

Any reason for not using enum? Becuase we need int64_t?

[amd-vkale]

I don't think we'd ever need an int64_t for a client id - I'd say it's a reasonably safe change to use enum.

[cursor]

The int64_t was load-bearing rather than arbitrary. It matches hsa::ClientID, which is how the tag is stored in inst_pkt_t. So, the answer is an enum that pins int64_t as its underlying type, which commit 6f56146 on the stacked #9586 does:

enum client_id : int64_t
{
    COUNTERS_CLIENT_ID     = 1,
    THREAD_TRACE_CLIENT_ID = 2,
    PC_SAMPLING_CLIENT_ID  = 3,
    SPM_CLIENT_ID          = 4,
};

Unscoped rather than enum class on purpose: it keeps the implicit conversion to ClientID, so no emplace site needs a cast, while still grouping the values under one type.

The one place that needed adjusting was the regression test, which now builds the pair piecewise instead of via std::make_pair — make_pair would deduce the enum type and lean on a pair-to-pair conversion, whereas piecewise converts the tag directly and matches what the production hook does.

Not in #8891's diff — it is upstack on #9586.

[amd-vkale]

See above for reference for where this fix is in the stacked PR and the specifics of the fix.

[bwelton]

Correctness / lifetime: in-flight counter completions are dropped when a context is stopped while dispatches are still executing

Thanks for splitting the counter-collection migration out — the enqueue path and the get_notifiers/should_batch/queue_id cleanup look clean. One blocking issue:

Completion routing changed from provenance-based to liveness-based. Previously the completion callback lived in the queue-controller's per-queue map and fired for an in-flight dispatch regardless of whether the context was still active, and the old stop_context relied on this ("keep callbacks registered to avoid dropping counter data for in-flight dispatches"). The new signal_completion_hook (counters/queue_hooks.cpp:83-89) instead re-queries get_active_contexts(...) at completion time, but context::stop_context nulls the active slot (context/context.cpp:371) before calling counters::stop_context (context/context.cpp:380).

So a kernel dispatched while the context was active but completing after rocprofiler_stop_context() returns finds zero active counter contexts, and completed_cb never runs. Result:

• counter records for those in-flight kernels are silently lost, and
• the packet_return_map entry (erased only in completed_cb, dispatch_handlers.cpp:162) is leaked, with possible misassociation if the packet address is later reused.

counters::stop_context (core.cpp:172-191) doesn't drain in-flight GPU work, so nothing compensates. Stop-while-in-flight is an ordinary lifecycle/finalize pattern, so this isn't an edge case.

One possible fix: route completion by packet provenance instead of current activeness. completed_cb already resolves ownership via pointer lookup in packet_return_map and self-filters, so iterating registered (rather than only active) counter contexts in signal_completion_hook would keep completions reachable after stop — provided the callback thread and counter_callback_info objects are kept alive until in-flight dispatches drain (stop_context currently calls callback_thread_stop() unconditionally at core.cpp:190).

Test coverage note: the new queue_hooks_test.cpp only checks is_any_active() == false with no contexts, and the trimmed tests/core.cpp start/stop tests just inspect the enabled flag — nothing would catch this. Please add a test that starts a counter context, enqueues kernels, stops the context while dispatches are in flight, and asserts every kernel produces a record and packet_return_map ends up empty.

[amd-vkale]

Correctness / lifetime: in-flight counter completions are dropped when a context is stopped while dispatches are still executing

Thanks for splitting the counter-collection migration out — the enqueue path and the get_notifiers/should_batch/queue_id cleanup look clean. One blocking issue:

Completion routing changed from provenance-based to liveness-based. Previously the completion callback lived in the queue-controller's per-queue map and fired for an in-flight dispatch regardless of whether the context was still active, and the old stop_context relied on this ("keep callbacks registered to avoid dropping counter data for in-flight dispatches"). The new signal_completion_hook (counters/queue_hooks.cpp:83-89) instead re-queries get_active_contexts(...) at completion time, but context::stop_context nulls the active slot (context/context.cpp:371) before calling counters::stop_context (context/context.cpp:380).

So a kernel dispatched while the context was active but completing after rocprofiler_stop_context() returns finds zero active counter contexts, and completed_cb never runs. Result:

• counter records for those in-flight kernels are silently lost, and
• the packet_return_map entry (erased only in completed_cb, dispatch_handlers.cpp:162) is leaked, with possible misassociation if the packet address is later reused.

counters::stop_context (core.cpp:172-191) doesn't drain in-flight GPU work, so nothing compensates. Stop-while-in-flight is an ordinary lifecycle/finalize pattern, so this isn't an edge case.

One possible fix: route completion by packet provenance instead of current activeness. completed_cb already resolves ownership via pointer lookup in packet_return_map and self-filters, so iterating registered (rather than only active) counter contexts in signal_completion_hook would keep completions reachable after stop — provided the callback thread and counter_callback_info objects are kept alive until in-flight dispatches drain (stop_context currently calls callback_thread_stop() unconditionally at core.cpp:190).

Test coverage note: the new queue_hooks_test.cpp only checks is_any_active() == false with no contexts, and the trimmed tests/core.cpp start/stop tests just inspect the enabled flag — nothing would catch this. Please add a test that starts a counter context, enqueues kernels, stops the context while dispatches are in flight, and asserts every kernel produces a record and packet_return_map ends up empty.

@bwelton Thanks for this - I am prioritizing fixing this gap right now.

[amd-vkale]

Status for reviewers: the counter collection migration, together with all the review fixes that were previously split into the stacked PR #11869, is now consolidated.

Where to review the code: #11970. It sits on users/vkale/remove-callbacks-counters--with-cursor-fixes, a branch in ROCm/rocm-systems rather than in my fork, because CodeQL does not run on pull requests from a fork. #11970 targets develop.

This PR (#8891) is now stacked on #11970 and holds only a marker commit with no file changes, so its "Files changed" tab will look empty. It stays open to keep this review history attached to the work.

#11869 has been closed as superseded — every commit that was on it is an ancestor of the consolidated branch, so the lifecycle, race, scope and hook tag fixes are all present. Two further fixes were made on top of it: the completion hook test fixture now uses a null session so the test does not require a live HSA runtime, and a wording fix in per_agent_scoping_app.cpp.

[github-actions]

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

• Add new commits
• Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

[github-actions]

This pull request has been automatically closed due to inactivity (30 days with no updates).

If you'd like to continue working on this, feel free to reopen the PR or create a new one.