Skip to content

[rocprofiler-sdk] Fix queue interposition completion waiter ordering - #7924

Merged
venkat1361 merged 5 commits into
developfrom
users/venkat1361/queue-interposition-fix
Jun 30, 2026
Merged

venkat1361 merged 5 commits into
developfrom
users/venkat1361/queue-interposition-fix

Conversation

@venkat1361

Copy link
Copy Markdown
Contributor

Motivation

  • transpose-sampling test from rocprofiler-systems can hang because queue interposition may start the async completion-signal waiter before the final real doorbell is published, so the waiter can block waiting for GPU completion while the GPU has not yet been told about the packet.

Technical Details

  • Defer completion-waiter enqueue from write_interceptor() to process_doorbell_impl(), after publish_submitted_packets() publishes the real write index and rings the doorbell.

JIRA ID

Test Plan

Test Result

Submission Checklist

Clamp the async signal handler pool size when GPU_MAX_HW_QUEUES resolves to zero, and defer completion-signal waiter enqueue until after process_doorbell_impl publishes the final doorbell. This prevents PTL async enqueue or inline execution from waiting on packets that the GPU cannot observe yet.
Copilot AI review requested due to automatic review settings June 29, 2026 05:47
@venkat1361
venkat1361 requested review from a team as code owners June 29, 2026 05:47
@therock-pr-bot

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
🌿 Branch Name ✅ Pass
📝 PR Title/Description ❌ Fail Error: Title does not follow Conventional Commits style.
Expected: start with a valid type (feat, fix, docs, …).
Desired format: type(optional-scope): short description
───
Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword.
Expected: include a JIRA ID / ISSUE ID line (separator : or -, or omitted; value may be a JIRA key, a number with/without #, or a link), OR a closing keyword + issue reference. Accepted examples:
JIRA ID : TESTAUTO-6039
JIRA ID - #330
JIRA ID #330
ISSUE ID : TESTUTO-3334
ISSUE ID #3334
ISSUE ID - TESTAUTO-3433
ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234
Closes #10
Fixes octo-org/octo-repo#100
Resolves: #123
#123
https://github.com/<org_name>/<repo_name>/issues/123
Current: no valid JIRA/ISSUE/closing-keyword reference found
Forbidden Files ✅ Pass
🧪 Unit Test ❌ Fail Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.hpp, projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/queue_interposition.cpp; no test file found
🔎 pre-commit ⏳ Pending ⏳ Still running…
🚫 Draft PR 🔜 To Be Enabled
🚩 Feature Flag 🔜 To Be Enabled
📊 Code Coverage 🔜 To Be Enabled

⚠️ 2 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a potential hang in rocprofiler-sdk’s inline HSA queue interposition path by ensuring completion-signal waiters are not enqueued until after the real write index has been published and the final doorbell has been rung.

Changes:

  • Defers async completion-waiter enqueue from write_interceptor() to process_doorbell_impl() (after publish_submitted_packets()).
  • Refactors async-signal-handler thread-count selection into get_async_signal_handler_thread_count().
  • Adds unit tests covering env-driven thread-count selection and clamping behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/tests/queue_interposition.cpp Adds unit tests for async-signal-handler thread-count selection via env vars.
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.hpp Exposes get_async_signal_handler_thread_count() (and includes <cstddef> for size_t).
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/queue_interposition.cpp Implements deferred waiter scheduling and centralizes thread-count env logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@minseobshin11 minseobshin11 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@venkat1361
venkat1361 requested a review from a team as a code owner June 29, 2026 15:54
@github-actions github-actions Bot added the github actions Pull requests that update GitHub Actions code label Jun 29, 2026
@bwelton

bwelton commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

While i don't have opposition to this change.

queue interposition may start the async completion-signal waiter before the final real doorbell is published. so the waiter can block waiting for GPU completion while the GPU has not yet been told about the packet.

This shouldn't matter. The async just wont trigger until the completion signal is set. Its not clear why this change fixes anything other than changing the ordering and holding the lock for potentially less time.

@rahulc-gh rahulc-gh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - wait for CI to Pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants