async api with work request tracking #1737
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Allow us to test the bandwidth of concurrent rdma operations. actual concurrency support will be added later in the stack. Differential Revision: D85724514
Summary: This diff adds a FIFO waiting mechanism for acquiring the queue pair. **Actual concurrency support will come in follow-up diff**: this will require we track wr_id and create a polling task for each queue pair, which will need further refactoring. **Not true concurrent support**: Only one request can be in-flight at a time per queue pair connection. However, subsequent requests now wait fairly (FIFO) instead of panicking with "already checked out" errors. Core changes: 1. **Fair Waiting via Semaphore**: Replaced Available/CheckedOut states with Connecting/Ready/ConnectionError. Added QueuePairEntry wrapping RdmaQueuePair + Arc<Semaphore>. request_queue_pair uses two-phase approach (get/create QP, then acquire semaphore permit for FIFO fairness). 2. **Refactor - Moved read_into/write_from to RdmaManagerActor**: Prevents deadlock in actor message queue. Old design had RdmaBuffer call request_queue_pair RPC → perform operation → call release_queue_pair RPC, causing release messages to queue behind waiting requests. Now entire operation (request → use → release) happens within single actor message handler. 3. **Refactored Connection Logic**: Extracted establish_connection helper handling both loopback and remote connections. 5. **Test**: Added create_buffer_pair method and concurrent tests. Differential Revision: D85627877
Summary: Introduces async RDMA API with background completion polling and work request tracking via CompletionTracker. Large operations are automatically chunked and tracked using oneshot channels. The new read_into() and write_from() methods now use async completion tracking for better concurrency. The request_queue_pair() method is preserved for backwards compatibility; however, using previous QP polling APIs won't work if a polling task is actively polling the queue pair, as the background task will consume completions. Differential Revision: D85962453
meta-cla
bot
added
the
CLA Signed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: Introduces async RDMA API with background completion polling and work request tracking via CompletionTracker. Large operations are automatically chunked and tracked using oneshot channels. The new read_into() and write_from() methods now use async completion tracking for better concurrency. The request_queue_pair() method is preserved for backwards compatibility; however, using previous QP polling APIs won't work if a polling task is actively polling the queue pair, as the background task will consume completions.
Differential Revision: D85962453