[release/1.3.2] libfabric: flush FI_MORE per rail - #1989
Merged
Merged
Conversation
FI_MORE WRITE batching (#1626) flushes the doorbell positionally (last of each 16-group / last of the transfer), which is only correct when every descriptor in a group is on the same rail. But each descriptor's rail comes from its own buffer's selected_rails, so a transfer spanning buffers with different rail lists (e.g. dynamo's many DRAM registrations) alternates physical rails within a group. The positional flush then rings only one rail's doorbell; the other rail's batch is never submitted and the transfer hangs. nixlbench doesn't hit this: one buffer per device = one rail list. Fix: postXfer precomputes the flush decision per physical rail (walking descriptors backwards, the first rail seen is its last post), and prepareAndSubmitTransfer takes a bool apply_fi_more instead of deciding positionally. Batching is opt-in, so unmarked descriptors (READ, striped, rail-less) flush. Also: pair the remote endpoint with the rail by the same index; hoist FI_MORE_BATCH_SIZE to libfabric_common.h; drop the unused desc_count param. Signed-off-by: Arun Karthik <akkart@amazon.com> (cherry picked from commit 78ece0c)
akkart-aws
approved these changes
Jul 23, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backports #1966 onto
release/1.3.2.This cherry-picks
78ece0c9897f6751052514c7cb37fde6ab7228ecas9dce5cb6615e45cfd71a45fcb6f42cfa5b28dfa9.Details
apply_fi_moredecision passed intoprepareAndSubmitTransfer.postXferflow while adapting the per-rail flush precompute from the original PR.Validation
git diff --check HEAD~1 HEADFull Meson build was not run locally because
meson/ Pythonmesonbuildis not installed in this environment.