Repository navigation
UCP/FT: rebuild failed lanes on recovery via LANES_ADDR exchange - #11562
Merged
evgeny-leksikov merged 3 commits intoJul 6, 2026
Merged
evgeny-leksikov merged 3 commits into
evgeny-leksikov merged 3 commits into
Conversation
evgeny-leksikov
changed the title
8 tasks done
Complete failed-lane recovery on top of the recovery foundation and the keepalive-driven state machine (openucx#11545): - Rebuild failed p2p/iface lanes: install wireup proxies over failed stubs, recreate the inner UCT endpoints, and drive them to READY so the lane resumes serving traffic. - Wire the LANES_ADDR_REQUEST/REPLY handlers to exchange peer addresses and rebuild the requested lanes, including local failover for asymmetric failures, and free the recovery state once all lanes recover. - Pin a recovered iface lane to its original peer device. - Resolve the remote id on the RMA multi path under UCP_ERR_HANDLING_MODE_FAILOVER. - Add end-to-end fault-tolerance gtests (initiator/target failure incl. the all-lanes-failed case) using the keepalive-driven recovery cadence.
evgeny-leksikov
force-pushed
the
ft-split/06-recovery-rebuild-upstream
branch
from
June 17, 2026 21:28
6e82a8e to
31d5003
Compare
evgeny-leksikov
marked this pull request as ready for review
June 17, 2026 21:29
evgeny-leksikov
commented
Jun 17, 2026
william-gallagher-nv
previously approved these changes
Jul 2, 2026
william-gallagher-nv
left a comment
william-gallagher-nv
left a comment
Contributor
There was a problem hiding this comment.
Discussed with @evgeny-leksikov in person. Looks good.
gleon99
reviewed
Jul 2, 2026
| * see as failed go through local failover first, so subsequent rebuild | ||
| * steps encounter proper failed stubs and the reply rides the new | ||
| * am_lane. */ | ||
| peer_unaware = lanes_info->provided_lane_map & ~ucp_ep_get_failed_lanes(ep); |
Contributor
There was a problem hiding this comment.
What if heterogenous EP config -> wrong lanes are rebuilt? It assumes here peer/local indices are identical..
Contributor
Author
There was a problem hiding this comment.
per offline discussion, heterogenous EP case will be handled in separate PR
gleon99
reviewed
Jul 2, 2026
gleon99
reviewed
Jul 2, 2026
evgeny-leksikov
force-pushed
the
ft-split/06-recovery-rebuild-upstream
branch
from
July 3, 2026 13:36
4fb7275 to
ef69773
Compare
- Added assertions to ensure proper state checks for UCT endpoints during recovery. - Simplified the recovery path by removing unnecessary checks and ensuring that wireup endpoints are correctly marked as ready. - Updated the randomization logic in the fault tolerance tests to use a more efficient method. - Improved error handling in wireup connection processes to ensure robust endpoint management.
evgeny-leksikov
force-pushed
the
ft-split/06-recovery-rebuild-upstream
branch
from
July 3, 2026 14:10
ef69773 to
458f386
Compare
gleon99
approved these changes
Jul 6, 2026
This was referenced Aug 24, 2026
This was referenced Sep 3, 2026
wilsonliu-b10
added a commit
to basetenlabs/ucx
that referenced
this pull request
Sep 21, 2026
Brings the fork's UCP fault-tolerance work under the device plane: probe-gated lane recovery (openucx#11563), lane rebuild via LANES_ADDR (openucx#11562), and the fix that stops progressing an exhausted recovery instead of asserting on it. The device work was forked before all three, so until now the two could not be pinned together and a trt-llm image had to choose. One conflict, in ucp_wireup_ep_connect_aux: both sides changed how the auxiliary transport is selected, for the same reason and by different means. The device plane narrows the tl_bitmap to the endpoint's device pin, so an auxiliary cannot land on a device no lane is on; master added local and remote device bitmaps to the same selector. Resolved as the device pin plus UINT64_MAX for both new bitmaps -- the tl_bitmap already excludes every other device's resources, so restricting again by device index would be redundant, and UINT64_MAX is what master passes at this site. Master's own new call site, in ucp_ep_recovery_create_aux_ep, needs nothing: it already sets local_dev_bitmap to UCS_BIT of the failed lane's dev_index, so recovery keeps an auxiliary on the lane's device. The two mechanisms encode the same invariant and do not fight. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Wilson Liu <wilson.liu@baseten.co>
This was referenced Sep 27, 2026
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Complete failed-lane recovery on top of the recovery foundation and the
keepalive-driven state machine (#11545):
How?
recreate the inner UCT endpoints, and drive them to READY so the lane
resumes serving traffic.
rebuild the requested lanes, including local failover for asymmetric
failures, and free the recovery state once all lanes recover.
all-lanes-failed case) using the keepalive-driven recovery cadence.