Add a FIFO ticket lock over a torch.distributed store - #2143
Open
fzyzcjy wants to merge 1 commit into
Open
Conversation
fzyzcjy
requested review from
Shi-Dong,
Zhichenzzz,
guapisolo,
jybsuper,
maocheng23 and
yueming-yuan
as code owners
August 4, 2026 07:59
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 6, 2026 07:03
adb1caa to
dd88fa0
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
2 times, most recently
from
August 9, 2026 23:31
3558c4e to
9697adf
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 9, 2026 23:33
dd88fa0 to
ff5d591
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 10, 2026 05:59
9697adf to
9ffac37
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 10, 2026 06:08
ff5d591 to
82d9cdb
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 10, 2026 16:06
9ffac37 to
9116b42
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 10, 2026 16:07
82d9cdb to
f490c5f
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 10, 2026 16:33
9116b42 to
f657fc7
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
2 times, most recently
from
August 12, 2026 02:48
5ed2c39 to
f5d8d99
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
2 times, most recently
from
August 12, 2026 14:31
5efda26 to
7dcf53b
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 12, 2026 14:31
f5d8d99 to
407c863
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
3 times, most recently
from
August 13, 2026 03:25
a960834 to
bfad33c
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 13, 2026 03:25
6e3788d to
3bf836e
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 13, 2026 06:00
bfad33c to
8b66f2c
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 13, 2026 06:01
3bf836e to
c10abc9
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 13, 2026 06:39
8b66f2c to
e43aa5c
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
2 times, most recently
from
August 14, 2026 04:45
084464f to
c4b79af
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 14, 2026 04:46
e43aa5c to
9c04d57
Compare
guapisolo
approved these changes
Aug 26, 2026
The rollout engines are guarded by a ray Lock actor, which costs a handle passed across processes and, worse, stays locked forever when its holder dies. A ticket lock over a store owned by the training world has neither problem: two monotonic counters, no rollback, no reset, and a store whose lifetime is rank 0's process. Ranks that never contend still join the address broadcast, which is collective, but open no connection to the store. A critical section that raises keeps the lock, since the failed holder's remote work may still be in flight; recovery there is the same cell teardown a crash gets. The tests exercise the queue over a real TCPStore, where it actually spans processes: separate connections share one queue, a holder that disappears keeps its ticket, and a rank polling a store whose host died raises instead of spinning. Blocking is proven by the counters rather than by a stopwatch, so a lock that never waited cannot pass, and every wait is bounded and every helper process reaped so a broken lock fails the suite instead of hanging it. A scripted store and a fake clock pin the wait loop's warning throttle, the poll-interval sleep, and the first-warning deadline; the factory side pins the gloo-group broadcast, kwarg forwarding, abstainer connection avoidance, and IPv6 addresses. This commit only adds the utility; the switch follows.
fzyzcjy
force-pushed
the
tom/refactor-miles/op12-1
branch
from
August 29, 2026 12:38
9c04d57 to
4618ca5
Compare
fzyzcjy
force-pushed
the
tom/refactor-miles/op9-15
branch
from
August 29, 2026 12:40
c4b79af to
d2a3147
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1837