[Distributed][Bug fix] Gate each TP step on every follower having read the request's inputs by arun-elr · Pull Request #754 · mstar-project/mstar

Fail only the request whose async (Mooncake) tensor read failed, and free its buffers once its transfers stop

arun-elr

committed

b8886ab

Hold READS_DONE while a failed read's transfers run, and read a torn-down read's outcome from its future

arun-elr

committed

5cb1f92

Poll a read's future through done() and result() only, as the SHM arena's event future offers no more

arun-elr

committed

7b05062

test: the output-failure teardown fake tensor manager reports no failed reads

arun-elr

committed

a646727

Transfer metrics: a failed read is dropped and reported, not raised, so the read delivered before it counts once

arun-elr

committed

94fe35c

test: the serving-metrics stub tensor manager reports no failed reads

arun-elr

committed

2b71c70

test: the fake Mooncake engine's report waits for the reader to submit the batch it targets

arun-elr

committed

90dc274

Count a read's retries under the device filter the Mooncake engine was initialized with

arun-elr

committed

9b5fb83

Log a failed output read in one line, as a failed input read is

arun-elr

committed

32035f8

Count the HCAs for read retries on the first failed tensor, not at startup

arun-elr

committed

8b5d2c7

test: over real Mooncake TCP, a failed read keeps its buffers until a transfer still running reports it stopped

arun-elr

committed

a2c97da

Gate each lockstep step on every follower having read the request's inputs

arun-elr

committed

77e49c8

Read gate: cheapest checks first and followers cached per request, one edge index per batch, only the lockstep node's inputs marked, a removed request's late reports dropped

arun-elr

committed

e0f9381

Read gate: a follower beside a node that does not feed the lockstep node is still gated; the instance's own persisted outputs never wait; a held fresh row queues behind the older backlog

arun-elr

committed

c97f773

Read gate: only this rank's own lockstep output returning skips the count, and loops' terminal and accumulated outputs count as feeding the lockstep node

arun-elr

committed

f988e42

Read gate: the module docstring names only feeding nodes beside a follower as ungated

arun-elr

committed

64715b4

test: the read gate's fake engine has the prefix-cache hooks the TP broadcast now calls (#835)

arun-elr

committed

ccee453

Read gate: a removed request's late reports go through the worker's removed-id drop

arun-elr

committed

8c80a4d

test: read gate at TP4, under speculation, on SP layouts, and a report replayed after NEW_REQUEST

arun-elr

committed

c02fd2d

Navigation Menu

Repository navigation

[Distributed][Bug fix] Gate each TP step on every follower having read the request's inputs - #754

[Distributed][Bug fix] Gate each TP step on every follower having read the request's inputs#754
arun-elr wants to merge 19 commits into
mstar-project:mainfrom
arun-elr:fix/tp-read-gate

Commits on Oct 25, 2026

Fail only the request whose async (Mooncake) tensor read failed, and free its buffers once its transfers stop

Hold READS_DONE while a failed read's transfers run, and read a torn-down read's outcome from its future

Poll a read's future through done() and result() only, as the SHM arena's event future offers no more

test: the output-failure teardown fake tensor manager reports no failed reads

Transfer metrics: a failed read is dropped and reported, not raised, so the read delivered before it counts once

test: the serving-metrics stub tensor manager reports no failed reads

test: the fake Mooncake engine's report waits for the reader to submit the batch it targets

Count a read's retries under the device filter the Mooncake engine was initialized with

Log a failed output read in one line, as a failed input read is

Count the HCAs for read retries on the first failed tensor, not at startup

test: over real Mooncake TCP, a failed read keeps its buffers until a transfer still running reports it stopped

Gate each lockstep step on every follower having read the request's inputs

Read gate: cheapest checks first and followers cached per request, one edge index per batch, only the lockstep node's inputs marked, a removed request's late reports dropped

Read gate: a follower beside a node that does not feed the lockstep node is still gated; the instance's own persisted outputs never wait; a held fresh row queues behind the older backlog

Read gate: only this rank's own lockstep output returning skips the count, and loops' terminal and accumulated outputs count as feeding the lockstep node

Read gate: the module docstring names only feeding nodes beside a follower as ungated

test: the read gate's fake engine has the prefix-cache hooks the TP broadcast now calls (#835)

Read gate: a removed request's late reports go through the worker's removed-id drop

test: read gate at TP4, under speculation, on SP layouts, and a report replayed after NEW_REQUEST