[Mooncake][Worker] Fail only the request whose async tensor read failed, at TP>1 too by arun-elr · Pull Request #419 · mstar-project/mstar

Fail only the request whose async (Mooncake) tensor read failed, and free its buffers once its transfers stop

arun-elr

committed

006f3c7

Hold READS_DONE while a failed read's transfers run, and read a torn-down read's outcome from its future

arun-elr

committed

b9622ba

Poll a read's future through done() and result() only, as the SHM arena's event future offers no more

arun-elr

committed

b15fba9

test: the output-failure teardown fake tensor manager reports no failed reads

arun-elr

committed

3e01445

Transfer metrics: a failed read is dropped and reported, not raised, so the read delivered before it counts once

arun-elr

committed

dc83e30

test: the serving-metrics stub tensor manager reports no failed reads

arun-elr

committed

c817311

test: the fake Mooncake engine's report waits for the reader to submit the batch it targets

arun-elr

committed

c4cbd40

Count a read's retries under the device filter the Mooncake engine was initialized with

arun-elr

committed

6bf3a03

Log a failed output read in one line, as a failed input read is

arun-elr

committed

a61c341

Count the HCAs for read retries on the first failed tensor, not at startup

arun-elr

committed

9ff9f8d

test: over real Mooncake TCP, a failed read keeps its buffers until a transfer still running reports it stopped

arun-elr

committed

9e3fd14

test: acceptance tests for a TP follower whose input read fails after rank 0 read its copy

arun-elr

committed

490b808

TP: take a row whose follower read failed out of the step on every rank, before preparing it

arun-elr

committed

3748ab9

TP read check: record every failed read for the check, and share the row vote

arun-elr

committed

8087222

test: the in-process Mooncake engine of the slice tests reads async too

arun-elr

committed

1628c8b

Navigation Menu

Repository navigation

[Mooncake][Worker] Fail only the request whose async tensor read failed, at TP>1 too - #419

[Mooncake][Worker] Fail only the request whose async tensor read failed, at TP>1 too#419
Gaurav-Shah05 merged 15 commits into
mstar-project:mainfrom
arun-elr:fix/failed-tensor-read

Commits on Oct 26, 2026

Fail only the request whose async (Mooncake) tensor read failed, and free its buffers once its transfers stop

Hold READS_DONE while a failed read's transfers run, and read a torn-down read's outcome from its future

Poll a read's future through done() and result() only, as the SHM arena's event future offers no more

test: the output-failure teardown fake tensor manager reports no failed reads

Transfer metrics: a failed read is dropped and reported, not raised, so the read delivered before it counts once

test: the serving-metrics stub tensor manager reports no failed reads

test: the fake Mooncake engine's report waits for the reader to submit the batch it targets

Count a read's retries under the device filter the Mooncake engine was initialized with

Log a failed output read in one line, as a failed input read is

Count the HCAs for read retries on the first failed tensor, not at startup

test: over real Mooncake TCP, a failed read keeps its buffers until a transfer still running reports it stopped

test: acceptance tests for a TP follower whose input read fails after rank 0 read its copy

TP: take a row whose follower read failed out of the step on every rank, before preparing it

TP read check: record every failed read for the check, and share the row vote

test: the in-process Mooncake engine of the slice tests reads async too