[arun-elr]

Handles TP>1 itself now, so #754 isn't needed (closed).

Fixes #418

What does this PR do?

On Mooncake TCP or RDMA, one failed async tensor read stalls every request on its worker or data worker: get_ready_tensors re-raises it on every poll and never removes it, so the requests behind it wait for its teardown (client abort or the request timeout), logging the same error each time (#332). This PR makes a failed read fail only its own request, and frees its buffers once Mooncake says the transfer stopped.

• The read fix. get_ready_tensors drops a failed or cancelled read and reports it once through take_failed_reads(). The worker fails the request with FAIL_REQUESTS; the data worker answers with one error chunk in the output slot of each tensor the read would have delivered.
• The buffer release. Tensor reads use TrackedMooncakeReader: one async batch per tensor, polled with transfer_check_status, with the sync read's retries and deadline. A failed read hands over the transfers still running; its buffers stay out of the tensor store until each one reports COMPLETED or FAILED, then they are unregistered and freed. Any other outcome keeps them for good. READS_DONE waits for those transfers too, so the producer keeps the memory they read.
• TP>1. Rank 0 schedules a lockstep step from its own copy of each input, so a follower whose read fails after rank 0 sent the step used to hang the instance. Now rank 0 marks a step that takes an input the followers read (ScheduleTPNode.read_check), a follower builds it w/ the rows it couldn't read left empty, and before prepare_inputs one gloo all-reduce of a flag per row takes those rows out on every rank. They're never prepared, committed or sent, so nothing of them reaches the client, the Talker or the prefix cache, and the request fails through the conductor like above.

How was it tested?

On 1628c8b (rebased onto main 3a4aedb): ruff check . clean; CPU Core (the ci.yml paths incl. the TP lockstep sim) 6,595 passed, 1,189 skipped, 6 failed, the same 6 as main 3a4aedb (6,531 passed, 1,187 skipped), all from packages missing on this host (ftfy, soundfile/torchcodec, prometheus_client). On the previous head cf3e84a merged into main 3d764d0: CPU Core (plus test/rust) 6,936 passed, 1,280 skipped, 9 failed (main: 6,876 passed, 1,278 skipped, the same 9 failed on this host; list below); TP lockstep sim 11 passed (7 on main).

Tests in detail

• test/modular/test_failed_tensor_read.py (new, 31 tests): 29 passed, 2 skipped on CPU; on an H200 both real Mooncake TCP cases pass, incl. a 2 GiB transfer still running after its read failed. The 6 read-fix tests fail on main.
• test/modular/test_tp_read_failure_acceptance.py (new, 7 tests): two real TP2 ranks in their own processes over gloo running Worker.run, the scheduler, the SHM manager, the engine and Qwen3.5's real prepare_inputs; the follower's image_grid_thw (or text) read fails after rank 0 read its copy. No hang, only that req fails, none of its outputs leave either rank, its pages are never leasable w/ prefix_cache_tp on, and the req beside it is bit-identical to a run w/o the failure. All 7 hang or fail on a2c97da.
• test/modular/test_tp_read_check.py (new, 22 tests): the same two ranks w/ an embedding that all-reduces inside prepare_inputs (as Qwen3.5's vocab-parallel one does) and w/ a step where every row failed; TP4 and TP2 x SP2 over four gloo ranks; the follower's serial and MSTAR_TP_ASYNC_SCHED builds; rank 0's marking; a read that fails after its request already failed for another reason. All 22 fail on a2c97da, and each fails when the logic it covers is removed (26 mutations).
• test/modular/test_tp_async_lockstep_sim.py: 4 model checks (tp_read_check_sim.py) over every interleaving of the read outcome, scheduling and the step.

Served by @merceod on 2x H200, TP2 Qwen3.5-0.8B over Mooncake TCP (comment): 11 injected follower-failure cells, only the target failed and main wedged; cost at c32 TTFT +2.7%, req/s -5.8% w/ ranges overlapping, c8 within noise.

Known behavior: a slow follower read stalls the whole instance in the all-reduce until it lands or fails (up to MC_TRANSFER_TIMEOUT), the same as main today. Read-checked steps are never pre-planned.

Qwen3-Omni over Mooncake TCP, main vs this PR on one H200, qwen3omni_2gpu's layout (the Talker reads the Thinker's states from another worker every step) with both workers on GPU 0; median of 3 fresh servers per arm, 0 errors in 360 requests per arm:

	c8 main	c8 #419	c16 main	c16 #419
TTFT p50, ms	121.8	123.8	155.3	150.9
first audio p50, ms	561	562	787	784
ITL p50, ms	26.9	27.1	38.5	37.4
audio s / s	33.2	32.7	48.2	48.2
requests / s	1.68	1.70	2.44	2.50

Notes on the numbers

The two arms' min to max ranges overlap in every cell; c8 audio s / s is 1.5% lower w/ requests per second the same, since this PR's sampled replies ran about 0.3 s shorter on average. With one injected Talker read failure per server, this PR failed only that request (in-band error) and the 20 others completed; main hung it until the 120 s timeout and failed or hung the 12 beside it.

Charts

e2e matrix, Mooncake TCP models (main vs f78bfbc)

The e2e matrix's tag:mooncake models on one H200, base and head back to back, 3 passes per point, 1 session per arm. These serve every node in one worker, so they cover the data worker's reads, not worker-to-worker hops. orpheus_mooncake did not start on either arm (its gated weights did not download), so it has no row.

Model	Modality	Correct	c=1 latency, base to head	c=8 throughput, base to head	Verdict
qwen3_omni_mooncake	text	yes	387 ms to 395 ms (-2.0%)	7.45 req/s to 7.24 req/s (-2.9%)	same
qwen3_omni_mooncake	image in	yes	450 ms to 452 ms (-0.4%)	289 tok/s to 289 tok/s (-0.1%)	same
qwen3_omni_mooncake	speech out	yes	1268 ms to 1264 ms (+0.3%)	42.4 audio s/s to 43.7 audio s/s (+3.2%)	same

CPU suite details

CPU Core paths from ci.yml plus test/rust, -n 4, OMP_NUM_THREADS=2. The 9 that also fail on main 3d764d0 on this host: test_lingbot_world_shell.py (3, no ftfy), test_chatterbox_model.py (2, no soundfile or torchcodec) and test_serving_metrics.py::test_refusals_count_from_when_metrics_are_turned_on (no prometheus_client), all of which CI installs; test/rust/test_runtime_lockstep.py::test_only_the_tp_leader_counts_new_tokens[1] (the Rust and Python runtimes disagree w/ this host's prebuilt mstar_rust); and two eval signal tests that time out on a loaded host. On the H200: test/distributed/test_tensor_transport.py 12 passed with Mooncake TCP.

Served run: harness and commands

One spot H200 (no 8-GPU capacity for two physical GPUs), Mooncake 0.3.13.post1 over TCP on loopback (MC_FORCE_TCP=1), torch 2.9.1+cu128, Rust graph runtime (the default). Each arm imports its own tree (PYTHONPATH, checked in each server's log).

• Server: python -m mstar.cli.main serve qwen3_omni --config qwen3omni_2gpu_1dev.yaml --tensor-comm-protocol TCP --enable-metrics, where the config is configs/qwen3omni_2gpu.yaml plus rank_devices: {1: 0} and the colocated config's KV sizes (thinker_kv and talker_kv 256 pages) so both workers fit on one GPU.
• Load: an out-of-tree closed-loop client, streamed /v1/chat/completions with modalities: [text, audio], temperature 0, max_tokens 96, eight fixed prompts in order; warm-up waves of 16 and 8, then 40 requests at c8 and 80 at c16. Replies average 68 tokens and 19 to 20 s of audio on both arms.
• Injection: a sitecustomize shim on the Talker's worker replaces the next thinker_states read's future with one that fails 1 s later, armed right before one target request; 12 more requests 0.5 s later, then 8 after those finish, all with a 120 s client timeout. On [Mooncake][Worker] Fail only the request whose async tensor read failed, at TP>1 too #419 the shim also logs each failed read take_failed_reads() hands back (86 for the target request, its later stream chunks) and the transfer-metrics series show TCP reads on both arms.

Not tested

• RDMA. What transfer_check_status reports for a failed RDMA read is untested: none of our VMs has an InfiniBand NIC (the 1-GPU presets have none). In Mooncake 0.3.13.post1's source, a task reports COMPLETED or FAILED only once every slice is accounted for (success + failed == slice_count, in both the RDMA and TCP transports), and the release frees only on those two values, so an unexpected RDMA status keeps the buffers rather than freeing them.
• qwen3omni_2gpu on two physical GPUs. 8x H200 spot had no capacity, so the same two-worker layout ran on one GPU; the Talker's per-step Mooncake read and its GIL use are the same.
• Served TP2 beyond the 0.8B pass. qwen3_5_9b_tp2, c64, qwen3omni_full_tp2 and bagel_xpu_cfg_tp2 (both checked on every step) have no served numbers.
• MSTAR_TP_ASYNC_SCHED=1 cost. The served failure cells ran w/ it on and off; the cost pair didn't (main's async launch failed in FlashInfer's multicast all-reduce setup).

Depends on and merge order

#629 is stacked on this. #471 (merged) added the abort of a request whose output failed and the cut of a stream at its first error; a failed read here goes through the same _fail_request, which aborts the request's walk once.

Design

• Why one batch per tensor. transfer_check_status looks at a batch's first task only, and frees the batch on COMPLETED (1) or FAILED (-1); it returns -2 on a slice timeout without freeing, else 0. Mooncake's sync batch read reports FAILED for the whole batch as soon as one task fails, while the other tasks may still be moving, and on its timeout frees nothing it can name. So only a per-tensor batch can say when every transfer into a buffer has stopped.
• Why not one call on success. No Mooncake 0.3.13.post1 call (also unchanged in 0.3.14-rc1) waits for a whole read in C++ and still leaves its tasks pollable after a failure: batch_transfer_sync_read and get_batch_transfer_status return only 0 or -1, and on a failure call freeBatchID themselves, which deletes the batches whose tasks finished and keeps, without saying which, those still running.
• Ownership of a batch id. The reader thread polls its ids until it raises AsyncReadError(running, unknown); from then on only the manager polls the ids it handed over, at the top of get_ready_tensors (no timer). A read torn down while still running keeps its future, and the manager adopts its ids only once the reader is done, so no id is polled by two threads or after Mooncake freed it.
• What is never freed. A failed submit (it may have posted part of the transfer and returns no id; 0, or uint64 -1 before 0.3.13), a status other than 1, 0, -1, -2, an error that is not an AsyncReadError, or a failed unregister: the buffers move to a list that is never freed, with one warning.
• Drain. has_inflight_reads counts a live request's failed reads whose transfers are still running, and the data worker's drained finish checks it, so READS_DONE (after which the producer frees its source) waits for them.
• Futures. A read's future is polled through done() and result() only, which is all the SHM arena's _CudaEventFuture offers.
• KV transfers keep the base reader's batched sync read; only tensor reads change.
• TP>1, which steps are checked. Rank 0 marks per request the inputs of its lockstep node that landed on it by a read (not its own persisted output coming back) or came from a node beside it outside the instance (an encoder on rank 0), and spends the marks once a step that took them ran. A step that takes a marked input, or any streamed input, is checked; decode steps aren't, so they pay nothing.
• TP>1, why before prepare_inputs. Qwen3.5's prefill_vision embeds each text part inside prepare_inputs w/ a vocab-parallel embedding, an all-reduce per part. A rank that prepares a row the others don't runs collectives w/ no partner, so the row has to leave every rank before any prepares it (test_a_prepare_that_runs_a_collective_per_row_stays_in_step).
• TP>1, the follower's step. It has rank 0's rows in rank 0's order, the unread ones w/ no inputs, so a step is built, submitted, voided and settled the same on every rank until the check, also under MSTAR_TP_ASYNC_SCHED. A read-checked step isn't pre-planned, so every rank admits it after the check.
• Poll pacing. Each status poll takes the GIL, which the sync read's wait in C++ never did. The reader polls every 100 us and waits on the first tensor still running instead of polling them all.
• Retry count. The number of HCAs (how often a failed tensor is retried, as the sync read does) is counted under the engine's device filter on the first failed tensor, once, behind a lock; a read that never fails never starts the second engine get_local_topology needs.

Performance (CPU, Mooncake TCP loopback)

Out-of-repo script: two TCP managers in one process (localhost, lo), host buffers; latency is start_read_tensors to the read's future done, 300 reads per shape; "torch loop" is a main thread running small torch.add ops while another thread reads back to back, as a fraction of its idle rate. Three runs each, main fa37d22 vs this PR before its rebase onto b4297bb (the commits since touch only failed and torn-down reads, the drain check and when the retry count is taken). The host ran other jobs, so read these as relative numbers.

Shape	p50 main	p50 this PR	torch loop main	torch loop this PR
1 x 4 KiB	0.180 to 0.185 ms	0.267 to 0.273 ms	0.25 to 0.28	0.34 to 0.35
1 x 1 MiB	0.83 to 1.23 ms	1.17 to 1.35 ms
1 x 4 MiB	3.31 to 3.59 ms	3.64 to 3.85 ms	0.78 to 0.95	0.84 to 0.86
8 x 512 KiB	3.56 to 3.81 ms	2.54 to 4.24 ms	0.68 to 0.84	0.63 to 0.74

A 4 KiB read pays about 0.09 ms at p50 and a 4 MiB read about 0.3 ms (8 percent), about one poll interval plus a GIL wait; 1 MiB and 8 x 512 KiB overlap with main. Beside 8-tensor reads the main thread runs about 0.1 slower on average, with overlapping ranges. Served, this does not show up in the Qwen3-Omni numbers above, where the Talker polls every step.

Tests in detail

test/modular/test_failed_tensor_read.py:

• Read fix (fail on main): a failed read does not block another request; a cancelled read counts as failed; later reads of a failed request are dropped (acked if they completed) until cleanup; a finished read waiting behind an earlier one is queried once; the worker fails the request once; the data worker puts the error in the failed tensor's slot, so the output after it is released; a future with only done() and result(), like the SHM arena's _CudaEventFuture, is polled like any other.
• Release, on a fake engine with scripted statuses: freed once the last running transfer reports COMPLETED or FAILED, held while it reports 0 or -2, through teardown; a read torn down mid-transfer is freed once it stops, with no id polled by both threads; a read that fails while teardown polls it keeps its buffers; a failed submit (0 and uint64 -1, first or retry submit), an unknown status (in the reader or the manager) and a failed unregister keep the buffers for good; a FAILED tensor is retried as often as the sync read retries; a read past its deadline hands over its running transfer; the drain waits for a failed read's running transfers and stops waiting once the request is torn down; KV reads keep the batched sync read; the HCA count is taken on the first failed tensor, once for three concurrent failures, and never on a good read.
• Real Mooncake TCP (skipped without Mooncake): a rejected read reports FAILED, fails only its request, and its buffer is unregistered and freed; a read whose first tensor is rejected while a 2 GiB second one is still moving fails at once, keeps both buffers registered and the request's read in flight until Mooncake reports the large transfer COMPLETED, then frees both.

Changed: test_empty_tensor_transport.py and test_mooncake_flat_layout.py fake the async read API; test_worker_drain.py's drained-finish test now has no read in flight; test_transfer_metrics.py expects a failed read to be dropped and reported, not raised; eight stub tensor managers gain take_failed_reads.

Earlier served runs

Qwen3.5-0.8B over Mooncake TCP on 2 x H200, on 4fbc57e (the production code before the two follow-ups), in a review: against main at c8, c32 and c64 with overlapping TTFT and ITL ranges and 0 errors, and injected read failures with 76 buffer keeps matched by 76 frees. The H200 runs with injected Mooncake TCP read failures in the earlier review threads (first review, later rounds) measured an earlier, wider version of this PR, which also carried the changes now in #471 and the read-buffer budget this version replaces with the release.

Checklist

• ruff check . passes
• Added or updated tests / docs where relevant

AI assistance: written with AI coding agent

[arun-elr]

I tested this on an H200 with Mooncake's TCP transport, including injected read failures (results above). I don't have RDMA hardware right now. I will test on RDMA when I get access to it and post the results here.

[arun-elr]

@merceod could you please take a look at this when you have a chance? It fixes the repeated "Mooncake read failed" loop pattern from #332. The details and H200 fault-injection results are above.

[merceod]

Read the diff and ran it: the 6 new tests pass here and fail on main, the five touched suites pass (77), the CPU core list passes (954). On H200s with Qwen3.5-0.8B, main against this branch. SHM: 112/112 greedy outputs identical and load within the noise of a shared node, as expected since SHM reads carry no future. Mooncake TCP (this Mooncake build needs MC_FORCE_TCP=1 on nodes with IB NICs, otherwise it registers buffers with RDMA and every read returns -1, which incidentally reproduced the #332 wedge on main for real: 8.4 million log lines in three minutes and the server never came up, while this branch came up and answered each request with a fast 500). With a working transport and every 5th Mooncake read raising in both processes: this branch answered 15 of 20 requests with a 500 in 23 to 328 ms and 5 correctly, no stall, no main-loop error, 10 of 10 correct once the flag came off; main under the same hook answered none of 20, every request hit the 90 s client timeout, and its log grew by 21.7 million lines (1.6 GB) in seven minutes. An abort storm of 80 image requests closed 0 to 200 ms after sending: no failed read and no quarantine on either tree, so client aborts do not feed this path. The worker side does what the description says: a failed read is dropped in one poll, its request reported once, other requests keep going, and the teardown keeps the buffers a transfer may still write into.

Three things on the data-worker side need a fix before this merges:

1. An output-read failure releases the client but never stops the request. The API server sets req.error and parks the rid when the error chunk arrives, and nothing calls abort_request, so the workers keep generating until the walk ends. Every later output of that request is read, dropped, and answered with one more error chunk plus a traceback. In the injection run the data worker logged one traceback per failed output read (5 for 5), and nothing told the conductor to stop those requests. Calling abort_request when the error chunk is recorded (or sending ABORT_REQUEST from the data worker) drains it at once.
2. Ready outputs of a failed request still go out after the error. _process_read_tensors handles failed reads first and then emits whatever ready holds for the same rid from the same poll, and the chat SSE loop keeps yielding after an error event. With injection on, 2 of 16 streams carried two error events (orders "EEd" and "cEEd" for content, error, done); I did not catch a content chunk after an error in that sample, but the code allows it. Skipping ready[rid] for rids with a failed read and ending the stream at the first error fixes both the order and the double error.
3. The quarantine has no accounting and no bound. Each failed Mooncake read moves the edge's buffers, GPU memory on the worker, into a list that is never trimmed, and the only trace is one warning per event. Under the #332 condition this grows until a restart with nothing a monitor can read: with the RDMA registration failing, the worker parked 1,229 reads and 117 MB in two and a half minutes, 1.5 MB per image request. A bytes and reads counter in the --log-stats output and a byte budget past which the process reports itself unhealthy would make the tradeoff visible. Asking Mooncake whether the batch drained, so the buffer can be freed later, is the longer-term fix.

Notes, none blocking:

• SHM read failures still raise inside start_read_tensors and fail every in-flight request on the worker through the main-loop handler. Not a regression, but the title reads as if any read failure is per request now.
• _fail_request logs the same traceback once per tensor of a dropped read.
• get_ready_tensors now calls future.done() twice and result() on waiting entries every poll. On the arena that is an extra event query per pending edge.
• _quarantine_read_buffers pops uuid_to_shard_dim but not uuid_to_edge_name.
• test/cpu-core.txt merges cleanly against main as of today.

[merceod]

Two ordering problems here. The failed reads are handled first and then ready still holds this rid's outputs from the same poll, so they are emitted after the error chunk; and the chat SSE loop in serving_chat.py keeps yielding after an error event, so a stream can carry error, content, error. With injection on, 2 of 16 streams carried two error events ("EEd", "cEEd"). Skipping ready[rid] for rids with a failed read (and ending the stream at the first error) fixes both.

[arun-elr]

Fixed in 5dc5ff4. Nothing goes out for a request after its error: outputs ready in the same poll are released without postprocessing, and outputs held for ordering are dropped. iter_result_chunks now ends the response at the first error. It no longer appends req.error after an in-band error, which was the second "E" in "EEd". On the H200 run with injection (before the #368 rebase), all 16 streams carried exactly one error and nothing after it (rcEd or rEd).

[merceod]

This releases the client but the request keeps running: the API server sets req.error and parks the rid when the error chunk arrives (entrypoint.py 642-656) and never calls abort_request, so the workers generate until the walk ends, and every later output of the rid is read, dropped, acked and answered with one more error chunk plus a traceback from _fail_request. In the injection run the data worker logged one traceback per failed output read, and nothing told the conductor to stop those requests. Aborting the request when the first error chunk is recorded (or sending ABORT_REQUEST from here) drains it at once.

[arun-elr]

Fixed in 5dc5ff4. On a request's first failed output read, the data worker aborts it through the client-abort path (ABORT_REQUEST plus its own drain). The conductor stops the walk at once, and outputs still on their way are acked unread. Later finish or abort notices for that request are ignored, so it gets one ABORT_REQUEST and one READS_DONE (tested across 19 orderings with REMOVE_REQUEST), and one traceback (a repeat failure is debug-only). On the H200 run before the #368 rebase, each of the 26 output-read failures logged one traceback and one self-abort. I didn't call APIServer.abort_request when the error chunk is recorded, because it pops the pending request and the client would lose the error before reading it.

[merceod]

This list is never trimmed and nothing counts it. Each failed Mooncake read parks the whole edge's buffers, GPU memory on the worker, still registered, and the only trace is the one warning per event. Under the #332 condition that grows until a restart with nothing a monitor can read; with Mooncake's RDMA registration failing on an IB node the worker parked 1,229 reads and 117 MB in two and a half minutes. A bytes and reads counter in the --log-stats output and a byte budget past which the process reports itself unhealthy would make the tradeoff visible; asking Mooncake whether the batch drained so the buffer can be freed later is the longer-term fix. Also uuid_to_edge_name is not popped here while uuid_to_shard_dim is.

[arun-elr]

Fixed in 5dc5ff4. Kept reads and bytes are counted per process. The --log-stats report has a Read quarantine section on every request that finishes or fails, so it shows up even when every request fails. Past MSTAR_READ_QUARANTINE_BUDGET_MB (default 1 GiB per process), /health returns 503 naming the process, on both frontends. At your ~47 MB/min, the default trips in about 22 minutes. uuid_to_edge_name is popped too. Two things are left for #431: freeing the buffers once Mooncake confirms the batch drained, and checking the receive-side register_memory return (a failed registration is currently kept and counted like a posted read).

[arun-elr]

@merceod I rebased onto #368 and addressed your review; ready for another look.

Thanks for the review and for reproducing #332 on the IB nodes. All three blocking items are fixed in 5dc5ff4, with replies on each thread. The PR grew for one reason: each submission now gets a unique internal id, so a reused request id can't reach another submission's state once failed requests are aborted and torn down asynchronously. You haven't reviewed that part yet. The description has a section on it, and clients still see only their own id.

Your three scenarios on one H200, run before I rebased onto #368 (Qwen3.5-0.8B, against main 99f5108b):

• Injected read failures (every 5th Mooncake TCP read, both processes): 10/20 chat requests correct and 10 fast 500s (14-46 ms), with no timeouts. All 16 streams ended with exactly one error and nothing after it. 9 of 10 /generate requests got fast 500s. Each failed request is stopped: one traceback and one self-abort per output failure, and one FAIL_REQUESTS per input failure. The run logged 760 lines during injection. Main: 0/20 chat, 0/16 streams, and 53 M log lines in 11 minutes.
• Abort storm (80 image requests closed 3-200 ms after sending): 59 drained mid-flight, with no failed reads, no kept buffers and no tracebacks. Answers were identical before and after the storm, and identical to main.
• SHM: 112/112 greedy outputs identical to main, with load within noise.
• /health returns 503 past the budget, the --log-stats block appears on failed requests, and every per-request map was empty after each run.
• After rebasing onto Host-side overhead: per-row Python out of the decode step #368, at 5dc5ff4: I reran the injected-failure smoke. Decode tokens now ride inline, so only 7 output reads failed; there were no timeouts and no loop errors, and each failed stream got one error. Inline tokens follow the same rules: under the admit lock, nothing after a request's error, and every error path wakes the stream.

The first H200 run turned up a bug in my changes, now fixed in 5dc5ff4. The data worker's teardown could drop a request's entry while the API message loop was reading it. That raised KeyError and skipped that loop pass (3 times in one run; no client saw it). Those maps now share a lock, and one bad request, message or chunk no longer skips the rest of the pass. The rerun had 0 KeyErrors and the same client-visible results.

On the notes:

• _fail_request logs one traceback per request; a repeat failure is debug-only.
• get_ready_tensors makes one done() call per pending read and checks result() once.
• SHM read failures still go through the worker's main loop, as on main. The description says so, and I retitled the PR to match.
• With this Mooncake build, --tensor-comm-protocol TCP on IB nodes needs MC_FORCE_TCP=1.
• At your ~47 MB/min, the default 1 GiB budget trips /health in about 22 minutes.

Freeing the kept buffers and the receive-side register_memory check are in #431.

[arun-elr]

@merceod rebased onto current main (no behavior change beyond resolving conflicts with #380); ready for another look whenever you have time.

[arun-elr]

@merceod could you take another look when you have time? All three points from your review are fixed, with a reply on each thread, and the branch merges cleanly with current main.

• A failed output read now aborts the request once, so the workers stop, and later outputs are acked unread.
• Nothing goes out for a request after its error; the stream ends at the first error.
• Kept reads and bytes show in --log-stats, and /health returns 503 past MSTAR_READ_QUARANTINE_BUDGET_MB (default 1 GiB).

The one part you have not reviewed yet is the per-submission internal id (its own section in the description). #471 and #481 conflict with this PR, so landing it first means each of them rebases once.

[arun-elr]

/review

[github-actions]

An earlier review run didn't complete; a later run finished: #419 (comment)

[arun-elr]

/review

[arun-elr]

/review continue

[github-actions]

Automated review

The PR stops one failed async (Mooncake TCP/RDMA) tensor read from stalling every other request. get_ready_tensors now drops the failed read and reports it through take_failed_reads(). The worker reports the request once with FAIL_REQUESTS. The data worker sends one error chunk and then aborts the request itself. The failed read's local buffers are kept for the life of the process, counted, and checked against MSTAR_READ_QUARANTINE_BUDGET_MB; past the budget, /health returns 503 on both frontends. The PR also gives every submit its own internal request id ({client}#{token}-{n}), and the API server sets the client-id-derived seed in model_kwargs itself, so the conductor's per-request noise stays the same. The code matches the description.

Findings: none. Points I traced and found correct:

• The seed injected in submit_request gives the same value the conductor's _req_id_to_seed gave before, and no model treats an explicitly set seed differently from a missing one.
• Worker-side failed reads stay keyed by handle and are converted to the wire string in _fail_requests. The new tensor-manager state is purged in cleanup_request, which runs before the handle is released.
• The SHM and arena managers inherit the changed get_ready_tensors and cleanup_request without overriding them.
• The Rust pong field msg is #[serde(default)].
• No code outside entrypoint.py reads pending_requests with a client id.

Description:

• The SHM load comparison (72.9–75.1 vs 72.0–75.1 req/s at c=16) was run before the rebase onto Host-side overhead: per-row Python out of the decode step #368. Since that rebase, the API message thread takes _admit_lock for every inline token (new_result_token) and every result chunk (get_result_chunks), on top of request_lock. That is a per-token cost on the hot path (AGENTS.md invariants 6 and 10). Please add a post-rebase throughput number at c=32/64 on a small model such as Qwen3.5 0.8B, or explain why it isn't needed.
• The test table doesn't list results for test_data_worker_wakeup.py or test_video_frame_protocol.py, which this PR also changes.
• Not blocking: the per-submission internal id changes request-id semantics across the API server, data worker and tensor keys, and hasn't been reviewed yet. Per AGENTS.md's "Splitting work across PRs", it could land separately, under or over the read-failure fix. The description explains why it's bundled.

Questions

• How does an input-read failure behave in a TP/SP lockstep instance when only one rank's read fails? Only that rank calls _fail_requests. If a follower's input never arrives while the leader schedules the request, the follower can't run the leader's broadcast batch, and _complete_drain_if_ready waits on pending_tp_follow_count. main behaves the same way, so this isn't a regression. But if it can hang, the claim "fails only its own request" should be limited to non-lockstep nodes.

---

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[arun-elr]

How does an input-read failure behave in a TP/SP lockstep instance when only one rank's read fails?

It depends on which rank fails:

• The leader's read fails: only that request fails. The leader never schedules it, _begin_drain fans the drain out to the followers, and nothing for it is queued in their TP FIFO, so every rank sends READS_DONE.
• A follower's read fails: only that request fails if the conductor's drain reaches the leader before the leader schedules the request. If the leader has already broadcast a batch with it (the usual order, since a failed read often reports at its timeout), the instance hangs. pop_ready_rids is all or none, so the follower can never build that batch, and the leader stays in the collective with its drain deferred behind the in-flight step. _complete_drain_if_ready then waits on pending_tp_follow_count, and the follower ignores the conductor's forced REMOVE_REQUEST after MSTAR_DRAIN_TTL_S. The other requests in that batch stall too. main hangs the same way.

A CPU repro with the real MicroScheduler and Worker drain code on fake ranks shows all three cases, and the hang test passes on both main and this PR. Fixing it needs a protocol change (a readiness gate before the leader schedules, or followers running the batch with placeholder inputs). It is tracked in #564 with the repro. The description now limits "fails only its own request" to the leader for lockstep instances.

Thanks for raising it.

[arun-elr]

Please add a post-rebase throughput number at c=32/64 on a small model such as Qwen3.5 0.8B, or explain why it isn't needed.

Measured on one H200 with the same SHM workload as before (Qwen3.5 0.8B, the 112 greedy chat requests), on the PR head and on its main base. There were three alternating server runs per tree and 15 reps of 448 requests per row. Median results:

Concurrent	main req/s	This PR req/s	p95, main / PR
16	86.5	84.9	396 / 408 ms
32	116.8	116.2	644 / 645 ms
64	146.7	142.8	833 / 853 ms

All three rows are within run-to-run noise: at 64 the per-run medians span 139.6-158.3 req/s on this PR and 141.0-150.4 on main. Every request succeeded, and the greedy outputs were identical on every run. Ranges and method are in the description's "Post-rebase load" toggle.

The test table doesn't list results for test_data_worker_wakeup.py or test_video_frame_protocol.py, which this PR also changes.

Added to the table, with the command: 5 and 31 pass on the head.

[arun-elr]

Hardening: a request_id that is still in flight now gets a 409 instead of superseding the earlier submission, so one client can't cancel or take over another's request. The live request keeps running and gets its full output, and a retry after it is answered, fails, is aborted or disconnects is still served. The Rust frontend behaves the same. Tests and the merge order with #519 are in the description.

[NSagan271]

🤖 Automated live test, AI-generated (Claude Code). Not checked line by line by a human. Numbers come from one allocation, so treat small deltas as provisional.

Live test: [API server][Data worker] Fail only the request whose async (Mooncake) tensor read failed

Works: with Mooncake TCP reads failing on purpose, only the affected requests fail, within tens of milliseconds, where base left most of the batch hanging until the client timed out.

1x H200 (fault injection, Qwen3.5 A/B), 3x H200 (Qwen3-Omni TP2) · a test hook outside the repo fails every 5th Mooncake read · PR head 0bac0c9 vs base 57bb5c1 (merge-base)

Claims

Claim	Result
A failed read no longer stalls other requests	✅ 20 chats, 4 at a time. Base: 2 ok, 2 × 500, 16 timeouts at 45 s. PR: 13 ok, 7 × 500 in 12–25 ms
A stream ends with one error	✅ 16 streams. Base: 16 timeouts. PR: 11 ok; the other 5 end content → error → [DONE] at 52–58 ms
Reported once, not re-raised every poll	✅ Base logged 9.3 M lines, 8,337 of them Mooncake read failed, in about 4 min; PR logged 629 lines
Recovers afterwards	✅ Both arms: 10/10 chats and 4/4 streams after injection stopped
/health turns 503 past the quarantine budget	✅ MSTAR_READ_QUARANTINE_BUDGET_MB=0.0001: 503 worker_0 keeps 1128 bytes … from 7 failed tensor reads, past its budget of 104 bytes
Rust frontend's deep /health	✅ test_rust_frontend.py: 31 passed (base 29)

Performance

Qwen3.5 0.8B, ABBA, text tok/s per restart:

arm	c=8	c=32	c=128
base	2792, 2858	6818, 7523	9919, 9850
PR	2716, 2761	7512, 6844	10010, 9905

• c=32 and c=128: no regression, inside the restart spread.
• c=8: 2.9% lower, and the two arms' ranges just separate. Single A/B jobs here have moved −1.4% to −5.5% with no code change, so treat this as provisional, not a finding. Qwen3-Omni TP2 audio-to-speech and text-to-speech at c=1/8: 0 failed.

Other

• Not tested: RDMA; TP follower failures (owned by Worker: a failed input read on one TP/SP follower hangs the whole lockstep instance #564); the Python-runtime smoke (MSTAR_RUST_ZMQ=0 MSTAR_RUST_GRAPH=0).

[merceod]

Re-review of 38113a4 on klone H200s, one PR server and one main server per lane, Mooncake TCP with every 5th read made to fail for the failure runs, SHM for the load cells. These nodes run at 1560 MHz under load, so the absolute numbers are low on both trees alike.

The three things from round one are fixed, in the code and in the served runs.

• Nothing goes out after a request's first error. All 20 failing streams ended with one error and [DONE], no second error and no content after it. Main timed out 16 of 16 streams.
• A failed output read stops the request. 7 output failures gave 7 self-aborts, each with one drain and one teardown 4 to 28 ms later, and 55 input failures were each reported once. Main logged 18.4 M lines (1.4 GB) in 7.5 minutes, the PR 1,123 for the whole run.
• Kept buffers are counted and budgeted. 5 of 5 bad MSTAR_READ_QUARANTINE_BUDGET_MB values fail at startup naming the setting, past a 104 B budget /health is 503 on both frontends with the reason, failed /generate reports carry the quarantine block, every per-request map was empty after the TTL, and uuid_to_edge_name is popped.

Load over SHM, main against main plus this PR, 3 launches per tree and 15 reps of 448 requests per cell: c=8 37.8 vs 39.4 req/s, c=16 58.1 vs 58.7, c=32 75.9 vs 79.1, c=64 91.2 vs 99.1, per-launch medians overlapping. The 112 greedy outputs were identical on every launch of both trees, and a fixed request_id at temperature 0.8 gave the same text on both. The Python runtime behaves like the Rust one under injection, and the abort storm left no failed reads and no kept buffers on either tree.

The 409 contract holds on the Python frontend and fixes real breakage on main (a duplicate cut the first stream after one line, 16 concurrent duplicates gave 15 empty 200s, drop and retry hung). What is left is the Rust frontend. Drop a stream and resubmit the same id at once and 9 of 15 rounds came back 200 with no output (comment inline). Main hangs there instead, so this turns a hang into a silent empty answer, and it contradicts "the Rust frontend behaves the same". The description lists the stale relay under not here (#519), so either land after #519 and rebase onto its per-submission tokens as you planned, or make the aborted relay skip its done. Requesting changes for that one thing.

Smaller points.

• The merge with #519 conflicts in 7 files and 29 hunks (entrypoint.py 17, bridge.rs 5), not the 4 files listed, and the two contracts differ a little. This PR blocks an id while the answer is unread and refuses at once. #519 frees it when the request finishes and waits 0.5 s first.
• A failed image read keeps about 600 KB here (55 reads, 33 MB), so the 1 GiB default trips after roughly 1,700 failed image reads rather than after the 22 minutes of text reads in the description (comment inline).
• _client_facing_error rewrites a verbatim internal id only. That is fine for # and regex characters (checked live), not for an id escaped inside a repr. Theoretical.

[merceod]

Here is the drop-and-retry case. abort removes the route first (line 277) and only then tells the backend, the aborted submission's relay then sends done for the client id (rust_frontend.py:157), and that done closes the route the retry just opened here under the same id (lines 192-198). The retry's output arrives with no route and its client already has an empty 200. On klone 9 of 15 rounds (7 of 7 when the retry rides a keep-alive connection opened before the drop, 2 of 8 on a new connection); main hangs to the client timeout on the same path. Keying routes and aborts per submission, as #519 does with sub, fixes it, and so would skipping the done of a relay whose submission was aborted.

[arun-elr]

Fixed in this PR, the same way as #519. Each submission now carries its own token (sub) between the Rust frontend and the bridge loop. Routes, replies and aborts go by it, and the 409 check stays per id. abort now sends the abort before it frees the id, so the backend sees it before the retry's submit. The aborted relay's done carries the old token and no longer finds a route. The bridge loop also binds each relay and abort to its own submission at submit time.

• New test_a_retry_right_after_a_dropped_stream_is_served, with the real mstar-server and bridge and a stub whose aborted stream ends only after the retry is submitted: 5 of 5 empty 200s before, 5 of 5 served after, and 600 of 600 in a longer CPU loop.
• Three Rust unit tests for the route table, including an older submission's done arriving after the retry.
• H200, Qwen3.5 0.8B through --rust-frontend, a streaming /generate dropped after its first line and the id resubmitted at once: 30 of 30 served (15 on a keep-alive connection, 15 on a new one), no 409. The previous commit gave 2 empty 200s in 30, both on keep-alive.

[merceod]

True on the Python frontend (5 of 5 retries served at once on klone). Through the Rust frontend a retry right after a disconnect is the empty-200 case above, so until #519 lands this should say Python frontend, or the bridge needs the fix.

[arun-elr]

With the bridge fix above the claim holds on both frontends, and docs/clients.rst now says so: the id is free again "on the default and the Rust (--rust-frontend) server alike, including a retry sent right after a disconnect". On an H200, immediate retries after a disconnect were served 30 of 30 through the Rust frontend and 10 of 10 through the Python one.

[merceod]

Worth saying the per-read cost depends on the tensor. Text reads keep about 190 B each, but a failed image input read kept about 600 KB here (55 reads, 33 MB), so the default trips after roughly 1,700 failed image reads, not after the 22 minutes quoted for text.

[arun-elr]

Added to docs/environment_variables.rst: what one failed read keeps depends on its tensor, about 190 B for a text read and about 600 KB for an image input read, so the default lasts about 1,700 failed image reads. The Resources section of the description says the same. No new bound: the budget and the /health 503 stay the limit, and freeing the buffers is #431. The injected run on the new head kept 9 worker reads in 1720 B, about 190 B each.

[arun-elr]

the fix is to record every failed read for the check and keep the filter for the report only

Great catch, done: every failed read goes to read_failed_rids now and the failed_rids filter only gates the report (worker.py#L1570-L1586). test_a_read_that_fails_after_its_request_failed_still_goes_to_the_check fails it first, then fails its read, then sends a read-checked head; before the fix the follower never builds that step (test).

also loosened the wall-clock bound in test_a_step_whose_every_row_failed_runs_empty_on_every_rank: it's now the no-failure run's time plus COLLECTIVE_S instead of a flat 8 s (L152).

[merceod]

Approve on cf3e84a. The push is the three things asked for and nothing else. Every failed read now goes to read_failed_rids and the failed_rids filter only gates the report, with a test that fails the request first, then its read, then sends a read-checked head. The async follower path builds unread rows through with_unread_rows, with the comment on why the push-back of a never-popped row stays harmless and a test for it. The per-row vote is flagged_on_any_rank in distributed/all_reduce.py, and the wall-clock bound is the no-failure run's time plus the collective allowance. Those three tests pass here in 24 s.

[Gaurav-Shah05]

@arun-elr you need to rebase before merge

[arun-elr]

you need to rebase before merge

@Gaurav-Shah05 rebased on main, CI's green. one conflict w/ #623 in cleanup_request, kept both (its slice pops go after the ack, which needs slice_origins), plus a test-only commit so #623's in-process Mooncake engine also reads async.