[xiaodouzi666]

Description

Fixes #4662

What was wrong.

• Every RDMA worker learns on its own. Each worker keeps its own RailMonitor per peer machine (WorkerContext::rails), so a NIC pair that cannot reach the peer is discovered by every worker separately.
• Every lesson costs an RC retry timeout. A work request on a dead pair fails only after the QP's retry timeout (about 3.7 s on our test hosts), and a worker pauses the pair only after three failures inside the error window.
  • Only the first non-flush error of a timeout reaches the monitor; the work requests it flushes do not count.
  • So under READ a worker usually sees one counted failure per timeout round, and most workers never pause the pair.
• One slice stalls the whole transfer. A transfer is split into slices over all workers. On hosts where a management NIC cannot reach the peer's rail NICs, main stalled 18–48 transfers per 30 s tebench run, and WRITE ran at 11–27 GB/s where these hosts reach about 60.

What this PR changes. The monitors of one peer machine share the failure count and the pause verdicts of each NIC pair through a process-wide table in rail_monitor.cpp:

• Count. markFailed takes the shared count, with the same window rule as before.
• Publish. A fresh pause and a re-pause after a failed Half-Open trial go into the table. A later pause extends an earlier one and never shortens it; the larger cooldown is kept until a success.
• Withdraw. A successful probe, or a successful transfer on a worker that has counted a failure on the pair or holds its pause, removes the entry.
• Sync. admit() and findBestRemoteDevice() first compare a generation counter, which costs one atomic load when nothing changed. On a change the monitor:
  • adopts new pauses;
  • extends shorter ones;
  • releases pairs whose entry is gone or only counting again;
  • leaves a pair alone while its own probe or trial is on the wire, and re-reads the table once that probe ends without success.
• Wiring. Workers set the peer machine id on the monitors they create: one line in getOrCreateRail. The allocation cache and monitors without a peer keep today's behaviour.

The table is keyed by NIC names, not ids, so monitors holding different topology snapshots agree. Probing and the Closed/Open/Half-Open state machine stay per worker. GdrReachability (#2966) already applies the same process-wide idea to NIC/GPU pairs.

Why this is safe.

• The endpoints of a pair are already shared by all workers, so a pair one worker proved dead is dead for the rest. The PR only shares what one worker has learned.
• The monitor's own lock ([TENT][FEAT] Reduce CQE Processing with Selective Signaling #4632) already serializes admission with teardown; the shared table takes its own lock below it. The table never calls back into a monitor or an endpoint.
• The completion fast path is unchanged and takes no shared lock.
• No new configuration key, no new file, no signature change. Log lines keep their prefixes and give the new reasons in brackets: (paused by another worker, cooldown=..s) and (recovered on another worker).

Limits.

• The first failure on each dead pair still costs one timeout. Knowing in advance is what [TENT] Exclude cross-subnet NIC pairs from the automatic rail map #4730 adds; the two are complementary.
  • [TENT] Exclude cross-subnet NIC pairs from the automatic rail map #4730 excludes pairs by subnet when the configured prefix covers the routing domain.
  • This PR learns from any failure, also on IB, IPv6, explicit rail files or an unconfigured prefix.
• READ learns in about six timeout rounds (about 23 s), not two. Only one failure per round is counted per dead pair, so the pooled count still needs three rounds per pair.
• A transient outage can pause a healthy pair on every worker.
  • One outage that times out several lanes at once is enough.
  • As on main, a paused pair comes back through a probe or a Half-Open trial, which only admit() arms. In a failure-free steady state the best path never admits that pair again.
  • Pooling widens this existing gap from one worker to all workers. A success on a worker that never saw the pause does not clear it.
• Selective signaling. The learn-once behaviour assumes the default signal_interval of 1. With selective signaling, successes inferred for unsignaled work requests ahead of an error CQE can withdraw a fresh pause.

Suggested Merge Priority

• P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
• P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
• P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
• P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Unit tests (tent_rail_monitor_test, no RDMA device needed): 15 new cases in RailMonitorShareTest; the 23 existing cases are unchanged. They cover:

• a pause adopted by another worker;
• failures counted across workers;
• recovery by a transfer and by a probe;
• a success resetting the count for all;
• other peers and monitors without a peer;
• expired pauses;
• re-escalation and extension, and a shorter pause not cutting a longer one;
• in-flight probes left alone;
• a withdrawal followed by a new failure, and re-reading the table after a probe that did not succeed;
• the cooldown kept until a success;
• a rebuilt monitor taking over pauses.

Each case uses its own peer id, so the process-wide table does not leak between cases or --gtest_repeat rounds. Each part of the change was reverted one line at a time (25 mutations), and every mutation made at least one case fail.

Test commands:

cmake -S . -B build -DUSE_TENT=ON -DBUILD_UNIT_TESTS=ON -DBUILD_BENCHMARK=ON
cmake --build build -j
ctest --test-dir build/mooncake-transfer-engine/tent/tests -j8

Test results:

• Unit tests pass (TENT ctest 66/66; tent_rail_monitor_test also with --gtest_repeat=3 --gtest_shuffle, and under ASan)
• Integration tests pass (if applicable)
• Manual testing done (described below)

Two hosts.

• Each host has five RoCE v2 NICs: a management NIC (MTU 1024) and four rail NICs (MTU 4096) on routed /28 subnets.
• The management NIC cannot reach the rail NICs, on either host.
• Tool: tebench --backend=tent --xport_type=rdma, 6 threads, 1 MiB unless noted; main and this PR alternate.
• The 30 s runs, the run with [TENT] Use each RDMA NIC's path MTU and match it with the peer #5061 and the soak used this commit. The 60 s, no-dead-pair and one-engine runs used an earlier revision that differs only in how pairs recover or are re-escalated.

Scenario	main	this PR
Both hosts open all five NICs, no rail file, 30 s (READ and WRITE, both directions)	READ 46.0–50.7 GB/s, WRITE 11.3–20.2 GB/s; 18–48 transfers stalled > 1 s; errors until the end	READ 49.9–57.7 GB/s, WRITE 57.3–63.5 GB/s; 9 stalled transfers; WRITE learns in two rounds (last error at 8.0–8.5 s), READ in six (about 23 s)
Same, 60 s	READ 44.8–48.7 GB/s, WRITE 40.0–40.6 GB/s; 34–48 stalled	READ 52.3–59.3 GB/s, WRITE 60.6–63.1 GB/s; 7–9 stalled
Same, with #4661 fixed as well (#5061), 30 s	—	the rail NICs run at 4096; READ 55.8–56.7 GB/s, WRITE 53.1–61.4 GB/s; 7–12 stalled
No dead pair in use: a rail file with 16 pairs and direct rails, 64 MiB, 30 s	0 errors	0 errors, 0 pauses; bandwidth −0.8% to +0.6% of main
One engine targeting its own segment, all five NICs, 30 s	WRITE 0.1 and 23.8 GB/s, READ 34.9 and 45.6 GB/s	WRITE 47.2 and 51.3 GB/s, READ 50.4 and 55.8 GB/s
About 10 min (11 sizes × 55 s) of WRITE/READ with data checks, 64 KiB–64 MiB	—	every check passes; all errors fall in six rounds at 175–242 s, none afterwards
This PR without the line in getOrCreateRail	—	the same as main (no pause is shared), which shows that line is the only entry point

Two observations from these runs:

• READ gains only 9–24% at 30 s, against 2.2–5.6× for WRITE. WRITE times out several lanes per round, so the pooled count reaches three in one round. READ counts one failure per round per dead pair, so the pause comes after three rounds, plus three more for the pair the remap moves to.
• Every run still pays the first rounds. WRITE pays two (about 4 s and 8 s; the second is the pair the remap after the first pause runs into); READ pays six (about 23 s). Without prior knowledge such as [TENT] Exclude cross-subnet NIC pairs from the automatic rail map #4730's subnet filter, the first failure on a pair costs one timeout.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Claude was used during this period, and human review was implemented as well.