Repository navigation
Conversation
An error callback can finish a partially submitted task while it remains on the QoS chain. The submit loop then completes it again when unlinking. Require the task to be off the chain before publishing completion and reclaiming it.
Add a per-worker control queue using the existing mutex and condition variable. Expose worker-local state through WorkerContext. Snapshot control and I/O queues under the same lock, then process control tasks before I/O tasks outside the lock. This prevents newly queued I/O from overtaking its preceding control task. Release callback captures outside the lock and drain queued control tasks before worker exit. Fail requests if task allocation fails after dequeueing. Signed-off-by: Cage Chen <cage@arcfra.com>
Detect connection failures through I/O and admin polling. Stop submission and drain qpairs on their owner workers before reconnecting the controller, then resume I/O after recovery. Bind segments during Open and let their owner workers create qpairs, advance asynchronous connections, and clean up on shutdown. Signed-off-by: Cage Chen <cage@arcfra.com>
Master heartbeat probes keep reusing cached controllers and qpairs after connection failures. Subsequent probes can keep failing after the target recovers, eventually causing healthy segments to be removed. Drain all probe qpairs on the affected controller before disconnecting it. Reconnect on the next heartbeat, refresh namespace handles, and recreate qpairs as needed. Use the existing heartbeat thread to advance recovery and finish outstanding I/O safely before reusing probe buffers. Signed-off-by: Cage Chen <cage@arcfra.com>
cage-goat
requested review from
XucSh,
YiXR,
stmatengss and
ykwd
as code owners
October 20, 2026 10:18
cage-goat
changed the title
This was referenced Oct 20, 2026
Open
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #4805
This PR depends on #4656, which introduces client-side NoF controller recovery. It reuses the qpair setup and safe teardown helpers introduced there.
Only the final commit contains the Master heartbeat recovery fix and its E2E tests. Earlier commits are inherited from the dependency branch. Please focus the review on the final commit.
Master NoF heartbeat probes can keep reusing failed controllers and qpairs after a connection failure. Subsequent probes may continue failing after the target recovers, eventually causing available segments to be removed.
This PR adds recovery for Master heartbeat connections:
Namespace registrations and the existing segment removal policy are preserved. Only successful probe reads refresh liveness.
Suggested Merge Priority
Module
mooncake-transfer-engine)mooncake-store)mooncake-conductor)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Built and tested on Linux with SPDK v23.01.1,
USE_NOF=ON, andBUILD_UNIT_TESTS=ON. Hugepages and access permissions were configured before running the E2E test.Test commands:
Test results:
The E2E starts an independent
mooncake_masterprocess and a TCP malloc-bdev target with two namespaces sharing one controller. Both namespaces use 32 MiB capacity and 4096-byte sectors, with unchanged configuration across restart.It registers both namespaces through RPC, pauses the target with
SIGSTOPuntil a probe read times out, and restarts the target at the same endpoint. Both namespaces must resume successful heartbeat reads without restarting Master or registering the segments again. RPC queries verify that the original segment UUIDs and registration information are preserved.A second, persistent outage verifies that both namespaces are removed through the existing heartbeat policy and that the registration list becomes empty. Master must then exit normally after
SIGTERM.Recovery evidence:
The following complete lines from one E2E run show the probe timeout, controller disconnect/reconnect, qpair creation, and successful heartbeat reads for both namespaces:
Checklist
./scripts/code_format.shAI Assistance Disclosure
Codex assisted with design analysis, implementation, code review, E2E testing, fault validation, and documentation. The human submitter is responsible for reviewing, understanding, and explaining all changes.