[cage-goat]

Description

Building on the NoF initiator interface introduced in #3424, this PR adds a Linux kernel NVMe-oF backend for client data I/O and master heartbeat probes. It reuses the kernel's connection management capabilities and supports deployment without SPDK/DPDK or hugepages.

Fixes #5001

Main changes:

• Automatically discover preconnected namespaces using the endpoint and NSID.
• Implement asynchronous I/O with O_DIRECT and io_uring, reusing the existing NoF workers and QoS.
• Use a separate io_uring queue for each namespace to prevent busy namespaces from exhausting a shared queue and reduce interference between namespaces.
• Add build-time and runtime selection for the Linux backend while retaining the existing SPDK backend.

Connection setup and reconnection are managed by the operating system and deployment tools. The current implementation supports one connected controller per subsystem on each host. If a device is deleted or recreated, the affected Mooncake processes must be restarted.

Suggested Merge Priority

• P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
• P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
• P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
• P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Validated with a Linux-only build using USE_NOF=ON, USE_NOF_LINUX=ON, USE_NOF_SPDK=OFF, and BUILD_UNIT_TESTS=ON.

Test commands:

cmake --build build-linux -j8 --target \
  nof_linux_initiator_test nvmeof_initiator_test \
  nof_heartbeat_test nof_linux_initiator

ctest --test-dir build-linux --output-on-failure --timeout 120 \
  -R '^(nof_linux_initiator_test|nvmeof_initiator_test|nof_heartbeat_test)$'

sudo BUILD_DIR="$PWD/build-linux" \
  bash mooncake-store/tests/e2e/run_nof_linux_initiator_e2e.sh

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

Unit tests cover endpoint parsing, device discovery, NoF workers, and heartbeat handling. The E2E test verifies real writes, reads, and payload consistency against a temporary NVMe/TCP target.

Additional network fault testing on Linux 6.17.1, with unlimited reconnection, a 10-second KATO, and fast_io_fail_tmo=30, verified that:

• Outstanding I/O returns an error during the outage.
• After reconnection, the original handle resumes reads and writes, and the heartbeat probe API recovers.
• A test process with outstanding I/O exits normally while the network remains unavailable.
• Temporary targets, connections, and fault-injection resources are cleaned up afterward.

Fault testing covers the Linux initiator and probe API, not the complete Store segment removal and re-registration workflow. Final-commit regression verification remains pending.

Performance testing

Validated the Linux initiator in a real NVMe/RDMA environment. The single-NS test accessed one namespace in one subsystem through a single 200 Gbit/s network port. The dual-NS test used two independent subsystems, each containing one namespace and accessed through a dedicated 200 Gbit/s network port, for an aggregate link bandwidth of 400 Gbit/s.

The logical I/O size was 1 MiB, with a 256 KiB submission chunk size and 256 KiB random LBA alignment. Single-NS random read/write throughput reached 22.77/22.68 GiB/s, while dual-NS aggregate throughput reached 44.81/45.05 GiB/s—approximately 98%–100% of the fio baseline measured in the same environment.

The tests used two additional benchmark-only changes that are not included in this PR:

• Initialize I/O buffers with random data once before timing, without refilling them on each write.
• Support configurable random LBA alignment, set to 256 KiB for these tests.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Codex assisted with implementation, testing, code review, and documentation.
The human submitter is responsible for understanding and defending all changes.