[Juhyun-Kim-Memphis]

Description

Ping can be served by a separate RPC server whose only thread never runs a data-plane handler, and clients find it by themselves. This is the thread half of RFC #4181. The lock half is #4180, which removes client_mutex_ and the guards from the Ping path but does not touch RPC threading. Production code: +114/−4 in six files. Default off.

Depends on #4180. With a single heartbeat thread, any lock wait inside Ping stalls every heartbeat at once. Without the lock half, one healthy client went OFFLINE in the RFC's join-storm run. Do not enable --heartbeat_rpc_port on a master without #4180.

Why

coro_rpc gives each io thread its own io_context, binds an accepted connection to one of them for its lifetime, and runs non-coroutine handlers inline on the thread that read the request. A Ping is therefore read only by the io thread that owns its connection. If that one thread is inside a long handler, the Ping waits for it, even when every other io thread is idle, so raising rpc_thread_num does not remove the wait, and removing locks from Ping cannot help because nothing is waiting on a lock.

On main, UnmountSegment runs its O(keys) ClearInvalidHandles sweep inline when HA, snapshot or CXL is enabled. On a non-HA rig with --enable_snapshot, 18M keys and 12 concurrent unmounts, healthy clients' heartbeats stopped for 19.8 s on main and the master marked 6 of them SUSPECTED. Every Ping that took longer than 2 s completed within 8 ms of an UnmountSegment returning. With this PR the worst gap was 1.0 s and nothing was suspected. A run with only the lock half (#4182, now closed in favour of #4180) showed 29.1 s; the lock half does not act here, and the spread most likely comes from which io thread a heartbeat's connection happened to share with the deepest unmount queue. Each figure is a single run (RFC #4181 §4).

What changes

• Master. --heartbeat_rpc_port (default 0; non-HA mode) starts a second coro_rpc_server with one io thread that serves only Ping and ServiceReady. Each coro_rpc_server owns its own io_context_pool, so the two servers share no io thread. A new GetHeartbeatRpcPort RPC on the main server returns the port (0 when disabled). In HA mode the flag logs a warning and is ignored.
• Client. MasterClient::Connect asks for the port, probes ServiceReady on it, and then sends Ping there. The heartbeat address gets its own connection pool, so a Ping also no longer shares connections with the client's own data-plane RPCs. Other RPCs are unchanged.

Scope

• Non-HA only. In HA the remount after a promotion walks every metadata shard under exclusive snapshot_mutex_, which is where a heartbeat server would matter most; HA support (the supervisor's server lifecycle) is the follow-up listed in the RFC.
• RFC [RFC]: [Store] The client heartbeat path must share no lock or thread with data-plane operations #4181 §7 asks the maintainers whether they would rather run every non-Ping handler on a worker pool instead; this PR will follow that answer.

Rollout needs no coordination

The main server keeps serving Ping. A client that predates this change, a master that does not advertise a port, and an advertised server that does not answer all leave Ping on the main server. This is the design of #3541 on the P2P branch applied to main's Ping, with the port discovered from the master instead of configured on every client, which removes the client/master mismatch cases #3541 had to add error codes for. (A new client connecting to an older master logs one RPC call failed line per Connect for the unknown RPC, then behaves as before.) A deployment that remaps ports between the client and the master (for example a Kubernetes Service whose port differs from its targetPort) must expose the heartbeat port under the same number; otherwise clients stay on the main server.

Module

• Mooncake Store (mooncake-store)

Type of Change

• Bug fix
• New feature

How Has This Been Tested?

Integration test (new) heartbeat_rpc_server_test, with real servers and a real MasterClient:

• PingIsServedWhileTheMainIoThreadIsBusy: with the heartbeat server, Ping returns within 2 s while the main server's only io thread is held by a blocking handler.
• PingWaitsForTheMainIoThreadWithoutTheServer: the control. Without it, Ping does not return within 2 s.
• PingFallsBackToTheMainServerWhenUnreachable: an advertised port nobody listens on leaves Ping on the main server.
• PingWaitsForABusyIoThreadEvenWhenAnotherIsIdle: the main server has two io threads and one of them is held. Of two connections opened before the hold, the Ping on the one that shares the held thread does not return within 2 s, while the other does. Raising rpc_thread_num leaves the wait in place for the connections on a busy thread.

Results: the three original tests 3/3, repeated 5× with no flake; master_service_test 58/58 (Release, GCC 13, Ubuntu 24.04). The new test: a standalone copy with the same fixture and the pinned yalantinglibs passes 50/50, and the same assertion fails with one or three io threads, so it does check the placement. On the real target it passes in this PR's CI (CTest unit tests).

Rig. Measurements for both dimensions, alone and together, are in RFC #4181 §4.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh (clang-format 20, changed lines)
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue ([RFC]: [Store] The client heartbeat path must share no lock or thread with data-plane operations #4181; this PR is well under 500 LOC)

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Claude (Anthropic) helped analyze the threading, write the change and its tests, and run the verification. The submitter reviewed every changed line.

[Juhyun-Kim-Memphis]

Closing as superseded by #4601, which landed on 10-16 and takes the worker-pool alternative from §5 of RFC #4181: every non-Ping handler now runs on a worker thread, so a slow request no longer delays Ping, without a second port or client-side discovery. That covers the thread half of the RFC, so this PR isn't needed upstream. If a gap remains that only a dedicated heartbeat server can close, we'll come back with a new PR measured against current main.