[Aionw]

Description

Part of RFC #4181 (thread dimension). This takes the "run every non-Ping handler on a worker pool" alternative from §5 of the RFC instead of a separate heartbeat server (#4183).

Problem. coro_rpc binds each connection to one io thread for its whole lifetime and runs a plain (non-coroutine) handler inline on that thread. A slow handler, such as GetReplicaListByRegex over millions of keys or an UnmountSegment that runs ClearInvalidHandles, therefore stops its io thread from reading any other connection bound to it. That includes the Ping of healthy clients, which the master then moves to SUSPECTED and eventually OFFLINE.

Change.

• New RpcWorkerPool (rpc_worker_pool.h). It registers a method as a coroutine handler under the same route key that coro_rpc derives from the method's name (coro_rpc::func_id<Method>()), so the wire format and the client side are unchanged. The coroutine only posts the call to an asio::thread_pool and awaits it, so the io thread goes back to reading right away. The pool is generic over the service type and does not depend on WrappedMasterService.
• RegisterRpcService takes an optional RpcWorkerPool*. With a pool, every handler except Ping and ServiceReady runs on its worker threads; those two stay on the io threads, which the pool now keeps free. Without one (embedded master, tests), every handler runs inline as before.
• New master flag and config key handler_worker_thread_num. 0 (the default) starts as many worker threads as rpc_thread_num, so the master runs data-plane handlers on worker threads in both HA and non-HA mode. The help text and the deployment guide now describe rpc_thread_num as the io thread count.
• HA mode: MasterServiceSupervisor starts a pool for each leader term. When the term's server stops, the pool finishes the calls it holds before the OpLog writer stops and leadership is released, matching what inline handlers guaranteed; it drains before the term's service is destroyed.

Compared with #4183, there is no second port and no client-side discovery. The cost is one thread hop per RPC (see the numbers below).

Behaviour changes worth reviewing

• Each data-plane RPC pays one hop to a worker thread and one back to the io thread for the response, about 4 µs at p50.
• coro_rpc keeps reading a connection while a coroutine handler is in flight, so pipelined requests on one connection can now run concurrently and complete out of order (responses are still matched by request ID). MasterClient pipelines on a connection only once a process has more than 160 requests in flight to the master (the ylt client_pool reuse limit). Below that, concurrent calls already use separate connections, which could be bound to different io threads and run concurrently, so handlers never had an ordering guarantee between concurrent calls.
• The default thread count doubles, from rpc_thread_num to 2 × rpc_thread_num.

Known limitation: the worker queue is unbounded and not observable. On main, a busy io thread stops reading its connections, so overload shows up as requests waiting in socket buffers and as late Pings. With the pool, the io threads keep accepting requests and queue them for the workers, so:

• Queued requests now wait in the master's memory instead of in the kernel. Every MasterClient call blocks its caller until the response arrives, so the queue depth stays bounded by the number of calls the clients have in flight; it does not grow on its own.
• There is no queue-depth or queue-wait metric yet, and Ping stays fast by design, so a saturated master can be told apart from a healthy one only by data-plane latency.

A bounded queue that rejects excess requests (for example with a busy error the client can retry), and metrics for queue depth and queue wait, are left for a follow-up.

Out of scope / follow-ups

• Lock dimension: Ping still waits on client_mutex_ behind exclusive holders such as ReMountSegment; that is [Store] Centralize client session lifecycle and resource ownership #4180.
• Worker queue bound and metrics, as above.

Benchmark. A throwaway benchmark (not included in this PR) started an in-process master with 2 io threads and N keys, then ran 4 clients calling GetReplicaListByRegex back to back with a pattern that matches nothing (a full key walk with a tiny response), 2 clients calling GetReplicaList back to back, and 1 client sending a Ping every 20 ms. Connections were opened one by one so that the Ping connection always shared an io thread with slow clients. 3 runs per row.

mode	keys	Ping p50	Ping max	GetReplicaList p50
inline (main)	1M	~0.49 s (one regex scan)
inline (main)	5M	~2.8 s
8 worker threads	1M	~0.025 ms	< 1 ms
8 worker threads	5M	~0.025 ms	~4 ms
2 worker threads (saturated by scans)	1M	unaffected		~0.5 s

On an idle master the hop costs about 4 µs per call (p50 of a point RPC 10 µs → 14 µs). With the pool saturated, Ping is still served promptly; data-plane calls wait in the worker queue instead.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

make -j14 mooncake_master rpc_worker_pool_test batch_replica_clear_tenant_rpc_test master_service_config_test high_availability_test master_bench
./mooncake-store/tests/rpc_worker_pool_test
./mooncake-store/tests/batch_replica_clear_tenant_rpc_test
./mooncake-store/tests/master_service_config_test
./mooncake-store/tests/high_availability_test
# smoke test of the real binary with the default worker pool
./mooncake-store/src/mooncake_master --rpc_port=$P --rpc_address=127.0.0.1 ...
./mooncake-store/benchmarks/master_bench --master_server=127.0.0.1:$P --operation=BatchPut --duration=5 ...
./mooncake-store/benchmarks/master_bench --master_server=127.0.0.1:$P --operation=BatchGet --duration=5 ...

Test results:

• Unit tests pass

• Integration tests pass (if applicable)

• Manual testing done (describe below)

• rpc_worker_pool_test (new, 2 cases): pooled handlers, including a zero-argument one, are reachable under their method names; and with a single io thread, a pooled handler that never returns leaves the io thread free to accept a new connection and answer an inline Ping.

• batch_replica_clear_tenant_rpc_test 1/1 and master_service_config_test 19/19 pass; a new case checks that handler_worker_thread_num reaches MasterServiceSupervisorConfig.

• high_availability_test passes locally (5 passed, 2 skipped), but this build has no etcd backend, so the case that runs the supervisor through a leader term (HaWithoutOplogRestoresEmptyContextAndServes) only runs in CI, where etcd is available.

• mooncake_master started with handler_worker_thread_num resolved to rpc_thread_num (16/16 in the log), served master_bench BatchPut and BatchGet without errors, and exited cleanly on SIGTERM.

• The changed translation units and the new test pass g++ 11.4 -fsyntax-only (Ubuntu 22.04, the CI compiler).

• The docs build with Sphinx, and the new flag row renders in the deployment guide.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Claude Code helped implement the worker pool, write the unit test and the throwaway benchmark, and draft this description. I have reviewed every changed line and can defend the change end to end.

[Icedcoco]

Thanks for adding this worker pool; moving the handlers off the RPC I/O threads looks useful for keeping Ping responsive when a handler is slow. I had two questions while reviewing the overload and configuration paths.

It looks like each request is submitted with coro_io::post, which queues work on the Asio thread pool. The worker count limits how many handlers run at once, but I couldn’t find a bound on how many requests can wait in the queue. If requests arrive faster than the workers can process them for a sustained period, queued tasks and the request state they retain could accumulate. Would it be worth adding a queue limit or another form of backpressure, with a clear overload response? I may be overlooking an existing admission control, so I’d be glad to hear how you expect this to behave under sustained load.

I also noticed that the CLI flag is a signed int32_t, while handler_worker_thread_num is stored as an unsigned value. A setting such as --handler_worker_thread_num=-1 therefore converts to a very large number. Asio rejects thread counts that large, and this construction path doesn’t appear to catch the exception, so the master would fail during startup. Could we validate that the value is non-negative and report a clear configuration error? The current 0 default behavior could remain unchanged.

[Aionw]

Thanks for the careful review!

Unbounded queue. You're right, and the PR body lists it as a known limitation, with the bound and metrics planned for a follow-up. One detail shapes what that bound can look like: coro_rpc starts a coroutine handler detached (directlyStart) and goes straight back to reading the connection, and ylt has no way to pause reads on a connection. Waiting inside the handler would only move the backlog from the asio queue to the waiters, so real TCP backpressure isn't available here. The bound has to be an admission limit that fails fast with a busy error the client can retry. That changes the client-visible error contract, so I'd rather do it in its own PR together with queue-depth and queue-wait metrics. In practice the depth is bounded today by in-flight client calls: MasterClient blocks each caller until the response arrives, and ylt's client_pool caps pipelining at 160 in-flight requests per connection.

Negative thread count. Good catch, confirmed. Fixed in 2894006: a gflags validator now rejects negative values at startup with a clear error, and 0 keeps its current meaning.

[Icedcoco]

LGTM pending CI.

One consideration for a follow-up: the worker pool’s queue is unbounded, so sustained overload could lead to growing memory usage. It may be worth documenting the expected overload behavior or tracking admission control separately.