[Yong-ee]

Description

Backport HA RPC recovery fixes needed by the v0.3.13.post1_l4 deployment branch.

This change is based on upstream PRs kvcache-ai#3743 and kvcache-ai#3943, adapted without importing unrelated OpLog, snapshot, or DFS changes.

The changes:

• Separate foreground Master RPCs from HA control and ServiceReady probe pools.
• Use one bounded Master connection policy in both HA and non-HA modes: no pool-level connection retries or reconnect wait, with a 1-second connection timeout. The outer heartbeat/reconnect loops own recovery.
• Wait for HA clients to observe a ready Master view before connecting.
• Publish an Etcd leader address only after the Master RPC service is ready, using a warming value and lease-preserving compare-and-put.
• Protect in-flight Master RPCs during client teardown with a drain guard.
• Keep retired RPC pools alive process-wide and distinguish pools by behavioral configuration, preventing reconnect coroutines from accessing freed pool state.
• Keep the request timeout at coro_io's 30-second default unless MC_RPC_TIMEOUT_MS is explicitly set; connection and retry overrides remain available through the existing environment variables.

References: kvcache-ai#3743, kvcache-ai#3943.

Module

• Mooncake Store (mooncake-store)
• Common (mooncake-common)
• Transfer Engine (mooncake-transfer-engine)
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement

How Has This Been Tested?

Checks completed:

git diff --check
cd mooncake-common/etcd && go test ./...

The Etcd wrapper Go tests passed ([no test files]). The RPC timeout test expectations cover the bounded HA/non-HA connection policy and the 30-second default request timeout. A usable CMake build directory was not available in the development environment, so C++ compilation and integration tests were not run locally.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (not applicable)
• I have added tests to prove changes are effective
• For changes >500 LOC: I have filed an RFC issue (not applicable; 532 additions / 44 deletions)

AI Assistance Disclosure

• No AI tools were used
• AI tools were used.

The changes were prepared with AI assistance. The human submitter is responsible for reviewing every changed line and defending the implementation end-to-end.