Skip to content

[Bugfix][Store] Exit standalone master without destroying the object index - #4865

Open
huangxiaobai-ydmy wants to merge 1 commit into
kvcache-ai:mainfrom
huangxiaobai-ydmy:fix/issue-4856-standalone-fast-exit
Open

huangxiaobai-ydmy wants to merge 1 commit into
kvcache-ai:mainfrom
huangxiaobai-ydmy:fix/issue-4856-standalone-fast-exit

Conversation

@huangxiaobai-ydmy

Copy link
Copy Markdown
Contributor

Description

Fixes #4856.

Standalone (non-HA) mooncake_master previously returned from main() after SIGTERM/SIGINT, so destructors walked metadata_shards_ object-by-object. At 12M–25M objects that took ~20–48 s after the RPC server had already stopped, delaying Kubernetes rollouts; the index is not reused across a restart after the legacy snapshot removal.

This change matches the HA shutdown path since #4745:

  1. Install the shared shutdown signal handlers and watchdog on the non-HA path.
  2. On a requested shutdown: stop the RPC server, admin server, RPC worker pool, and HTTP metadata server.
  3. Call ExitAfterShutdown(0) so the object index is reclaimed by the kernel without a user-space walk.

Standalone has no etcd lease to revoke; the HTTP metadata server is stopped explicitly before _exit.

Suggested Merge Priority

  • P0 — Urgent
  • P1 — High
  • P2 — Normal
  • P3 — Low

Module

  • Mooncake Store (mooncake-store)

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

# Rely on CI Build & Test (Linux). Full multi-million-object SIGTERM timing
# needs the reporter's load setup from #4856 and was not re-run locally
# (Windows host without Mooncake toolchain).

Test results:

  • Unit tests pass
  • Integration tests pass (if applicable)
  • Manual testing done (code review against HA ExitAfterShutdown path; destructor audit for non-HA external effects)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (if applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specify below)

Cursor AI assisted with issue analysis against #4856 / HA #4745, implementing the non-HA ExitAfterShutdown path, and PR drafting. The submitter reviewed the changes.

…index

On SIGTERM, skip walking metadata_shards_ via ExitAfterShutdown after
stopping RPC, admin, worker pool, and HTTP metadata (kvcache-ai#4856).

(Rebased onto main after kvcache-ai#5030; packaging commits dropped.)
@huangxiaobai-ydmy
huangxiaobai-ydmy force-pushed the fix/issue-4856-standalone-fast-exit branch from 4b7cb16 to 29bc657 Compare October 28, 2026 01:37
@github-actions github-actions Bot added size/S 0-99 changed lines (additions + deletions) Tests labels Oct 28, 2026
huangxiaobai-ydmy added a commit to huangxiaobai-ydmy/Mooncake that referenced this pull request Oct 28, 2026
@huangxiaobai-ydmy

Copy link
Copy Markdown
Contributor Author

Previous build-wheel failure was an apt Failed to update package lists flake (not packaging). Pushed an empty commit to retrigger CI.

@staryxchen

Copy link
Copy Markdown
Collaborator

Thanks for picking up the test-wheel-ubuntu failure as well.
The _structured_ranges.py staging belongs in a dedicated wheel PR so this one can stay scoped to the standalone-master SIGTERM path (#4856). We opened that as #5030: it only adds the module to MIGRATED_PYTHON_MODULES and the packaging asserts, without touching master.cpp or CUDA EP JIT packaging.
The current scripts/build_wheel.sh diff on this branch also drops CUDA_EP_JIT_DIR staging relative to main (#4581). After #5030 merges, please rebase onto main and drop the wheel/packaging commits (_structured_ranges staging plus the packaging-test edits) so this PR only contains the mooncake-store/src/master.cpp shutdown change.

@huangxiaobai-ydmy

Copy link
Copy Markdown
Contributor Author

Thanks @staryxchen — agreed. Will wait for #5030 to land, then rebase onto main and drop the wheel/packaging commits so this PR only keeps the master.cpp shutdown change.

@staryxchen

Copy link
Copy Markdown
Collaborator

#5030 is now on main (3fbd075). Please rebase this branch onto current main and drop the wheel/packaging commits (scripts/build_wheel.sh, python/tests/packaging/test_project.py, python/tests/packaging/test_installed_wheel.py) so the remaining diff is only the standalone-master SIGTERM change in mooncake-store/src/master.cpp.

@huangxiaobai-ydmy
huangxiaobai-ydmy force-pushed the fix/issue-4856-standalone-fast-exit branch from 59f34c0 to a8926f9 Compare October 28, 2026 11:48
@huangxiaobai-ydmy

Copy link
Copy Markdown
Contributor Author

Thanks @staryxchen — rebased onto main (3fbd075) and dropped the wheel/packaging commits. The remaining diff is only mooncake-store/src/master.cpp.

@huangxiaobai-ydmy

Copy link
Copy Markdown
Contributor Author

Friendly Monday ping for review when you have a moment — already rebased onto main after #5030 and trimmed to only the master.cpp shutdown change; CI is green.

@ykwd @stmatengss @YiXR @Aionw @XucSh — thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-ci size/S 0-99 changed lines (additions + deletions) Store Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: [Store] Standalone master spends 20–48 s destroying its index after SIGTERM (12M–25M objects)

2 participants