Repository navigation
[Bugfix][Store] Interrupt in-flight snapshot children during shutdown - #3942
Open
waizuichougou wants to merge 1 commit into
Open
waizuichougou wants to merge 1 commit into
waizuichougou wants to merge 1 commit into
Conversation
waizuichougou
requested review from
00fish0,
Aionw,
Icedcoco,
Libotry,
XucSh,
YiXR,
stmatengss and
ykwd
as code owners
September 8, 2026 06:53
waizuichougou
force-pushed
the
fix/snapshot-child-shutdown
branch
from
September 8, 2026 10:28
2bee3a3 to
922fabe
Compare
11 of 14 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
MasterSnapshotManager::Stop()stopped future snapshot scheduling, but it didnot interrupt a snapshot child that was already running. The manager destructor
then joined the snapshot thread while that thread remained in
WaitForSnapshotChild(). A standalone master shutting down during a slow orstalled periodic snapshot could therefore block until the configured child
timeout (five minutes by default).
This change makes the child wait responsive to
Stop()and bounds childtermination:
condition variable, preserving the normal polling interval while allowing
shutdown to wake it immediately;
waitpid()during the existing five-second SIGTERM grace period so aresponsive child is reaped promptly, then retain SIGKILL as the fallback;
standalone master installs handlers that only update parent-process state;
inheriting those handlers otherwise prevents the child from terminating on
SIGTERM.
Trigger
unavailable so the child remains in progress.
Before this change,
MasterService::~MasterService()calledStop()and laterjoined the snapshot thread, but the in-flight child wait continued until the
snapshot completed or
snapshot_child_timeout_secondselapsed. With thedefault configuration, that can delay shutdown by up to 300 seconds.
Behavior
The payload and publication order are unchanged: metadata, segments, task
state, and the manifest are uploaded before the snapshot descriptor/latest
marker is published. Cancelling before publication can leave unreferenced
partial objects, as the existing timeout path already can, but restore does not
select them. Cancelling after publication is safe because all snapshot payloads
and the manifest have already completed.
No steady-state transfer, allocation, snapshot serialization, or restore code
is changed. This path is used by standalone periodic snapshots; HA batch-OpLog
snapshot ownership is not changed.
Module
mooncake-transfer-engine)mooncake-store)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
The regression test forks a real child, runs the production wait path on a
separate thread, calls
Stop(), and verifies both bounded completion and thatthe child has already been reaped. The existing timeout test verifies the shared
termination helper, and the auto-snapshot test verifies that a normal child can
still serialize, publish, exit, and be reaped.
Test commands:
cmake --build build --target snapshot_child_process_test -j 4 build/mooncake-store/tests/snapshot_child_process_test build/mooncake-store/tests/snapshot_child_process_test \ --gtest_filter='SnapshotChildProcessTest.StopInterruptsRunningSnapshotChild' \ --gtest_repeat=20 --gtest_break_on_failure pre-commit run --files \ mooncake-store/include/master_snapshot_manager.h \ mooncake-store/src/master_snapshot_manager.cpp \ mooncake-store/tests/ha/snapshot/snapshot_child_process_test.cppTest results:
snapshot_child_process_testtests passed.with real fork, signals, and waitpid in the unit test).
grace period; the SIGKILL fallback was also exercised with a child that
ignored SIGTERM.
Checklist
./scripts/code_format.shconfiguration changes)
AI Assistance Disclosure
Codex assisted with this contribution.