Skip to content

[Store] Fix snapshot test eviction race - #3813

Merged
Aionw merged 1 commit into
kvcache-ai:mainfrom
Icedcoco:fix/snapshot-eviction-test-race
Sep 1, 2026
Merged

Aionw merged 1 commit into
kvcache-ai:mainfrom
Icedcoco:fix/snapshot-eviction-test-race

Conversation

@Icedcoco

@Icedcoco Icedcoco commented Sep 1, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Stop the MasterService eviction worker before snapshot/restore validation in
the shared snapshot test fixture, and serialize manual snapshot writes under
snapshot_mutex_. This prevents background eviction from changing replica
metadata while the test compares the original and restored services.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

ctest --test-dir /tmp/mooncake-build-fix \
  -R '^master_service_ssd_test_for_snapshot$' \
  --output-on-failure --repeat until-fail:2

/tmp/mooncake-build-fix/mooncake-store/tests/master_service_ssd_test_for_snapshot \
  --gtest_filter='MasterServiceSSDSnapshotTest.EvictObject'

ctest --test-dir /tmp/mooncake-build-fix \
  -R '^(master_service_test_for_snapshot|master_service_promotion_test_for_snapshot)$' \
  --output-on-failure

pre-commit run --files \
  mooncake-store/tests/ha/snapshot/master_service_test_for_snapshot_base.h

Test results:

  • Unit tests pass
  • Integration tests pass (not applicable)
  • Manual testing done (not applicable)

The SSD snapshot target passed twice, EvictObject passed ten consecutive
runs, and the other snapshot fixtures passed. Pre-commit passed all applicable
hooks.

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (not applicable)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed an RFC issue (not applicable)

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (analysis and implementation assistance). The human
    submitter must review and defend every changed line end-to-end.

@Aionw Aionw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Aionw
Aionw merged commit 8b8ec70 into kvcache-ai:main Sep 1, 2026
23 checks passed
huangxiaobai-ydmy pushed a commit to huangxiaobai-ydmy/Mooncake that referenced this pull request Sep 9, 2026
EvictObject TearDown failed because the restored MasterService kept
BatchEvict running above the watermark before snapshot comparison.
Disable eviction on the restore config and join its worker, extending kvcache-ai#3813.

Co-authored-by: Cursor <cursoragent@cursor.com>
huangxiaobai-ydmy pushed a commit to huangxiaobai-ydmy/Mooncake that referenced this pull request Sep 10, 2026
EvictObject TearDown failed because the restored MasterService kept
BatchEvict running above the watermark before snapshot comparison.
Disable eviction on the restore config and join its worker, extending kvcache-ai#3813.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants