[SuperGoodGame]

Description

Failover currently loses drain jobs, client task history, and in-flight COPY/MOVE reservations. Recover them through the native batch OpLog materialized store and bounded v3 snapshot chunks, preserving task IDs, retry budgets, progress, segment maintenance state, source pins, pending quota, and existing reader leases on moved sources.

Supersedes #4481 after #4406 removed the legacy snapshot stack. Related RFC: #4431. The existing batch boundary, verified artifacts, fallback, GC/pruning, and move-only promotion handoff carry the new records; no separate job database or legacy snapshot implementation is added. Promotion also retains this state when periodic snapshots are disabled.

Ordinary metadata updates retain in-flight targets; completion, revoke, and timeout explicitly settle them. Copy/Move End keep their native visibility-before-durability contract, and source reclamation uses the native durable finalizer plus the actual outstanding reader lease. Recovery validates before publishing jobs/tasks. Unremounted segments retain exact owner/address identity and use the existing unmount/offboarding lifecycle.

Compatibility: v1/v2 snapshots remain readable. The new OpLog operations and v3 snapshots require upgraded HA master/standby consumers. MoveStart rejects an existing incomplete target with UNAVAILABLE_IN_CURRENT_STATUS; Client::Move surfaces that error, while restored in-flight operations retain their End/Revoke paths after remount.

Priority

• P2 — Normal

Module

• Mooncake Store (mooncake-store)

Type of Change

• New feature
• Bug fix

How Has This Been Tested?

18 test executables: 416 passed, 3 existing environment skips, 0 failures. This includes all 139 HA cases and the 15-case focused check of recovery plus existing completion/finalization contracts. Three skipped cases require etcd/verification integration support absent from the CPU build.

ctest --test-dir build --output-on-failure -R '^master_service_ha_test$'
ctest --test-dir build --output-on-failure -j2 -R '^master_service_(tenant_quota|task_scenario|replication_scenario|segment_scenario)_test$'
ctest --test-dir build --output-on-failure -j2 -R '^(task_manager_test|batch_oplog_snapshot_(types|codec|writer|provider|promotion|gc|rebootstrap)_test|batch_oplog_pruning_coordinator_test|oplog_(applier|batch_codec)_test|hot_standby_service_test|segment_test)$'

Regressions cover double promotion without duplicated progress, retry/cancel/missing history, LOCAL_DISK updates during COPY/MOVE, quota/pins after settlement, actual reader-lease retention, orphan metrics and exact remount, unremounted unmount/offboarding, chunk corruption/fallback, suffix replay, and GC/pruning.

CPU build, four compile jobs and one link job. No live etcd/RDMA cluster test.

Checklist

• Self-review and independent correctness/simplicity reviews; findings addressed
• Formatted changed C++ lines
• PR-scoped pre-commit passes
• Added recovery and failure-boundary regressions
• RFC for >500 LOC: [RFC]: [Store] Completing the cross-client move/migration control plane (DrainJob HA recovery, task lifecycle APIs, surface parity) #4431

AI Assistance Disclosure

• AI tools were used: Codex implemented the migration, tests, and independent reviews. The human submitter remains responsible for line-by-line review and the final change.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-20T06:18:35.725806Z	d13ba4a	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.