[Store] Remove legacy master snapshots by Icedcoco · Pull Request #4406 · kvcache-ai/Mooncake

Icedcoco · 2026-09-30T02:05:36Z

Description

Remove the legacy Master-generated catalog snapshot and restore path now that HA recovery uses standby-generated batch OpLog snapshots.

This removes the legacy snapshot manager, repository, codec, catalog backends, provider, configuration flags, and legacy-only tests. The existing Master state coordination lock is retained under the clearer master_state_mutex_ name, and batch OpLog snapshot, promotion, pruning, and rebootstrap paths remain available.

Module

Mooncake Store

Type of change

Refactor / cleanup
HA behavior change

Testing

cmake --build build -j32 --target mooncake_master master_service_ha_test
MasterServiceHATest.*Batch: 30/30 passed
pre-commit run --files $(git diff --name-only --diff-filter=ACMR)

Checklist

Removed legacy Master snapshot generation and restore code
Removed legacy catalog backends and configuration
Preserved batch OpLog snapshot configuration and recovery paths
Updated documentation with removal-in-progress notices
Reviewed the full diff for unrelated changes

AI assistance disclosure

This change was prepared with AI assistance. The human submitter must review every changed line and be able to defend the implementation end to end.

The rebase onto main removed the LegacyRestoreContext phase declarations from master_service.h together with the legacy snapshot helpers, but their definitions in master_service.cpp come from kvcache-ai#3806 and back the hot-standby promotion restore path, so master_service.cpp no longer compiled. Restore the declarations, and drop a duplicate blank line in master_service_ha_test.cpp that fails the format check.

Removing hot_standby_snapshot_bootstrap_test.cpp also dropped every StandbyControllerTest case, including the Redis promotion gate that kvcache-ai#4295 added. Move the cases that do not depend on the legacy catalog snapshot path into standby_controller_test.cpp, and point the real-etcd snapshot smoke back at the lifecycle case. Cases that configure the removed enable_snapshot_restore and snapshot_catalog_store_type options stay deleted.

The removed PromoteStandbyReturnsStartFailure and PromoteStandbyAndExport_ReturnsErrorWhenNotStarted cases forced a start failure through the deleted snapshot catalog options, which also dropped coverage of PromoteStandbyAndExport's guard that the supervisor relies on. Force the start failure with an OpLog backend that has no endpoint instead, and check that both promotion APIs reject a never-started standby and report the start error.

Aionw

LGTM

Drop the snapshot-only bootstrap hunk and its test, removed upstream by kvcache-ai#4406. Also set the applied-sequence gauge on the preserved promotion cursor restart path added by kvcache-ai#4559, which installs a baseline without going through the applier.

First slice of RFC kvcache-ai#4700 (Rethinking the Master Global State Lock). PR kvcache-ai#4406 removed the legacy Master snapshot path but kept snapshot_mutex_ (renamed master_state_mutex_) and its ~69 guard sites, many of which now protect lifecycle interactions that merely accumulated around the original barrier. This slice adds the explicit Master lifecycle proposed in RFC Section 7.1 without changing any locking behavior: - RestoreLifecycleState {kConstructed, kRecovering, kReady, kServing, kFailed} with transitions wired where they exist today: the HA promotion restore moves Constructed -> Recovering -> {Ready, Failed}; the supervisor's ActivateServingState takes Ready -> Serving; the non-HA path opens Constructed -> Ready -> Serving explicitly. - RestoreFromStandbyState becomes a lifecycle wrapper around the existing implementation (renamed RestoreFromStandbyStateCore). Entry from Serving is refused per the RFC's proposed no-cycles-out-of-Serving constraint; the existing restore retry semantics (replacement restore from Ready or Failed, exercised by the HA tests) are preserved. - AdmitsBusinessWorkers() pauses the business background workers (eviction, client monitor, task cleanup, job dispatch, NoF heartbeat, dynamic replication, replica cleanup) while a promotion restore is in flight, independent of master_state_mutex_. kConstructed keeps workers admitted so tests and embedded users keep the constructor-starts-workers behavior; tightening that is left to the follow-up slices. master_state_mutex_ stays exactly where it was; this slice changes no locking behavior. The follow-up slices (remount prepare/rebind/publish, endpoint readiness ownership, LOCAL_DISK registration admission) build on this state machine before the global lock is removed from ordinary paths. RFC: kvcache-ai#4700

Resolve the conflicts with 225 commits of main and port the code main added against SegmentManager onto SegmentPool. - Legacy master snapshots are gone (#4406): drop the branch's edits to the deleted snapshot codec, catalog-backed provider, snapshot manager, RestoreState/ApplySnapshotState and MetadataSerializer, and their tests. snapshot_mutex_ is now master_state_mutex_. - Client liveness bindings lost their atomics (#4516). A memory buffer still reads its owner through its SegmentLifetime, whose owner is rebound on remount and stays atomic; only the local-disk binding becomes a plain field. - Dynamic replication moved into a controller (#4589). DynamicReplicationController::SelectPlan now takes a list of SegmentCandidate values (segment, allocatable, used, capacity) that MasterService builds from the pool, so the controller depends on neither segment manager. - The admin segment-status API (#4343) and the drain-job registration lock use SegmentPool write access; CreateDrainJob validates targets before taking job_mutex_. - Object length comes from ObjectMetadata::TotalLength() (#4615). - The HTTP metadata prefix comes from MasterMetadataConfig.

…emoval kvcache-ai#4406 removed the Master-generated snapshot and restore path, but left behind the code that only that path called. None of it has a caller outside its own tests: - serialize/serializer.{h,cpp}: the msgpack Serializer<T> specializations for allocators, buffers, replicas and mounted segments, SerializationHelper and SerializationError. - ha/snapshot: StoreResourceSnapshotCodec, AllocatorSnapshotCodec and LocalSsdCodec. - SegmentSerializer and SegmentView in segment.h, TaskManagerSerializer with the restore_task/clear_all helpers only it used, and LocalSsdManager::ExportPersistedState/RestorePersistedState with LocalSsdPersistedState. - The detached snapshot path added by kvcache-ai#3706: SegmentPool::CaptureSnapshot and RestoreSnapshot, OffsetBufferAllocator and OffsetAllocator CaptureSnapshot/Restore with their snapshot structs and layout validation, AdoptMode::ReplacePool and IsKnownSegmentStatus. SegmentPool::WriteAccess::PrepareAdopt stays in its pre-kvcache-ai#3706 form: the read-lock test still uses it to install a blocking allocator. The OffsetBufferAllocator node cap goes back to the literal 64Mi it had before kvcache-ai#3706 named it OffsetAllocatorSnapshot::kMaxNodes.

…emoval kvcache-ai#4406 removed the Master-generated snapshot and restore path, but left behind the code that only that path called. None of it has a caller outside its own tests: - serialize/serializer.{h,cpp}: the msgpack Serializer<T> specializations for allocators, buffers, replicas and mounted segments, SerializationHelper and SerializationError. - ha/snapshot: StoreResourceSnapshotCodec, AllocatorSnapshotCodec and LocalSsdCodec. - SegmentSerializer and SegmentView in segment.h, TaskManagerSerializer with the restore_task/clear_all helpers only it used, and LocalSsdManager::ExportPersistedState/RestorePersistedState with LocalSsdPersistedState. - The detached snapshot path added by kvcache-ai#3706: SegmentPool::CaptureSnapshot and RestoreSnapshot, OffsetBufferAllocator and OffsetAllocator CaptureSnapshot/Restore with their snapshot structs and layout validation, AdoptMode::ReplacePool and IsKnownSegmentStatus. SegmentPool::WriteAccess::PrepareAdopt stays in its pre-kvcache-ai#3706 form: the read-lock test still uses it to install a blocking allocator. The OffsetBufferAllocator node cap goes back to the literal 64Mi it had before kvcache-ai#3706 named it OffsetAllocatorSnapshot::kMaxNodes. DestroyedPoolAndRetainedAllocatorRemoveMetricLabels stays, rewritten to mount directly instead of restoring a snapshot. kvcache-ai#3706 also made buffer allocator destructors remove their segment's metric labels and made remove_segment_metrics keep a label that still has usage; both are on the ordinary mount path and lost their only coverage with the snapshot tests.

First slice of RFC kvcache-ai#4700 (Rethinking the Master Global State Lock). PR kvcache-ai#4406 removed the legacy Master snapshot path but kept snapshot_mutex_ (renamed master_state_mutex_) and its ~69 guard sites, many of which now protect lifecycle interactions that merely accumulated around the original barrier. This slice adds the explicit Master lifecycle proposed in RFC Section 7.1 without changing any locking behavior: - RestoreLifecycleState {kConstructed, kRecovering, kReady, kServing, kFailed} with transitions wired where they exist today: the HA promotion restore moves Constructed -> Recovering -> {Ready, Failed}; the supervisor's ActivateServingState takes Ready -> Serving; the non-HA path opens Constructed -> Ready -> Serving explicitly. - RestoreFromStandbyState becomes a lifecycle wrapper around the existing implementation (renamed RestoreFromStandbyStateCore). Entry from Serving is refused per the RFC's proposed no-cycles-out-of-Serving constraint; the existing restore retry semantics (replacement restore from Ready or Failed, exercised by the HA tests) are preserved. - AdmitsBusinessWorkers() pauses the business background workers (eviction, client monitor, task cleanup, job dispatch, NoF heartbeat, dynamic replication, replica cleanup) while a promotion restore is in flight, independent of master_state_mutex_. kConstructed keeps workers admitted so tests and embedded users keep the constructor-starts-workers behavior; tightening that is left to the follow-up slices. master_state_mutex_ stays exactly where it was; this slice changes no locking behavior. The follow-up slices (remount prepare/rebind/publish, endpoint readiness ownership, LOCAL_DISK registration admission) build on this state machine before the global lock is removed from ordinary paths. RFC: kvcache-ai#4700

…tore it A standalone master loses its whole index on every restart, planned or not: the legacy snapshot is gone (kvcache-ai#4406) and the HA OpLog is the only other way to keep it. For a planned restart the old master can hand its index over instead, once nothing changes it any more (QuiesceBackgroundWork): a snapshot taken then cannot point at memory that is reused after it, which is what made periodic snapshots unsafe (kvcache-ai#3771). This adds the snapshot and its restore, not yet wired into the master binary. Format. The snapshot is a batch OpLog snapshot (object chunks, segment records, weight metadata and the manifest indexing them, written with the existing codecs under the root "handoff") plus a handoff manifest and a quarantine file. The handoff manifest vouches for the batch manifest by size and CRC, carries what a remount compares that a segment record does not (protocol, host id, status), object and replica counts, the next replica id and the struct_pack fingerprints of the writer; it is written last. Each segment's identity (segment id, base, owner) is in its StandbySegmentInfo record. Schema 1; a reader accepts only the schema and codec layout it was built with. Export (WriteHandoffSnapshot). Metadata shards are exported and written in parallel, each file written as it is made, with the writeback started at once and finished while the next chunk is encoded, so the writer holds almost no extra memory and leaves no dirty page cache. Exported are the completed memory replicas on OK, DRAINING and DRAINED segments. Puts still in flight and incomplete replicas are left out and their ranges are quarantined for the full put_start_release_timeout; ranges the master still holds back for discarded replicas, and ranges a previous restore still quarantines, are carried with the time they had left. Only memory replicas are carried: LOCAL_DISK, DISK, NoF and DFS replicas are left out, so an object with a memory and a LOCAL_DISK replica comes back with its memory replica only, and an object whose only replica is on disk is not exported (its store registers its disk tier again with the new master, as after a cold start). Restore (RestoreFromHandoff), into an empty master before it serves: the restored segments are mounted with their identity and owner and an empty real offset allocator, closed for reads (read gate) and for allocation. The chunks are read and installed in parallel with placeholder buffers (the helpers of the parallel promotion restore), then each segment's allocator is rebuilt from the restored and quarantined ranges and every placeholder is swapped for its imported buffer. Restored objects keep the read-lease floor of the old master (export time + kv lease TTL); expired leases are not cleaned up. Any error fails the restore; the caller then discards the master and starts from an empty one (cold start), so no partial index is ever served. Remount. A restored segment opens only when its owner remounts it with exactly the restored identity; then the owner's objects are readable and the segment allocatable, without a scan of the index. A mount or remount that proves a restored segment stale (another client presents its endpoint, the owner presents another id or range at it, or no longer lists it) unmounts it, which makes all of its objects unreadable at once; the cleanup worker erases them. A restored segment that is unmounted (e.g. its owner never came back) is forgotten by the task cleanup thread. Nothing changes the status of a restored segment before its owner confirmed it. Allocator: ImportOffsetBufferAllocations imports live allocations into an existing allocator and hands out each buffer as it is imported; ImportOffsetBufferAllocator is built on it.

ReRouteRestoredObjectsByKey and RebuildGroupState ran only after a master snapshot was loaded, which kvcache-ai#4406 removed. Since then only tests call them: standby restore files each object under its own tenant and key and wires its group as it publishes it. Remove both, with their tests and test hooks, and correct the design doc, which still described the master-snapshot behavior.

First slice of RFC kvcache-ai#4700 (Rethinking the Master Global State Lock). PR kvcache-ai#4406 removed the legacy Master snapshot path but kept snapshot_mutex_ (renamed master_state_mutex_) and its ~69 guard sites, many of which now protect lifecycle interactions that merely accumulated around the original barrier. This slice adds the explicit Master lifecycle proposed in RFC Section 7.1 without changing any locking behavior: - RestoreLifecycleState {kConstructed, kRecovering, kReady, kServing, kFailed} with transitions wired where they exist today: the HA promotion restore moves Constructed -> Recovering -> {Ready, Failed}; the supervisor's ActivateServingState takes Ready -> Serving; the non-HA path opens Constructed -> Ready -> Serving explicitly. - RestoreFromStandbyState becomes a lifecycle wrapper around the existing implementation (renamed RestoreFromStandbyStateCore). Entry from Serving is refused per the RFC's proposed no-cycles-out-of-Serving constraint; the existing restore retry semantics (replacement restore from Ready or Failed, exercised by the HA tests) are preserved. - AdmitsBusinessWorkers() pauses the business background workers (eviction, client monitor, task cleanup, job dispatch, NoF heartbeat, dynamic replication, replica cleanup) while a promotion restore is in flight, independent of master_state_mutex_. kConstructed keeps workers admitted so tests and embedded users keep the constructor-starts-workers behavior; tightening that is left to the follow-up slices. master_state_mutex_ stays exactly where it was; this slice changes no locking behavior. The follow-up slices (remount prepare/rebind/publish, endpoint readiness ownership, LOCAL_DISK registration admission) build on this state machine before the global lock is removed from ordinary paths. RFC: kvcache-ai#4700

…tore it A standalone master loses its whole index on every restart, planned or not: the legacy snapshot is gone (kvcache-ai#4406) and the HA OpLog is the only other way to keep it. For a planned restart the old master can hand its index over instead, once nothing changes it any more (QuiesceBackgroundWork): a snapshot taken then cannot point at memory that is reused after it, which is what made periodic snapshots unsafe (kvcache-ai#3771). This adds the snapshot and its restore, not yet wired into the master binary. Format. The snapshot is a batch OpLog snapshot (object chunks, segment records, weight metadata and the manifest indexing them, written with the existing codecs under the root "handoff") plus a handoff manifest and a quarantine file. The handoff manifest vouches for the batch manifest by size and CRC, carries what a remount compares that a segment record does not (protocol, host id, status), object and replica counts, the next replica id and the struct_pack fingerprints of the writer; it is written last. Each segment's identity (segment id, base, owner) is in its StandbySegmentInfo record. Schema 1; a reader accepts only the schema and codec layout it was built with. Export (WriteHandoffSnapshot). Metadata shards are exported and written in parallel, each file written as it is made, with the writeback started at once and finished while the next chunk is encoded, so the writer holds almost no extra memory and leaves no dirty page cache. Exported are the completed memory replicas on OK, DRAINING and DRAINED segments. Puts still in flight and incomplete replicas are left out and their ranges are quarantined for the full put_start_release_timeout; ranges the master still holds back for discarded replicas, and ranges a previous restore still quarantines, are carried with the time they had left. Only memory replicas are carried: LOCAL_DISK, DISK, NoF and DFS replicas are left out, so an object with a memory and a LOCAL_DISK replica comes back with its memory replica only, and an object whose only replica is on disk is not exported (its store registers its disk tier again with the new master, as after a cold start). Restore (RestoreFromHandoff), into an empty master before it serves: the restored segments are mounted with their identity and owner and an empty real offset allocator, closed for reads (read gate) and for allocation. The chunks are read and installed in parallel with placeholder buffers (the helpers of the parallel promotion restore), then each segment's allocator is rebuilt from the restored and quarantined ranges and every placeholder is swapped for its imported buffer. Restored objects keep the read-lease floor of the old master (export time + kv lease TTL); expired leases are not cleaned up. Any error fails the restore; the caller then discards the master and starts from an empty one (cold start), so no partial index is ever served. Remount. A restored segment opens only when its owner remounts it with exactly the restored identity; then the owner's objects are readable and the segment allocatable, without a scan of the index. A mount or remount that proves a restored segment stale (another client presents its endpoint, the owner presents another id or range at it, or no longer lists it) unmounts it, which makes all of its objects unreadable at once; the cleanup worker erases them. A restored segment that is unmounted (e.g. its owner never came back) is forgotten by the task cleanup thread. Nothing changes the status of a restored segment before its owner confirmed it. Allocator: ImportOffsetBufferAllocations imports live allocations into an existing allocator and hands out each buffer as it is imported; ImportOffsetBufferAllocator is built on it.

…tore it A standalone master loses its whole index on every restart, planned or not: the legacy snapshot is gone (kvcache-ai#4406) and the HA OpLog is the only other way to keep it. For a planned restart the old master can hand its index over instead, once nothing changes it any more (QuiesceBackgroundWork): a snapshot taken then cannot point at memory that is reused after it, which is what made periodic snapshots unsafe (kvcache-ai#3771). This adds the snapshot and its restore, not yet wired into the master binary. Format. The snapshot is a batch OpLog snapshot (object chunks, segment records, weight metadata and the manifest indexing them, written with the existing codecs under the root "handoff") plus a handoff manifest and a quarantine file. The handoff manifest vouches for the batch manifest by size and CRC, carries what a remount compares that a segment record does not (protocol, host id, status, NUMA/HCA locality), object and replica counts, the next replica id and the struct_pack fingerprints of the writer; it is written last. Each segment's identity (segment id, base, owner) is in its StandbySegmentInfo record. Schema 1; a reader accepts only the schema and codec layout it was built with. Export (WriteHandoffSnapshot). Metadata shards are exported and written in parallel, each file written as it is made, with the writeback started at once and finished while the next chunk is encoded, so the writer holds almost no extra memory and leaves no dirty page cache. Exported are the completed memory replicas on OK, DRAINING and DRAINED segments. Puts still in flight and incomplete replicas are left out and their ranges are quarantined for the full put_start_release_timeout; ranges the master still holds back for discarded replicas, and ranges a previous restore still quarantines, are carried with the time they had left. Only memory replicas are carried: LOCAL_DISK, DISK, NoF and DFS replicas are left out, so an object with a memory and a LOCAL_DISK replica comes back with its memory replica only, and an object whose only replica is on disk is not exported (its store registers its disk tier again with the new master, as after a cold start). Restore (RestoreFromHandoff), into an empty master before it serves: the restored segments are mounted with their identity and owner and an empty real offset allocator, closed for reads (read gate) and for allocation. The chunks are read and installed in parallel with placeholder buffers (the helpers of the parallel promotion restore), then each segment's allocator is rebuilt from the restored and quarantined ranges and every placeholder is swapped for its imported buffer. Restored objects keep the read-lease floor of the old master (export time + kv lease TTL); expired leases are not cleaned up. Any error fails the restore; the caller then discards the master and starts from an empty one (cold start), so no partial index is ever served. Remount. A restored segment opens only when its owner remounts it with exactly the restored identity; then the owner's objects are readable and the segment allocatable, without a scan of the index. A mount or remount that proves a restored segment stale (another client presents its endpoint, the owner presents another id or range at it, or no longer lists it) unmounts it, which makes all of its objects unreadable at once; the cleanup worker erases them. A restored segment that is unmounted (e.g. its owner never came back) is forgotten by the task cleanup thread. Nothing changes the status of a restored segment before its owner confirmed it. Allocator: ImportOffsetBufferAllocations imports live allocations into an existing allocator and hands out each buffer as it is imported; ImportOffsetBufferAllocator is built on it.

Every object now lives in its tenant's namespace (metadata::NamespaceTable, one route per tenant) instead of the 1024 metadata shards. The primitives the previous steps moved MasterService onto keep their shape and change what is behind them: - MetadataAccessorRW and MetadataAccessorRO hold the key's own lock through a route guard instead of the key's shard lock. A plain or ObjectRef accessor holds the key only while it holds an object; a CreateKey accessor holds it either way, so the caller can publish. - The tenant cursors give way to the routes' own cursors: a walk over every tenant is VisitNamespaces followed by ns.objects.ReadCursor(), and the in-flight sweep reads the namespace's in-flight list. - Group membership and the shared group lease live in each route's group index, which Publish and TearDown keep in step, so GroupDomain and the register/unregister calls go. MetadataShard, its accessors, GroupDomain, TenantState and the cursor slices are deleted. So is the per-shard disk object counter, which nothing read since kvcache-ai#2286, and the test-only ClearCandidatesForReload with the tracker's Reset, both orphaned by kvcache-ai#4406. A tenant's quota account is looked up per request instead of being cached on its TenantState; the quota cutover replaces it with the namespace policy. ClearStaleHandles and BatchEvict keep the bucket shrink of kvcache-ai#4103 and whose maps have grown sparse, and ShrinkBucketsIfSparse moves to common/shrink_buckets.h. Until the eviction cutover, BatchEvict's census runs one namespace per thread rather than slices of one namespace. Tests that reached into the shards hold keys through the route instead, and the ones that pinned shard routing or slices are dropped. Co-authored-by: CAICAIIs <3360776475@qq.com>

First slice of RFC kvcache-ai#4700 (Rethinking the Master Global State Lock). PR kvcache-ai#4406 removed the legacy Master snapshot path but kept snapshot_mutex_ (renamed master_state_mutex_) and its ~69 guard sites, many of which now protect lifecycle interactions that merely accumulated around the original barrier. This slice adds the explicit Master lifecycle proposed in RFC Section 7.1 without changing any locking behavior: - RestoreLifecycleState {kConstructed, kRecovering, kReady, kServing, kFailed} with transitions wired where they exist today: the HA promotion restore moves Constructed -> Recovering -> {Ready, Failed}; the supervisor's ActivateServingState takes Ready -> Serving; the non-HA path opens Constructed -> Ready -> Serving explicitly. - RestoreFromStandbyState becomes a lifecycle wrapper around the existing implementation (renamed RestoreFromStandbyStateCore). Entry from Serving is refused per the RFC's proposed no-cycles-out-of-Serving constraint; the existing restore retry semantics (replacement restore from Ready or Failed, exercised by the HA tests) are preserved. - AdmitsBusinessWorkers() pauses the business background workers (eviction, client monitor, task cleanup, job dispatch, NoF heartbeat, dynamic replication, replica cleanup) while a promotion restore is in flight, independent of master_state_mutex_. kConstructed keeps workers admitted so tests and embedded users keep the constructor-starts-workers behavior; tightening that is left to the follow-up slices. master_state_mutex_ stays exactly where it was; this slice changes no locking behavior. The follow-up slices (remount prepare/rebind/publish, endpoint readiness ownership, LOCAL_DISK registration admission) build on this state machine before the global lock is removed from ordinary paths. RFC: kvcache-ai#4700

…the tenant quota policy (#4887) * [Store] Serve MasterService metadata from the namespace table Every object now lives in its tenant's namespace (metadata::NamespaceTable, one route per tenant) instead of the 1024 metadata shards. The primitives the previous steps moved MasterService onto keep their shape and change what is behind them: - MetadataAccessorRW and MetadataAccessorRO hold the key's own lock through a route guard instead of the key's shard lock. A plain or ObjectRef accessor holds the key only while it holds an object; a CreateKey accessor holds it either way, so the caller can publish. - The tenant cursors give way to the routes' own cursors: a walk over every tenant is VisitNamespaces followed by ns.objects.ReadCursor(), and the in-flight sweep reads the namespace's in-flight list. - Group membership and the shared group lease live in each route's group index, which Publish and TearDown keep in step, so GroupDomain and the register/unregister calls go. MetadataShard, its accessors, GroupDomain, TenantState and the cursor slices are deleted. So is the per-shard disk object counter, which nothing read since #2286, and the test-only ClearCandidatesForReload with the tracker's Reset, both orphaned by #4406. A tenant's quota account is looked up per request instead of being cached on its TenantState; the quota cutover replaces it with the namespace policy. ClearStaleHandles and BatchEvict keep the bucket shrink of #4103 and whose maps have grown sparse, and ShrinkBucketsIfSparse moves to common/shrink_buckets.h. Until the eviction cutover, BatchEvict's census runs one namespace per thread rather than slices of one namespace. Tests that reached into the shards hold keys through the route instead, and the ones that pinned shard routing or slices are dropped. Co-authored-by: CAICAIIs <3360776475@qq.com> * [Store] Run memory and quota eviction through the Evictor BatchEvict and EvictTenantMemoryForQuota hand the choice of what to evict to eviction::Evictor, which walks each namespace's lease index, oldest lease first, and holds one key at a time. What used to be the census, the reserve frontier, the two passes and EvictGroupOrObject goes: a group comes out of the index as one entry, and each namespace gives up its own share of the target. MasterService::MemoryEviction holds what the evictor is built from: - the policy: completed memory replicas nobody holds; a soft-pinned object is deferred to make up the minimum when soft pins may be evicted, and skipped otherwise; - the reclaimer: on the spot (cache, quota and dynamic-replication bookkeeping as before), through the OpLog under HA, or behind the offloading reclaimer with offload-on-evict; - the observers: metrics, the per-run log, and the KV removal event when the OpLog is off. BatchEvict keeps its housekeeping (expired writes, deferred releases), the route shrink after a round that evicted and the allocation-pressure signal. Quota eviction now also goes through the OpLog under HA, which it bypassed before. NoF eviction keeps its own pass, since NoF replicas are not in the lease index. The eviction index ranks leases in 100 ms buckets, so the group test puts its ungrouped objects a bucket after the group instead of a nanosecond. * [Store] Attach tenant quotas to the namespaces as their policy Multi-tenancy is now TenantQuotaPolicy, the NamespacePolicy every namespace is created under; with multi-tenancy off there is no policy at all. MasterService no longer books quota itself: - Admission and growth: a write asks the policy before it allocates (MetadataAccessorRW::Grow, at the five places memory grows: PutStart, UpsertStart, CopyStart, MoveStart and PromotionAllocStart), and a retained-client AddReplica holds the policy's admission token for the whole request instead of taking the manager's policy lock by hand. - Cost: every write lock released on a key settles what the account holds for it towards what the object owes now (TenantQuotaPolicy::MemoryCharge, plus a replaced object's charge while the upsert that replaced it is still being written). UpsertStart tells the policy before it replaces an object; EraseMetadata tells it before a teardown. - Background: restores, capacity changes and each eviction-thread round go to the policy (OnRestored, OnCapacityChanged, OnMaintenance), which owns the per-tenant watermark eviction and asks for it through StoreControl. EvictTenantMemoryForQuota becomes EvictNamespaceMemory. That covers what TenantQuotaLedger tracked as pending, committed and replaced bytes, so the ledger, ObjectMetadata::quota_ledger, the pending_quota_charge_bytes task fields, the charge/release/settle helpers and UpsertStart's replacement-charge transfer with its rollback paths are gone. QuotaEraseMode, now only about KV events, becomes EraseMode. The RPC layer asks the policy directly for the quota admin calls (tenant_quota_policy()); MasterService keeps IsTenantQuotaEnabled() for request tenant resolution. Error codes are unchanged. * [Store] Copy an eviction bucket's keys into one buffer per cursor A lease cursor copied each bucket it read into one Entry per key, each with its own vector and its own key string, under the index lock. With every eviction going through the index, that was most of a round's time: with a million leases in one 100 ms bucket (a bulk load or a restore), 72% of a BatchEvict that evicted a tenth of them went to those two allocations per key, for keys the round never reached as much as for those it did. The cursor now copies a bucket's keys back to back into one buffer it reuses (LeaseEvictionIndex::Copy), and Entry::keys views it: the keys stay valid until the cursor is asked for the next entry or destroyed, which is how the evictor uses them. batch_evict_bench, 1M objects, target 0.1: 0.44 s -> 0.21 s with every lease in one bucket (main 0.16 s), 0.13 s -> 0.10 s with 10k keys per bucket (main 0.17 s). * [Store] Keep pending session tags and soft-pin actions off the object ObjectMetadata held its pending session tag and pending soft-pin action inline, 96 and 80 bytes, though each exists only between a write's start and its end. They are now HeapOptional, a pointer while absent, which takes 158 bytes off every object (1M objects: 1245 -> 1087 bytes allocated per object). * [Store] Clean up after the cutover Two fixes: - A rejected PutStart or UpsertStart counted toward mooncake_tenant_quota_reject_total twice: TenantQuotaPolicy::OnGrow counts every rejection, and the two writes still counted it again. - The route shrink after an eviction round skipped a namespace the round emptied, since VisitNamespaces only visits namespaces that hold an object. That namespace has the most buckets to give back; the shrink now visits every namespace. And what the cutover left behind: - TenantHasObjects, which lost its last caller with the ledger; the accessor's friend for a rollback that no longer reaches its guard; a friend naming a class that does not exist; the <bitset> includes the shard bitsets needed. - The erase mode and media hint of the durable finalizers, which every caller passed as kFull and nothing. - MasterService members for the offload cap ratio and force-evict flag, read only to build the eviction settings. - The cursor's Clear and Seal around ReadBucket move into ReadBucket, so the copy is sealed by the call that fills it. - One comment on Grow instead of the same one at five call sites; comments, log lines and test names that still talked about metadata shards or the ledger. - Tests: the test peer's ledger-era wrappers (tests call the policy hooks directly), the helpers that searched for group ids hashing to another metadata shard, and QueueLimitRejectsCrossShard, which became the same test as QueueLimitRejectsBeyondCap once there were no shards. * [Store] Refuse unregistered tenants before taking the key, and scope eviction upkeep - PutStart and UpsertStart took the key, and so created the tenant's namespace, before Grow(0) refused a tenant that is not registered. A namespace stays once created, so every write from an unknown tenant id left one behind, with its 1024 stripes, for every later eviction round to walk. PrepareWriteStart now refuses the tenant first; Grow(0) under the key keeps the last word. - The route shrink after eviction took every stripe lock of every namespace whenever a round released anything, and released replicas alone, with the key kept, were enough. The evictor now shrinks the route of a namespace only once its part of the run dropped an object. An object dropped through the OpLog goes once its record is durable, so a later run's shrink gives its key back, as on main. - A BatchEvict round that only released expired discarded replicas counted as a failure: MetricsObserver saw the evictor's report and nothing else. The pool-wide success and failure counters move back to BatchEvict, which counts that release as progress; the observer keeps the per-tenant bytes. - The namespace policy was declared after the namespaces and so destroyed before them, against the Namespace contract that the policy outlives it. It is now declared first. * [Store] Refuse to delete a tenant while a write to it is in flight - A PutStart with NoF replicas only charges the tenant nothing, and its object counts only once it is published. DeletePolicy looked at those two and nothing else, so it could delete the tenant while such a write was allocating, and the write then published under a tenant that no longer had a policy. On main the metadata scan waited on the key the write held. - The tenant's account now counts the writes in flight: the first growth asked under a write lock begins one, and releasing that lock ends it once the charge is settled and the object, if any, is published. Disabling the policy is refused while one is in flight. The writer and the disable each write one word and then read the other's, sequentially consistent, so a racing pair cannot both miss. - The key marks the begun write in the top bit of the word that holds its charge, which the charge never reaches. - The USE_NOF test now expects DeletePolicy to refuse at once rather than wait for the write. * [Store] Fall back to a group's deferred members together The fallback came back to each deferred object on its own and asked whether the minimum was met between them, so it could stop after one member of a soft-pinned group and keep the rest, which readers of the group miss on anyway. It now asks only between entries, as main's second pass did between groups, at the cost of evicting more soft-pinned members than the minimum strictly needs. * [Store] Shrink a route stripe as its slots go The route shrink ran after an eviction pass and after ClearStaleHandles, so an object dropped through the OpLog, which goes only once its record is durable, was erased after that pass had already looked. A namespace left idle then kept its stripes' high-water buckets until some later pass happened to shrink it. The route now shrinks a stripe as it drops a slot: SlotHandle::reset already holds the stripe lock exclusively to erase it, and ShrinkBucketsIfSparse costs a bucket count check unless the map has grown sparse. Every erase gives its buckets back on the spot, whichever path did it, so ObjectRoute::ShrinkSparseStripes and the passes that called it are gone. --------- Co-authored-by: CAICAIIs <3360776475@qq.com>

First slice of RFC kvcache-ai#4700 (Rethinking the Master Global State Lock). PR kvcache-ai#4406 removed the legacy Master snapshot path but kept snapshot_mutex_ (renamed master_state_mutex_) and its ~69 guard sites, many of which now protect lifecycle interactions that merely accumulated around the original barrier. This slice adds the explicit Master lifecycle proposed in RFC Section 7.1 without changing any locking behavior: - RestoreLifecycleState {kConstructed, kRecovering, kReady, kServing, kFailed} with transitions wired where they exist today: the HA promotion restore moves Constructed -> Recovering -> {Ready, Failed}; the supervisor's ActivateServingState takes Ready -> Serving; the non-HA path opens Constructed -> Ready -> Serving explicitly. - RestoreFromStandbyState becomes a lifecycle wrapper around the existing implementation (renamed RestoreFromStandbyStateCore). Entry from Serving is refused per the RFC's proposed no-cycles-out-of-Serving constraint; the existing restore retry semantics (replacement restore from Ready or Failed, exercised by the HA tests) are preserved. - AdmitsBusinessWorkers() pauses the business background workers (eviction, client monitor, task cleanup, job dispatch, NoF heartbeat, dynamic replication, replica cleanup) while a promotion restore is in flight, independent of master_state_mutex_. kConstructed keeps workers admitted so tests and embedded users keep the constructor-starts-workers behavior; tightening that is left to the follow-up slices. master_state_mutex_ stays exactly where it was; this slice changes no locking behavior. The follow-up slices (remount prepare/rebind/publish, endpoint readiness ownership, LOCAL_DISK registration admission) build on this state machine before the global lock is removed from ordinary paths. RFC: kvcache-ai#4700

github-actions Bot added documentation Improvements or additions to documentation run-ci Store labels Sep 30, 2026

Icedcoco marked this pull request as ready for review October 12, 2026 07:03

Icedcoco requested review from 00fish0, Aionw, Libotry, ShangmingCai, UNIDY2002, XucSh, YiXR, alogfans, staryxchen, stmatengss and ykwd as code owners October 12, 2026 07:03

Yuchen Kou added 7 commits October 12, 2026 15:12

[Store] Remove legacy master snapshots

820ecf7

[Store] Fix OpLog snapshot export test size

d8073db

[Store] Remove stale legacy snapshot smoke dependency

e42694c

[Store] Remove stale legacy snapshot references

94c226c

[Store] Update snapshot architecture documentation

978f0f5

[Store] Clarify batch snapshot authority

db6890c

[Store] Remove stale legacy snapshot references after rebase

2676194

Icedcoco force-pushed the retire_legacy_snapshot branch from 0a50d6f to 2676194 Compare October 12, 2026 07:18

Aionw added 3 commits October 12, 2026 15:28

Aionw approved these changes Oct 12, 2026

View reviewed changes

Yuchen Kou added 2 commits October 13, 2026 11:18

[Store] Inject standby startup failures in controller tests

b2454d7

[Store] Format standby controller factory call

db96964

ykwd approved these changes Oct 13, 2026

View reviewed changes

Aionw mentioned this pull request Oct 16, 2026

[Store] Add the namespace layer over the object route #4721

Merged

10 of 13 tasks

pjdurden mentioned this pull request Oct 16, 2026

[Store] Sync HA applied sequence metric with recovered standby baseline #4402

Open

SuperGoodGame mentioned this pull request Oct 17, 2026

[Store] Persist and restore DrainJobs in master snapshots #4481

Open

15 of 32 tasks

kunkunblueberry mentioned this pull request Oct 17, 2026

[Store] Add restore lifecycle state machine and worker admission #4736

Merged

7 of 36 tasks

Aionw mentioned this pull request Oct 19, 2026

[Store] Remove snapshot serializers orphaned by the legacy snapshot removal #4774

Merged

8 of 14 tasks

SuperGoodGame mentioned this pull request Oct 20, 2026

[Store] Recover drain jobs through batch OpLog #4794

Open

10 tasks done

Juhyun-Kim-Memphis mentioned this pull request Oct 20, 2026

[INF-522] Carry upstream #4183: separate heartbeat RPC server (default off) vessl-ai/Mooncake#21

Merged

Aionw mentioned this pull request Oct 22, 2026

[Store] Move reads, offboarding and standby restore onto per-key accessors and tenant cursors #4844

Merged

10 of 12 tasks

Juhyun-Kim-Memphis mentioned this pull request Oct 22, 2026

[Bug]: [Store] Standalone master spends 20–48 s destroying its index after SIGTERM (12M–25M objects) #4856

Open

1 task done

ShangmingCai mentioned this pull request Oct 23, 2026

Bump version to 0.3.14 in pyproject.toml #4880

Open

8 of 36 tasks

Aionw mentioned this pull request Oct 23, 2026

[Store] Cut MasterService over to tenant namespaces, the Evictor and the tenant quota policy #4887

Merged

12 of 36 tasks

Navigation Menu

Repository navigation

[Store] Remove legacy master snapshots - #4406

[Store] Remove legacy master snapshots#4406
ykwd merged 12 commits into
kvcache-ai:mainfrom
Icedcoco:retire_legacy_snapshot

Icedcoco commented Sep 30, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Aionw left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Icedcoco commented Sep 30, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Module

Type of change

Testing

Checklist

AI assistance disclosure

Uh oh!

Aionw left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Icedcoco commented Sep 30, 2026 •

edited by github-actions Bot

Loading