[CAICAIIs]

Description

The MasterService half of the tenant-first metadata model (RFC #3670), on top of the layer that is already in main: per-object metadata and the object route from #4127 and #4161, the tenant directory, catalog and registry from #4215.

MasterService keeps one metadata::TenantRegistry in place of the 1024 metadata shards. A tenant's objects live on that tenant's own route, its groups on the tenant's group table, and its quota account is bound by the registry's factory, so an object is excluded only while it is itself being read, written or evicted, and two unrelated objects never contend.

This PR carries the model swap and the route striping that keeps the object path's lock granularity, and nothing else: the correctness and cost work the migration made reachable is listed under Follow-ups.

What the migration does

• Every object-level path resolves its tenant through the registry and runs its read-modify-write under the entry's own lock: put, upsert, remove, remove-all, the point and batch reads, eviction, offload, promotion, dynamic replication, client offboarding, the durable finalizers and the stale-handle sweeps. EraseMetadata is the single teardown point and claims the publication through the entry's own state, so a second eraser of the same object bails out instead of releasing its refcounts, quota charges and KV events twice.
• What stays on the service: the per-tenant replica-action records, the client leases and the promotion-candidate key index stay keyed by tenant id, because each belongs to the subsystem that keeps it and that subsystem validates what it holds against the entry the route publishes. Per-object task state lives on the entry, so the parallel per-key maps have no successor.
• The snapshot payload keeps its layout: one zstd-compressed entry per tenant holding objects with [tenant_id, key, metadata] items, and a reader still accepts the legacy two-element item. Decoding replaces what the routes held, so a key the payload does not carry cannot survive from the state the decode started in. The weight metadata object main added to the payload is written and validated here too.
• The object route is striped. A tenant's route was one map under one lock, and growing that map rehashes every node it holds under the exclusive lock, so every read and write of the tenant waited behind that walk. The route is 64 maps, each under its own lock, with a key always landing in the same stripe: every per-key operation still takes exactly one lock and sees the same map, growth is charged to one stripe's keys, and readers of different keys stop sharing a lock. A sweep of the route at 1, 4, 16, 64 and 256 stripes, at 1, 4, 16 and 32 writer threads, put the count at 64: one stripe reached 8.5M lookups and 250k publishes per second, 64 reached 79M and 6.7M, and 256 reached 114M and 9.1M for four times the memory, so each further split buys less than the memory it costs.
• The old shard containers, the four accessor classes and the group domain are gone. The metadata layer itself is untouched: metadata/tenant.h, metadata/tenant_registry.h, object_entry.h, object_index.h, group_index.h and dynamic_replication_lease_table.h keep the API this migration consumes, and the features main gained since the branch point (managed weight revisions, ProbeKey/BatchProbeKey, the DFS immutable bucket allocator) are integrated on the tenant model rather than dropped.

Module

• Mooncake Store (mooncake-store)

Type of Change

• Refactor
• Performance improvement

How Has This Been Tested?

Test commands:

cmake --build build -j$(nproc)
ctest -R "master_service|master_scenario|tenant|object_entry|object_index|group_index|dynamic_replication|promotion|eviction" -j8
./scripts/code_format.sh --changed-lines --check
pre-commit run --files $(git diff --name-only Origin/main..HEAD)

Test results:

• Unit tests pass

• Integration tests pass (if applicable)

• Manual testing done (describe below)

• Build: the tree builds with no errors; the 18 test targets that carry these paths link and run.

• Suites: 349 cases pass across master_service_test (65), master_service_tenant_quota_test (35), master_service_group_test (12), promotion_on_hit_test (55), master_service_evict_scenario_test (22), master_service_concurrent_scenario_test (10), master_service_group_scenario_test (13), master_service_tenant_scenario_test (10), master_service_offload_scenario_test (25), master_service_scenario_test (32), offload_on_evict_test (1), dynamic_replication_test (27), master_service_processing_key_double_erase_test (1), master_service_weight_management_test (12), metadata_tenant_test (12), object_index_test (6), group_index_test (7) and eviction_strategy_test (4). The three op-log cases of master_service_evict_scenario_test talk to an external etcd and flake against a shared instance: each of them passes on its own, and the same flake reproduces on main.

• Formatting: ./scripts/code_format.sh --changed-lines --check reports no violation with clang-format 20.1.8, and pre-commit run --files passes over the changed files.

• ThreadSanitizer: master_service_test (65), master_service_tenant_quota_test (35), promotion_on_hit_test (55), dynamic_replication_test (27), master_service_group_test (12) and master_service_evict_scenario_test (22) pass under -DENABLE_TSAN=ON with no sanitizer report and no lock-order inversion.

Performance. Synthetic master workload: 8 segments of 400 GiB that stay far below the eviction watermark, and the same 8 segments of 4 GiB held at a 0.5 watermark so the background evictor works throughout; 448 KiB values, group_size=5, fixed arrival, replica_num=1, preferred placement, 60 s per run, Release builds. The two versions run interleaved inside each round with the order flipped every round, six rounds per version and rate. Rates per segment: exist 6.3359, put 3.1431, get 2.0242 at the default, doubled at 2×.

workload	rate	version	p50 (ms)	p99 (ms)	p999 (ms)
8 × 400 GiB	default	main	0.454	3.264	8.055
8 × 400 GiB	default	this branch	0.471	3.316	7.720
8 × 400 GiB	2×	main	0.427	3.165	6.925
8 × 400 GiB	2×	this branch	0.474	3.155	9.637
8 × 4 GiB, watermark 0.5	default	main	0.435	3.153	6.860
8 × 4 GiB, watermark 0.5	default	this branch	0.434	3.011	5.231
8 × 4 GiB, watermark 0.5	2×	main	0.402	3.104	5.374
8 × 4 GiB, watermark 0.5	2×	this branch	0.400	2.882	3.551

• Means of the six rounds. Where the evictor runs throughout (the 8 × 4 GiB pool) the two versions are level on p50, within 0.002 ms, and this branch is lower on p99 and p999, by 0.14–0.22 ms and 1.6–1.8 ms. Where it stays idle (the 8 × 400 GiB pool) p50 is 3.7% and 11% higher on this branch, 0.017 and 0.047 ms, p99 moves by at most 0.05 ms, and p999 does not separate at all: per-round p999 spans 3.6–12.4 ms on this branch and 3.9–15.8 ms on main, so read it as neither a win nor a regression. The 8 × 4 GiB runs take the eviction path throughout, 8 passes per run at the default rate and 33–34 at 2×.
• Both versions complete the same work: completed_tasks = scheduled_tasks (5512 and 11032), identical committed_keys and retained_keys, no overload events, and refused puts within 0.6% in the 8 × 4 GiB runs (432 on both sides at the default rate, 1392 against 1400 at 2×).
• The read-path and batch-read cost work — resolving the read tenant once per chunk of a batch, moving the read response out of the access result, dropping the per-key owning identity — is listed under Follow-ups rather than here, and the p50 gap in the 8 × 400 GiB runs is the cost this branch carries until that work lands. The route striping is the one cost change this PR keeps, and the sweep behind its count is in the striped-route bullet above.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective (the existing suites are ported to the tenant model)
• For changes >500 LOC: I have filed an RFC issue (RFC [RFC]: Tenant-First Object-Level Metadata Locking #3670)

AI Assistance Disclosure

• AI tools were used (specify below)

AI assistance was used for parts of this migration and to draft this description. The submitter has reviewed every changed line and can defend the change end to end.

Follow-ups

Work the migration made reachable, left out of this PR. Each item needs its own measurement or test before it is worth doing, and none of them changes what this PR guarantees.

Correctness on paths the migration rewrote

• Bind every durable cleanup to the publication it was armed for. FinalizeRemovedReplicasAfterDurable resolves the object by key and then releases quota, emits the removal delta and tears the object down on whatever that key publishes when it runs, so a same-key recreate that landed while the cleanup was in flight can be charged for it. It should act only on the publication the cleanup was armed with, as FinalizeMetadataEraseAfterDurable and FinalizeExpiredProcessingReplicasAfterDurable already do.
• Settle expired dynamic replication by proposal identity. The expired-pending sweep records only the key and the entry it saw, so a proposal that took the key between the scan and the cleanup can have its task failed and its lease dropped by a sweep that never saw it. The sweep should record the proposal (proposal, lease and task ids, version epoch, the expiry it observed) and act only on the proposal it recorded.
• Drop a tenant's replica-action record once it holds nothing. The record now lives as long as the tenant, so a tenant that has gone quiet keeps an entry with no lease and no candidate.
• Tear an evicted group's members down in place. EvictGroupOrObject collects the members it invalidated and erases them in a second pass, so a replica another path adds to one of those entries in between goes down with it. The teardown belongs inside the member's own callback, which already holds that entry's lock.
• Bound one promotion retry round. A round walks the candidate index without a per-round bound, so one tenant with a large index can spend the round on itself.
• Do not register a tenant for a request that publishes nothing. PutStart resolves through get-or-create before it allocates, so a rejected or failed request leaves an empty tenant behind. It should resolve the tenant instead and let the tenant be created when the object is about to be published.
• Stop rebuilding group state in the snapshot decode. The decode wires each object's group membership and shared lease as it routes it, and then walks the tenants again to rebuild what it has already wired. The walk is what raises the group lease to the latest deadline among its members, so removing it needs the wiring to keep that guarantee.

Cost

• Let the stripe count follow a tenant's object count. 64 stripes is the measured ratio for a tenant of about thirty thousand objects; a tenant holding far more keys wants more stripes, and one holding fewer cannot use more than it has, so the count belongs to the tenant rather than to the index.
• The batch reads hold the snapshot lock longer than they need to. BatchGetReplicaList keeps the shared lock across the promotion and dynamic-replication candidate queues, so a reload waits for the whole batch; the lock only has to cover the metadata reads, as the single-key read path already does.
• The eviction walks copy every handle of the population. BatchEvict holds every tenant's route as one vector of handles for the whole census, and the tenant watermark pass does the same for one tenant; the second pass of BatchEvict materialises every candidate identity. A walk that takes one route stripe at a time and a bounded chunk of it holds a chunk instead. The deadline vectors the census accumulates (no_pin, soft pins, candidates) still grow with the population and need their own decision.
• Bound the decompressed size of one payload entry. The snapshot writes one zstd-compressed entry per tenant, so a tenant whose objects alone exceed the intended entry size puts them all in one entry. Spreading the tenants over a fixed number of entries bounded that, and the reader has to keep accepting the shape either way.
• Split replica_action_mutex_ into per-tenant state and locks. The one lock over the action records serializes promotion and dynamic-replication action paths across unrelated tenants. Judge a split on the p50, p99 and p999 of those paths, the wait on that lock and their throughput, with promotion and dynamic replication on and off, and with two tenants against many.
• Shorten how long the eviction passes hold snapshot_mutex_. BatchEvict and NoFBatchEvict take it shared for the whole census and scan, so a reload waits for a pass. Anything that replaces that has to keep what the lock buys today: no tenant is removed under a running scan.
• Revisit the two passes of ObjectIndex::SnapshotObjects(). It counts the stripes before it copies them, which is what keeps the handles in one allocation; one pass would trade that back for repeated growth. Worth changing only with a measurement that shows the count pass costs more.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.