[Icedcoco]

Description

When the ordered OpLog writer temporarily runs out of admission slots, memory eviction discards its scan progress and starts another full metadata scan on the next attempt. With millions of objects, this makes reclamation spend substantial time rediscovering candidates while submitting relatively few eviction operations. This PR preserves eviction progress across attempts and fixes two additional problems identified while validating that implementation: quadratic deadline bookkeeping and failure to resume after a transient writer error.

Background and original symptoms

MasterService::BatchEvict first performs a metadata census: it scans the object population, determines eviction eligibility, and collects timestamps used to select candidates. It then uses the existing nth_element selection to choose a candidate frontier and submits eviction records to the ordered OpLog.

Each submission needs an admission slot. Previously, when reservation returned TASK_PENDING_LIMIT_EXCEEDED (-1401), eviction stopped and its local candidate state was destroyed. After the eviction thread's existing 10 ms sleep, the next attempt repeated the census and candidate selection.

In the original large-object-count reproduction, approximately 9.46 million objects were present. A typical attempt evicted only about 1,025 objects, or 0.0108% of the population, against a target near 10%. Across 1,984 completed attempts, every attempt encountered reservation backpressure; the median eviction count was 1,025 and the average attempt duration was approximately 1.72 seconds. The count was not a fixed eviction limit: progress depended on when writer slots became available and when the next reservation failed.

This creates a feedback loop under memory pressure: slow reclamation causes allocation failures and retries, while repeated scans delay the eviction records needed to free space. Shortening the 10 ms interval does not remove the repeated scan cost.

Initial implementation: preserve an eviction episode

The first part of this change introduces an in-memory continuation for OpLog-backed memory eviction. An episode retains:

• The original target, lower bound, census base, and cumulative eviction accounting.
• Copied candidate identities, including shard, tenant, key, deadline, and object generation, together with the next candidate index.
• The current phase, second-pass cutoff, next shard, pending candidates for that shard, and remaining target.
• A copied group-member snapshot and the first member that has not completed, allowing a partially submitted group to resume without resubmitting earlier members.

Here, a deadline is the timestamp used by the existing lease/pin eligibility and eviction-cutoff logic. The census separates objects with and without soft-pin protection so that the fallback pass can retain the existing policy when selecting additional candidates.

When admission is full, eviction returns with its cursor on the unsubmitted candidate or group member. A later tick resumes that episode. Repeated ticks while blocked do not rebuild the full census. The existing 10 ms cadence and approximate selection order are preserved.

Because metadata may change during a pause, candidates carry an in-memory object generation. New insertions and in-place republication receive fresh generations. Before processing a saved candidate, eviction looks it up again and checks its generation, current lease, pin state, and eligible replicas. A stale candidate cannot evict a replacement object that happens to use the same key.

The continuation stores values and cursors; it retains no metadata iterator, shard lock, or OpLog reservation across ticks. It is discarded on restore, loss of the owning leadership context, shutdown, writer replacement, or an unrecoverable error. Generations and continuation state do not change the persisted snapshot or OpLog format.

Follow-up findings and supplementary fixes

1. Remove per-object searches and erases from the deadline census

The initial continuation implementation preserved scan progress, but after each successful primary submission it searched the census deadline vector and erased a matching element. This happened in both the initial and resumed paths. A vector erase shifts the remaining elements, and the preceding search is also linear. For N census entries and E submissions, this added approximately O(N × E) bookkeeping, even when eviction never paused.

A comparison workload exposed the resulting performance problem. With the same 16-worker, 1 GiB synthetic-segment workload, the observed rates were approximately:

Mode	Successful PutEnd objects/s	Durable PUT_END entries/s
HA and OpLog disabled	89,900	Not applicable
HA enabled, OpLog disabled	96,000	Not applicable
HA and OpLog enabled	2,520	2,520
HA, OpLog, and snapshots enabled	2,550	2,550

With OpLog enabled and snapshots disabled, increasing segment capacity from 256 MiB to 1 GiB reduced durable PUT_END throughput from approximately 8,587 to 2,523 entries/s. Profiling identified the eviction call chain through vector::erase into memmove; copying accounted for approximately 89.7% of sampled eviction-thread stacks in the larger case. Completed eviction episodes took approximately 62 seconds at the median.

These observations identify a substantial algorithmic hotspot. They do not establish that every part of the throughput gap comes from that hotspot. The comparison exercised real metadata processing and OpLog persistence with synthetic segments; it did not measure payload-transfer throughput. Successful responses in modes without an OpLog are also not evidence of durability.

The supplementary fix replaces incremental vector maintenance with deferred accounting:

• Each successful primary result appends only its trigger deadline and newly submitted object count. A resumed group records only the additional members submitted on that attempt.
• If the eviction lower bound has already been met, the episode finishes without compacting the census.
• If a second pass is needed, a shared PrepareEvictionSecondPass helper aggregates submitted counts by native clock ticks and compacts the no-soft-pin deadline census once.
• Compaction preserves duplicate-deadline multiplicity and submillisecond precision. It removes at most the available matching entries, so a missing match cannot remove an unrelated deadline.
• Existing nth_element cutoff selection remains in place. Once the cutoff is selected, both deadline vectors and the submission vector release their allocated capacity.
• Candidate refill uses its known vector offset instead of searching for each candidate again.

For this bookkeeping, expected total work becomes O(E) when no second pass is needed, or O(N + E) when it is. Both initial and resumed paths use the same preparation helper.

2. Allow automatic recovery after a transient writer retry

The initial implementation also treated IsAccepting() == false as a permanent writer failure. However, the writer temporarily stops accepting new reservations while retrying a recoverable backend error and restores admission when that retry succeeds.

If eviction ran during that interval, it could set the writer-replacement latch and clear its continuation. The writer would subsequently recover, but eviction would keep returning because the latch remained set. Fault injection reproduced this state: admission had recovered and there was no terminal error, yet repeated eviction calls performed no census until a new writer was installed.

The supplementary fix exposes a mutex-protected writer admission snapshot that distinguishes accepting, retrying, stopped, terminal, and invalid states. Eviction now:

• Preserves its continuation and any unconsumed allocation-pressure signal while the writer is retrying.
• Treats reservation rejection caused by temporary unavailability as resumable, including the race where the writer recovers before its state is queried.
• Propagates the pause result explicitly through group processing.
• Keeps replacement requirements for stopped, terminal, and invalid writers.
• Distinguishes reservation backpressure from an actual commit rejection. A raw PERSISTENT_FAIL code alone does not permanently latch writer replacement.

Final behavior and durability

Regression tests verify that admission backpressure preserves the episode, repeated blocked attempts avoid another census, and available capacity advances the saved candidate or group-member cursor. Transient backend retries recover automatically with the same writer. Stale or newly protected candidates are skipped after revalidation.

The existing durable-before-finalize contract is preserved: eviction reserves admission before marking replicas for removal, submits the existing REMOVE or PUT_END record, and performs final cleanup and accounting through the durable callback. Commit rejection rolls back removal marks. Pausing does not bypass persistence or hold a reservation while waiting.

Additional counters expose census runs, continuation creation and resumes, backpressure pauses, drops by reason, and stale-candidate skips. Existing eviction outcome accounting is settled once per episode rather than once per paused tick.

Suggested Merge Priority

• P2 — Normal: Correctness and performance fixes for OpLog-backed memory eviction.

Module

• Mooncake Store (mooncake-store)

Type of Change

• Bug fix
• Performance improvement

How Has This Been Tested?

The affected targets were built serially. The completed regression runs recorded 250 passing tests:

Test suite	Passing tests
master_service_ha_test	150
ordered_oplog_writer_test	54
master_metrics_test	19
master_service_test, filtered to eviction and soft-pin tests	14
master_service_group_test	13

Test commands, from the directory containing the test executables:

./master_service_ha_test
./ordered_oplog_writer_test
./master_metrics_test
./master_service_test --gtest_filter='*Evict*:*SoftPin*'
./master_service_group_test

Coverage includes repeated queue saturation without another census; primary, second-pass, and group cursors; candidate replacement and concurrent upsert; soft-pin selection; durable cleanup and commit rollback; restore and shutdown; fencing and terminal writers; and retry recovery before a census, during a scan, and with an existing continuation.

Deadline-specific regressions cover duplicate timestamps, partial group progress, submillisecond differences, missing census matches, skipping unnecessary second-pass preparation, and releasing vector capacity.

The full HA run exposed two test-isolation races involving background eviction/reaping. The tests now stop those workers when driving synchronous state transitions and explicitly age metadata instead of relying on a sleep. Both affected tests passed ten repeated runs after the correction.

An optimized local microbenchmark compared the old bookkeeping with the new submission-recording and second-pass preparation logic. Each case used 4,096 submissions, identical inputs, and five trials; the reported values are medians. Old and new cutoffs were checked for equality.

Census deadlines	Old find/erase	New recording + preparation	Speedup
100,000	51.23 ms	1.53 ms	~34×
500,000	511.58 ms	7.84 ms	~65×
1,000,000	1,192.87 ms	15.33 ms	~78×

This measures the local deadline algorithm, not cluster throughput. The complete distributed comparison matrix has not been rerun with the supplementary fixes, so this PR does not claim a measured end-to-end throughput recovery. The separate HA-002 successful-response/durable-admission discrepancy remains outside this change; durable PUT_END and REMOVE must continue to be measured separately during performance acceptance.

Test results:

• The regression suites above passed.
• Service/writer fault-injection tests verify automatic recovery after retryable errors.
• Local deadline microbenchmark completed with matching cutoff results.
• Scoped pre-commit checks passed on the final files.
• Distributed performance matrix rerun after the supplementary fixes.

Checklist

• Human submitter has reviewed every changed line and can explain the change end-to-end.
• Changed C/C++ lines were checked using ./scripts/code_format.sh --staged through pre-commit during implementation.
• Pre-commit was run on the files changed by this PR.
• Regression tests were added for the changed behavior.
• Documentation: design and investigation notes remain local; this PR contains code and tests only.
• Public issue/RFC reference supplied where required by the contribution guidelines; problem above is the investigation identifier.

AI Assistance Disclosure

• AI tools were used.

OpenAI Codex assisted with root-cause analysis, implementation, regression tests, local performance analysis, code review, and this PR description. The human submitter is responsible for reviewing and understanding all submitted changes.