[Aionw]

Description

Follow-up to #4887, fixing the two P2 findings from its review. Stacked on #4887: only the last two commits are new (d9c738601, 0f7ee57ec); I'll rebase onto main once #4887 merges.

1. Offload before evicting through the OpLog under HA. Memory eviction picked one reclaimer among three: the OpLog under HA, otherwise OffloadingReclaimer with offload_on_evict, otherwise immediate removal. With HA on, the OpLog won, so an object whose only copy was in memory was dropped without the LOCAL_DISK copy offload_on_evict asks for. PutEnd does not offload early in that mode either, so nothing ever reached disk. Memory-pressure eviction has behaved this way since #2826 (reported as #4830). #4887 put tenant-quota eviction on the same Evictor; on main that path offloaded but wrote no OpLog record.

Whether to offload first and how a replica comes off are separate choices. MemoryEviction now holds the removal (the OpLog under HA, immediate otherwise), and offload_on_evict wraps it in OffloadingReclaimer instead of replacing it. The managed-weight cold path reuses the same removal.

#4868 fixes the same bug in main's BatchEvict, which #4887 deletes; this is the equivalent on the Evictor.

2. Settle a torn-down object's quota charge at teardown. An UpsertStart that takes over an unfinished write with no completed replica erases the old object and allocates the replacement under the same key lock. The account still held the old write's reservation until the lock was released, so the replacement was charged on top of it: with a 100-byte quota, PutStart(K, 100) then UpsertStart(K, 100) failed with TENANT_QUOTA_EXCEEDED. A torn-down key owes nothing, so OnTearDown now settles the key's charge to zero on the spot. A size-changing upsert replaces rather than tears down, so it still carries the old charge until the replacement settles. The write stays begun until the lock is released, so tenant deletion is still held off.

Behavior change for HA deployments with offload_on_evict. Eviction now queues the offload and keeps the source replica until it lands; the memory is freed on a later round, through the OpLog. One round therefore frees less memory than before. Deployments that prefer freeing memory fast should enable offload_force_evict and size offloading_queue_limit / offload_cap_ratio. HA deployments without offload_on_evict are unaffected.

Known gap, not addressed here. When an eviction-queued offload completes, NotifyOffloadSuccess adds the LOCAL_DISK replica without an OpLog record. The standby learns of it only from the next eviction's PUT_END, which lists the remaining replicas. A failover in between loses only the standby's knowledge of the disk copy; the memory replica is still recorded.

Suggested Merge Priority

• P0 — Urgent: Critical security vulnerability, data loss/corruption, or widespread production outage requiring immediate action
• P1 — High: Significant correctness or stability issue in existing functionality, or a regression blocking users without a practical workaround
• P2 — Normal: Routine bug fix, validated performance/compatibility improvement, or new feature for a confirmed use case
• P3 — Low: Experimental feature, optional enhancement, refactoring, or cleanup with no urgent user impact

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

make tenant_quota_policy_test master_service_tenant_quota_test master_service_ha_test offload_on_evict_test \
  master_service_evict_scenario_test evictor_test metadata_namespace_test master_service_test
# each binary run in parallel

Test results:

• Unit tests pass: 335 tests across those 8 binaries. New tests:
  • MasterServiceHATest.OffloadOnEvictOffloadsBeforeEvictingThroughOpLog
  • MasterServiceTenantQuotaTest.UpsertTakingOverAnUnfinishedWriteIsNotChargedForIt (the reviewer's sequence)
  • TenantQuotaPolicyTest.ATornDownObjectIsNotChargedAgainstItsSuccessor
• Integration tests pass (if applicable)
• Manual testing done (describe below)

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Claude Code helped trace the eviction and quota paths, write the fix and tests. I reviewed every changed line.