[Store] Add synthetic master workload benchmark by zhangzuo21 · Pull Request #3147 · kvcache-ai/Mooncake

zhangzuo21 · 2026-07-28T02:53:21Z

Description

This PR adds master_synthetic_bench, a standalone benchmark that generates a configurable, stateful workload for an already-running Mooncake master.

The existing master_bench focuses on driving one selected operation at maximum throughput. The new benchmark is intended for mixed and rate-controlled workloads:

independent Exist, Put, and Get streams for each segment;
Poisson or fixed open-loop request arrivals;
configurable per-segment QPS and batch sizes;
stateful BatchPutStart / BatchPutEnd transactions;
monotonically increasing, tagged key namespaces;
configurable key size, value size, replica count, and lookup hit ratios;
per-segment worker pools and bounded pending queues;
segment mount, heartbeat, remount, and cleanup handling;
overload detection without silently dropping or rate-limiting requests.

Only keys that successfully complete both PutStart and PutEnd are added to the committed-key pool used by later Exist and Get requests.

The benchmark connects to the master through --master_server. It does not start or modify the master, does not require a trace file, and does not introduce additional master or sender metrics.

This PR contains only:

the new benchmark executable;
its English usage and parameter documentation; and
the minimal CMake target required to build it.

There are no changes to existing master behavior or production APIs.

Module

Type of Change

How Has This Been Tested?

Copilot

Pull request overview

This PR introduces a new standalone benchmark, master_synthetic_bench, intended to generate configurable mixed (Exist/Put/Get) open-loop workloads against an already-running Mooncake master, plus accompanying usage documentation and the minimal CMake target to build it.

Changes:

Add master_synthetic_bench C++ benchmark implementing per-segment open-loop scheduling, worker pools, segment mount/heartbeat/remount, and committed-key tracking for hit-ratio sampling.
Add English usage and parameter documentation for the new benchmark (MASTER_SYNTHETIC_BENCH.md).
Register the new benchmark executable in mooncake-store/benchmarks/CMakeLists.txt.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
mooncake-store/benchmarks/MASTER_SYNTHETIC_BENCH.md	Adds build/run instructions, workload semantics, parameters, and exit-code documentation for the new benchmark.
mooncake-store/benchmarks/master_synthetic_bench.cpp	Implements the synthetic benchmark: segment lifecycle + heartbeat, open-loop request scheduling, task queues, and RPC execution logic.
mooncake-store/benchmarks/CMakeLists.txt	Adds the `master_synthetic_bench` executable target and links it against `mooncake_store`.

Comments suppressed due to low confidence (1)

mooncake-store/benchmarks/master_synthetic_bench.cpp:412

BatchGetReplicaList failure accounting similarly ignores short responses. If fewer results are returned than requested keys, the benchmark should treat the missing entries as failures so rpc_failure_events reflects partial/bad responses.

            const auto counts =
                CountExpectedResults(master_client_.BatchGetReplicaList(keys));
            if (counts.second != 0) {
                stats_.rpc_failure_events.fetch_add(1,
                                                    std::memory_order_relaxed);

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+cmake -S . -B build \
+  -DCMAKE_BUILD_TYPE=Release \
+  -DBUILD_UNIT_TESTS=ON
+cmake --build build --target master_synthetic_bench -j


+        const long double nanoseconds = seconds * 1'000'000'000.0L;
+        return std::chrono::nanoseconds(
+            std::max<int64_t>(1, static_cast<int64_t>(nanoseconds)));
+    }


+            const auto results = master_client_.BatchExistKey(keys);
+            uint64_t failed = 0;
+            uint64_t found = 0;
+            uint64_t missing = 0;
+            for (const auto& result : results) {
+                if (!result.has_value()) {
+                    ++failed;
+                } else {
+                    if (*result) {
+                        ++found;
+                    } else {
+                        ++missing;
+                    }
+                }
+            }
+            stats_.exist_true_items.fetch_add(found, std::memory_order_relaxed);
+            stats_.exist_false_items.fetch_add(missing,
+                                               std::memory_order_relaxed);
+            if (failed != 0) {
+                stats_.rpc_failure_events.fetch_add(1,
+                                                    std::memory_order_relaxed);
+            }


codecov-commenter · 2026-07-28T03:49:36Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 0% with 507 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...oncake-store/benchmarks/master_synthetic_bench.cpp	0.00%	507 Missing ⚠️

📢 Thoughts on this report? Let us know!

…nch (PR kvcache-ai#3147) read_path_bench: in-process master metadata read-path microbenchmark used for the kvcache-ai#3670/kvcache-ai#3671 fast-path A/B (GetReplicaList throughput; 100k objects, 32 threads, 160k reqs -> +0.12% vs ca09c18 baseline). master_synthetic_bench (PR kvcache-ai#3147): configurable synthetic master workload benchmark against a live mooncake_master (mooncake::MasterClient, gRPC). Used to measure the eviction scenario (medium + high load). Adapted BatchPutEnd to the current vector<ObjectMeta> client API and added per-RPC latency percentiles (p50/p99/p999, us) for exist/get/put. Measured high-load eviction p99/p999 drop to ~1.5-7 ms after kvcache-ai#3671 route sharding (vs ~130-280 ms baseline).

[Store] Add synthetic master workload benchmark

28223ae

zhangzuo21 requested a review from ykwd as a code owner July 28, 2026 02:53

Copilot AI review requested due to automatic review settings July 28, 2026 02:53

zhangzuo21 requested review from XucSh, YiXR and stmatengss as code owners July 28, 2026 02:53

github-actions Bot added run-ci Store labels Jul 28, 2026

Copilot started reviewing on behalf of zhangzuo21 July 28, 2026 03:07

View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Aionw approved these changes Aug 10, 2026

View reviewed changes

This was referenced Aug 26, 2026

Expose production master capacity tuning Aionw/cakemaster#24

Merged

以 shared-nothing 分片 metadata plane，并评估 Compio thread-per-core runtime Aionw/cakemaster#25

Open

CAICAIIs mentioned this pull request Aug 27, 2026

[Store] Decouple object routing from grouped-object semantics #3589

Merged

8 of 18 tasks

CAICAIIs mentioned this pull request Sep 1, 2026

[Store] Tenant-first object-level metadata locking (flat group) #3822

Closed

9 of 28 tasks

This was referenced Sep 26, 2026

[RFC][Store] Offline timestamped master RPC trace replay #4337

Closed

[Store] Add offline master RPC trace replay benchmark #4338

Open

stmatengss mentioned this pull request Oct 13, 2026

[CI/Build] Gate CPU and T-One RDMA performance regressions #4639

Open

15 of 18 tasks

Navigation Menu

Repository navigation

[Store] Add synthetic master workload benchmark - #3147

[Store] Add synthetic master workload benchmark#3147
zhangzuo21 wants to merge 1 commit into
kvcache-ai:mainfrom
zhangzuo21:agent/master-synthetic-bench

zhangzuo21 commented Jul 28, 2026 •

edited by github-actions Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

codecov-commenter commented Jul 28, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zhangzuo21 commented Jul 28, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Module

Type of Change

How Has This Been Tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

codecov-commenter commented Jul 28, 2026

Codecov Report

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhangzuo21 commented Jul 28, 2026 •

edited by github-actions Bot

Loading