[2xdevv]

Description

store_kv_bench.py can benchmark memory and NoF replicas but not DFS replicas. This PR adds two flags so it can write to the descriptor-based DFS backend:

• --dfs-replica-num sets ReplicateConfig.dfs_replica_num. Defaults to 0.
• --enable-ssd-offload passes enable_ssd_offload to setup(). Off by default.

The second flag is needed because the client attaches the DFS backend (FileStorage) only when enable_ssd_offload is true. Without it, every DFS put fails with -1601 (DFS_SERVICE_UNAVAILABLE). It is kept as a separate flag, not derived from --dfs-replica-num, so that a process which only reads DFS data can still enable it.

Both flags default to off, so existing runs are unchanged. store_kv_bench.md documents both under Key Rules.

Related: #3251 also edits store_kv_bench.py (NoF unaligned I/O). It is unrelated in purpose but touches nearby code (the setup() call), so whichever lands second needs a small rebase.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake Conductor (mooncake-conductor)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

# Unit tests
cd mooncake-store/benchmarks && python3 -m unittest test_store_kv_bench

# Master (DFS enabled); same MOONCAKE_DFS_* exported for the client
export MOONCAKE_DFS_ROOT_DIR=<root> MOONCAKE_DFS_FS_ADAPTER=<posix|hf3fs> \
       MOONCAKE_DFS_SHARD_COUNT=4 MOONCAKE_DFS_SHARD_CAPACITY=$((256*1024*1024))
MOONCAKE_ENABLE_DFS=1 mooncake_master --enable_http_metadata_server=true &

# Client
export MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR=distributed_storage_backend \
       MOONCAKE_OFFLOAD_FILE_STORAGE_PATH=/tmp/mooncake_file_storage
python3 mooncake-store/benchmarks/store_kv_bench.py --scenario verify_write \
  --io-api <plain|zcopy> --enable-ssd-offload --nr-objects 64 --batch-size 4 \
  --value-size 65536 --memory-replica-num 1 --dfs-replica-num 1 --verify --pattern 0xab

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

Manual end-to-end runs (verify_write, 64 × 64 KiB objects, write + verified read):

DFS backend	plain	zcopy
posix adapter, local XFS	pass	—
posix adapter, 3FS via FUSE	pass	pass
hf3fs adapter (USRBIO), live 3FS, -DUSE_3FS=ON	pass	pass

For each backend I also checked that the data really is on DFS:

• Without the new flag: the same run without --enable-ssd-offload fails every put with -1601.
• Read back after the writer exits: a separate client started after the writer exited sees only the DFS replica for each key (no memory replica) and reads and verifies all 64 objects.
• Raw shard scan: scanning the shard files under the DFS root finds exactly the expected number of 4 KiB pattern blocks.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh (N/A: no C/C++ changes; Python formatted via pre-commit ruff-format)
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Claude Code helped trace why DFS puts failed from the benchmark, draft the flag and doc changes, and run the end-to-end checks above. I reviewed every changed line and the test results.

[fcczzz]

LGTM. One small documentation suggestion: please add the DFS setup prerequisites and a runnable example to store_kv_bench.md, including enabling DFS on the master, selecting distributed_storage_backend on the client, and configuring MOONCAKE_DFS_* as shown in the PR description.

[fcczzz]

@2xdevv Hi, could you please update the branch with the latest main and rerun CI? The PyTorch/PG version mismatch has been fixed in #4440.

Also, please add the DFS setup prerequisites and a runnable example to store_kv_bench.md, thanks!

[2xdevv]

Thanks for checking this out, @fcczzz, @ykwd ! I’ve rebased onto the latest main and added the DFS prerequisites and runnable example to store_kv_bench.md.
I also rebuilt and verified both plain and zcopy against our local 3FS cluster. The updated branch has triggered a new CI run.