[staryxchen]

Description

High-concurrency PD disaggregation registered 256 independent 64KiB notify
recv MRs per TENT RDMA endpoint. Endpoint count scales with
#session × #NIC, so ibv_reg_mr / ibv_create_qp returned ENOMEM while
HCA max_qp was still far from full.

Match the notify send path: one contiguous 256 × 64KiB recv buffer, one
ibv_reg_mr, then slice by slot. Slot encode/decode is shared with send so
adjacent slots cannot mix payloads.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Reshard (mooncake-reshard)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

./tent_endpoint_lifecycle_test
./tent_rdma_transport_test
# A/B on 8×H20 / 8×mlx5, same-machine PD, experiment R (memory-force-rdma),
# LOAD=ci (gsm8k 200/128/512) + `rdma resource show`

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

tent_endpoint_lifecycle_test (20) and tent_rdma_transport_test (24) passed,
including slot-adjacency / no-crosstalk / oversized-payload tests.

A/B on the same host and load:

	A unfixed	B this patch
peak QP	3118	3608
peak MR	116554	3984
notify recv ENOMEM	962	0
ibv_create_qp ENOMEM	20	0
gsm8k score	0.88	0.89

B peak MR matches data MR + #endpoint × (1 recv + 1 send).

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit on the files changed in this PR and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Cursor Grok 4.6 implemented the recv-MR coalescing, slot encode/decode tests,
and the A/B run. The human submitter reviewed every changed line.

Made with Cursor

[catyans]

LGTM. Re-reviewed the current head: all recv slots retain stable 64 KiB offsets inside one contiguous backing buffer and share the single MR's lkey; wr_id remains the slot index, decoding is bounded by the completion length, and teardown keeps the backing buffer alive until QP destruction and MR deregistration succeed. Partial-construction cleanup remains intact. CI is green, and the A/B results validate the intended MR reduction and removal of the observed ENOMEM failures.

suggestion (non-blocking): add a mock-verbs lifecycle test asserting that endpoint construction registers exactly one notify recv MR and cleanup deregisters it exactly once. The current hardware-free tests cover slot arithmetic and payload isolation well, but do not directly pin the registration-count invariant.

[alogfans]

LGTM