Skip to content

[Store] Migrate lightweight Python utilities to unified package - #3586

Merged
Aionw merged 5 commits into
kvcache-ai:mainfrom
Aionw:codex/migrate-python-utilities
Sep 18, 2026
Merged

Aionw merged 5 commits into
kvcache-ai:mainfrom
Aionw:codex/migrate-python-utilities

Conversation

@Aionw

@Aionw Aionw commented Aug 21, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Implements another Phase 2 increment from #3534 by moving lightweight Store-facing Python utilities into the unified root Python package.

  • move mooncake.buffer_pool and mooncake.mooncake_config into python/mooncake as their single authoritative source;
  • move their tests under python/tests;
  • preserve the existing public import paths;
  • update direct CMake installation and the temporary legacy wheel staging path;
  • add source-location and installed-wheel checks;
  • update ownership and labeling for the migrated paths.

This migration is independent of #3585. The branch is based directly on main and contains no Reshard migration changes.

Module

  • Transfer Engine (mooncake-transfer-engine)
  • Mooncake Store (mooncake-store)
  • Reshard (mooncake-reshard)
  • Mooncake EP (mooncake-ep)
  • Mooncake PG (mooncake-pg)
  • Integration (mooncake-integration)
  • P2P Store (mooncake-p2p-store)
  • Python Wheel (mooncake-wheel)
  • Common (mooncake-common)
  • Mooncake RL (mooncake-rl)
  • CI/CD
  • Docs
  • Other

Type of Change

  • Bug fix
  • New feature
  • Refactor
  • Breaking change
  • Documentation update
  • Performance improvement
  • Other

How Has This Been Tested?

Test commands:

PYTHONPATH="$PWD/python" python -S python/tests/unit/test_mooncake_config.py
python -m pytest -q python/tests/packaging/test_project.py
CMAKE_BUILD_PARALLEL_LEVEL=128 python -m build --wheel
MOONCAKE_TEST_WHEEL="$PWD/dist/mooncake_transfer_engine-0.3.12.post1-cp310-cp310-linux_x86_64.whl" python -m pytest -q python/tests/packaging/test_installed_wheel.py
python -m twine check dist/mooncake_transfer_engine-0.3.12.post1-cp310-cp310-linux_x86_64.whl
python -m pre_commit run --files <files changed against main>

Test results:

  • Unit tests pass (35 configuration tests and 5 packaging contract tests)
  • Integration tests pass (isolated installed-wheel smoke test)
  • Manual testing done (wheel manifest contains both migrated modules and twine check passes)

Checklist

  • I have performed a self-review of my own code
  • I have formatted my code using ./scripts/code_format.sh (no C/C++ source changes; Python and CMake formatting passed through pre-commit)
  • I have run pre-commit on the files changed in this PR and all hooks pass
  • I have updated the documentation (not applicable: public APIs and behavior are unchanged)
  • I have added tests to prove my changes are effective
  • For changes >500 LOC: I have filed RFC issue [RFC]: Unify Mooncake's Python package, build, and release architecture #3534

AI Assistance Disclosure

  • No AI tools were used
  • AI tools were used (specified below)

OpenAI Codex assisted with migration selection, implementation, packaging validation, and PR preparation. The human submitter remains responsible for reviewing and defending every changed line.

Aionw added 2 commits August 28, 2026 16:05
…utilities

# Conflicts:
#	.github/CODEOWNERS
#	python/tests/packaging/test_installed_wheel.py
#	python/tests/packaging/test_project.py
#	scripts/build_wheel.sh
@Aionw
Aionw force-pushed the codex/migrate-python-utilities branch from f0a0016 to f07bd07 Compare September 8, 2026 08:12
@Icedcoco

Icedcoco commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Could you please confirm whether the direct CMake installation path also needs to install buffer_pool.py?

Since the module was moved from mooncake-wheel/mooncake to python/mooncake, the install(DIRECTORY ...) rule in python/CMakeLists.txt will no longer include it. However, mooncake/init.py still imports mooncake.buffer_pool, so users installing through the direct CMake path may encounter ModuleNotFoundError.

The wheel staging path appears to handle this module correctly. Would it make sense to add buffer_pool.py to the direct CMake install list and include a corresponding installation test?

@Icedcoco Icedcoco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending CI

@ShangmingCai ShangmingCai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just some harmless test file migration and clean up.

@Aionw
Aionw merged commit 78465c5 into kvcache-ai:main Sep 18, 2026
23 checks passed
@Aionw
Aionw deleted the codex/migrate-python-utilities branch September 18, 2026 05:02
DHX98 added a commit to DHX98/Mooncake that referenced this pull request Sep 23, 2026
Adapt to upstream changes since the PR opened:
- MasterService header split (kvcache-ai#4127): PromotionTask::from_prefetch moves
  to object_runtime_state.h; friend list follows the MasterServiceTestPeer
  form (kvcache-ai#4135) plus PrefetchTaskMasterTest.
- probeKey/BatchProbeKey RPCs (kvcache-ai#3801) coexist with ExistOptions.
- Python unified-package layout (kvcache-ai#3586): keep buffer_pool.py at its
  pre-existing path; migration is a separate follow-up.
DHX98 added a commit to DHX98/Mooncake that referenced this pull request Sep 28, 2026
Adapt to upstream changes since the PR opened:
- MasterService header split (kvcache-ai#4127): PromotionTask::from_prefetch moves
  to object_runtime_state.h; friend list follows the MasterServiceTestPeer
  form (kvcache-ai#4135), plus PrefetchTaskMasterTest.
- probeKey/BatchProbeKey RPCs (kvcache-ai#3801) coexist with ExistOptions.
- Python unified-package layout (kvcache-ai#3586): keep buffer_pool.py at its
  pre-existing path; migration is a separate follow-up.
- Design doc moved to docs/source/design/store/ssd-prefetch.md per the
  kvcache-ai#4196 module-grouped layout; drop stale toctree entries for moved docs.
Aionw added a commit that referenced this pull request Sep 28, 2026
main's /python/tests/store/ entry (#3586) would otherwise drop @zxpdemonio from the two migrated structured-object tests. Place the per-file entries after it so they take effect.
Aionw added a commit that referenced this pull request Oct 14, 2026
* [Wheel] Migrate structured object storage to unified package

* [Wheel] Format migrated structured object store sources and keep legacy test fallback

The migrated files count as fully changed for the changed-lines ruff check, so format them with ruff 0.6.9 (the result equals ruff format of main's legacy-path copies). Let mooncake-wheel/tests/conftest.py extend mooncake.__path__ for both the unified source package and the legacy tree, so legacy-only helpers such as mooncake._partial_read stay importable from source.

* [Wheel] Keep structured-object test owners after merging main

main's /python/tests/store/ entry (#3586) would otherwise drop @zxpdemonio from the two migrated structured-object tests. Place the per-file entries after it so they take effect.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants