[dghosef]

pip install mstar-ai installs a pure-Python wheel (py3-none-any), so PyPI users never get M*'s compiled Rust parts: the mstar_rust extension module (Rust transport, graph runtime, SHM arena) and the mstar-server HTTP frontend. Today only people who build rust/ from a checkout with maturin and cargo have them. This PR adds platform wheels: manylinux_2_28 wheels for CPython 3.12, 3.13 and 3.14 on x86_64 and aarch64, built with cibuildwheel, each carrying mstar_rust, mstar-server and the license notices for everything compiled into them. pip picks a matching platform wheel and falls back to the pure wheel everywhere else (macOS, other architectures, other Pythons), which runs the Python transport and graph runtime as it does today.

The wheels statically link libzmq 4.3.4, which they ship under LGPL-3.0-or-later with libzmq's static-linking exception.

How it works:

1. setup.py adds setuptools-rust targets (RustExtension for rust/, RustBin for rust/server/) only when CIBUILDWHEEL=1. Every other build (pip install -e ., python -m build, an install from the sdist) stays pure Python. The wheel builds pass --locked and strip both binaries. The sdist now carries rust/ and rust-toolchain.toml, so CIBUILDWHEEL=1 pip wheel <sdist> builds the Rust wheel too.
2. pyo3 and the code built on it (the bindings, now in bindings.rs, plus graph and tensors) sit behind a new python cargo feature. extension-module implies it and stays the default. rust/server takes no features, so mstar-server links no libpython: manylinux has no shared libpython to link, and a linked one would tie the binary to one interpreter. cargo test for rust/ is now cargo test --release --no-default-features --features python.
3. cargo-about (rust/about.toml, rust/about.hbs) writes LICENSES-THIRD-PARTY-mstar_rust.txt and LICENSES-THIRD-PARTY-mstar-server.txt into each platform wheel's dist-info/licenses/. The C/C++ that crates vendor (libzmq via zeromq-src, Oniguruma via onig_sys, esaxx via esaxx-rs) is declared by hand. A license not in accepted fails the job.
4. Before anything can upload, each wheel is installed in a clean venv and has to import mstar_rust at the version EXPECTED_MSTAR_RUST_VERSION names, list both license notices, have no libpython in mstar-server's NEEDED, and run mstar-server --help with LD_LIBRARY_PATH unset. Then it's installed with its dependencies (CPU torch) and runs test/rust from a copy of test/, so mstar and mstar_rust both import from the wheel. That covers CPython 3.13, 3.14 and aarch64, which rust-transport never runs.
5. release.yml runs on a published release, on workflow_dispatch, and on PRs that touch rust/**, rust-toolchain.toml, .cargo/config.toml, setup.py, pyproject.toml, MANIFEST.in or release.yml. PR and manual runs build cp312 per arch, publish nothing and cancel when superseded; they use the same license config as a release. A release builds all three CPythons. pypi publishes the sdist, the pure wheel and the alias packages as before. pypi-wheels uploads the platform wheels only after wheels and pypi both pass, so a failing wheels holds back only the platform wheels. The workflow token is read-only except id-token: write on the two publish jobs.
6. libzmq comes from zmq-sys, which compiles zeromq-src's vendored 4.3.4 without HAVE_POSIX_MEMALIGN, so libzmq mallocs its command queues (16-byte aligned) though command_t is aligned(64). GCC on GitHub's ubuntu-26.04 runners comes from Ubuntu's amd64v3 builds and targets x86-64-v3, so it copies command_t with aligned 32-byte AVX stores, and the first one segfaults (zmq::mailbox_t::send, on the first communicator test). main passes only because its rust-transport cache still holds a libzmq built on ubuntu-24.04. A root .cargo/config.toml passes the defines libzmq's own build uses on Linux (HAVE_POSIX_MEMALIGN=1, ZMQ_CACHELINE_SIZE=64) to every cargo run in the repo, including setup.py's wheel builds, and the sdist ships it.
7. rust-toolchain.toml pins Rust 1.99.1 (with rustfmt and clippy) for CI, the release wheels, the CI and XPU Docker images and local builds. pyo3 goes from 0.23 to 0.26 because CPython 3.14 needs 0.25 or later; 0.26 renames allow_threads to detach, which is the only change in graph/runtime.rs and tensors.rs.

Behaviour change: MSTAR_RUST_ZMQ and MSTAR_RUST_GRAPH default to AUTO, which picks the Rust runtime wherever mstar_rust imports. So once the platform wheels publish, every pip install mstar-ai on Linux x86_64/aarch64 with CPython 3.12–3.14 runs the control mesh and the graph runtime on Rust with no config change. MSTAR_SHM_ARENA defaults to 0 and doesn't change. Each process logs control mesh transport: rust <version> from <path> (MSTAR_RUST_ZMQ=AUTO), and MSTAR_RUST_ZMQ=0 pins pyzmq.

A maturin build of rust/ installs a separate mstar-rust distribution as a mstar_rust/ package directory, while the wheel ships a top-level mstar_rust.cpython-3XX-*.so. They're different files, so uninstalling one never removes the other, but the package directory takes precedence on import and shadows the wheel's copy. The log line above names the loaded file, the version-mismatch warning gives the fix for a checkout (maturin develop) and for a pip install (upgrade, and pip uninstall mstar-rust if a leftover build shadows the wheel), and PACKAGING.md and installation.rst say so. find_server_binary also checks the running environment's scripts directory before $PATH, so an unactivated venv doesn't launch an older mstar-server from $PATH.

New dependencies: setuptools-rust>=1.11 in [build-system] requires, because it lets the existing setuptools build (which also copies configs/ into the package) compile the extension and the binary into the same wheel, which maturin can't do. cargo-about 0.9.2, in the release workflow only, to generate the license notices.

How it was tested

• Wheels built with cibuildwheel 4.2.1 in the manylinux_2_28 image for x86_64 and aarch64, cp312/313/314, then put through the workflow's check and test steps. test/rust against the installed wheel: 534 passed, 5 skipped (CUDA, sudo) on each of 3.12, 3.13 and 3.14. aarch64 wheels: import and mstar-server --help under qemu.
• CPU Core on the Eras CPU box (as CI, -n 4, OMP_NUM_THREADS=2): 8131 passed, with the same 2 failures as main (test_dump_stacks_retries_with_sudo_only_when_asked[False/True], from py-spy in the box's venv).
• cargo test --release --no-default-features --features python in rust/: 77 passed. cargo test --release in rust/server: 56 passed.
• actionlint 1.7.12 on every workflow: clean.
• The segfault: reproduced in an Ubuntu 26.04 amd64v3 userland (its GCC 15 defaults to -march=x86-64-v3), building the same test binary CI built (mstar_rust-fc38c2648720017d). It crashed 3 of 3 runs on vmovdqa %ymm0,(%rax) in zmq::mailbox_t::send, with %rax 16-byte aligned. With .cargo/config.toml the lib tests and rust/server's 56 tests pass there. A plain 26.04 amd64 userland, whose GCC targets baseline x86-64, doesn't crash.
• License step run as a release would, on the Eras box with cargo-about 0.9.2: exits 0, writes both notices, and each carries libzmq's whole COPYING.LESSER, static-linking exception included. libzmq's compiled sources (src/, include/, external/) match the upstream v4.3.4 tarball.
• This PR's CI on b299afc: rust-transport built libzmq fresh on ubuntu-26.04 (no cache) and passed: 77 + 56 cargo tests, interop 522 passed, 3 skipped. release.yml built both cp312 wheels; the clean-venv check passed and test/rust against each gave 518 passed, 3 skipped on x86_64 and on aarch64. pypi and pypi-wheels were skipped (PR run). CPU Core 8083 passed, the TP lockstep sim 11 passed, build and dynamo-smoke passed.
• test/rust/, test/modular/test_row_output_store.py and test/ci on the Eras box with --rust: 584 passed, 5 skipped.

Not in this PR / follow-ups

• setuptools-rust is fetched into every sdist build even though only cibuildwheel uses it. Avoiding that needs an in-tree build backend, or cibuildwheel --no-isolation with the build deps preinstalled.
• abi3 wheels (one per arch instead of one per CPython): an idea only.

🤖 Generated with Claude Code

[vasilevklart]

Yes we need to not forget about this for the next Pypi release, rust needs to be built into the wheel, was mentioning this to @merceod

[vasilevklart]

I tried it and it works, I know it is in draft but looks mostly right.. We want python 3.14 as well, right? Also probably need tests, packaging.md chagnes, check on the third party licenses + pinning rust compiler. Also rust HTTP frontend binary

[vasilevklart]

Oh and also since it can come from the mstar-ai wheel and rust/'s mstar-rust, if both get installed in one environment I believe uninstalling either deletes the other copy

[dghosef]

Taking out of draft mode. It was draft because I haven't had the chance to look at it but perhaps it's ready to receive comments already

[mstar-review]

Automated review

🟡 No findings on 6f9da1b, light review

Description

• The PR description is empty: it doesn't say what the change does, and it doesn't say which tests were run or what they returned, which AGENTS.md requires of every PR.
• The description doesn't say whether the new wheels job was ever run. It only fires on a published release, and CI never exercises it, so the first real test would be the next PyPI release.
• The description doesn't mention the user-visible behaviour change: with MSTAR_RUST_ZMQ and MSTAR_RUST_GRAPH defaulting to AUTO, every pip install mstar-ai on Linux x86_64/aarch64 with CPython 3.12/3.13 now runs the Rust transport and graph runtime instead of the Python ones.
• The description doesn't mention the new build dependency setuptools-rust>=1.11 in [build-system].requires, which every install from source now fetches, including pip install -e ..

Questions

• Was the cibuildwheel job run end to end (for example a release on a fork, or cibuildwheel run locally with the same CIBW_* settings), and was a resulting cp312/cp313 wheel installed and checked with import mstar_rust and pytest test/rust? The workflow sets no CIBW_TEST_COMMAND. If the build or auditwheel repair fails (vendored libzmq through zeromq-src, the rustup bootstrap inside the manylinux container, the pypa/cibuildwheel@v4.2.1 tag), pypi now needs: wheels, so the whole release, aliases included, publishes nothing.

What the review checked

• setup.py only adds RustExtension("mstar_rust", path="rust/Cargo.toml") when CIBUILDWHEEL=1, so the python -m build in the pypi job still produces the sdist and a pure py3-none-any wheel.
• The extension name matches the crate: [lib] name = "mstar_rust" and #[pymodule] fn mstar_rust in rust/src/lib.rs, so the top-level mstar_rust module that maturin develop installs is what the wheel installs too.
• Default cargo features stay on under setuptools-rust, so pyo3/extension-module is enabled and the cdylib doesn't link libpython, which manylinux needs.
• libzmq comes from the vendored zeromq-src (Cargo.lock: zeromq-src 0.2.6, zmq-sys 0.12.0), so the manylinux container needs no system libzmq.
• CIBW_BUILD cp312-manylinux_* cp313-manylinux_* leaves out musllinux and free-threaded builds, covers requires-python >=3.12, and is within what pyo3 0.23 supports.
• The per-matrix artifact names (wheels-${{ matrix.os }}) are unique, and download-artifact with merge-multiple into dist/ puts them next to the sdist and pure wheel for the mstar-ai publish step, while the alias dists stay in their own dist-* directories.
• The docs and README only say the mstar_rust extension ships in the wheel, not the separate rust/server binary, which matches the build.

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[Gaurav-Shah05]

LGTM. One thought on the wheels job.

CI builds and tests mstar_rust with maturin, but the release wheels build it with setuptools-rust and upload it without importing it. If that build can't load, users get no error, because M* falls back to the Python graph runtime and pyzmq.