Integration with poetry's pyproject.toml

[nbigaouette]

There is something I want to achieve but I am unsure as to what is the best approach. I was suggested on Gitter to post my usecase here in the hope of coming up with an example solution.

I have a current 100% Python project that is managed using poetry. It uses a pyproject.toml file to manage the project's dependencies.

Also, this project is deployed to an internal pypy repository using poetry publish --repository internalpypy as a wheel that gets pulled down by other projects (themselves using poetry too).

I would like to port one function of the Python project to Rust since it is quite heavy. I want to minimize changes to the project's organization. As such, how can I introduce the rust code to be built and included in the final wheel using only poetry build (and poetry publish)?

I did worked previously with milksnake but unfortunately:

• it doesn't support poetry (it requires a setup.py file to be configured properly but poetry's pyproject.toml replaces the setup.py file).
• it is abandoned

I checked maturin but:

• it seems to expect a specific directory structure which would be too disruptive for my project
• it seems to only build a 100% Rust wheel. This means that for a mixed rust-python project the main project would need to be split into two, introducing extra release complications.

Some poetry related issues I've found when searching this:

• Add support for non pure python projects: Add support for non pure python projects python-poetry/poetry#11
• Allow optional compilation of C extensions: Allow optional compilation of C extensions python-poetry/poetry#2330

I am looking at setuptools-rust right now.

Thanks!

[nbigaouette]

I've made an example repo that mimics my project's organization: https://github.com/nbigaouette/python-poetry-rust-wheel/tree/80496bf38e1d6b7a0218341338a6db0c0c7b5634

Running poetry build will actually use setuptools-rust to build the rust library. Unfortunately as of 80496bf38e1d6b7a0218341338a6db0c0c7b5634 the built library is not included in the wheel...

[kngwyu]

Integration with poetry build is interesting, but should be resolved by maturin rather than the PyO3 itself?

[davidhewitt]

Yes I asked @nbigaouette to open the issue here; I'll move it where I think is appropriate at the weekend.

[konstin]

it doesn't support poetry (it requires a setup.py file to be configured properly but poetry's pyproject.toml replaces the setup.py file).

poetry can run build python build scripts, see e.g. pendulum.

it seems to only build a 100% Rust wheel. This means that for a mixed rust-python project the main project would need to be split into two, introducing extra release complications.

The readme specifically explains how to build mixed projects.

As long as poetry's plugin interface is not finished and there is not standardized way to specify dependencies (https://discuss.python.org/t/pep-621-how-to-specify-dependencies/4599), I expect that you will have to duplicate at least the dependencies between poetry and any other tool, maybe even other metadata (until PEP 621 is widely used) and won't be able to use poetry publish.

[davidhewitt]

I'm going to close this question as I think @konstin 's answer is very good and there's not much we can do here in pyO3 for now. Thanks for the question and answers all!

[saolof]

It may be worth quickly mentioning that Maturin is a headache for Nix in the same way that Poetry is a godsend. Maturin is very reliant on procedural modifications of venvs compared to something Cargo or Poetry where everything is controlled by a requirements file and a separate lockfile.

As a result, installing polars with nix is currently impossible for example, because of an error thrown by Maturin

[konstin]

@saolof please don't comment on closed issues with new problems, if there is a specific thing that doesn't work please open an issue in the maturin repo with instructions to reproduce the problems and we we'll try to make your usecase work.