Add support for non pure python projects

[sdispater]

We should have a way to build non pure-python projects (from instance if they need to build C extensions).

[feluxe]

Hi!

Does this work with poetry?

Impressive work by the way! :)

[sdispater]

Thanks for your interest in poetry!

Yes, it works even though it's not currently documented because I do not know if it will stay this way since it relies on adding kwargs to the setup.py.

Anyway, it you want to test this, you have to specifiy a build script in your pyproject.toml:

[tool.poetry]
build = 'build.py'

that will define a build function that will add the necessary build keyword arguments to the call to setup() in the generated setup.py.

An example of this is in my pendulum project:

• pyproject.toml
• build.py

[feluxe]

This look promising! Thanks!

I'll give it a shot for my next package.

[bruxisma]

I know this issue is closed at this point, but I wanted to say I'm so glad this is here. I've been struggling with pipenv so much that I've stared into the abyss and we're now on a first name basis.

I'm currently working a library called brujería, that lets you treat your C and C++ modules as if they were python modules (both for REPL imports as well as setuptools builds). I'm still working out the kinks (I have to write another library so it can be decoupled from all of CMake's garbage), but I think it might help others in this situation. It's not in a working state at the moment as I'm doing janitorial work on both repos, but I still wanted to mention it also, @sdispater thank you for poetry. It's been so nice and I've only been using it for 10 minutes so far. :)

[salotz]

Here is an example for building a cython module https://github.com/ADicksonLab/geomm/blob/master/build.py.

[shimil]

Hey @sdispater, I love Poetry!
We want to adopt it as part of our tech stack, but consider support for C extensions an important feature. Can we count on continued support for it?

Many thanks

[AlJohri]

How can I use this build.py idea for external Cython dependencies? Currently I want to install (from source) the spaCy package. I have to currently resort to using poetry run to achieve this.

# install non-cython dependencies
poetry install

# install spaCy from source using `build_ext`
git clone https://github.com/explosion/spaCy.git spaCy
poetry run python spaCy/setup.py build_ext --inplace
poetry run pip install -e spaCy

[zoj613]

How can I use this build.py idea for external Cython dependencies? Currently I want to install (from source) the spaCy package. I have to currently resort to using poetry run to achieve this.

# install non-cython dependencies
poetry install

# install spaCy from source using `build_ext`
git clone https://github.com/explosion/spaCy.git spaCy
poetry run python spaCy/setup.py build_ext --inplace
poetry run pip install -e spaCy

you could bundle these steps as one Makefile command? Something like

poetry:
     poetry install
     git clone https://github.com/explosion/spaCy.git spaCy
     poetry run python spaCy/setup.py build_ext --inplace
     poetry run pip install -e spaCy

Then run all them using make poetry

[AlJohri]

@zoj613 and I've certainly done exactly that (see screenshot). but I wish I could specify spaCy as a git project directly in the pyproject.toml with some markers specifying that the cython extension needs to be built. is there anyway to do this?

[zoj613]

@zoj613 and I've certainly done exactly that (see screenshot). but I wish I could specify spaCy as a git project directly in the pyproject.toml with some markers specifying that the cython extension needs to be built. is there anyway to do this?

does it not build and install the cython packages by default when you install spaCy? If that is the case then you can just add the line
spaCy = {git = "https://github.com/explosion/spaCy.git"}
in your package list of pyproject.toml.

If it doesnt build it by default then I would clone the repo, modify the setup file to automatically build the Cython packages, then use your fork as the source in the pyrpoject.toml file like:
spaCy = {git = "https://github.com/AIJohri/spaCy.git"}

[AlJohri]

It doesn’t, it says that it can’t find the package version?

For some reason when pip tries to install it, it installs it on some sort of isolated environment without Cython which is why it fails to parse the setup.py.

I tried forking as you suggested but couldn’t figure out how to get past the Cython not found issue. I can post a dockerized example if you’re interested in debugging further.

[AlJohri]

@zoj613 in case you're curious, I found the root case was that the spaCy package has a pyproject.toml without the build dependencies defined: explosion/spaCy#4829

[teknico]

We're also using this feature in tinydecred with Cython. Any chance in stabilizing it? Thanks.

[abn]

@teknico I'd recommend creating a new issue detailing short-commings of the current implementations with as much details as possible please. We are already aware of a few issues. Including #1516.

[geronm]

salotz said:

Here is an example for building a cython module https://github.com/ADicksonLab/geomm/blob/master/build.py.

This comment seems to have fallen out of date.

I think this tag (v0.1.7) in the repo gives a stable reference with working build.py: https://github.com/ADicksonLab/geomm/tree/v0.1.7

[zoj613]

I have a mordern approach that avoids using an explicit setup.py file. It contains a build script that takes wheel building into account. See: https://github.com/zoj613/htnorm/blob/main/build.py

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.