Support generation of poetry manged setup.py file

[abn]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

As things stand today, poetry packages cannot be installed using pip or any other package managers from a VCS. This is useful when working with unreleased/unpublished versions of dependencies, where dependencies are managed by 'pip' or another tool. This was briefly touched on in #34 for a similar use case.

With this in mind, it could be great if poetry provided a solution, in a limited capacity, similar to what is implemented by poetry-setup. In order to achieve this, poetry could provide a command, eg: poetry setuptools:setupfile, that generates a usable setup.py file using the mechanism used today by SdistBuilder.

In theory, this could be migrated to be a plugin once #693 is implemented.

As an example. this, if implemented, could be used by projects as a pre-commit hook to generate setup.py file on commit. This should allow projects using pip etc. to install unpublished projects developed using poetry (if the maintainer wants it).

In addition to the above use case, this also allows for tools like PyCharm etc., to auto detect requirements and metadata.

Can follow this up with a proposed implementation PR.

Relates-to: #34

[webknjaz]

@abn other way to do it is to actually publish each commit to Test PyPI :)

[abn]

@webknjaz that would definitely work for a subset of use cases. The generation of setup.py and treating it similar to a lock file allows for a broader range of compatibility.

[cjw296]

Won't pip 19.0 remove the need for this? (as it understands pyproject.toml) - you'll need to add some manual stuff to it though, until #744 is resolved.

[abn]

@cjw296 I will see if I can validate that, if pip understands poetry projects, that will most likely cover a large chunk of the use cases. However, without this change, the other tools need to either support PEP-518 or poetry projects explicitly. With that in mind, I do think this is still useful even if pip support is completed.

[radek-sprta]

Snaps require setuptools and setup.py, at least for now, so having way to generate setup.py from poetry would help with that.

[aptlin]

Any updates on that?

[stevegore]

dephell will do this for you:

dephell deps convert --from pyproject.toml --from-format poetry --to setup.py --to-format setuppy

Or you can add this to pyproject.toml and just run dephell deps convert:

[tool.dephell.main]
from = {format = "poetry", path = "pyproject.toml"}
to = {format = "setuppy", path = "setup.py"}

[abn]

I have updated the PR to be a new export format instead of new command.

[KelSolaar]

I would be super glad to have that feature out-of-the-box. I looked quickly at dephell but the fact it added almost 30 dependencies to my project made me uncomfortable.

[danielhoherd]

One great reason to have this feature is that you cannot do pip install -e . unless there is a setup.py file.

ERROR: File "setup.py" not found. Directory cannot be installed in editable mode: /home/username/python-tool
(A "pyproject.toml" file was found, but editable mode currently requires a setup.py based build.)

For now the dephell command up above did what I was looking for, but requried attrs>=19.2.0. Also dephell has 49 dependencies.

[orsinium]

Also dephell has 49 dependencies.

DepHell is intended to be an outside tool like you have no poetry in your dependencies. The recommended way to install dephell is curl -L dephell.org/install | python3 that will put dephell into its own jail.

Also, most of the dependencies are part of dephell's own ecosystem (https://github.com/dephell/). And even after this, there are much fewer dependencies than even in pip (https://github.com/pypa/pip/tree/master/src/pip/_vendor), the only difference is they aren't vendorized for good reasons. Installation of all dependencies from the ground zero takes 17 seconds on TravisCI. So, it's more about decoupling than having a fat dependency tree.

[stevegore]

I feel I should point out that the pyproject.toml addition I've mentioned above doesn't actually required you to add dephell as a dependency of your project. Instead, I've got poetry and dephell both installed as part of the Docker image I use to build our Python projects.

[albireox]

For what it's worth, here is a small script that uses poetry itself to generate the setup.py. It seems to work fine so far.

[pawamoy]

@albireox nice, thank you for the script.

Here are two other versions to find Poetry's lib.

• In any virtualenv (pyenv, pipx, etc.):

  from pathlib import Path
  from shutil import which
  
  which_poetry = which("poetry")
  with open(which_poetry) as fd:
      shebang = fd.readline()
  poetry_python_lib = next(
      Path(shebang[2:]).parent.parent.glob("lib/python*")
  ) / "site-packages"

• More specifically, when installed with pipx, with support for Windows as well:

  import os
  from pathlib import Path
  
  if os.name == "nt":
      poetry_python_lib = Path("~/.local/pipx/venvs/poetry/Lib/site-packages").expanduser()
  else:
      poetry_python_lib = next(
          Path("~/.local/pipx/venvs/poetry/lib").expanduser().glob("python*")
      ) / "site-packages"

[nim65s]

@pawamoy: thanks, but you have an extra ) after .parent.parent

[kolypto]

In fact, poetry generates a setup.py when you build the project, and places it into one of the dist/*.tar.gz files :)
Let's extract it:

$ poetry build
$ tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1

now your poetry project is pip-editable ;)

[abn]

now your poetry project is pip-editable ;)

You can also just use a setup-shim for most cases.

cat "import setuptools; setuptools.setup()" > setup.py

Note that having a setup file can cause some side-effects.

[JasonAtallah]

In fact, poetry generates a setup.py when you build the project, and places it into one of the dist/*.tar.gz files :)
Let's extract it:

$ poetry build
$ tar -xvf dist/*.tar.gz --wildcards --no-anchored '*/setup.py' --strip=1

now your poetry project is pip-editable ;)

I was getting errors with --wildcards and --no-anchored.

Running just tar -xvf dist/*.tar.gz '*/setup.py' seems to work for me.
I then run cp $(find . -name setup.py) setup.py to copy the setup.py over to the root of the my repo.

[Aathish04]

Has there been any consensus on whether such a script/tool for keeping the generated setup.py file should be added to poetry itself?

I imagine it would be extremely useful, at least until pypa finally decide what to do when you need an editable install with only a pyproject.toml .

[ThatXliner]

I have a problem with readthedocs.io and I will require a setup.py for the time being (until pip supports installing poetry-based projects via pyproject.toml)

[webknjaz]

Why do you think pip doesn't?

[JasonAtallah]

Created a PR to allow exporting to setup.py

[jfaleiro]

Another implication for not having a setup.py under source control is you can't refer git repository dependencies:

$ poetry add git+https://gitlab.com/jfaleiro/quantlet-streaming.git#XXX

TypeError

  expected string or bytes-like object

  at .venv/lib/python3.8/site-packages/poetry/core/utils/helpers.py:27 in canonicalize_name
       23│ _canonicalize_regex = re.compile(r"[-_]+")
       24│ 
       25│ 
       26│ def canonicalize_name(name):  # type: (str) -> str
    →  27│     return _canonicalize_regex.sub("-", name).lower()
       28│ 
       29│ 
       30│ def module_name(name):  # type: (str) -> str
       31│     return canonicalize_name(name).replace(".", "_").replace("-", "_")

The message "expected string or bytes-like object" means poetry could not find a setup.py under source control at https://gitlab.com/jfaleiro/quantlet-streaming.git on tag XXX. It has a pyproject.toml but no setup.py. Hopefully this is helpful to someone else.

[finswimmer]

@jfaleiro there must be another reason for the error you receive. poetry doesn't need a setup.py within the git repo if there is a pyproject.toml and it is a poetry project.

[abn]

Since I am the original author of this issue, I will go ahead and close this. Since 2018, the status quo has changed. This is no longer a useful change or a core feature for poetry. Additionally, my original need is no longer a concern as there is better developer tooling support for poetry now.

For cases where a "legacy" editable mode is desireable you can simply make use of a setup.py shim as originally described in this article.

import setuptools; setuptools.setup()

Note: add name and version if you want sensible metadata that is not UNKNOWN-0.0.0.

Poetry itself now implements an editable install support for projects it manages. That might coer a few use cases. Furhtermore, poetry is in the process of removing the internal supprot for generating setup.py files and hence keeping the export around is added maintanence burden without much return. If such an export is desirable, it can be implemented by the community via the upcoming plug-in system as a plug-in to the export command.

[abersheeran]

I also have this problem. After reviewing the poetry source code, I found a solution and packaged it into a pypi package. https://github.com/abersheeran/poetry2setup

In short, you only need to execute two commands in the project to generate the setup.py file.

pip install poetry2setup

poetry2setup > setup.py

[ThatXliner]

Can't you install from a pyproject.toml now?

[ThatXliner]

@jfaleiro there must be another reason for the error you receive. poetry doesn't need a setup.py within the git repo if there is a pyproject.toml and it is a poetry project.

Like this?

[abersheeran]

@ThatXliner Yes, due to some open secrets (in China). When using pip install . directly to install, the download of poetry-core is a painful burden. I think someone might also have this need, so I posted it under this issue. If it disturbs you, I'm sorry.

[Purg]

For anyway that stumbles into this thread like I have, note that the latest version of poetry-core, version 1.0.8, supports PEP-660 to allow a pip install -e . of your package. I was able to update my pyproject.toml [build-system] section to use this version to do this.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.