A way to specify build-dependencies

[con-f-use]

Pipenv might need a way to specify built-dependencies and tooling packages that need to be installed in order for pipenv to do it's job correctly. Even a pyporject.toml is not sufficient, because you'd have to pass pip the option --no-use-pep517 in order to install build dependencies and then have pip build from setup.py the old-fashioned way.

Example

In my company we use a wrapper around setuptools to simplify writing setup.py scripts for our packaging. Let's call this wrapper stwrapper.

A typical setup.py would look like this:

# setup.py for "package-that-uses-stwrapper-in-its-setup-py"
from stwrapper import setup

setup(
   name='package-that-uses-stwrapper-in-its-setup-py',
   # ... some more arguments
)

Where the function stwrapper.setup parses arguments, does some extra work and then calls the real setuptools.setup. Effectively, the stwrapper package is sort of a built dependency for us.

Lets say we have the following Pipfile:

[dev-packages]
stwrapper = "*"
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, requires=["stwrapper"] }

and do a

pipenv lock --verbose

then, with the current version of Pipenv, we get:

Traceback:
...
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in '.'

Because stwarpper is not installed at the time pipenv tries to parse the setup.py of package-that-uses-stwrapper-in-its-setup-py. That is even the case, when stwrapper is installed in the system's python installation. Pipenv wants to have it inside the new virtual environment it creates. It's a Catch-22.

Our current workaround is to let pipenv create the virutal environment. Then do a pipenv run pip install stwrapper inside it and then the pipenv lock. That way it works.

Another, better workaround will be possible one pypa/pip#6370 is merged into pip. Then the behavior of pip-option --no-use-pep517 will be to install requirements specified in a pyproject.toml and proceed with pre-pep517 installation using setup.py.

Possible Solutions

1.) Pipenv gets an equivalent to setup_requires in setup files or requires=[...] in pyproject.toml's [backend] section that is global or we use the existing one that is currently only used for python_version:

   [requires]
   python_version = "2.7"
   stwrapper = "*"

   [packages]
   ...

2.) Pipenv uses the system python's installation of stwrapper (I'm aware of the --site-packages option but that may have unwanted side effects).
3.) Specify a per-package requires, i.e.

[packages]
mypackage = { version="2.0.1", requires=['stwrapper', 'gitpython'] }

4.) Pass --no-use-pep517 to pip for certain packages (similar syntax to 3.) and use that feature in conjunction with pyproject.toml for these packages

[packages]
mypackage = { version="2.0.1", pip_options=['--no-use-pep517'] }

5.) Specify an order/priority of package installation, so that the dependency is fullfilled when the dependent package is being installed/

Scope

I guess the problem applies to all projects that use non-standard-libaray packages in their setup.py files and/or use setuptools entry points to modify its behavior. You can not put such dependencies in the Pipfile, because they are needed for successfully locking.

Related Issues

• Pipfile needs a way to manually specify package installation order #2274
• Fix editable-mode logic when pyproject.toml present pip#6370
• new 'pre' syntax in Pipfile #1760

[uranusjr]

The canonical way is to fix packaging metadata in package-that-uses-stwrapper-in-its-setup-py (in other words, the package maintainers should be responsible to fixing this, not the packaging tools).

I believe the canonical way now is to enable build isolation (PEP 517), and leverage against PEP 518 (which is available in pip 19.x):

[build-system]
requires = ['setuptools', 'wheel', 'stwrapper']

[con-f-use]

Okay, thanks answering. I have not heard too much of PEP 517/518 yet. I thought they were still under consideration, a.k.a. not accepted yet. How exactly could I use that solution today with my above example?

Edit:
A pyproject.toml next to setup.py seems to be enough.

[uranusjr]

They are already accepted, and supported by pip and setuptools.

[con-f-use]

Okay, I just learned that the build-system solution, does not work in case of editable installs, unless pip is called with --no-use-pep517.

See PEP 517:

This PEP [517] originally specified another hook, install_editable, to do an editable install (as with pip install -e). It was removed due to the complexity of the topic, but may be specified in a later PEP.

Briefly, the questions to be answered include: what reasonable ways existing of implementing an 'editable install'? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so.

So the issue persists. It seems something like this, might be useful:

[dev-packages]
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, requires=["stwrapper"] }
# or
package-that-uses-stwrapper-in-its-setup-py = { path = ".", editable = true, pip_options=["--no-use-pep517"] }  # needs a pyproject.toml with the build-dependencies and no build-backand

[con-f-use]

Passing options down to pip like this

[dev-packages]
mypackage = { path = ".", editable = true, pip_options=["--no-use-pep517"] } 

seems to have been discussed here #287

[uranusjr]

I wonder if this can be fixed more simply by passing --no-use-pep517 during the retry phase of installation.

[con-f-use]

Since pypa/pip#6370 will implement a editable mode is not supported for pyproject.toml-style projects [...] you may pass --no-use-pep517 [...] for pip, Pipenv could look for that output and retry with --no-use-pep517. I so no reason not to.

[uranusjr]

@techalchemy What do you think about this, should the change require a PEEP? I’ll submit a fix if that’s not necessary.

[con-f-use]

Any work done yet?

[pawelswiecki]

What is the current status of this issue?

[con-f-use]

The more I use pipenv, the more I'd like to pass pip-options on a per-package basis (i.e. no. 4 of my proposed solutions).

passing --no-use-pep517 during the retry phase

Not that this wouldn't be a welcome improvement.

@uranusjr @kenneth-reitz @techalchemy @nateprewitt

[kennethreitz]

This seems like a potential good idea.

[con-f-use]

Is there any progress here? When is the next release scheduled? At least this seems straight forward to implement:

Since pypa/pip#6370 will implement a editable mode is not supported for pyproject.toml-style projects [...] you may pass --no-use-pep517 [...] for pip, Pipenv could look for that output and retry with --no-use-pep517. I so no reason not to.

[con-f-use]

Btw. passing options down to pip generally seems to be a good idea because otherwise, locking fails on Windows for packages like pycurl, because they try to build but require some additional options (see: https://github.com/pycurl/pycurl/blob/master/setup.py#L388):

    def configure_windows(self):
        # Windows users have to pass --curl-dir parameter to specify path
        # to libcurl, because there is no curl-config on windows at all.
        curl_dir = scan_argv(self.argv, "--curl-dir=")
        if curl_dir is None:
            fail("Please specify --curl-dir=/path/to/built/libcurl")

Generally, it might be a good idea not to fail catastrophically at locking, when pipenv can't build a source distribution on windows (or whatever it does with the source distribution at locking time).

Full Traceback

[techalchemy]

I agree this would be a good idea, someone just needs to write up a proposal (if you've done this already just point me at it)

essentially there's no disagreement about whether this should exist, more just no clear picture of how it should look if we are going to implement it. I don't have any particular uses for it so I would probably have a hard time designing the interface. For instance, would it just be an additional key in the Pipfile entry? What would it say?

That would be outlined in a PEEP as a PR to the peeps/ directory with a new number just saying the basics and following the templates that some of the other enhancements in there have followed. Just outline the desired behavior and how it should look, plus if there are any concerns or considerations that might make this hard to implement. Ideally, we want this to solve as general a case as possible so we don't have to reimplement this a bunch of different ways

Hope that provides a good path forward on this

[con-f-use]

Hi Dan, as I said, I'd be happy with something like this:

[dev-packages]
mypackage = { path = ".", editable = true, extra_pip_options=["--no-use-pep517", "--retries=10"] } 

But I do not know whether it's consistent with how other things work in pipenv. Also I have no idea wow to write a PEEP properly. And there is the question of how to handle it if pipenv would pass an argument to pip and it's also specified in extra_pip_options. Should the extras overwrite pipenvs setting? Should it be the other way arround? Should we get an "overwriting pip option" error? Should we get an error and have the option to just "prefer extras" or "prefere pipenv options"? I'd argue for warn and overwrite.

And just to be clear, we have two good ideas here. The one with extra_pip_options that I just outlined again and the one where pipenv retries with the pip option --no-use-pep517, whenever there is something like this in the output of pip:
editable mode is not supported for pyproject.toml-style projects [...] you may pass --no-use-pep517 [...] (during any stage, e.g. locking and installation). Should that be different PEEPs? I guess so.