Drop poetry's dependency on pip and setuptools in project virtual environment

[abn]

An initial attempt at using the embedded pip wheel from virtualenv package directly. This avoids the need for any critical packages in the project's virtual environment.

Resolves: #1584

Testing this pull request

Since this is a fundamental shift in how we setup virtual environments, testing a broad range of use cases would be great.

Using pipx

pipx install --suffix=@2826 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/2826/head'

Once this is done, you can use this version of poetry as shown here.

poetry@2826 new foobar
pushd foobar
poetry@2826 add pycowsay
poetry@2826 run pycowsay

If you want to do a pip list, there exists a convinience wrapper that will use the embedded pip if the environment does not contain one.

$ poetry@2826 run pip list
Package        Version
-------------- -------
attrs          20.3.0
more-itertools 8.7.0
packaging      20.9
pluggy         0.13.1
py             1.10.0
pycowsay       0.0.0.1
pyparsing      2.4.7
pytest         5.4.3
wcwidth        0.2.5

You will notice here that none of wheel, setuptools nor pip exists in the virtual environment.

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/2826/head
poetry new foobar
pushd foobar
poetry add pycowsay
poetry run pip list
EOF

[mikenerone]

Some sort of approach to let me lock pip, setuptools, and wheel for my own project is pretty important to me. Either removing Poetry's dependence on them as this branch does, or a config syntax that lets me bless overrides of Poetry's norms, or something. Right now, my team's dev environments are broken by default because the installed deps don't match the requirements in the package metadata. We workaround it for the moment with poetry run pip install ... (and the versions we need seem to work fine for Poetry), but it would be much, much better if these package deps could be treated per normal.

[gsemet]

We are dealing with exactly the same issue using pipenv, it always wants the latest version even if it breaks something. Please ensure at least to let the ability of user to freeze the version somehow. Ideally yes, it would use its own, independent setuptools/wheel, but for some complex projets that needs certain version of setuptools, especially with the version 50.x breaking much stuff with pyinstaller packaging, I am looking to a better alternative than p[ipenv|oetry] run pip install 'setuptools<50'

[abn]

@gsemet can you provide a bit more context please? This change only means that poetry's use is isolated to specific versions of pip/setuptools and decoupled. If your project depends on it, you can specify it as either a dev-dependency or build-system.requires. PEP 517 builds still take place in isolated environments, this means the build-system.requires of the project will be respected.

[gsemet]

Yes, this change is actually what i would to see. I am using poetry on my newer projects and does not face yet this issue I have on older projects still using pipenv. Pipenv always updates to the latest setuptools available on pypi within the project's virtualenv (also pip and wheels packages), and it occurs several times that the new version broke something. For reproductibility, there should not be these "moving" parts.

So yes, I am pretty much very interesting is seeing the right behavior in poetry, and maybe it already does it: poetry either vendors its own setuptools/wheel package so that they change with poetry updates, but if my package wants some specific version of setuptools for any reason, it should not cause any incompatibility with poetry. From out point of view, setuptools, pip and wheels are 3 packages like any others.

TL;DR: I need to update to poetry, it seems like you already do the right thing :)

[abn]

@gsemet ah, I think I misunderstood your first message. Glad that this is something that is useful. However, do note that poetry today treats setuptools, pip and wheel as "unsafe" packages in the environment. This means that, while, we do not explicitly update these versions, they cannot be managed by project.

This is what I (putting on my poetry user hat), want to change in this PR.

Poetry (putting on my maintainer hat again), has been gradually reducing reliance on pip and setuptools. Right now, we only use pip for wheel installations (wheels poetry downloads) and editable installs pip install -e /path/to/something. The reliance on setuptools is a side-effect of supporting editable dependency installations.

[gsemet]

So, I hope this PR lands in the next version of poetry :)

[apollo13]

I just saw this PR. This would be a huge improvement for us if it ends up fixing #1584 -- Thank you for your work on this @abn

[apollo13]

@abn Looking mostly good so far. I have left a few question, would be glad if you could get back on those.

[sdispater]

@abn

This is a solved problem and we should really use https://github.com/pypa/pep517

Well, not really, since pep517 makes subprocess calls to pip (https://github.com/pypa/pep517/blob/master/pep517/envbuild.py#L98) which we want to avoid. It also doesn't have reusable environments, i.e. you need to create a brand new environment for each sdist which is costly.

That's why I said it's no small task to make it pip agnostic.

[apollo13]

It seems as if https://github.com/pypa/build would be a pep517 compatible installer that doesn't utilize pip at all -- at least that is what a quick look through the source suggested.

EDIT:// That might have been to fast: https://github.com/pypa/build/blob/main/src/build/env.py#L157-L168

[abn]

At present it does use pip, see https://github.com/pypa/build/blob/2c1da830f214580e8e4b94e4c88345963fa32d2e/src/build/env.py#L157-L170

That said, if we do have support for wheel installations, and access to setuptools wheels (bundled via virtualenv), I suspect doing so will only require poetry to override the build lib implementation to use poetry's own wheel installation. But I can see what you mean by it not being a simple task. Although, I would definitely have to ask if that is worthwhile for poetry to implement and manage. Additionally, the "new environment" for each sdist is probably a good thing to be honest as it ensures build isolation. I doubt that will be an issue if you have local caching available.

[TBBle]

You don't actually override pypa/build's env-setup, since the caller is the one calling it, i.e. pip is only sub-called by IsolatedEnvBuilder.install from _build_in_isolated_env in __main__.py which is part of the CLI for the pypa/build package:

def _build_in_isolated_env(builder, outdir, distributions, config_settings):
    # type: (ProjectBuilder, str, List[str], ConfigSettings) -> None
    for distribution in distributions:
        with IsolatedEnvBuilder() as env:
            builder.python_executable = env.executable
            builder.scripts_dir = env.scripts_dir
            # first install the build dependencies
            env.install(builder.build_dependencies)
            # then get the extra required dependencies from the backend (which was installed in the call above :P)
            env.install(builder.get_dependencies(distribution))
            builder.build(distribution, outdir, config_settings)

So you just want to create a venv with the necessary dependencies yourself, and do this (_build_in_current_envfrom __main__.py, this is the --no-isolation path for their CLI tool)

def _build_in_current_env(builder, outdir, distributions, config_settings, skip_dependencies=False):
    # type: (ProjectBuilder, str, List[str], ConfigSettings, bool) -> None
    for dist in distributions:
        if not skip_dependencies:
            missing = builder.check_dependencies(dist)
            if missing:
                _error(
                    'Missing dependencies:'
                    + ''.join('\n\t' + dep for deps in missing for dep in (deps[0], _format_dep_chain(deps[1:])) if dep)
                )

        builder.build(dist, outdir, config_settings)

So for Poetry, I expect that if we want to use pypb/build, we just populate a venv as we do now, and use pypa/build's ProjectBuilder(srcdir) (builder above, this wraps/exposes the PEP-517 hooks) to build the wheel we need using that venv.

This also puts Poetry directly in control of whether it uses a single scratch venv to build all necessary wheels for the "real" project venv, or uses an isolated venv for each wheel, or some other combination thereof.

[bmw]

I'm eagerly awaiting this change as a resolution to #1584. When do people expect this PR to be included in a release?

[abn]

@bmw the next planned release is 1.2.0a0, which should be landing in the next month or so. In the meantime, you can use the version from master.

pipx install --force --suffix=@master 'poetry @ git+https://github.com/python-poetry/poetry.git@master'

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.