Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental: Add support for building pinned artifacts #71

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

abn
Copy link
Member

@abn abn commented Aug 27, 2020

This change allows poetry projects to be configured via pyproject.toml
to optionally enable version pinning and also to include nested
dependencies for application wheels.

NOTE: This is only a proposal, intended to serve as a basis for internal discussion.

Requires: #74 #75

There are some obvious deficiencies to this appraoch.

  1. The markers are computed on the fly and can be a bit fragile as we rely on weak relations to determine package/dependency hierarchy. This could be improved if the solver dumps the "net" markers into the lock file instead of just the pockage markers. Although, it can be argued that we can "compose" this information from available information. /shrug
  2. Path/Git/URL depednencies are unverified in this appraoch.
  3. The lack of a "root level" dependency notion and range does make this implementation dependent in the pyproject.toml file. So this means no install from lockfile option. Although, we are a step closer to that now.
  4. The way we handle condtional dependencies works, but I am not sure if it is sufficient. We take the "dumb" appraoch and assume that (1) is correct and that the PEP 517 front-end knows what it is doing.

An additional beneifit here is that we can improve the requirements.txt export using this mechanism.

Relates-to: python-poetry/poetry#2778 python-poetry/poetry#2331

To test this from poetry, you can install this branch into your poetry environment, patch the Locker implementation by inherting from poetry.core.lock.locker.Locker ony keeping the _locked_repository method; and then add the following to your pyproject.toml.

[tool.poetry.build.metadata]
dependencies = { lock = true, nested = true}

Once that is done, you can simply use poetry build.

Example Generated Metadata

The content below was generated from the poetry's lock file.

dependencies = { lock = false, nested = false}

Requires-Dist: cachecontrol[filecache] (>=0.12.4,<0.13.0)
Requires-Dist: cachy (>=0.3.0,<0.4.0)
Requires-Dist: cleo (>=0.8.1,<0.9.0)
Requires-Dist: clikit (>=0.6.2,<0.7.0)
Requires-Dist: crashtest (>=0.3.0,<0.4.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: functools32 (>=3.2.3,<4.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: futures (>=3.3.0,<4.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: glob2 (>=0.6,<0.7); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: html5lib (>=1.0,<2.0)
Requires-Dist: importlib-metadata (>=1.6.0,<2.0.0); python_version < "3.8"
Requires-Dist: keyring (>=18.0.1,<19.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (>=20.0.1,<21.0.0); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (>=21.2.0,<22.0.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: packaging (>=20.4,<21.0)
Requires-Dist: pathlib2 (>=2.3,<3.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: pexpect (>=4.7.0,<5.0.0)
Requires-Dist: pkginfo (>=1.4,<2.0)
Requires-Dist: poetry-core (>=1.0.0a9,<2.0.0)
Requires-Dist: requests (>=2.18,<3.0)
Requires-Dist: requests-toolbelt (>=0.8.0,<0.9.0)
Requires-Dist: shellingham (>=1.1,<2.0)
Requires-Dist: subprocess32 (>=3.5,<4.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: tomlkit (>=0.7.0,<1.0.0)
Requires-Dist: typing (>=3.6,<4.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: virtualenv (>=20.0.26,<21.0.0)

dependencies = { lock = true, nested = false}

Requires-Dist: cachecontrol[filecache] (==0.12.6)
Requires-Dist: cachy (==0.3.0)
Requires-Dist: cleo (==0.8.1)
Requires-Dist: clikit (==0.6.2)
Requires-Dist: crashtest (==0.3.1); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: functools32 (==3.2.3-2); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: futures (==3.3.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: glob2 (==0.6); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: html5lib (==1.1)
Requires-Dist: importlib-metadata (==1.7.0); python_version < "3.8"
Requires-Dist: keyring (==18.0.1); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (==20.0.1); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (==21.3.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: packaging (==20.4)
Requires-Dist: pathlib2 (==2.3.5); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: pexpect (==4.8.0)
Requires-Dist: pkginfo (==1.5.0.1)
Requires-Dist: poetry-core (==1.0.0a9)
Requires-Dist: requests (==2.24.0)
Requires-Dist: requests-toolbelt (==0.8.0)
Requires-Dist: shellingham (==1.3.2)
Requires-Dist: subprocess32 (==3.5.4); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: tomlkit (==0.7.0)
Requires-Dist: typing (==3.7.4.3); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: virtualenv (==20.0.30)

dependencies = { lock = true, nested = true}

Requires-Dist: SecretStorage (==3.1.2); python_version >= "3.6" and python_version < "4.0" and sys_platform == "linux"
Requires-Dist: appdirs (==1.4.4)
Requires-Dist: cachecontrol[filecache] (==0.12.6)
Requires-Dist: cachy (==0.3.0)
Requires-Dist: certifi (==2020.6.20)
Requires-Dist: cffi (==1.14.2)
Requires-Dist: chardet (==3.0.4)
Requires-Dist: cleo (==0.8.1)
Requires-Dist: clikit (==0.6.2)
Requires-Dist: configparser (==4.0.2); python_version == "2.7"
Requires-Dist: contextlib2 (==0.6.0.post1); python_version < "3"
Requires-Dist: crashtest (==0.3.1); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: cryptography (==3.0)
Requires-Dist: distlib (==0.3.1)
Requires-Dist: entrypoints (==0.3); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: enum34 (==1.1.10); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: filelock (==3.0.12)
Requires-Dist: functools32 (==3.2.3-2); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: futures (==3.3.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: glob2 (==0.6); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: html5lib (==1.1)
Requires-Dist: idna (==2.10)
Requires-Dist: importlib-metadata (==1.7.0); python_version < "3.8"
Requires-Dist: importlib-resources (==3.0.0); python_version < "3.7"
Requires-Dist: ipaddress (==1.0.23); python_version < "3"
Requires-Dist: jeepney (==0.4.3); python_version >= "3.6" and python_version < "4.0" and sys_platform == "linux"
Requires-Dist: keyring (==18.0.1); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (==20.0.1); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (==21.3.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: lockfile (==0.12.2)
Requires-Dist: msgpack (==1.0.0)
Requires-Dist: packaging (==20.4)
Requires-Dist: pastel (==0.2.0)
Requires-Dist: pathlib2 (==2.3.5); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: pexpect (==4.8.0)
Requires-Dist: pkginfo (==1.5.0.1)
Requires-Dist: poetry-core (==1.0.0a9)
Requires-Dist: ptyprocess (==0.6.0)
Requires-Dist: pycparser (==2.20)
Requires-Dist: pylev (==1.3.0)
Requires-Dist: pyparsing (==2.4.7)
Requires-Dist: pywin32-ctypes (==0.2.0); python_version >= "2.7" and python_version < "2.8" and sys_platform == "win32"
Requires-Dist: requests (==2.24.0)
Requires-Dist: requests-toolbelt (==0.8.0)
Requires-Dist: scandir (==1.10.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: secretstorage (==2.3.1); (sys_platform == "linux2" or sys_platform == "linux") and python_version < "2.8" and python_version >= "2.7"
Requires-Dist: shellingham (==1.3.2)
Requires-Dist: singledispatch (==3.4.0.3); python_version < "3.4"
Requires-Dist: six (==1.15.0)
Requires-Dist: subprocess32 (==3.5.4); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: tomlkit (==0.7.0)
Requires-Dist: typing (==3.7.4.3); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: typing-extensions (==3.7.4.2); python_version >= "3.5" and python_full_version < "3.5.4"
Requires-Dist: urllib3 (==1.25.10)
Requires-Dist: virtualenv (==20.0.30)
Requires-Dist: webencodings (==0.5.1)
Requires-Dist: zipp (==1.2.0); python_version < "3.8"

dependencies = { lock = false, nested = true}

Requires-Dist: SecretStorage (>=3); python_version >= "3.6" and python_version < "4.0" and sys_platform == "linux"
Requires-Dist: appdirs (>=1.4.3,<2)
Requires-Dist: cachecontrol[filecache] (>=0.12.4,<0.13.0)
Requires-Dist: cachy (>=0.3.0,<0.4.0)
Requires-Dist: certifi (>=2017.4.17)
Requires-Dist: cffi (>=1.8,<1.11.3 || >1.11.3)
Requires-Dist: chardet (>=3.0.2,<4)
Requires-Dist: cleo (>=0.8.1,<0.9.0)
Requires-Dist: clikit (>=0.6.2,<0.7.0)
Requires-Dist: configparser (>=3.5); python_version == "2.7"
Requires-Dist: contextlib2; python_version < "3"
Requires-Dist: crashtest (>=0.3.0,<0.4.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: cryptography
Requires-Dist: distlib (>=0.3.1,<1)
Requires-Dist: entrypoints; python_version >= "2.7" and python_version < "2.8"
Requires-Dist: enum34 (>=1.1,<2.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: enum34 (>=1.1.10,<2.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: enum34; python_version < "3"
Requires-Dist: filelock (>=3.0.0,<4)
Requires-Dist: functools32 (>=3.2.3,<4.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: futures (>=3.3.0,<4.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: glob2 (>=0.6,<0.7); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: html5lib (>=1.0,<2.0)
Requires-Dist: idna (>=2.5,<3)
Requires-Dist: importlib-metadata (>=1.6.0,<2.0.0); python_version < "3.8"
Requires-Dist: importlib-resources (>=1.0); python_version < "3.7"
Requires-Dist: ipaddress; python_version < "3"
Requires-Dist: jeepney (>=0.4.2); python_version >= "3.6" and python_version < "4.0" and sys_platform == "linux"
Requires-Dist: keyring (>=18.0.1,<19.0.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: keyring (>=20.0.1,<21.0.0); python_version >= "3.5" and python_version < "3.6"
Requires-Dist: keyring (>=21.2.0,<22.0.0); python_version >= "3.6" and python_version < "4.0"
Requires-Dist: lockfile (>=0.9)
Requires-Dist: msgpack (>=0.5.2)
Requires-Dist: packaging (>=20.4,<21.0)
Requires-Dist: pastel (>=0.2.0,<0.3.0)
Requires-Dist: pathlib2 (>=2.3,<3.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: pexpect (>=4.7.0,<5.0.0)
Requires-Dist: pkginfo (>=1.4,<2.0)
Requires-Dist: poetry-core (>=1.0.0a9,<2.0.0)
Requires-Dist: ptyprocess (>=0.5)
Requires-Dist: pycparser
Requires-Dist: pylev (>=1.3,<2.0)
Requires-Dist: pyparsing (>=2.0.2)
Requires-Dist: pywin32-ctypes (<0.1.0 || >0.1.0,<0.1.1 || >0.1.1); python_version >= "2.7" and python_version < "2.8" and sys_platform == "win32"
Requires-Dist: requests (>=2.1</b>8,<3.0)
Requires-Dist: requests-toolbelt (>=0.8.0,<0.9.0)
Requires-Dist: scandir; python_version >= "2.7" and python_version < "2.8"
Requires-Dist: secretstorage (<3); (sys_platform == "linux2" or sys_platform == "linux") and python_version < "2.8" and python_version >= "2.7"
Requires-Dist: secretstorage; python_version >= "3.5" and python_version < "3.6" and sys_platform == "linux"
Requires-Dist: shellingham (>=1.1,<2.0)
Requires-Dist: singledispatch; python_version < "3.4"
Requires-Dist: six
Requires-Dist: six (>=1.4.1)
Requires-Dist: six (>=1.9)
Requires-Dist: six (>=1.9.0,<2)
Requires-Dist: subprocess32 (>=3.5,<4.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: tomlkit (>=0.7.0,<1.0.0)
Requires-Dist: typing (>=3.6,<4.0); python_version >= "2.7" and python_version < "2.8"
Requires-Dist: typing-extensions (>=3.6,<4.0); python_version >= "3.5" and python_full_version < "3.5.4"
Requires-Dist: urllib3 (>=1.21.1,<1.25.0 || >1.25.0,<1.25.1 || >1.25.1,<1.26)
Requires-Dist: virtualenv (>=20.0.26,<21.0.0)
Requires-Dist: webencodings
Requires-Dist: zipp (>=0.4); python_version < "3.8"
Requires-Dist: zipp (>=0.5); python_version < "3.8"

@abn abn requested review from sdispater and a team August 27, 2020 01:27
@abn abn force-pushed the curious/dependency-version-pinning branch 3 times, most recently from 935dda6 to f4225e2 Compare August 30, 2020 21:57
This change allows poetry projects to be configured via pyproject.toml
to optionally enable version pinning and also to include nested
dependencies for application wheels.
@franziskakuehn
Copy link

is this going to be accepted?

is this a fat wheel approach?

@radoering
Copy link
Member

is this going to be accepted?

This is quite outdated, so probably not in the near future. Without taking a closer look, I assume that python-poetry/poetry#9428 is a more recent variant of this, which produces more reliable markers but is only implemented in poetry, not in poetry-core; in other words: not in the build-backend.

is this a fat wheel approach?

No, it is about pinning dependencies instead of allowing version ranges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants