Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lock PEP-518 build-system requires for each sdist in a lock. #2100

Open
jsirois opened this issue Mar 23, 2023 · 2 comments
Open

Lock PEP-518 build-system requires for each sdist in a lock. #2100

jsirois opened this issue Mar 23, 2023 · 2 comments

Comments

@jsirois
Copy link
Member

jsirois commented Mar 23, 2023

Currently a Pex generated lockfile can contain sdists from various sources. When building a PEX from such a lockfile and one of the sdist artifacts is selected, it must be built into a wheel. If the sdist utilizes PEP-518 build-system requires, that resolve is not locked. As a result building the same PEX on two different dates can pick up changes in the build system resolve and lead to a wheel with different content.

On the harmless side, we've seen this build-system requires:

[build-system]
requires = ["pdm-pep517>=1.0.0"]

Lead to built wheels with differing contents due to:

$ unzip -qc emote_rl-23.0.0-py3-none-any.whl emote_rl-23.0.0.dist-info/WHEEL
Wheel-Version: 1.0
Generator: pdm-pep517 1.1.3
Root-Is-Purelib: True
Tag: py3-none-any

In other words, even if the rest of the generated wheel has the same content, the Generator: pdm-pep517 1.1.3 line in the WHEEL metadata will be different when the build system releases an update.

Ideally, Pex could guaranty reproducibility here by including an extra lock per sdist that locks in its build-system requires and then Pex could use these extra locks when building wheels from the sdists later to form a reproducible PEX from the lock.

jsirois added a commit to jsirois/pex that referenced this issue Mar 23, 2023
Ideally Pex would be immune to this, but there is alot of work required
there. That work would be worthwhile - I think Pex would become the 1st
locker in the Python ecosystem robust in this way - but that is for
another day.

C.F.: pex-tool#2100
@jsirois
Copy link
Member Author

jsirois commented Mar 23, 2023

All Pip versions Pex supports support the following for pip wheel:

--no-build-isolation        Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used.

That seems to provide the crease on the build side of the equation for setting up a PEXed build environment from the lock thus re-introducing isolation along with using the locked the build-system requires.

jsirois added a commit that referenced this issue Mar 23, 2023
Ideally Pex would be immune to this, but there is alot of work required
there. That work would be worthwhile - I think Pex would become the 1st
locker in the Python ecosystem robust in this way - but that is for
another day.

C.F.: #2100
@zmanji
Copy link
Collaborator

zmanji commented Apr 27, 2023

I also had a problem like this with gevent-21.1.2.

A sublock for the build environment would be ideal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants