Pants should be distributed on Github Releases

[stuhood]

There are two large problems with our use of PyPI:

1. The https://github.com/pantsbuild/setup script resolves using PIP for the specified pantsbuild.pants wheel version, but without using a lockfile. (relates to Verifying Pants installations)
2. We have hit PyPI's size limits in the past, and are close to doing so again. See Manage pantsbuild.pants release size on PyPI #11614.

---

To ensure that using a particular version of Pants is reproducible over time, and to reduce load on PyPI, we should either:

1. distribute Pants as a pre-built PEX
2. distribute Pants as a lockfile that fully pins transitive dependencies for supported platforms (with the same considerations as Interpreter constraints do not play well with lock files. #12200)
   • EDIT: Does not resolve Manage pantsbuild.pants release size on PyPI #11614.
3. have the pants script dynamically generate and recommend committing a lockfile EDIT: Does not resolve Manage pantsbuild.pants release size on PyPI #11614.
4. switch the distribution model of Pants to a native binary
   • EDIT: Switch pants distribution model from "python embeds rust" to "rust embeds python" #7369 was resolved without removing use of PyPI.
   • It's still possible that we could actually use PyOxidizer or "a scie containing a PEX and an interpreter" to release a Pants binary, at which point scie-pants would not need to bootstrap an interpreter.

[stuhood]

#13234 (comment) discusses how to accomplish this with PEX: we should prioritize it to resolve #11614.

[stuhood]

I've updated the title (but not the description, yet) to reflect the fact that we should almost certainly do this by shipping a PEX, as explained in #13234 (comment). The reasoning is that shipping a PEX resolves #11614, while also being a natural incremental step toward shipping a rust binary in #7369.

[jsirois]

Noting that none of this solves the pantsbuild.pants release size on PyPI since we must live there to satisfy plugin dependencies. We can cheat and etc, but we'll be fighting reality a bit so we should be fully aware there are issues here.

[stuhood]

Noting that none of this solves the pantsbuild.pants release size on PyPI since we must live there to satisfy plugin dependencies. We can cheat and etc, but we'll be fighting reality a bit so we should be fully aware there are issues here.

That is addressed in the comment that I linked: #13234 (comment)

[jsirois]

It's not, that only partially addresses, and you can do it easier withPEX_TOOLS=1 pants.pex repository .... You aren't considering plugin development for example. We need some Pants distribution out there for that, at least type stubs. My point stands, we're cheating here and need to be careful and consider all the angles.

[jsirois]

Expanded there. There are vague mentions of developing a plugin, but fwict they are not fleshed out.

[Eric-Arellano]

@chrisjrn is currently working on #7369 and going directly for PyOxizider, rather than this intermediate Pex step. That's preferable to avoid this middle step.

Let's re-open this if PyOxidizer ends up not working out.

[stuhood]

I've update the title and description a bit, since #7369 didn't address #11614, or the reproducibility issue.

This ticket lightly dupes #11614, but I view it as more strategic ("how do we stop using PyPI") whereas #11614 is more tactical ("how do we (ab)use PyPI less").

[stuhood]

Huge thanks to @thejcannon for this one.