Manage pantsbuild.pants release size on PyPI

[jsirois]

We just hit a 20GB limit with the 2.4.0.dev0 release.

HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
Project size too large. Limit for project 'pantsbuild.pants' total size is 20 GB. See https://pypi.org/help/#project-size-limit

I worked around this by deleting the 5 dev releases in the 1.16.0 series, which is the 1st series where we started chewing up a bunch of space with several rust wheels per release. I then:

(twine) ^jsirois@gill ~/dev/pantsbuild/pants/dist/deploy/wheels/pantsbuild.pants/b2fbce097ab9f13cfaf710488f813cdf5aeadde4/2.4.0.dev0 (master) $ rm *.asc
(twine) ^jsirois@gill ~/dev/pantsbuild/pants/dist/deploy/wheels/pantsbuild.pants/b2fbce097ab9f13cfaf710488f813cdf5aeadde4/2.4.0.dev0 (master) $ twine upload --skip-existing -s *.whl
Uploading distributions to https://upload.pypi.org/legacy/
Signing pantsbuild.pants-2.4.0.dev0-cp37-cp37m-macosx_10_11_x86_64.whl
Signing pantsbuild.pants-2.4.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl
Signing pantsbuild.pants-2.4.0.dev0-cp38-cp38-macosx_10_11_x86_64.whl
Signing pantsbuild.pants-2.4.0.dev0-cp38-cp38-manylinux2014_x86_64.whl
Signing pantsbuild.pants.testutil-2.4.0.dev0-py37.py38-none-any.whl
  Skipping pantsbuild.pants-2.4.0.dev0-cp37-cp37m-macosx_10_11_x86_64.whl because it appears to already exist
  Skipping pantsbuild.pants-2.4.0.dev0-cp37-cp37m-manylinux2014_x86_64.whl because it appears to already exist
  Skipping pantsbuild.pants-2.4.0.dev0-cp38-cp38-macosx_10_11_x86_64.whl because it appears to already exist
Uploading pantsbuild.pants-2.4.0.dev0-cp38-cp38-manylinux2014_x86_64.whl
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39.4M/39.4M [01:00<00:00, 679kB/s]
Uploading pantsbuild.pants.testutil-2.4.0.dev0-py37.py38-none-any.whl
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 23.5k/23.5k [00:02<00:00, 9.83kB/s]

View at:
https://pypi.org/project/pantsbuild.pants/2.4.0.dev0/
https://pypi.org/project/pantsbuild.pants.testutil/2.4.0.dev0/

We need to come up with a strategy for managing our release storage consumption over time as well as come up with a short term plan for freeing up space used by existing releases.

As an initial idea it seems to make sense to me to:

1. Delete all .devX releases for all time prior to 2.4.0.dev0
2. Establish a (presumably automated) procedure for deleting all prior dev releases when a new dev series starts.

[stuhood]

I think that we should try to minimize the deleting we do, because there is still a chance that folks are pinned to dev releases.

So I could imagine this looking like choosing an age window for dev releases that:

1. we're comfortable with risking from a "delete something that someone might be using" perspective (two/three years?)
2. frees up sufficient headroom moving forward

For pantsbuild.pants, dev releases make up:

• 155 out of 418 for all time
• 100 out of 248 for releases before March 3rd 2019 (more than two years old)
• 60 out of 162 for releases before March 3rd 2018 (more than three years old)

The https://pypi.org/project/pypi-cleanup/ project seems like it would work for this.

But as https://blog.ovalerio.net/archives/1971 points out, deleting valid versions is always a bit problematic. While I expect that we're fairly safe deleting things older than X years, there is definitely an argument to be made for just requesting the capacity increase. Or perhaps doing a one time cleanup AND requesting the capacity increase.

[Eric-Arellano]

there is definitely an argument to be made for just requesting the capacity increase

There is, but I agree with John's instinct that we should be good Python citizens and not do this. PyPI hosting costs real money that we're taking away from the Python community and PSF so that users pinned to a stale dev release don't need to make the one line change from "1.20.0.dev3" to "1.20.0".

I agree with not deleting stable releases of course, but upgrading from a dev to stable release is little work for our users and gives them a better experience. I would also be shocked if there were people using a dev release from >1 year ago.

[stuhood]

In order to unblock tonight's releases I deleted dev releases for 1.2.0 through 1.6.0 (56 releases).

[stuhood]

I've manually deleted 1.7.0.dev* through 1.19.0.dev*. This is only pantsbuild.pants, rather than any of the plugin packages (which I expect are smaller).

[stuhood]

This is still an issue: I've only been doing manual cleanups, and should have invested in automating it. Sorry.

For posterity, the way to use pypi-cleanup==0.0.3 (NB: not the most recent version: had trouble with that one) to delete a range is:

pypi-cleanup -u ${pypiusername} -p pantsbuild.pants -n -r '1.1[0-9].0.dev[0-9]+'

The -n flag (or leaving -n off) is dryrun mode and will list what would be deleted. To actually execute the cleanup, replace with -y.

[jsirois]

I'd love to take care of the issue and implement my proposal. If you still object Stu then I'll hand over to you to document / automate what you see fit.

[stuhood]

I continue to think that we should avoid deleting "recent" dev releases, and instead have some age window. But I'm unlikely to have the time to implement that in the next few weeks.

[stuhood]

I've filed pypi/support#1376.

[stuhood]

Opened #13234 to begin automating this a bit.

[stuhood]

As discussed in the comments on #13234 (comment), our path forward is likely going to be implementing #12397. Will add more information there.

[stuhood]

We're getting close to hitting our 40GB limit again, even having deleted all dev releases up to 2.16.0dev1 (from November 2022), so we'll need to resolve #12397 and/or open another PyPI limit increase ticket in the next few weeks.

[Eric-Arellano]

We're getting close to hitting our 40GB limit again, even having deleted all dev releases up to 2.16.0dev1 (from November 2022), so we'll need to resolve #12397 and/or open another PyPI limit increase ticket in the next few weeks.

Another option could be deleting rcs. We have them starting from 2.5.

[stuhood]

Resolved by @thejcannon's work to move to Github releases. Thanks so much!

[thejcannon]

Also, for posterity ALL our v2 wheels are now in GitHub Releases, and are indexed at https://wheels.pantsbuild.org/simple/

Including dev, alpha, and RCs 🙂