Allow projects to store pre-releases on external file hosts

[leezu]

What's the problem this feature will solve?
https://pypi.org/stats/ shows that a small number of projects creates large storage and data transfer costs by uploading daily nightly releases to Pypi servers. If projects want to avoid imposing those costs, they need to ask users to specify the -f option to pip to select an external file host. This is inconvenient for users, thus projects may not be inclined to use this option.

Describe the solution you'd like
To avoid straining Pypi's ressources, such projects can host the nightly releases themselves. Currently users then need to install via pip install --pre X -f example.org/nightly.html.
Let projects specify example.org/nightly.html somewhere in Pypi, so that pip install --pre X will automatically use -f example.org/nightly.html as a source.

Alternative Solutions
Ask users to specify the -f option.

Additional context
This would simplify pypi/support#50 (comment)

[uranusjr]

This is probably more suited for the Warehouse issue tracker (the current PyPI implementation) than pip. pip by default does not discover file URLs, but simply asks the index server for them (which is PyPI by default). So this kind of information needs to be relayed by PyPI to pip.

PyPI actually used to allow external storage (both for pre-release and stable), but eventually dropped that feature entirely. I’m not sure about the reason behind it, but with this in mind, it’s probably better to start a discussion on a mailing list or forum before submitting a straight-up feature request. Some places to start:

• Distutils-SIG (mailing list)
• pypa-dev (mailing list, Google Group)
• discussion.python.org (Discourse)

[RonnyPfannschmidt]

also extra note - this was dropped for multiple reasons,

• security
• having control on the pypi side
• no longer having packages go away by taken down server

i would strongly suggest using a own index + extra_index_urls, or setting up a public devpi thats set up to crosslink pypi annd putting dev versions there

[xavfernandez]

Cf PEP-0470 for motivations.

[pradyunsg]

(I'm travelling, apologies for the terseness/typos)

If you want to request PyPI to allow for such links, do take note that PyPI removed this functionality as documented in PEP 470 linked above.

If you want to request for pip to add any other sources "out of the box", that's a no-go -- at minimum you'd need concensus to do that from discuss.python.org (Packaging category) and I don't think that'd be very easy to get.

None the less, given that pip isn't the only project affected by the requested change and there's an accepted PEP that has the exact opposite position, I am going to close this issue.

---

I suggest that you mention that the aim here is to move mxnet pre-releases off of PyPI so that the size of storing them on PyPI is less, which in turn would allow increasing your upload limits.

A good way IMO would be host your own index that has only the mxnet projects (with nighties/pre-releases) on that package index. This can be a bunch of static files somewhere linking to the S3 buckets y'all have. (PyPI doesn't have external links, doesn't mean other indexes can't have them). Only "stable" releases would make it to PyPI then.

This way, users who want to use the stable releases can get them from PyPI and users who want the pre-releases can add/pass "extra-index-urls" to pip's configuration or CLI, to be able to install those pre-releases.

If you want to have further discussion on this, please start a topic on discuss.python.org.