HOMEBREW_PIP_INDEX_URL is misleading and does not work-as-intended with some mirrors

[ctaintor]

brew doctor output

Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Your Xcode (15.2) is outdated.
Please update to Xcode 15.4 (or delete it).
Xcode can be updated from the App Store.

Verification

• My "brew doctor output" above says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update twice and am still able to reproduce my issue.
• This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

brew config output

HOMEBREW_VERSION: 4.3.14-35-g87b0800
ORIGIN: https://github.com/Homebrew/brew
HEAD: 87b0800a6cee961e5343a306f8566879bf121bb3
Last commit: 5 hours ago
Core tap JSON: 11 Aug 07:25 UTC
Core cask tap JSON: 11 Aug 07:25 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_ARTIFACT_DOMAIN: <SET TO MY ORGANIZATIONS MIRROR>
HOMEBREW_CASK_OPTS: []
HOMEBREW_DOCKER_REGISTRY_BASIC_AUTH_TOKEN: set
HOMEBREW_MAKE_JOBS: 10
HOMEBREW_PIP_INDEX_URL: <SET TO MY ORGANIZATIONS MIRROR>
HOMEBREW_SORBET_RUNTIME: set
Homebrew Ruby: 3.3.4 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/3.3.4_1/bin/ruby
CPU: 10-core 64-bit arm_blizzard_avalanche
Clang: 15.0.0 build 1500
Git: 2.46.0 => /opt/homebrew/bin/git
Curl: 8.7.1 => /usr/bin/curl
macOS: 14.6-arm64
CLT: 15.3.0.0.1.1708646388
Xcode: 15.2 => /Applications/Xcode-15.2.0.app/Contents/Developer
Rosetta 2: false

What were you trying to do (and why)?

I'm trying to install Python formulas where they download using my organization's mirror (set via HOMEBREW_PIP_INDEX_URL)

What happened (include all command output)?

brew install ly --build-from-source
==> Fetching ly
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/00b755d9a58976b570a25b006d7c3d4124ec954b/Formula/l/ly.rb
############################################################################################################################################################################################################################################## 100.0%
==> Downloading OUR-INTERNAL-MIRROR/packages/9b/ed/e277509bb9f9376efe391f2f5a27da9840366d12a62bef30f44e5a24e0d9/python-ly-0.9.7.tar.gz
curl: (56) The requested URL returned error: 404

Trying a mirror...
==> Downloading https://files.pythonhosted.org/packages/9b/ed/e277509bb9f9376efe391f2f5a27da9840366d12a62bef30f44e5a24e0d9/python-ly-0.9.7.tar.gz
############################################################################################################################################################################################################################################## 100.0%

What did you expect to happen?

I expected the mirroring to work correctly and not return a 404 and fallback.

I started digging into this and realized that Homebrew's mirror-handling code assumes that https://pypi.org/ and https://files.pythonhosted.org/ are both mirrored by a single URL and are interchangeable. However, the mirror we use (Artifactory) seems to have logic like:

• MIRROR_URL/simple/* --> equivalent to https://pypi.org/simple/*
• MIRROR_URL/packages/* --> equivalent to https://files.pythonhosted.org/*

In other words, the rewrite for a file like https://files.pythonhosted.org/packages/9b/ed/e277509bb9f9376efe391f2f5a27da9840366d12a62bef30f44e5a24e0d9/python-ly-0.9.7.tar.gz should be OUR-INTERNAL-MIRROR/packges/packages/9b/ed/e277509bb9f9376efe391f2f5a27da9840366d12a62bef30f44e5a24e0d9/python-ly-0.9.7.tar.gz Note the duplicated /packages!

So, this comes to the problem with HOMEBREW_PIP_INDEX_URL. As best I can tell, Homebrew never uses this variable for its intended purpose. Instead it uses this variable to then assume that there is a relationship between the PIP_INDEX_URL and where packages are hosted... and uses that assumption to incorrectly rewrite URLs.

I'm now left wondering how to fix this. As a workaround, I can change my HOMEBREW_PIP_INDEX_URL to be MIRROR_URL/packages/. This is, of course, not an acceptable PIP_INDEX_URL but it does accidentally work. It feels like there are a few options:

• do nothing, update the ENV description to be specific that this should be the mirror prefix for files.pythonhosted.org
• deprecate HOMEBREW_PIP_INDEX_URL, add a new environment variable like HOMEBREW_FILES_PYTHONHOSTED_ORG_MIRROR. If it's set, ignore HOMEBREW_PIP_INDEX_URL. If it's not set, keep the current logic of parsing HOMEBREW_PIP_INDEX_URL
• fix so HOMEBREW_PIP_INDEX_URL is used in the update-python-resources and add a HOMEBREW_FILES_PYTHONHOSTED_ORG_MIRROR.

I want to contribute – but just want to know what is the preferred approach from the brew team

Step-by-step reproduction instructions (by running brew commands)

brew cleanup --prune-all # to get a blank cache
brew install ly --build-from-source # could be any Python formula – just needs to include a dependent resource