Vendored pex in pants does not know how to find arm64 compatible packages on mac OS

[chrisjrn]

PyPI supports a new universal2 binary format for Mac OS 11 installs that work on both ARM64 and x86_64 architectures.

Attempting to run pytest on an architecture-aware install of Pants gives an error of the following form:

1.) The wheel tags for setproctitle 1.2.2 are cp39-cp39-macosx_11_0_arm64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/bin/python3.9', PythonIdentity('/usr/local/opt/[email protected]/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 5)))):
cp39-cp39-macosx_11_0_x86_64
cp39-cp39-macosx_11_0_intel
cp39-cp39-macosx_11_0_fat64
cp39-cp39-macosx_11_0_fat32
cp39-cp39-macosx_11_0_universal2
cp39-cp39-macosx_11_0_universal
… snip …

We should be downloading the universal2 package in this case, but we aren't.

I think the problem may be here: this is the relevant vendored copy of sysconfig.py from pex:

https://github.com/pantsbuild/pex/blob/main/pex/vendor/_vendored/pip/pip/_vendor/distlib/_backport/sysconfig.py#L729-L740

and here's the corresponding code from an up-to-date version of the relevant file from Python 3.9:

https://github.com/python/cpython/blob/3.9/Lib/_osx_support.py#L542-L555

[jsirois]

There are 0 Mac specific wheels on PyPI for setproctitle 1.2.2: https://pypi.org/project/setproctitle/1.2.2/#files

So, Pex must be building the sdist into a wheel (via pip wheel) and then later trying to resolve the wheel and failing.

[jsirois]

@chrisjrn to confirm this is a Pex problem and not a Pants problem, does pex --python /usr/local/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/bin/python3.9 setproctitle==1.2.2 fail similarly?

[jsirois]

I have no access to this hardware, even via CI; so I appreciate all the digging you can do to help debug this.

[Eric-Arellano]

I do not reproduce this on my M1: pex --interpreter-constraint='CPython==3.9.*' setproctitle. Note that my Py39 is built via Pyenv in ARM mode, and I've never explicitly enabled Rosetta via the arch command.

I'm wondering if perhaps Chris's Python interpreter being used was an Intel interpreter trying to run in Arm mode, for example. I agree it'll be helpful for Chris to run Pex directly and remove Pants from the equation.

[chrisjrn]

As suspected by @Eric-Arellano, I was able to reproduce this in pex, for an Intel binary from Homebrew (v3.9.5 or v3.9.6). It does not occur on an ARM binary from Homebrew (v3.9.6).

[chrisjrn]

The specific error was:

1.) The wheel tags for setproctitle 1.2.2 are cp39-cp39-macosx_11_0_arm64 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/bin/python3.9', PythonIdentity('/usr/local/opt/[email protected]/bin/python3.9', 'cp39', 'cp39', 'macosx_11_0_x86_64', (3, 9, 6)))):

[jsirois]

OK. I'm pretty sure this is an instance of #1020. Namely, the sequence is:
*1. At build time, the IC picks out an arm64 arch Python and so an arm64 wheel is built and included in the PEX file.
2. At run time, the IC picks out an x86_64 arch Python and so the PEX bootstrap resolve fails.

As outlined in #1020 - Pex currently 1st picks a runtime interpreter using the IC constraint, then tries a resolve, then fails. Instead, it should iterate through all interpreters matching the IC and try each one against a bootstrap resolve picking one from among those that work and onl failing if none do.

*Note that Pants is probably picking out the arm64 interpreter in practice and running Pex build with that. Since Pex always tests the current interpreter for compliance with constraints and uses that when applicable (for perf - skips a re-exec), this is the wedge in which the build time and runtime interpreter selection result could be different on the same machine.

[jsirois]

This is techinically a dup, but I'll leave it open to circle back and get a test from you @chrisjrn or @Eric-Arellano once #1020 is fixed since the whole arm thing is so bespoke right now.

[jsirois]

With the introduction of InterpreterTest in #1770, which confirms an interpreter can resolve the PEX before committing to it, this issue should be resolved. I usually like to test these things out for myself, but without access to a Mac ARM machine, I can't. @chrisjrn if you care to test with modern Pex and find there is still this same issue, please re-open with the test details.