distutils: pass -rpath on macOS when requested

[acolinisi]

Resurrecting a patch to an old issue that is still relevant: distutils does not set the runtime search path when linking the shared object on macOS, passing a -L argument instead of an -rpath argument, like on the other systems. The code in question states that OS X linker doesn't accept -rpath, but that code was written 19 years ago, and that comment is no longer true (certainly not on macOS 11.2, but I do not know what is the newest system version for which -rpath is not accepted, if any).

The runtime_library_dirs configuration option is vital for extensions that need to build against libraries that temporarily exist in a location different from their final installation location (such as during the build of a package for a distro).

Submitted to setuptools, but got instructions to submit to distutils instead for later merging to setuptools:
pypa/setuptools#2627

Fix -R option of build_ext for macOS (darwin)

Resolves this old bug against distutils that expired due to PIP 632:
https://bugs.python.org/issue36353

Applies patch originally submitted to CPython:
python/cpython#12418

[jmroot]

I do not know what is the newest system version for which -rpath is not accepted, if any

10.5 and later support @rpath. So MACOSX_DEPLOYMENT_TARGET should be checked before using this, given that cpython can still target Mac OS X versions older than 10.3.

[acolinisi]

Pushed check of MACOSX_DEPLOYMENT_TARGET.

Yeah, I didn't know about this variable. Setting this to an older version and then attempting to pass -rpath, we get the version info we needed:

ld: -rpath can only be used when targeting Mac OS X 10.5 or later

[jmroot]

Be aware that if #36 is merged, the value in the environment at runtime will be relevant, as it overrides sysconfig if present, and will no longer be restricted to being >= the sysconfig value.

[acolinisi]

So, you want me to get the value from os.environ instead? Or, from os.environ with fallback to sysconfig?

Btw, that mismatch error was triggering for me when I was trying to test this by overriding the env var. So, where does sysconfig get the value from? It's not from environment -- "hardcoded" per system? So, testing this PR on a 11.2 by setting the env var to 10.3 won't work, correct?

[jmroot]

If #36 is merged, the effective target would need to be discovered here the same way that spawn.py does it, i.e.

cur_target = os.environ.get('MACOSX_DEPLOYMENT_TARGET', _cfg_target)

[jmroot]

So, where does sysconfig get the value from? It's not from environment -- "hardcoded" per system? So, testing this PR on a 11.2 by setting the env var to 10.3 won't work, correct?

The value is recorded into sysconfig when building Python. So yes, using an older target currently doesn't work unless you also build Python with that same target or even older.

[acolinisi]

If #36 is merged, the effective target would need to be discovered here the same way that spawn.py does it, i.e.

cur_target = os.environ.get('MACOSX_DEPLOYMENT_TARGET', _cfg_target)

Ok, and _cfg_target should be gotten again in unixccompiler.py with

            _cfg_target = sysconfig.get_config_var(
                                  'MACOSX_DEPLOYMENT_TARGET') or ''

or is there some way to get it directly from spawn.py without duplicating this piece?

[jmroot]

I guess a common helper function might make sense.

[acolinisi]

Ok, then, we'll have to block on #36. I'll revisit once that is merged/notmerged.

[jaraco]

#36 is now merged. Also, the main branch now runs the tests in CI. If you merge/rebase with main, you'll see there's a test failure with this patch. Please have a look.

[acolinisi]

Rebased, factored into helper function used from the two places (as discussed earlier in this PR), updated and expanded the test case.

There is only one behavior change that I deliberately introduced: before this PR, if sysconfig did not return a version, then the MACOSX_DEPLOYMENT_TARGET env var would be ignored; after this PR, the env var is honored regardless of whether or not sysconfig returns a value (the validation check of version from env var for compatibility with version from sysconfig still happens only when sysconfig returned a version, of course). So, this change only matters for systems where sysconfig does not return a value (there may be no such systems), on all other normal systems, the behavior is entirely unchanged by the refactoring changes (sans the new -Wl,-rpath change).

Tested on macOS 11 on a real extension, with various combinations of the env var value.

[acolinisi]

Suggested changes to the patch (renames) have been pushed.