Delocate error when building Pillow on macOS

[freakboy3742]

What did you do?

When trying to build macOS wheels with cibuildwheel on a personal machine, an error is raised during the delocate step.

What did you expect to happen?

A cibuildwheel build should successfully complete.

What actually happened?

$ CIBW_ARCHS=arm64 cibuildwheel
...
(logs of build of dependencies and the unrepaired Pillow wheel redacted)
...
Repairing wheel...

+ delocate-wheel --require-archs arm64 -w /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/repaired_wheel -v /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/built_wheel/pillow-11.2.0.dev0-cp39-cp39-macosx_11_0_arm64.whl
Fixing: /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/built_wheel/pillow-11.2.0.dev0-cp39-cp39-macosx_11_0_arm64.whl
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libfreetype.6.dylib
  Search path:
    
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libpng16.16.dylib
  Search path:
    
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libpng16.16.dylib
  Search path:
    
ERROR:delocate.libsana:@rpath/libz.1.dylib not found, requested by /Users/rkm/projects/pillow/build/deps/darwin/lib/libpng16.16.dylib
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libfreetype.6.dylib
  Search path:
    
ERROR:delocate.libsana:@rpath/libz.1.dylib not found, requested by /Users/rkm/projects/pillow/build/deps/darwin/lib/libfreetype.6.dylib
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/tmp8h0w3b6k/wheel/PIL/_imaging.cpython-39-darwin.so
  Search path:
    
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libtiff.6.dylib
  Search path:
    
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /Users/rkm/projects/pillow/build/deps/darwin/lib/libtiff.6.dylib
  Search path:
    
ERROR:delocate.libsana:@rpath/libz.1.dylib not found, requested by /Users/rkm/projects/pillow/build/deps/darwin/lib/libtiff.6.dylib
ERROR:delocate.libsana:
@rpath/libz.1.dylib not found:
  Needed by: /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/tmp8h0w3b6k/wheel/PIL/_imaging.cpython-39-darwin.so
  Search path:
    
ERROR:delocate.libsana:@rpath/libz.1.dylib not found, requested by /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/tmp8h0w3b6k/wheel/PIL/_imaging.cpython-39-darwin.so
Traceback (most recent call last):
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/bin/delocate-wheel", line 8, in <module>
    sys.exit(main())
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/cmd/delocate_wheel.py", line 116, in main
    copied = delocate_wheel(
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/delocating.py", line 1057, in delocate_wheel
    copied_libs = delocate_path(
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/delocating.py", line 511, in delocate_path
    lib_dict = tree_libs_from_directory(
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/libsana.py", line 377, in tree_libs_from_directory
    return _tree_libs_from_libraries(
  File "/private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/build/venv/lib/python3.9/site-packages/delocate/libsana.py", line 320, in _tree_libs_from_libraries
    raise DelocationError("Could not find all dependencies.")
delocate.libsana.DelocationError: Could not find all dependencies.

                                                              ✕ 5.22s
Error: Command delocate-wheel --require-archs arm64 -w /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/repaired_wheel -v /private/var/folders/b_/khqk71xd45d049kxc_59ltp80000gn/T/cibw-run-_1yd3ael/cp39-macosx_arm64/built_wheel/pillow-11.2.0.dev0-cp39-cp39-macosx_11_0_arm64.whl failed with code 1. 

What are your OS, Python and Pillow versions?

• OS: macOS 15.2, 14.7
• Python: 3.9, 3.12
• Pillow: 11.1+; main (github hash 2486361)

Additional details

The problem appears to have been caused by the introduction of zlib-ng. On my 2 test machines, the compiled version of this library has an install_name of @rpath/libz.1.dylib. This is fine, but it requires that DYLD_LIBRARY_PATH includes the build dependencies path.

When cibuildwheel invokes delocate-wheel, DYLD_LIBRARY_PATH is set to include the build dependencies lib directory. However, delocate-wheel is invoked with shell(), and macOS does not pass DYLD_LIBRARY_PATH to child subprocesses.

All the other binary dependencies appear to be compiled with absolute paths in the install_name.

This clearly isn't an issue with CI, as wheel builds are succeeding there. However, I can't work out why they are succeeding. I can only assume that there is something on the CI machine that puts a libz.1.dylib compatible library on the linking path which is able to satisfy delocate.

I've been able to work around the issue locally by manually rewriting the install name of libz.dylib after the build-zlib-ng completes (there doesn't appear to be a compilation option to explicitly guarantee an absolute install_name). However, it's not clear to me if I've missed something obvious in my setup, or if this is an issue that others are likely to see.

[radarhere]

DYLD_LIBRARY_PATH was indeed added to pyproject.toml in #8500, and if I remove DYLD_LIBRARY_PATH from pyproject.toml, I see the error that you're reporting - https://github.com/radarhere/Pillow/actions/runs/12664298313/job/35292210395#step:5:8924

I wonder if you're hitting https://cibuildwheel.pypa.io/en/stable/faq/#macos-passing-dyld_library_path-to-delocate, and the solution is radarhere@27dd70a

diff --git a/pyproject.toml b/pyproject.toml
index 2c6c7bcd0..4b9bdd5d6 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -96,15 +96,17 @@ before-all = ".github/workflows/wheels-dependencies.sh"
 build-verbosity = 1
 
 config-settings = "raqm=enable raqm=vendor fribidi=vendor imagequant=disable"
-# Disable platform guessing on macOS
-macos.config-settings = "raqm=enable raqm=vendor fribidi=vendor imagequant=disable platform-guessing=disable"
 
 test-command = "cd {project} && .github/workflows/wheels-test.sh"
 test-extras = "tests"
 
+[tool.cibuildwheel.macos]
+repair-wheel-command = "DYLD_LIBRARY_PATH=$(pwd)/build/deps/darwin/lib delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"
+# Disable platform guessing on macOS
+config-settings = "raqm=enable raqm=vendor fribidi=vendor imagequant=disable platform-guessing=disable"
+
 [tool.cibuildwheel.macos.environment]
 PATH = "$(pwd)/build/deps/darwin/bin:$(dirname $(which python3)):/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin"
-DYLD_LIBRARY_PATH = "$(pwd)/build/deps/darwin/lib"
 
 [tool.black]
 exclude = "wheels/multibuild"

[freakboy3742]

@radarhere That does appear to the cluster of issues that I'm hitting.

I didn't know that GitHub Actions had disabled SIP, but there's some suggestion that it has been disabled on the GitHub Actions images as of macOS-13. That would explain why this problem isn't occurring in CI.

It surprises me a little that I'm the first person to hit this - but given the zlib_ng patch doesn't have any macOS-specific pieces, and the CI config works (producing viable wheels), I might just be very lucky in being the first person to try rebuilding the whole of Pillow on macOS since the zlib-ng PR landed.

Regardless - As for a fix - the approach you've described would work - this is the other possible approach:

diff --git a/.github/workflows/wheels-dependencies.sh b/.github/workflows/wheels-dependencies.sh
index d721c750c..3c9cc20f9 100755
--- a/.github/workflows/wheels-dependencies.sh
+++ b/.github/workflows/wheels-dependencies.sh
@@ -139,6 +139,14 @@ function build_zlib_ng {
         && ./configure --prefix=$BUILD_PREFIX --zlib-compat \
         && make -j4 \
         && make install)
+
+    if [ -n "$IS_MACOS" ]; then
+        # Ensure that on macOS, the library name is an absolute path, not an
+        # @rpath, so that delocate picks up the right library (and doesn't need
+        # DYLD_LIBRARY_PATH to be set). The default Makefile doesn't have an
+        # option to control the install_name.
+        install_name_tool -id $BUILD_PREFIX/lib/libz.1.dylib $BUILD_PREFIX/lib/libz.1.dylib
+    fi
     touch zlib-stamp
 }
 

The argument I'd make for this approach is that it doesn't require redefining "default" behavior of cibuildwheel. However, I could go either way. If you'd like me to turn this diff into a PR, let me know.

[freakboy3742]

@radarhere I couldn't help myself - see #8673 - but it looks like you've gone the explicit route with #8672.