Gotchas when installing on MacOS #106
Comments
|
Is there a way we could activate openmp support at runtime? so that it's compiled with it, but if not present at runtime it issues a warning and runs without it? |
|
In bjlittle/geovista#620 the ImportError: dlopen(/.../venv/lib/python3.12/site-packages/pykdtree/kdtree.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace '___kmpc_for_static_fini'`into a much more helpful error message which may point the user to a sensible solution? (e.g. This way, although the runtime error is still present, at least the user can easily try one of the two solutions hinted from the error message, and resolve this issue very quickly and easily. Perhaps this isn't ideal, but at least the user will still be able to take advantage of the performance improvements with |
|
I had a similar thought. I was wondering if it would be possible at build time to build both versions of the extensions. We always build the one without OpenMP, but if we detect OpenMP we build that version too and have two possible extensions to import. Then we try/except on the OpenMP extension import and warn and fallback if we have to use the non-OpenMP version. If the OpenMP extension file(s) don't even exist then we know that the build process wasn't even able to find OpenMP so there is no reason to warn in that case...I think. |
|
So I played around with my idea mentioned above with having two extensions, one compiled with OpenMP or at least attempted and one explicitly not including OpenMP. It all kind of worked except it seemed that Cython only compiled the cython module for a specific name once so when trying to import the second no-OpenMP version it gets confused by what the module name is. Subject: [PATCH] Dual extensions with and without openmp
---
Index: setup.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/setup.py b/setup.py
--- a/setup.py (revision 75804c99cc873ccc74b19e47c3a90003fd6fce4b)
+++ b/setup.py (date 1705517886814)
@@ -20,8 +20,9 @@
import re
import numpy as np
-from Cython.Build import cythonize
-from setuptools import setup, Extension
+from Cython.Build import build_ext
+from Cython.Distutils import Extension
+from setuptools import setup
from setuptools.command.build_ext import build_ext
@@ -56,17 +57,20 @@
comp = self.compiler.compiler_type
omp_comp, omp_link = _omp_compile_link_args(comp)
if comp in ('unix', 'cygwin', 'mingw32'):
- extra_compile_args = ['-std=c99', '-O3'] + omp_comp
+ extra_compile_args = ['-std=c99', '-O3']
extra_link_args = omp_link
elif comp == 'msvc':
- extra_compile_args = ['/Ox'] + omp_comp
+ extra_compile_args = ['/Ox']
extra_link_args = omp_link
else:
# Add support for more compilers here
raise ValueError('Compiler flags undefined for %s. Please modify setup.py and add compiler flags'
% comp)
- self.extensions[0].extra_compile_args = extra_compile_args
+ # Only apply OpenMP specific flags to OpenMP
+ self.extensions[0].extra_compile_args = extra_compile_args + omp_comp
self.extensions[0].extra_link_args = extra_link_args
+ # No OpenMP version of the extension
+ self.extensions[1].extra_compile_args = extra_compile_args
build_ext.build_extensions(self)
@@ -186,12 +190,19 @@
readme = readme_file.read()
extensions = [
- Extension('pykdtree.kdtree', sources=['pykdtree/kdtree.pyx', 'pykdtree/_kdtree_core.c'],
+ Extension('pykdtree.kdtree_tryomp', sources=['pykdtree/kdtree.pyx', 'pykdtree/_kdtree_core.c'],
+ include_dirs=[np.get_include()],
+ define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
+ cython_directives={"language_level": "3"},
+ ),
+ # NOTE: Compilation flags handled in build_ext_subclass (assumes second extension is noomp)
+ Extension('pykdtree.kdtree_noomp', sources=['pykdtree/kdtree.pyx', 'pykdtree/_kdtree_core.c'],
include_dirs=[np.get_include()],
define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
- compiler_directions={"language_level": "3"},
+ cython_directives={"language_level": "3"},
),
]
+
setup(
name='pykdtree',
@@ -206,7 +217,7 @@
install_requires=['numpy'],
tests_require=['pytest'],
zip_safe=False,
- ext_modules=cythonize(extensions),
+ ext_modules=extensions,
cmdclass={'build_ext': build_ext_subclass},
classifiers=[
'Development Status :: 5 - Production/Stable',
Index: pykdtree/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/pykdtree/__init__.py b/pykdtree/__init__.py
--- a/pykdtree/__init__.py (revision 75804c99cc873ccc74b19e47c3a90003fd6fce4b)
+++ b/pykdtree/__init__.py (date 1705518033734)
@@ -1,0 +1,15 @@
+"""Simple wrapper around importing pykdtree for OpenMP use or not."""
+
+try:
+ from . import kdtree_tryomp as kdtree
+ raise ImportError("pykdtree")
+except ImportError:
+ import warnings
+ warnings.warn(
+ """Pykdtree failed to import its C extension. This usually means it
+was built with OpenMP (C-level parallelization library) support but could not
+find it on your system. Pykdtree will use a less performant version of the
+algorithm instead. To enable better performance OpenMP must be installed
+(ex. ``brew install omp`` on Mac with HomeBrew)."""
+ )
+ from . import kdtree_noomp as kdtree
\ No newline at end of file
$ python -c "from pykdtree import kdtree"
/home/davidh/repos/git/pykdtree/pykdtree/__init__.py:8: UserWarning: Pykdtree failed to import it's C extension. This usually means it
was built with OpenMP (C-level parellelization library) support but could not
find it on your system. Pykdtree will use a less performant version of the
algorithm instead. To enable better performance OpenMP must be installed
(ex. ``brew install omp`` on Mac with HomeBrew).
warnings.warn(
Traceback (most recent call last):
File "/home/davidh/repos/git/pykdtree/pykdtree/__init__.py", line 5, in <module>
raise ImportError("pykdtree")
ImportError: pykdtree
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/davidh/repos/git/pykdtree/pykdtree/__init__.py", line 15, in <module>
from . import kdtree_noomp as kdtree
ImportError: dynamic module does not define module export function (PyInit_kdtree_noomp) |
|
FYI: I ended up following these instructions. Copied here for convenience:
The above didn't work for me. I also tried adding Any ideas? Edit: I only get the |
There is a fourth option that you could consider:
Obviously, that also works in reverse: |
The CI and wheel building process had to change between these versions because GitHub deprecated their old mac runners and if I remember correctly homebrew bumped their minimum build target to only work for newer versions of MacOS. What version of MacOS are you using now? |
Ugh I can't believe I got that wrong. Would you mind submitting a pull request to fix the typo? |
I'm on ARM and OSX
Sure, can do! I've also had a look at the build logs for version Installing previous versions of Sorry, I know this is a pain in the neck to debug, especially if you're not even on the same system, but it would be good to incorporate any findings into the error message, right? |
|
Hm. So I don't think you ever specified what error you are currently getting. What's the message and traceback (if there is one)? When pykdtree is built and turned into a wheel, it builds with whatever OpenMP it can find. The pykdtree As for PATH, pykdtree or rather python itself is using whatever dynamic library linking is available on Mac. The "hack" way of manipulating this at run time is to specify |
|
Sorry, should have shared it in my first post: I get the >>> from pykdtree.kdtree import kdtree
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
File ~/.pyenv/versions/3.11.8/lib/python3.11/site-packages/pykdtree/__init__.py:4
3 try:
----> 4 from . import kdtree
5 except ImportError as err:
ImportError: dlopen(/Users/philipps/.pyenv/versions/3.11.8/lib/python3.11/site-packages/pykdtree/kdtree.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '___kmpc_for_static_fini'
The above exception was the direct cause of the following exception:
ImportError Traceback (most recent call last)
Cell In[1], line 1
----> 1 from pykdtree.kdtree import kdtree
File ~/.pyenv/versions/3.11.8/lib/python3.11/site-packages/pykdtree/__init__.py:6
4 from . import kdtree
5 except ImportError as err:
----> 6 raise ImportError(
7 "Pykdtree failed to import its C extension. This usually means it "
8 "was built with OpenMP (C-level parallelization library) support but "
9 "could not find it on your system. To enable better performance "
10 "OpenMP must be installed (ex. ``brew install omp`` on Mac with "
11 "HomeBrew). Otherwise, try installing Pykdtree from source (ex. "
12 "``pip install --no-binary pykdtree --force-install pykdtree``)."
13 ) from err
ImportError: Pykdtree failed to import its C extension. This usually means it was built with OpenMP (C-level parallelization library) support but could not find it on your system. To enable better performance OpenMP must be installed (ex. ``brew install omp`` on Mac with HomeBrew). Otherwise, try installing Pykdtree from source (ex. ``pip install --no-binary pykdtree --force-install pykdtree``).I just quickly tried setting In case that's helpful: $ brew ls --verbose libomp
/opt/homebrew/Cellar/libomp/19.1.0/INSTALL_RECEIPT.json
/opt/homebrew/Cellar/libomp/19.1.0/.brew/libomp.rb
/opt/homebrew/Cellar/libomp/19.1.0/include/ompx.h
/opt/homebrew/Cellar/libomp/19.1.0/include/ompt.h
/opt/homebrew/Cellar/libomp/19.1.0/include/omp.h
/opt/homebrew/Cellar/libomp/19.1.0/include/omp-tools.h
/opt/homebrew/Cellar/libomp/19.1.0/sbom.spdx.json
/opt/homebrew/Cellar/libomp/19.1.0/lib/libomp.dylib
/opt/homebrew/Cellar/libomp/19.1.0/lib/libomp.a |
|
This must be an OpenMP version thing. It isn't complaining that it can't find libopenmp, but a specific symbol ( A stackoverflow answer suggests you can list all symbols for a library on mac with: Does that return anything? |
|
|
A fourth option is to copy the libomp.dylib from the build environment into the distributed binary wheel and ensure that it gets used, since libomp is not a system feature. A good basic intro to some of the extended rpath linking on macOS is at https://itwenty.me/posts/01-understanding-rpath/ Something like this might be what's needed to pull the dependency into the wheel and situate it next to the compiled module: Repos/git/pykdtree master ✗
▶ file /opt/local/lib/libomp/libomp.dylib
/opt/local/lib/libomp/libomp.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|WEAK_DEFINES|BINDS_TO_WEAK|NO_REEXPORTED_DYLIBS|HAS_TLV_DESCRIPTORS>
Repos/git/pykdtree master ✗
▶ otool -L build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so
build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so:
/opt/local/lib/libomp/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
Repos/git/pykdtree master ✗
▶ cp /opt/local/lib/libomp/libomp.dylib build/lib.macosx-15.0-arm64-cpython-311/pykdtree/
Repos/git/pykdtree master ✗
▶ install_name_tool -change /opt/local/lib/libomp/libomp.dylib @loader_path/libomp.dylib build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so
Repos/git/pykdtree master ✗
▶ otool -L build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so
build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so:
@loader_path/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)Update: I did confirm that this results in a co-resident libomp that gets used: |
|
I read this and thought "oh I bet I could mimic what the pyproj library does since I know that bundles the PROJ library in wheels but still prefers system PROJ if available". But it looks complicated enough that I'm scared, especially for a lower-level library like OpenMP. For reference here is pyproj's wheel building job: And the script it is calling before wheel building to build PROJ inside the source directory: So I'm not sure I understand what would be a good next step. I guess I (only one maintainer of this package) am OK bundling OpenMP but I definitely don't have the time to figure it out. I'd also be OK doing another release of pykdtree so it builds with the newer OpenMP on homebrew and would hopefully work for more people (we assume). Just to be sure, @rayg-ssec if you have any time, could you try installing pykdtree from PyPI with pip with old (assuming you haven't updated homebrew) and new (assuming you're OK upgrading) OpenMP from homebrew and see if you get the same errors as @schlegelp? I wonder if the OpenMP formula/recipe in homebrew changed enough that we need to somehow get pykdtree/wheel building to be smarter about where to look and what RPATH to use to find OpenMP. Like is there a generic non-versioned path to the latest install of OpenMP that homebrew creates that we could use instead of the versioned one? Lastly, I'm nervous that if we bundle OpenMP and pykdtree starts sourcing it over any system OpenMPs then we may break much more "important" libraries that use OpenMP (ex. numba) because pykdtree is imported first. |
|
Complication: I'm using macports, not homebrew. In its case, the paths for (currently version 18) libomp are: |
|
It looks like R project solved this by maintaining a set of libomp signed libraries intended to match the version of the system clang, to be shipped with binaries. https://mac.r-project.org/openmp/ . More recent ones of universal binaries (i.e. usable on arm64 and x86-64 builds). |
|
@schlegelp Could you run the equivalent of this command with your installed pykdtree and tell us what the output is? Edit: As another check, in your code, is pykdtree the first thing you are importing? If not, can you try importing it first and see what happens. |
|
Hi. Sorry for the radio silence - I've had a few busy days. For these tests I always import with $ otool -L <...>/lib/python3.11/site-packages/pykdtree/kdtree.cpython-311-darwin.so
<...>/lib/python3.11/site-packages/pykdtree/kdtree.cpython-311-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)with $ otool -L <...>/lib/python3.11/site-packages/pykdtree/kdtree.cpython-311-darwin.so
<...>/lib/python3.11/site-packages/pykdtree/kdtree.cpython-311-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
|
|
@schlegelp That output is a little surprising. Notice how @rayg-ssec's output mentions OpenMP and yours doesn't. @rayg-ssec I'm unfamiliar with otool output, but shouldn't it at least list openmp even if it can't find it? |
|
The pykdtree/build/lib.macosx-15.0-arm64-cpython-311 master ✗
▶ otool -L pykdtree/kdtree.cpython-311-darwin.so
pykdtree/kdtree.cpython-311-darwin.so:
/usr/local/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1351.0.0)
pykdtree/build/lib.macosx-15.0-arm64-cpython-311 master ✗
▶ PYTHONPATH=$PWD python3.11 -m pykdtree.test_tree
Traceback (most recent call last):
File "/Users/keoni/Repos/git/pykdtree/build/lib.macosx-15.0-arm64-cpython-311/pykdtree/__init__.py", line 4, in <module>
from . import kdtree
ImportError: dlopen(/Users/keoni/Repos/git/pykdtree/build/lib.macosx-15.0-arm64-cpython-311/pykdtree/kdtree.cpython-311-darwin.so, 0x0002): Library not loaded: /usr/local/lib/libomp.dylib
---8<---Basically it looks like the module being probed does not have any OMP dependency at all to be satisfied. So now the question is whether there's another preferred pykdtree module somewhere in the python path that has an unsatisfied OMP dependency. One of the easiest ways to cause that problem might be via "pip install --user" adding to ~/.local , instead of using a virtualenv. I've seen too many cases of collateral damage from |
|
Not sure if that's actually what you're suggesting @rayg-ssec but I did double check and there is no additional Perhaps as another data point: if I build pykdtree myself using pykdtree/kdtree.cpython-311-darwin.so:
/opt/homebrew/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)After building with pykdtree/kdtree.cpython-311-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2) |
|
Given the above, now I'm wondering if the module is being built with OMP dependencies/header usage but not linked to libomp, resulting in symbol resolution problem - though that shouldn't pass a CI import test presumably? |
|
@rayg-ssec would it make sense to download any of the macos wheels here on PyPI and run otool on the |
|
Done using the python 3.13 wheel: ~/tmp
▶ unzip ~/Downloads/pykdtree-1.3.13-cp313-cp313-macosx_12_0_arm64.whl
Archive: /Users/keoni/Downloads/pykdtree-1.3.13-cp313-cp313-macosx_12_0_arm64.whl
creating: pykdtree/
creating: pykdtree-1.3.13.dist-info/
inflating: pykdtree/test_tree.py
inflating: pykdtree/__init__.py
inflating: pykdtree/kdtree.cpython-313-darwin.so
inflating: pykdtree/render_template.py
inflating: pykdtree-1.3.13.dist-info/RECORD
inflating: pykdtree-1.3.13.dist-info/WHEEL
inflating: pykdtree-1.3.13.dist-info/top_level.txt
inflating: pykdtree-1.3.13.dist-info/LICENSE.txt
inflating: pykdtree-1.3.13.dist-info/METADATA
~/tmp
▶ PYTHONPATH=$PWD python3.13
Python 3.13.0 (main, Oct 12 2024, 04:39:52) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pykdtree
Traceback (most recent call last):
File "/Users/keoni/tmp/pykdtree/__init__.py", line 4, in <module>
from . import kdtree
ImportError: dlopen(/Users/keoni/tmp/pykdtree/kdtree.cpython-313-darwin.so, 0x0002): symbol not found in flat namespace '___kmpc_for_static_fini'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<python-input-0>", line 1, in <module>
import pykdtree
File "/Users/keoni/tmp/pykdtree/__init__.py", line 6, in <module>
raise ImportError(
...<6 lines>...
) from err
ImportError: Pykdtree failed to import its C extension. This usually means it was built with OpenMP (C-level parallelization library) support but could not find it on your system. To enable better performance OpenMP must be installed (ex. ``brew install omp`` on Mac with HomeBrew). Otherwise, try installing Pykdtree from source (ex. ``pip install --no-binary pykdtree --force-install pykdtree``).
>>>
~/tmp
▶ otool -L pykdtree/kdtree.cpython-313-darwin.so
pykdtree/kdtree.cpython-313-darwin.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)
~/tmp
▶ nm -gU pykdtree/kdtree.cpython-313-darwin.so
0000000000005598 T _PyInit_kdtree
000000000001cb18 S ___pyx_module_is_main_pykdtree__kdtree
00000000000049e0 T _calc_dist_double
0000000000003250 T _calc_dist_float
0000000000004298 T _construct_subtree_double
0000000000002b04 T _construct_subtree_float
0000000000004458 T _construct_tree_double
0000000000002cc4 T _construct_tree_float
0000000000004410 T _create_node_double
0000000000002c7c T _create_node_float
00000000000048c8 T _delete_subtree_double
0000000000003134 T _delete_subtree_float
0000000000004904 T _delete_tree_double
0000000000003170 T _delete_tree_float
0000000000003f64 T _get_bounding_box_double
00000000000027d0 T _get_bounding_box_float
0000000000004a78 T _get_cube_offset_double
00000000000032fc T _get_cube_offset_float
0000000000004ab4 T _get_min_dist_double
0000000000003338 T _get_min_dist_float
0000000000003f08 T _insert_point_double
000000000000274c T _insert_point_float
0000000000004094 T _partition_double
0000000000002900 T _partition_float
000000000000493c T _print_tree_double
00000000000031a8 T _print_tree_float
0000000000004b1c T _search_leaf_double
0000000000004db4 T _search_leaf_double_mask
00000000000033a0 T _search_leaf_float
00000000000036b0 T _search_leaf_float_mask
000000000000505c T _search_splitnode_double
00000000000039c8 T _search_splitnode_float
0000000000005298 T _search_tree_double
0000000000003c04 T _search_tree_float[edit: Removed some URL artifacts due to rich text pasting from my logbook] |
|
How can it be looking for/using OpenMP symbols (error message), but not be linked against it (otool)? We could try adding an import test to the CI, but I'm still not sure how this happens. |
|
The simplest way I think (read: venture a guess) this could be made to happen is if the |
Modern versions of pykdtree are released on PyPI as both binary wheels and a source distribution (sdist). Pykdtree includes C extensions that require compilation if installed from the sdist, but if installed from the binary wheel then no C compiler is required on the user's machine as the shared libraries (ex.
.sofiles on linux) are bundled with the wheel. Pykdtree also optionally depends on OpenMP. If the extensions are built on a system with OpenMP installed then the extensions are linked to and therefore require OpenMP to exist at runtime. This means if pykdtree's wheels on PyPI were built with OpenMP then any user installing the wheels must also have OpenMP installed or they will get an error when importing pykdtree.This gets complicated in the case of MacOS where the build image used in pykdtree's CI environment includes OpenMP, but it only exists because it is a dependency of one of the other tools/libraries that GitHub Actions includes (via homebrew) in their runner image. Since this isn't a "standard" package when users do
pip install pykdtreeand get the wheel built with OpenMP, they get an error at import time. See bjlittle/geovista#620 for one of these cases.The workarounds are to:
brew install ompI think is the command - I don't have a mac)pip install --force-reinstall --no-binary pykdtree pykdtree)So our (as pykdtree maintainers) options are:
We could (and should) of course document all of these gotchas.
The text was updated successfully, but these errors were encountered: