Apple Silicon: cannot import name '_ccallback_c'

[danielbrown2]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

I'm having some issues with the Apple silicon scipy package for my conda-forge feedstock CI and I am struggling to see how to fix it. All the other OSes run fine, including MacOS Intel. Some maybe relevant versions:

scipy:             1.11.3-py310hd1cfc7d_1 
cython:          3.0.5-py310hd5a4765_0

It fails for Python 3.9, 3.10, and 3.11.

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=827833&view=logs&j=15f8dc28-9f6f-5e3a-5b3c-2b169071e5be&t=63fa899e-c160-5185-75c4-c5fd30c90374&l=235

  Traceback (most recent call last):
    File "/Users/runner/miniforge3/conda-bld/finesse_1700561621237/_build_env/venv/lib/python3.10/site-packages/scipy/__init__.py", line 143, in <module>
      from scipy._lib._ccallback import LowLevelCallable
    File "/Users/runner/miniforge3/conda-bld/finesse_1700561621237/_build_env/venv/lib/python3.10/site-packages/scipy/_lib/_ccallback.py", line 1, in <module>
      from . import _ccallback_c
  ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (/Users/runner/miniforge3/conda-bld/finesse_1700561621237/_build_env/venv/lib/python3.10/site-packages/scipy/_lib/__init__.py)

The ImportError: cannot import name '_ccallback_c' looking around online seems to be from an improperly installed package, but considering the CI is installing all of this from scratch I don't see how that applies here.

The conda forge package I'm trying to build is a more complicated one using Cython. I tried lower versions of scipy, 1.11 and 1.10 and Cython 0.29 and 3 but I get the same error.

Any tips on things to try/check would be appreciated!

Installed packages

bzip2:             1.0.8-h93a5062_5           conda-forge
    ca-certificates:   2023.11.17-hf0a4a13_0      conda-forge
    colorama:          0.4.6-pyhd8ed1ab_0         conda-forge
    cython:            3.0.5-py310hd5a4765_0      conda-forge
    gmp:               6.3.0-h965bd2d_0           conda-forge
    libblas:           3.9.0-19_osxarm64_openblas conda-forge
    libcblas:          3.9.0-19_osxarm64_openblas conda-forge
    libcxx:            16.0.6-h4653b0c_0          conda-forge
    libffi:            3.4.2-h3422bc3_5           conda-forge
    libgfortran:       5.0.0-13_2_0_hd922786_1    conda-forge
    libgfortran5:      13.2.0-hf226fd6_1          conda-forge
    liblapack:         3.9.0-19_osxarm64_openblas conda-forge
    libopenblas:       0.3.24-openmp_hd76b1f2_0   conda-forge
    libsqlite:         3.44.0-h091b4b1_0          conda-forge
    libzlib:           1.2.13-h53f4e23_5          conda-forge
    llvm-openmp:       17.0.5-hcd81f8e_0          conda-forge
    metis:             5.1.0-h13dd4ca_1007        conda-forge
    mpfr:              4.2.1-h9546428_0           conda-forge
    ncurses:           6.4-h463b476_2             conda-forge
    bzip2:                  1.0.8-h10d778d_5           conda-forge
    ca-certificates:        2023.11.17-h8857fd0_0      conda-forge
    cctools_osx-64:         973.0.1-ha1c5b94_15        conda-forge
    cctools_osx-arm64:      973.0.1-h7bb7a8e_15        conda-forge
    clang:                  16.0.6-hc177806_2          conda-forge
    clang-16:               16.0.6-default_h762fdd7_2  conda-forge
    clang_impl_osx-arm64:   16.0.6-ha81d56d_6          conda-forge
    clang_osx-arm64:        16.0.6-h40be2b1_6          conda-forge
    clangxx:                16.0.6-default_h762fdd7_2  conda-forge
    compiler-rt:            16.0.6-he1888fc_1          conda-forge
    compiler-rt_osx-64:     16.0.6-he1888fc_1          conda-forge
    cross-python_osx-arm64: 3.10-41_cpython            conda-forge
    crossenv:               1.4.0-pyhd8ed1ab_0         conda-forge
    cython:                 3.0.5-py310had63691_0      conda-forge
    gettext:                0.21.1-h8a4c099_0          conda-forge
    icu:                    73.2-hf5e326d_0            conda-forge
    ld64_osx-64:            609-ha20a434_15            conda-forge
    ld64_osx-arm64:         609-h6aa3cc8_15            conda-forge
    libblas:                3.9.0-19_osx64_openblas    conda-forge
    libcblas:               3.9.0-19_osx64_openblas    conda-forge
    libclang-cpp16:         16.0.6-default_h762fdd7_2  conda-forge
    libcxx:                 16.0.6-hd57cbcb_0          conda-forge
    libffi:                 3.4.2-h0d85af4_5           conda-forge
    libgfortran:            5.0.0-13_2_0_h97931a8_1    conda-forge
    libgfortran5:           13.2.0-h2873a65_1          conda-forge
    libiconv:               1.17-hac89ed1_0            conda-forge
    liblapack:              3.9.0-19_osx64_openblas    conda-forge
    libllvm16:              16.0.6-he4b1e75_2          conda-forge
    libopenblas:            0.3.24-openmp_h48a4ad5_0   conda-forge
    libsqlite:              3.44.0-h92b6c6a_0          conda-forge
    libxml2:                2.11.6-hc0ae0f7_0          conda-forge

Environment info

Not sure how to get this from CI pipeline

[rgommers]

Something is indeed going wrong there, but that particular build of scipy 1.11.3 is over a month old and we've had no bug reports about it being broken. So it most likely is related to something else installed in the build env.

[h-vetinari]

Can you please try with cython <3? Scipy 1.11 is not yet ready for Cython 3, and our 1.11.3 builds were done with 0.29.x as well.

[danielbrown2]

What's confusing me is that our main project CI running Apple silicon macos runners on Gitlab SaaS seem ok. All the build and tests pass for making a wheel which is done with conda-forge packages:

https://gitlab.com/ifosim/finesse/finesse3/-/jobs/5577026130

  + cython                             3.0.5  py310hd5a4765_0          conda-forge/osx-arm64        3MB
  + scipy                                1.11.3  py310hd1cfc7d_1          conda-forge/osx-arm64       14MB

which are the same versions I mention in the first post.

[danielbrown2]

Limiting it to less than Cython 3 still results in the same error:

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=830153&view=logs&j=15f8dc28-9f6f-5e3a-5b3c-2b169071e5be&t=63fa899e-c160-5185-75c4-c5fd30c90374

[danielbrown2]

Which also has now started using Scipy 1.11.4-py310h2b794db_0. This isn't feeling like a scipy package issue to me...

[danielbrown2]

I noticed the numpy version was different, our CI was running 1.26, but fixing that to 1.22.4 works fine with this same scipy package.