Skip to content

Conversation

@abathur
Copy link
Member

@abathur abathur commented Jan 12, 2024

In the course of testing out the fix that ultimately landed in #278945, I ran into the following missing-function error in cffi's test suite:
Executing pytestCheckPhase
============================= test session starts ==============================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
rootdir: /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0
collecting ... ^Mcollecting 229 items                                                           ^Mcollecting 641 items                                                           ^Mcollecting 872 items 

==================================== ERRORS ====================================
_____________ ERROR collecting testing/cffi1/test_parse_c_type.py ______________
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/nix/store/riwkg7zfhj0fzn918s8yiblnvyp0vcmd-python3-3.11.7/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
testing/cffi1/test_parse_c_type.py:75: in <module>
    @ffi.callback("int(unsigned long long *)")
src/cffi/api.py:396: in callback_decorator_wrap
    return self._backend.callback(cdecl, python_callable,
E   SystemError: ffi_prep_closure_loc() is missing
------------------------------- Captured stderr --------------------------------
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
=============================== warnings summary ===============================
src/cffi/_imp_emulation.py:4
  /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0/src/c/../cffi/_imp_emulation.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in P
    from imp import *

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR testing/cffi1/test_parse_c_type.py - SystemError: ffi_prep_closure_loc() is missing
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 1 error in 7.96s ==========================

The missing function is in turn caused by these defines: https://github.com/python-cffi/cffi/blame/feaff378873d8a3f52bbb63f7d5e24f9ab18f1e9/src/c/_cffi_backend.c#L106-L113

If I understand correctly (I may not), cffi is assuming the system libffi here, which doesn't apply in our case. (cc @reckenrode)

(The underlying defines (which affect macOS before 10.15) have been in cffi for 4+ years now, so I'm not quite sure how we haven't picked up on it already.)

Description of changes

I edited the patch for this behavior (pkgs/development/python-modules/cffi/darwin-use-libffi-closures.diff) to remove the version check.

I went ahead and did this for all 3 checks, even though I suspect we didn't technically need all of them to pass the test failure above.

Things done

I was able to successfully build and use nix with this patch present.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

If I understand correctly (I may not), cffi is assuming the system
libffi here, which doesn't apply in our case.

The underlying defines (which affect macOS before 10.15) have been in
cffi for 4+ years now, so I'm not quite sure how we haven't picked up
on it already.
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jan 12, 2024
@ofborg ofborg bot added 6.topic: darwin Running or building packages on Darwin 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. labels Jan 12, 2024
@abathur abathur marked this pull request as ready for review January 19, 2024 03:47
@reckenrode
Copy link
Contributor

lgtm, but I’d like to get some more eyes on it to make sure.

@abathur abathur requested review from mweinelt, tjni and toonn February 6, 2024 15:56
Copy link
Contributor

@toonn toonn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vcunat vcunat merged commit 579b24c into NixOS:staging Feb 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: darwin Running or building packages on Darwin 6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-darwin: 5001+ This PR causes many rebuilds on Darwin and must target the staging branches. 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants