Skip to content

Conversation

@abathur
Copy link
Member

@abathur abathur commented Jan 12, 2024

In the course of testing out the fix that ultimately landed in #278945, I ran into the following missing-function error in cffi's test suite:
Executing pytestCheckPhase
============================= test session starts ==============================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
rootdir: /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0
collecting ... ^Mcollecting 229 items                                                           ^Mcollecting 641 items                                                           ^Mcollecting 872 items 

==================================== ERRORS ====================================
_____________ ERROR collecting testing/cffi1/test_parse_c_type.py ______________
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/nix/store/riwkg7zfhj0fzn918s8yiblnvyp0vcmd-python3-3.11.7/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
/nix/store/xmn75r6ddzyqbky5zkaryl8djskgakd1-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
testing/cffi1/test_parse_c_type.py:75: in <module>
    @ffi.callback("int(unsigned long long *)")
src/cffi/api.py:396: in callback_decorator_wrap
    return self._backend.callback(cdecl, python_callable,
E   SystemError: ffi_prep_closure_loc() is missing
------------------------------- Captured stderr --------------------------------
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.3/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.5/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.44.2/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p7/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.12/lib'
ld: warning: directory not found for option '-L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-configd-453.19/lib'
ld: warning: directory not found for option '-L/nix/store/zd2q4dbddgw06nx4cv8qkzs4s24i4rh8-tzdata-2023d/lib'
=============================== warnings summary ===============================
src/cffi/_imp_emulation.py:4
  /private/tmp/nix-build-python3.11-cffi-1.16.0.drv-0/cffi-1.16.0/src/c/../cffi/_imp_emulation.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in P
    from imp import *

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
ERROR testing/cffi1/test_parse_c_type.py - SystemError: ffi_prep_closure_loc() is missing
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 1 error in 7.96s ==========================

The missing function is in turn caused by these defines: https://github.com/python-cffi/cffi/blame/feaff378873d8a3f52bbb63f7d5e24f9ab18f1e9/src/c/_cffi_backend.c#L106-L113

If I understand correctly (I may not), cffi is assuming the system libffi here, which doesn't apply in our case.

(The underlying defines (which affect macOS before 10.15) have been in cffi for 4+ years now, so I'm not quite sure how we haven't picked up on it already.)

Description of changes

I edited the patch for this behavior (pkgs/development/python-modules/cffi/darwin-use-libffi-closures.diff) to remove the version check.

I'm not certain this is correct, so I'll open this as a draft for now. I went ahead and did this for all 3 checks, even though I suspect we didn't need all of them to pass the test failure above.

Things done

I was able to successfully build and use nix with this patch present.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

r-ryantm and others added 30 commits December 26, 2023 07:41
citations: fix build on darwin
https://gitlab.gnome.org/GNOME/folks/-/compare/0.15.6...0.15.7

Also re-enable doCheck on Linux, and workaround Clang 16 build issue.
bobby285271 and others added 22 commits January 12, 2024 20:45
morewaita-icon-theme: init at 42.3
Other kernels have their kernel at top level, but zen's are missing.
python311Packages.aiosomecomfort: 0.0.24 -> 0.0.25
java-service-wrapper: enable aarch64-linux platform
…splus-mark-broken-musl

pkgsMusl.dbus_cplusplus: mark broken for Musl
…Compiler-mark-broken-musl

pkgsMusl.perl536Packages.POSIXstrftimeCompiler: mark broken for Musl
linux_zen, linux_lqx: add missing top-level attributes
lualatex assumes a writeable font cache relative to `$HOME`, for nix this has two implications.
First, the cache might diverge from the nix store if users use LuaLaTeX.
Second, `$HOME` needs to be set to a writable path in derivations.
If I understand correctly (I may not), cffi is assuming the system
libffi here, which doesn't apply in our case.

The underlying defines (which affect macOS before 10.15) have been in
cffi for 4+ years now, so I'm not quite sure how we haven't picked up
on it already.
@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Jan 12, 2024
@abathur abathur changed the base branch from master to staging January 12, 2024 16:54
@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: GNOME GNOME desktop environment and its underlying platform 8.has: documentation This PR adds or changes documentation 6.topic: TeX Issues regarding texlive and TeX in general labels Jan 12, 2024
@abathur
Copy link
Member Author

abathur commented Jan 12, 2024

ick

@abathur abathur closed this Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: GNOME GNOME desktop environment and its underlying platform 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: TeX Issues regarding texlive and TeX in general 8.has: documentation This PR adds or changes documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.