python312Packages.pymupdf: fix "mupdf not found" import error

[bennofs]

Description of changes

Fixes the following error:

$ nix shell --impure -I nixpkgs=flake:github:NixOS/nixpkgs/nixos-24.05 --expr 'with import <nixpkgs> {}; python3.withPackages (p: [p.pymupdf])' -c python3 -c 'import fitz'
Traceback (most recent call last):
  File "/nix/store/73znvwd538dglfsk2wdixj106i5infjn-python3-3.11.9-env/lib/python3.11/site-packages/fitz/__init__.py", line 140, in <module>
    from . import mupdf
ImportError: cannot import name 'mupdf' from partially initialized module 'fitz' (most likely due to a circular import) (/nix/store/73znvwd538dglfsk2wdixj106i5infjn-python3-3.11.9-env/lib/python3.11/site-packages/fitz/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/73znvwd538dglfsk2wdixj106i5infjn-python3-3.11.9-env/lib/python3.11/site-packages/fitz/__init__.py", line 142, in <module>
    import mupdf

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[SuperSandro2000]

In 4a3db5a (#298783) @dotlambda proposed to move it to buildInputs. What is more correct? I am undecided.

[bennofs]

Are buildInputs even included in the environment when using pythonWithPackages envs? Mupdf itself provides a python module, so it needs to be part of the env, and I don't see how moving this to buildInputs changes that?

[bennofs]

But I am not 100% familiar with the python nix packaging. From the documentation, I think the most correct would be to put it in "dependencies"?

[FliegendeWurst]

Yes, dependencies would be best. However it is currently roughly equivalent to propagatedBuildInputs anyway.

[dotlambda]

dependencies sounds correct to me.

[dotlambda]

Please fix the commit message.

[bennofs]

Should we merge this?

[l0b0]

How do I test this? I tried to set

nixpkgs.overlays =
  let
    owner = "bennofs";
    branchname = "fix-pymupdf";
    pkgsReview = pkgs.fetchzip {
      url = "https://github.com/${owner}/nixpkgs/archive/${branchname}.tar.gz";
      sha256 = "sha256-i6rDWzvPHFl143VzJmBzhRzlz10SKewe4uuRarPUygY=";
    };
  in
  [
    (self: super: {
      review = import pkgsReview {
        overlays = [ ];
        inherit (super) config;
      };
      python312Packages.pymupdf = self.review.python312Packages.pymupdf;
    })
  ];

but that results in errors "attribute 'pip' missing" and "attribute 'python' missing".

[elohmeier]

This can be tested using nix-build -E 'with import ./. {}; python3.withPackages (ps: [ ps.pymupdf ])' and then ./result/bin/python3 -c 'import pymupdf; print(pymupdf.__version__)'.

Before the PR:

$ nix-build -E 'with import ./. {}; python3.withPackages (ps: [ ps.pymupdf ])'
/nix/store/37mb5xk23v4jrzy46acpval4wwhiiw91-python3-3.12.7-env
$ ./result/bin/python3 -c 'import pymupdf; print(pymupdf.__version__)'
Traceback (most recent call last):
  File "/nix/store/37mb5xk23v4jrzy46acpval4wwhiiw91-python3-3.12.7-env/lib/python3.12/site-packages/pymupdf/__init__.py", line 196, in <module>
    from . import mupdf
ImportError: cannot import name 'mupdf' from partially initialized module 'pymupdf' (most likely due to a circular import) (/nix/store/37mb5xk23v4jrzy46acpval4wwhiiw91-python3-3.12.7-env/lib/python3.12/site-packages/pymupdf/__init__.py)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/nix/store/37mb5xk23v4jrzy46acpval4wwhiiw91-python3-3.12.7-env/lib/python3.12/site-packages/fitz/__init__.py", line 2, in <module>
    from pymupdf import *
  File "/nix/store/37mb5xk23v4jrzy46acpval4wwhiiw91-python3-3.12.7-env/lib/python3.12/site-packages/pymupdf/__init__.py", line 198, in <module>
    import mupdf
ModuleNotFoundError: No module named 'mupdf'

After the PR:

$ nix-build -E 'with import ./. {}; python3.withPackages (ps: [ ps.pymupdf ])'
/nix/store/k6xajrhjnyp9g6fglxxazqw62awm240j-python3-3.12.7-env
$ ./result/bin/python3 -c 'import pymupdf; print(pymupdf.__version__)'
1.24.10

This PR actually fixes the import error, since withPackages now picks up the Python module from mupdf-cxx, but I would still like to see another approach in the pymupdf module which just copies the mupdf module into the output of the pymupdf derivation. This also ensures compatibility with pyproject.nix / uv2nix (since that uses its own approach for dependency resolution) and would be more in line with how the upstream PyPI package works. See this PR for that solution: #352845

I've tested this on x86_64-linux, aarch64-linux and aarch64-darwin.

(If you want to test this on Darwin make sure to apply the related tesseract-codesigning-fix from here first: #352788)

[dotlambda]

ModuleNotFoundError: No module named 'mupdf'

Any idea why that's not caught by pythonImportsCheck?

[Samuel-Martineau]

Out of curiosity, is there anything blocking this from moving forward?

[phanirithvij]

there is a merge conflict @bennofs

Edit: I just noticed you are in Nixpkgs/retired-nixpkgs-contributors, can you clarify here once if you intend to see this pr through or whether someone else can pick it up.

[bennofs]

there is a merge conflict @bennofs

Edit: I just noticed you are in Nixpkgs/retired-nixpkgs-contributors, can you clarify here once if you intend to see this pr through or whether someone else can pick it up.

I'm happy if someone else picks it up, no need to wait for me :)

[bennofs]

This PR actually fixes the import error, since withPackages now picks up the Python module from mupdf-cxx, but I would still like to see another approach in the pymupdf module which just copies the mupdf module into the output of the pymupdf derivation. This also ensures compatibility with pyproject.nix / uv2nix (since that uses its own approach for dependency resolution) and would be more in line with how the upstream PyPI package works. See this PR for that solution: #352845

Wouldn't uv2nix then have the same problem for any other python package that use the nixpkgs way to depend on native packages? Or is there something special about pymupdf here.

My opinion is that we should be consistent within nixpkgs, and I believe the rest of nixpkgs uses the toPythonModule way of depending on python modules provided by native packages.

[bennofs]

Actually, this change has already been applied on master independently of this PR, as commit 36354c7. So this PR is no longer relevant.