pkgconfig files are installed into "site-packages/.*.mesonpy.libs/pkgconfig"

[mgorny]

When the meson.build specifies a pkgconfig file, this file gets installed into a weird .*.mesonpy.libs/pkgconfig subdirectory of site-packages. This is e.g. happening with dbus-python.

$ find .venv | grep pc$
.venv/lib/python3.11/site-packages/.dbus_python.mesonpy.libs/pkgconfig/dbus-python.pc

[rgommers]

Thanks for the report @mgorny. Where did you expect them to be installed, inside the package or in the system pkg-config location? And did the previous build system for dbus-python handle this?

For reference, this should be the code: https://gitlab.freedesktop.org/dbus/dbus-python/-/blob/master/meson.build#L206-215

[eli-schwartz]

The deprecated autotools build for dbus-python places all files in the system locations, both for pkg-config and for documentation, headers, the python site-packages directory probe.... This should be functionally equivalent to installing dbus-python with meson setup builddir && ninja -C builddir && meson install -C builddir, without any .dist-info metadata.

The old setuptools setup.py simply ran the autotools build pointed to install files at

'pythondir=' + os.path.join(srcdir, self.build_lib), 
'pyexecdir=' + os.path.join(srcdir, self.build_lib),

And discarded all files other than the *.py and *.so, then let setuptools see the files that were manually seeded into setuptools' own build directory and package them up.

[rgommers]

Ah fun. I guess putting everything outside of site-packages can be achieved by the user when explicitly using install_dir: ... for Python extensions and files. But meson-python won't be aware of that. I feel like we recently had a relevant discussion on this involving pkg-config files and pybind11-global, but I'm failing to dig up the most pertinent comments about installing outside of site-packages being discouraged.

The question remains what meson-python does, and should be doing, regarding .pc files when meson.build includes:

pc_mod = import('pkgconfig')

pc_mod.generate(
    description: 'Python bindings for D-Bus',
    filebase: 'dbus-python',
    name: 'dbus-python',
    requires: ['dbus-1 >= 1.8'],
    subdirs: ['dbus-1.0'],
    variables: {
        'exec_prefix': '${prefix}',
        'datarootdir': '${prefix}' / get_option('datadir'),
    },
)

.*.mesonpy.libs/pkgconfig/ does not look like the correct answer.

[mgorny]

Where did you expect them to be installed, inside the package or in the system pkg-config location?

Well, I don't think installing inside the package would serve any purpose — so they'd either have to land in the correct system directory, or not be installed at all.

The larger problem with this is that wheels technically don't support installing outside the few standard Python directories (and PEP 491 is deferred). To some degree we can do that by relying on UB but I have mixed feelings about this.

I'm not sure but it's also possible that we'd need a way to install headers in top-level system header directory instead of python directory but perhaps pkgconfig can work around that. That is, if generated file contained the correct paths (is this even possible with wheels?)

[mgorny]

CC @smcv (dbus-python maintainer)

[dnicolodi]

I don't understand if the issue you are reporting is about meson-python installing files in an unexpected location or about the files being present at all in the wheel distribution.

For the first, IMO there is very little that can be done: the wheel package format does not allow to do that. Installing a wheel does not allow to run code and the only way to determine some system paths is to run code. Even if that would be possible, virtual Python environments need to be supported, and there is no specification for where, for example, pkgconfig definitions should be installed in a venv. If you need to install files outside the python paths wheels are not the right solution. And meson-python is about building wheels, thus this is outside the scope of the tool.

For the latter, I think some way of having some files installed by meson install end up in the wheel is required. This can be some form of heuristic or could be based on install tags or something different.

[eli-schwartz]

I had been thinking that for a build backend merged directly into Meson that install tags would be the way to go, and the project would simply specify what files other than python-runtime should be included in the wheel.

More generally, simply treating some files (pkg-config, headers, docs) as installed by a system software builder and some files (.py files, C extensions) as installed via the wheel spec, would probably be the best way to handle it.

People e.g. installing binary wheels precompiled and published on PyPI, are not interested in the docs at least -- and it matches what setuptools did so there's that. :p

[mgorny]

Well, in my opinion the current behavior is wrong but I don't have a strong opinion what the right behavior should be. If we can't get them working, then I suppose mesonpy should either skip them or (perhaps even better) error out entirely, so people don't fall into the trap of declaring mesonpy as the builder when the package can't be built correctly using it.

I'm perfectly happy with skipping wheels entirely and installing the package using meson. However, that implies losing .dist_info and therefore breaking the dependency resolution in its reverse dependencies.

[henryiii]

pybind11-global installs into the data directory of the wheel (/include and /share, IIRC). That produces "nice" linux-style structure. It's also surprising / dangerous to write to your root if you are not in a virtual environment. If you install twice on two different Python versions, they overwrite each other. Homebrew complains about files it doesn't manage showing up. Etc. It can be better to install to your package in site-packages, and then teach your tool about that. For CMake, if you give site-packages as a search directory, then pybind11/share/cmake/pybind11Config.cmake (IIRC) is discoverable. Something like that might work for pkgconfig. Or, you could define an entrypoint for pkg-config and add everything in that entry point when you run pkg-config. Etc.

That's why we split pybind11 (the "unsurprising" package) and pybind11-global (the data directory) into different packages, and to the best of my knowledge, the usage of pybind11-global is relatively low.

[rgommers]

One thing we should not forget is that wheels are not just for putting on PyPI. If you want to produce some other packaging format, that often goes through a wheel.

More generally, simply treating some files (pkg-config, headers, docs) as installed by a system software builder and some files (.py files, C extensions) as installed via the wheel spec, would probably be the best way to handle it.

I'm afraid you're right about that. The trouble with it is that for Python packages that do ship headers, pkg-config files, etc. we cannot rely on a system builder. So with things like pybind11 and numpy, they default to installing those headers inside site-packages after all. And then document to run Python code (e.g. numpy.get_include()) in order to get at that location. Which is bad for headers (breaks cross-compiling), and worse for pkg-config probably.

The "solution" for those packages is then to encode the information on paths directly into Meson (and CMake), so we can do dependency('numpy') and dependency('pybind11'). Which is okay for the most popular packages, but not very scalable. I don't see a better way though, because the limitations of Python packaging and virtualenvs kind of make the pybind11-global way non-recommended. While it would work perfectly fine with, for example, conda envs.

I am pondering whether there should be a standard CLI flag to switch between "put them inside site-packages/<pkgname>" (default) and "put them in the correct place for system installs". We'd then have a way to build conda-forge packages with pip install . --use-system-install-for-headers-and-pkgconfig. Kinda standardizing how to do the pybind11-global approach. I don't actually know if this will get traction, but it could make sense for Linux distros, conda, etc. - numpy headers could then land in /usr/include/numpy rather than site-packages/numpy/core/include/numpy. Things already go into /usr/bin/, /usr/share/ etc., (example: Ubuntu file list) so why not for headers?

Probably too much trouble and takes years to change, but it'd be nice ....

[smcv]

The deprecated autotools build for dbus-python places all files in the system locations, both for pkg-config and for documentation, headers, the python site-packages directory probe.... This should be functionally equivalent to installing dbus-python with meson setup builddir && ninja -C builddir && meson install -C builddir, without any .dist-info metadata.

That sounds right. The dbus-python Meson build (just plain Meson, without going via meson-python) is meant to be essentially equivalent to the old Autotools build, and suitable for producing distro packages (.deb, .rpm and equivalent). My high-level goal here is to keep dbus-python alive on minimal maintenance because I have too many other responsibilities, while using a better build system than distutils: in particular, it is a requirement to be able to detect dbus via pkg-config, and it is a requirement to have an equivalent of meson dist or Autotools make distcheck that verifies that the release actually works, because if I accidentally make a release that doesn't work (as has happened in the past), then recovering from that results in spending more time on dbus-python than I can justify.

At the moment, the Meson build doesn't generate a .egg-info or .dist-info directory, and the Debian packaging fills in the gaps. Ideally the Meson packaging should maybe be doing this, when not invoked by meson-python? If people who know about the Python packaging ecosystem have a plan for how this should work, I'd consider merge requests.

Similarly, the PKG-INFO etc. in the source release are currently populated by an Autotools dist-hook, and when I drop the Autotools build altogether, I'll presumably need to migrate those to a Meson add_dist_script. Again, if people who know about the Python packaging ecosystem have a plan for how this should work, I'd consider merge requests.

The version of dbus-python on PyPI is there because pip users expect/demand that all Python libraries appear on PyPI (possibly against my better judgement, because dbus-python requires a pre-existing system installation of libdbus, which is straightforward for Meson or a distro-specific packaging system but is out-of-scope for PyPI). I used meson-python for the PyPI build, to avoid having to either NIH the build system, or wrap Meson using arbitrary shell commands (the way the PyPI build for older versions wrapped Autotools).

If you want finer control over its installation or if you want to build third-party extensions to dbus-python (like PyQt does to provide Qt main loop integration), then I would suggest using Meson directly without going via meson-python.

[FFY00]

At the moment, the Meson build doesn't generate a .egg-info or .dist-info directory, and the Debian packaging fills in the gaps. Ideally the Meson packaging should maybe be doing this, when not invoked by meson-python? If people who know about the Python packaging ecosystem have a plan for how this should work, I'd consider merge requests.

This is a common issue. I have considered adding a command to install the missing metadata, or a wrapper that installs the metadata additionally, but those seem like really poor options.

@eli-schwartz now that we are in the Meson organization, would it be possible for Meson to consider detecting when meson-python is used and install the metadata in those cases? I am fine making this a Python >=3.11 feature, avoiding adding any dependencies to Meson.

[FFY00]

@mgorny unfortunately, this is a limitation of meson-python. I opened mesonbuild/meson#11462 to try to better integrate it into Meson directly, and solve this issue, but it didn't seem to go anywhere.

I would recommend you to use Meson directly to install the project (ninja install), and copy just the metadata from the wheel onto the file-system. If that is an issue, I'd be willing to add a CLI to generate the metadata, so that projects can use it in their Meson definitions.

cc @jpakkane