[BUG] Editable installs don't include all folders

[Martmists-GH]

setuptools version

62.6.0

Python version

3.10.5

OS

Arch Linux

Additional environment information

No response

Description

I have an issue with my project where certain modules are not installed. here's a link to the repo

The core is written in native code, and built as a C extension (named _kaudio) from build/generated/ksp/{target}/{target}Main/resources/entrypoint.cpp.
An autogenerated module with python stubs is also generated, located at build/generated/ksp/{target}/{target}Main/resources/. This module is called kaudio.
Lastly there's an app bundled built on top of this native code, with the code located at src/main/kaudio/app, and is fittingly called kaudio.app

Here's the relevant code for how setup.py is structured

stub_root = f"{build_dir}/generated/ksp/{target}/{target}Main/resources/"

def extensions():
    native = Extension("_kaudio",
                       sources=[f"{build_dir}/generated/ksp/{target}/{target}Main/resources/entrypoint.cpp"],
                       # Temporary workaround for [KT-52303](https://youtrack.jetbrains.com/issue/KT-52303)
                       # When resolved, use `{build_dir}/` instead of `build/`
                       include_dirs=[f"build/bin/{target}/{folder}/"],
                       library_dirs=[f"build/bin/{target}/{folder}/"],
                       libraries=["kaudio"])  # Some of the native code is precompiled into a static library called `kaudio`

    return [native]


setup(
    name="kaudio",
    python_requires=">=3.10",
    ext_modules=extensions(),
    data_files=[
        ("share/applications", ["src/main/resources/com.martmists.kaudio.desktop"]),
    ],
    entry_points={
        "console_scripts": ["kaudio=kaudio.app.__main__:main"],
        "kaudio.app.nodes": ["builtins=kaudio.app.metadata:load_builtins"]
    },
    packages=find_packages(where=stub_root) + 
        ["kaudio.app"] + 
        ["kaudio.app." + it for it in find_packages(where="src/main/kaudio/app")],
    package_dir={"": stub_root, "kaudio.app": "src/main/kaudio/app"}
)

In this setup, the kaudio package is installed just fine in both editable and non-editable state, but kaudio.app is unavailable in editable state.

---

While this may seem like an issue with pip, as pip was used to install it, they reported it as an issue with setuptools: pypa/pip#10810 (comment)

Expected behavior

kaudio.app can be used when installed with pip install -e just like when not using the -e flag.

How to Reproduce

1. Clone the linked repo
2. Run pip install -e .
3. Try to run python -m kaudio.app

Output

/home/mart/.pyenv/versions/3.10.5/bin/python: No module named kaudio.app

[abravalheri]

Hi @Martmists-GH, data_files is not currently supported by the existing implementation of the editable install.

This is how our docs explain the existing process:

You can perform a pip installation passing the -e/--editable flag (e.g., pip install -e .). It works very similarly to pip install ., except that it doesn’t actually install anything. Instead, it creates a special .egg-link file in the target directory (usually site-packages) that links to your project’s source code. It may also update an existing easy-install.pth file to include your project’s source code, thereby making it available on sys.path for all programs using that Python installation.

TL;DR: it extends the Python's sys.path to include the relevant parts of the project directory.

We might add support for that in the upcoming PEP 660 implementation, however, non-Python modules will be installed normally and will not be "automatically updated" on changes.

[abravalheri]

There is another existing limitation: complex package_dir are not currently supported.
setuptools will only support no package_dir or pacakge_dir = {"": ...} (no extra folders).

Support for extra folders might also be added when tackling PEP 660. But that currently is under consideration.

[Martmists-GH]

If data_files aren't supported in editable I don't think that's an issue, but ideally I don't have to set up symlinks to every folder I want to use that isn't in the main source directory. I'll take a look at PEP 660 👍

[abravalheri]

I understand. I am currently working on the implementation of PEP 660. Would you be interested in participating in some beta tests? I could ping you when it is available...

(It is not finished yet, but hopefully it will be soon).

[abravalheri]

@Martmists-GH, could you please checkout if v64.0.3+ solves your problem?

[Martmists-GH]

Still the same result, unfortunately.

As for the contents of kaudio.egg-link:

The first line is correct; That is the folder where kaudio and _kaudio (native part) are located.
Note that the second line points to the root of the project, rather than the specified src/main/kaudio/app, which is where kaudio.app is supposed to point to.

[abravalheri]

Um... this is weird.

The log message "Running setup.py develop" and the fact that you still have a .egg-link means that you are still using an old installation style, not the new one.

Any chance you are running on Ubuntu and it is related to pypa/pip#6264?

Another possibility (but I don't think this is the case) is that you don't have a pyproject.toml file and therefore pip simply don't run the PEP 660 hooks. However, I did a quick test in my PC with a project without pyproject.toml and pip does seem to use PEP 660 hooks without any problem.

[abravalheri]

This is a simplified reproducer I used in a Debian system:

cd /tmp
rm -rf proj
mkdir -p proj/src/main/kaudio/app
touch proj/src/main/kaudio/app/__init__.py
echo "print('hello world')" > proj/src/main/kaudio/app/__main__.py
mkdir -p proj/src/main/kaudio/app/pkg1
touch proj/src/main/kaudio/app/pkg1/__init__.py
mkdir -p proj/other/generated/ksp/x/xMain/resources/pkg2
touch proj/other/generated/ksp/x/xMain/resources/pkg2/__init__.py
mkdir -p proj/other/generated/ksp/x/xMain/resources/pkg3
touch proj/other/generated/ksp/x/xMain/resources/pkg3/__init__.py

cd proj

cat <<EOF > setup.py
from setuptools import find_packages, setup

stub_root = "other/generated/ksp/x/xMain/resources"

setup(
    name="kaudio",
    packages=find_packages(where=stub_root) +
             ["kaudio.app"] +
             ["kaudio.app." + it for it in find_packages(where="src/main/kaudio/app")],
    package_dir={"": stub_root, "kaudio.app": "src/main/kaudio/app"},
)
EOF

python3.10 -m venv .venv
.venv/bin/python -m pip install -U pip setuptools
.venv/bin/python -m pip install -e .
.venv/bin/python -m kaudio.app

Indeed this reproducer does not install "kaudio.app". I can see in the logs that pip ignores the PEP 660 hooks and uses develop.

logs

Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.2.2 setuptools-64.0.3
Obtaining file:///tmp/proj
  Preparing metadata (setup.py) ... done
Installing collected packages: kaudio
  Running setup.py develop for kaudio
Successfully installed kaudio-0.0.0
/tmp/proj/.venv/bin/python: Error while finding module specification for 'kaudio.app' (ModuleNotFoundError: No module named 'kaudio')

If I add the following pyproject.toml, however, it will work:

[build-system]
requires = ["setuptools>=64.0.3"]
build-backend = "setuptools.build_meta:__legacy__"
# `__legacy__` is useful if you are importing stuff from the local directory
# otherwise you could also use "setuptools.build_meta"

logs

Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-22.2.2 setuptools-64.0.3
Obtaining file:///tmp/proj
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: kaudio
  Building editable for kaudio (pyproject.toml) ... done
  Created wheel for kaudio: filename=kaudio-0.0.0-0.editable-py3-none-any.whl size=2285 sha256=2013de0dcd15953e78c530bf1e9ef9f0e44ba76e1c62064b26470f8b452643d0
  Stored in directory: /tmp/pip-ephem-wheel-cache-wj7uc2no/wheels/40/90/76/e0b638bfb2222a58bdd40cbfe82dc44c826b5e1ad26ad37687
Successfully built kaudio
Installing collected packages: kaudio
Successfully installed kaudio-0.0.0
hello world

This means that the second possibility I previously dismissed is actually what is happening 😅.

The develop command has limitations about what it can handle in terms of package_dir. If you need to use complex package_dir, it is recommended to add a pyproject.toml file.

This will make pip opt into the setuptools' implementation of PEP 660 hooks (which are able to handle complex package_dir).

[Martmists-GH]

Is there any way to enforce this behavior without adding yet another configuration file to the project or is a pyproject.toml file the only way to do it?

[abravalheri]

As pointed out by pip maintainers, pip install -e . --use-pep517 might be the another way.

[Martmists-GH]

that seemed to do the trick 👍

>>> import kaudio
>>> kaudio
<module 'kaudio' from '/home/mart/git/kaudio-python/build/kaudio/generated/ksp/native/nativeMain/resources/kaudio/__init__.py'>
>>> import kaudio.app as app
>>> app
<module 'kaudio.app' from '/home/mart/git/kaudio-python/src/main/python/kaudio/app/__init__.py'>