pip editable installs doesn't copy data_files defined in setup.py

[shrishinde]

virtualenv

• pip version: 19.1.1
• Python version: 2.7.6
• OS: CentOS 6.9

Description
pip editable installs doesn't copy data_files defined in setup.py

Expected behavior
data_files defined in setup.py should be copied to virtualenv irrespective of editable or non-editable mode installs.

How to Reproduce
Use any python module for this which has setup.py defined with data_files param for setup. Here setup is from setuptools import setup

I faced this issue with my private code base but for giving example here I am using jsnapy

1. activate virtualenv and install jsnapy
   pip install jsnapy
2. Now check data_files copied under 'etc' or 'var/logs` under virtualenv dir.

[shrishinde@chi106230vm site-packages]$ ls -al ~/virtualenv/dev_bouncer/var/logs/
total 12
drwxrwxr-x. 3 shrishinde shrishinde 4096 Jun 11 16:11 .
drwxrwxr-x. 3 shrishinde shrishinde 4096 Jun 11 16:11 ..
drwxrwxr-x. 2 shrishinde shrishinde 4096 Jun 11 16:11 jsnapy

3. Now uninstall jsnapy and also remove these data_files if not removed implicitly.
4. Now clone git repo for jsnapy under /opt/
git clone [email protected]:Juniper/jsnapy.git
5. Now install jsnapy within virtualenv by making use of local git repo setup.py with -e option.
   pip install -e /opt/jsnapy/
6. data_files under virtualenv etc or var/logs are not copied.

[chrahunt]

The best approach here depends on your purpose, and there are two factors I can see:

1. Are these data files supposed to be accessed at runtime by the package scripts?
2. Do these data files live outside the package?

If the files live inside the package, then this currently works with setuptools and pip for editable installs, sdists, and wheels, it just requires using package_files instead of data_files. The files can be accessed at runtime without any issues. See:

example.sh

#!/bin/sh
cd "$(mktemp -d)"
project_dir="$PWD/project"
echo "See project at $project_dir"
mkdir -p project/project

# Example data file.
echo "hello world" > project/project/hello.txt

# Example usage of data file.
# See also https://stackoverflow.com/a/20885799/1698058
cat <<EOF > project/project/__main__.py
try:
    import importlib.resources as importlib_resources
except ImportError:
    import importlib_resources


if __name__ == '__main__':
    print(importlib_resources.read_text('project', 'hello.txt'))
EOF

touch project/project/__init__.py

# setup.py required to install package data.
# See also https://stackoverflow.com/a/14159430/1698058
cat <<EOF > project/setup.py
from setuptools import find_packages, setup


setup(
    name='project',
    version='0.1.0',

    include_package_data=True,
    install_requires=[
        'importlib_resources; python_version<"3.7"'
    ],
    package_data={
        'project': ['hello.txt'],
    },
    packages=find_packages(),
)
EOF

echo "include project/hello.txt" > project/MANIFEST.in

make_venv() {
    dir="$(mktemp -d)"
    # Change directory to show that data file is found even without package
    # being in sys.path.
    cd "$dir"
    python -m venv _venv
    . _venv/bin/activate
    pip install --upgrade pip >/dev/null 2>&1
}

reset_venv() {
    deactivate
    rm -rf _venv
}

echo "Building packages."
make_venv
pip install wheel
cd "$project_dir"
python setup.py bdist_wheel sdist
cd -
reset_venv

echo "From editable install."
make_venv
pip install -e "$project_dir"
python -m project
reset_venv

echo "From sdist."
make_venv
pip install "$project_dir/dist/project-0.1.0.tar.gz"
python -m project
reset_venv

echo "From wheel."
make_venv
pip install "$project_dir/dist/project-0.1.0-py3-none-any.whl"
python -m project
reset_venv

If the files live outside the package (like some of the jsnapy files) - this is not suggested in general, even for non-editably installed packages. For files meant to be accessed at runtime there is a specific statement in the setuptools docs here suggesting to take a different approach. For files not meant to be accessed at runtime, it may be better to bring the issue up on the setuptools issue tracker along with the specific use case. I could not find any existing issue there that seemed related.

[chrahunt]

Hi @shrishinde, I hope the above helped. I'll close this issue now, but please let us know if you have any other problems.