Packaged script execute permissions lost from v46.1.0 onwards

[dhallam]

Issue

We use pipenv and, as a result, are forced to use the latest version of setuptools as it internally pins to the latest version.

We observed that executable scripts that are part of python packages have lost their execute flag from setuptools v46.1.0 onwards. The example to demonstrate this bug uses pyspark which includes a number of executable scripts in its package.

The issue was introduced by commit 7843688 where the copy_file() function is now called with preserve_mode=False. The change log states the reason for the change as:

Prevent keeping files mode for package_data build. It may break a build if user's package data has read only flag.

Unfortunately, this has the side effect of stripping all execute permissions from files, meaning users can't use the scripts "out of the box" - they have to set the execute permissions manually.

Demonstration Script

#!/bin/bash
set -eu

wget -nc https://files.pythonhosted.org/packages/9a/5a/271c416c1c2185b6cb0151b29a91fff6fcaed80173c8584ff6d20e46b465/pyspark-2.4.5.tar.gz

for version in "46.0.0" "46.1.0"; do
    rm -rf .venv pyspark-2.4.5
    tar xzf pyspark-2.4.5.tar.gz
    virtualenv -q -p /usr/bin/python3.7 .venv
    . .venv/bin/activate
    python3 -m pip install --upgrade setuptools="==${version}" wheel
    pushd pyspark-2.4.5
    python3 setup.py -q bdist_wheel
    pushd dist
    unzip -q pyspark-2.4.5-py2.py3-none-any.whl
    echo -e "\n\n${version}: Here are the permissions for spark-submit:\n"
    ls -l ./pyspark/bin/spark-submit
    echo -e "\n\n"
    popd
    popd
done

Expected result

-rwxr-xr-x 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit

Actual Result

-rw-rw-r-- 1 dave dave 1040 Feb  2 19:35 ./pyspark/bin/spark-submit

[tibdex]

I'm facing this issue too on macOS 10.15.3 and Pipenv 2018.11.26. This code was working fine in 46.0.0 but started failing in 46.1.0:

from pyspark.sql import SparkSession # pyspark==2.4.5

spark = SparkSession.builder.appName("Test").getOrCreate()

[mattclay]

In the Ansible community we've started seeing this issue testing Ansible Collections, since ansible-test depends on internal executable scripts which are not entry points.

See #1607 (comment)

[leekillough]

The bug seems to have been introduced in the last 3 days, when we started seeing permission problems, most notably the lack of execution permissions in scripts.

Someone briefly mentioned 512565e, but that seems to only be whitespace changes, and the original change in 7843688 goes back to 2018, which was merged a long time ago.

We can workaround this issue by changing the permissions of installed scripts, but I'm not sure that this project is the root cause of this recent regression.

@jaraco @KOLANICH

[mattclay]

@leekillough #1607 (which contained 7843688) was merged into master only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:

https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0

#1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.

[leekillough]

@leekillough #1607 (which contained 7843688) was merged into master only 2 days ago with the change. The change can be seen in the release history for version v46.1.0, released on March 21:

https://setuptools.readthedocs.io/en/latest/history.html#v46-1-0

#1424: Prevent keeping files mode for package_data build. It may break a build if user’s package data has read only flag.

Sorry, I missed that PR. I thought I went through them all, but for some reason I could never find commit 7843688.

[HyukjinKwon]

Hi @jaraco and @jorikdima, Is it going to be fixed?

Apache Spark community faced this issue and CI started to break. We're setting the upper bound of setuptools to work around CI break for now.

[HyukjinKwon]

I also filed this issue in Spark side to track SPARK-31231.