[SPARK-29802][BUILD] Use python3 in build scripts

[srowen]

What changes were proposed in this pull request?

Use /usr/bin/env python3 consistently instead of /usr/bin/env python in build scripts, to reliably select Python 3.

Why are the changes needed?

Scripts no longer work with Python 2.

Does this PR introduce any user-facing change?

No, should be all build system changes.

How was this patch tested?

Existing tests / NA

[Fokko]

LGTM

fokkodriesprong@Fan spark % grep -R "env python" . 
./python/pyspark/version.py:#!/usr/bin/env python3
./python/pyspark/find_spark_home.py:#!/usr/bin/env python3
./python/setup.py:#!/usr/bin/env python3
./python/run-tests.py:#!/usr/bin/env python3
./dev/create-release/releaseutils.py:#!/usr/bin/env python3
./dev/create-release/generate-contributors.py:#!/usr/bin/env python3
./dev/create-release/translate-contributors.py:#!/usr/bin/env python3
./dev/github_jira_sync.py:#!/usr/bin/env python3
./dev/merge_spark_pr.py:#!/usr/bin/env python3
./dev/run-tests-jenkins.py:#!/usr/bin/env python3
./dev/run-tests.py:#!/usr/bin/env python3

[holdenk]

LGTM, just ran into needing this ~30 minutes ago on one of my laptops :)

[SparkQA]

Test build #126071 has finished for PR 29151 at commit ce84970.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Jenkins retest this please

[SparkQA]

Test build #126079 has finished for PR 29151 at commit ce84970.

• This patch fails PySpark pip packaging tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

The pip packaging failure is interesting. Let me know if you want me to take a deeper look at our packaging tests.

On Fri, Jul 17, 2020 at 7:00 PM UCB AMPLab ***@***.***> wrote: Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126079/ Test FAILed. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29151 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOT5PGIN2XAW77ULAR3DDR4D64HANCNFSM4O6ZS5KQ> .

-- Cell : 425-233-8271

[HeartSaVioR]

LGTM. This should be helpful when you encounter the issue on merging PR who has non-ascii char in their name/email. Possibly also for PR title/description as well.

[srowen]

Hm, yeah that's a weird failure. I'm guessing it's related but not sure why. @holdenk @shaneknapp @HyukjinKwon do you know why that might happen?

AssertionError: Egg-link /home/jenkins/workspace/SparkPullRequestBuilder@2/python does not match installed location of pyspark (at /home/jenkins/workspace/SparkPullRequestBuilder@3/python)

[dongjoon-hyun]

Give the following log, it seems to be related to virtualenv-side bug.

Constructing virtual env for testing
Using conda virtual environments
Testing pip installation with python 3.6
Using /tmp/tmp.2RCoqQqNP8 for virtualenv
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /tmp/tmp.2RCoqQqNP8/3.6:
...

[HyukjinKwon]

@srowen, I am investigating it in #29117. I will be able to fix soon I guess.

[HyukjinKwon]

Merged to master.

[HyukjinKwon]

With the test results from Jenkins and GitHub Actions, seems all tests pass.