Skip to content

[SPARK-31483][PySpark] Use SPARK_PYTHON or 'python' to run find_spark_home.py #28256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mzhang-code wants to merge 2 commits into from
Closed

[SPARK-31483][PySpark] Use SPARK_PYTHON or 'python' to run find_spark_home.py #28256

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8fab69b
Update find-spark-home
mzhang-code Apr 18, 2020
4376ae7
Merge remote-tracking branch 'upstream/master'
mzhang-code Jul 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 1 addition & 4 deletions bin/find-spark-home
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,5 @@ elif [ ! -f "$FIND_SPARK_HOME_PYTHON_SCRIPT" ]; then
else
# We are pip installed, use the Python script to resolve a reasonable SPARK_HOME
# Default to standard python interpreter unless told otherwise
if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"python"}"
fi
export SPARK_HOME=$($PYSPARK_DRIVER_PYTHON "$FIND_SPARK_HOME_PYTHON_SCRIPT")
export SPARK_HOME=$(${PYSPARK_PYTHON:-"python"} "$FIND_SPARK_HOME_PYTHON_SCRIPT")
Copy link
Member

@HyukjinKwon HyukjinKwon Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm .. can we strip the non-printable characters instead?
Respecting PYSPARK_DRIVER_PYTHON falling back to PYSPARK_PYTHON is expected.

Copy link
Member

@HyukjinKwon HyukjinKwon Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least I can come up with one way although it's hacky. e.g.)

a=$(ipython -c "import sys; print('/User', file=sys.stderr)" 2>&1 >/dev/null)
ls $a

Copy link
Author

@mzhang-code mzhang-code Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a workaround for ipython, but not for jupyter, because jupyter doesn't support jupyter find_spark_home.py. I think PYSPARK_DRIVER_PYTHON is more meant for "frontend". This fix enables PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS=notebook  pyspark too.

Copy link
Author

@mzhang-code mzhang-code Apr 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, PYSPARK_DRIVER_PYTHON's falling back to PYSPARK_PYTHON happens after find-spark-home too:

spark/bin/pyspark

Line 45 in f1fde0c

PYSPARK_DRIVER_PYTHON=$PYSPARK_PYTHON

Copy link
Member

@HyukjinKwon HyukjinKwon Jul 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mzhang-code, can we just add a bandaid fix like: if PYSPARK_DRIVER_PYTHON ends with jupyter or ipython, uses PYSPARK_PYTHON or python for now with some comments about why we're using PYSPARK_PYTHON instead of PYSPARK_DRIVER_PYTHON?

fi