More changes to PySpark python executable configuration:

- Introduce PYSPARK_DRIVER_PYTHON
- Attempt to use python2.7 as the default Python version.
- Refuse to launch IPython without Python 2.7 if PYSPARK_PYTHON isn't set.

Author: JoshRosen

bin/pyspark

@@ -51,33 +51,43 @@ fi
 . "$FWDIR"/bin/load-spark-env.sh
 
 # In Spark <= 1.1, setting IPYTHON=1 would cause the driver to be launched using the `ipython`
-# executable, while the worker would still be launched using PYSPARK_PYTHON.  This allowed users to
-# launch IPython notebooks with PySpark without requiring IPython to be installed on the workers.
-# Unfortunately, this approach had a few drabacks:
+# executable, while the worker would still be launched using PYSPARK_PYTHON.
 #
-#   - It wasn't easy to use a custom IPython executable (SPARK-3265).
-#   - There was a risk that the `ipython` and `PYSPARK_PYTHON` executables might run different
-#     Python versions (e.g. 2.6 on driver and 2.7 on the workers), which might lead to issues
-#     when using certain Python serializers that are incompatible across releases (e.g. marshal).
-#
-# In Spark 1.2, we removed the documentation of the IPYTHON and IPYTHON_OPTS variables, since
-# we've made the necessary changes to allow `ipython` to be used on the workers, too.  Now,
-# users can simply set PYSPARK_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to
-# pass options when starting the Python driver (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').
+# In Spark 1.2, we removed the documentation of the IPYTHON and IPYTHON_OPTS variables and added
+# PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS to allow IPython to be used for the driver.
+# Now, users can simply set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set
+# PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver
+# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython
+# and executor Python executables.
 #
 # For backwards-compatibility, we retain the old IPYTHON and IPYTHON_OPTS variables.
 
-# If IPython options are specified, assume user wants to run IPython (for backwards-compatibility)
-if [[ -n "$IPYTHON_OPTS" ]]; then
-  IPYTHON=1
-  # For backwards-compatibility:
+# Determine the Python executable to use if PYSPARK_PYTHON or PYSPARK_DRIVER_PYTHON isn't set:
+if hash python2.7 2>/dev/null; then
+  # Attempt to use Python 2.7, if installed:
+  DEFAULT_PYTHON="python2.7"
+else
+  DEFAULT_PYTHON="python"
+fi
+
+# Determine the Python executable to use for the driver:
+if [[ -n "$IPYTHON_OPTS" || "$IPYTHON" == "1" ]]; then
+  # If IPython options are specified, assume user wants to run IPython
+  # (for backwards-compatibility)
   PYSPARK_DRIVER_PYTHON_OPTS="$PYSPARK_DRIVER_PYTHON_OPTS $IPYTHON_OPTS"
+  PYSPARK_DRIVER_PYTHON="ipython"
+elif [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
+  PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"$DEFAULT_PYTHON"}"
 fi
 
-# Figure out which Python executable to use.
-# If we're not running in the legacy IPYTHON mode, then use a default PYSPARK_PYTHON
-if [[ "$IPYTHON" != "1" && -z "$PYSPARK_PYTHON" ]]; then
-  PYSPARK_PYTHON="python"
+# Determine the Python executable to use for the executors:
+if [[ -z "$PYSPARK_PYTHON" ]]; then
+  if [[ $PYSPARK_DRIVER_PYTHON == *ipython* && $DEFAULT_PYTHON != "python2.7" ]]; then
+    echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2
+    exit 1
+  else
+    PYSPARK_PYTHON="$DEFAULT_PYTHON"
+  fi
 fi
 export PYSPARK_PYTHON
 
@@ -108,9 +118,9 @@ if [[ -n "$SPARK_TESTING" ]]; then
   unset YARN_CONF_DIR
   unset HADOOP_CONF_DIR
   if [[ -n "$PYSPARK_DOC_TEST" ]]; then
-    exec "$PYSPARK_PYTHON" -m doctest $1
+    exec "$PYSPARK_DRIVER_PYTHON" -m doctest $1
   else
-    exec "$PYSPARK_PYTHON" $1
+    exec "$PYSPARK_DRIVER_PYTHON" $1
   fi
   exit
 fi
@@ -126,9 +136,5 @@ if [[ "$1" =~ \.py$ ]]; then
 else
   # PySpark shell requires special handling downstream
   export PYSPARK_SHELL=1
-  if [[ "$IPYTHON" = "1" ]]; then
-    exec "${PYSPARK_PYTHON:-ipython}" $PYSPARK_DRIVER_PYTHON_OPTS
-  else
-    exec "$PYSPARK_PYTHON" $PYSPARK_DRIVER_PYTHON_OPTS
-  fi
+  exec "$PYSPARK_DRIVER_PYTHON" $PYSPARK_DRIVER_PYTHON_OPTS
 fi

core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala

@@ -34,7 +34,8 @@ object PythonRunner {
     val pythonFile = args(0)
     val pyFiles = args(1)
     val otherArgs = args.slice(2, args.length)
-    val pythonExec = sys.env.get("PYSPARK_PYTHON").getOrElse("python") // TODO: get this from conf
+    val pythonExec =
+      sys.env.getOrElse("PYSPARK_DRIVER_PYTHON", sys.env.getOrElse("PYSPARK_PYTHON", "python"))
 
     // Format python file paths before adding them to the PYTHONPATH
     val formattedPythonFile = formatPath(pythonFile)

docs/programming-guide.md

@@ -211,17 +211,17 @@ For a complete list of options, run `pyspark --help`. Behind the scenes,
 
 It is also possible to launch the PySpark shell in [IPython](http://ipython.org), the
 enhanced Python interpreter. PySpark works with IPython 1.0.0 and later. To
-use IPython, set the `PYSPARK_PYTHON` variable to `ipython` when running `bin/pyspark`:
+use IPython, set the `PYSPARK_DRIVER_PYTHON` variable to `ipython` when running `bin/pyspark`:
 
 {% highlight bash %}
-$ PYSPARK_PYTHON=ipython ./bin/pyspark
+$ PYSPARK_DRIVER_PYTHON=ipython ./bin/pyspark
 {% endhighlight %}
 
 You can customize the `ipython` command by setting `PYSPARK_DRIVER_PYTHON_OPTS`. For example, to launch
 the [IPython Notebook](http://ipython.org/notebook.html) with PyLab plot support:
 
 {% highlight bash %}
-$ PYSPARK_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --pylab inline" ./bin/pyspark
+$ PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook --pylab inline" ./bin/pyspark
 {% endhighlight %}
 
 </div>