Skip to content

[SPARK-33155][K8S] spark.kubernetes.pyspark.pythonVersion allows only '3' #30049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 5 commits into from
Closed

[SPARK-33155][K8S] spark.kubernetes.pyspark.pythonVersion allows only '3' #30049

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a37344b
[SPARK-XXX][K8S][DOC] spark.kubernetes.pyspark.pythonVersion allows o…
dongjoon-hyun Oct 15, 2020
a66f408
fix ut
dongjoon-hyun Oct 15, 2020
cfa9a2c
update entrypoint.sh
dongjoon-hyun Oct 15, 2020
a7ccc1c
rename test case
dongjoon-hyun Oct 15, 2020
d500290
Address comments
dongjoon-hyun Oct 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/running-on-kubernetes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1087,7 +1087,7 @@ See the [configuration page](configuration.html) for information on Spark config
<td><code>spark.kubernetes.pyspark.pythonVersion</code></td>
<td><code>"3"</code></td>
<td>
This sets the major Python version of the docker image used to run the driver and executor containers. Can either be 2 or 3.
This sets the major Python version of the docker image used to run the driver and executor containers. Can be 3.

@viirya viirya Oct 15, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think Only 3 is available for Python3 like below looks better.

</td>
<td>2.4.0</td>
</tr>
Expand Down
6 changes: 3 additions & 3 deletions resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,11 +285,11 @@ private[spark] object Config extends Logging {

val PYSPARK_MAJOR_PYTHON_VERSION =
ConfigBuilder("spark.kubernetes.pyspark.pythonVersion")
.doc("This sets the major Python version. Either 2 or 3. (Python2 or Python3)")
.doc("This sets the major Python version. Only 3 is available for Python3.")
.version("2.4.0")
.stringConf
.checkValue(pv => List("2", "3").contains(pv),
"Ensure that major Python version is either Python2 or Python3")
.checkValue(pv => List("3").contains(pv),
"Ensure that major Python version is Python3")
.createWithDefault("3")

val KUBERNETES_KERBEROS_KRB5_FILE =
Expand Down
4 changes: 1 addition & 3 deletions ...e/src/test/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStepSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ class DriverCommandFeatureStepSuite extends SparkFunSuite {
test("python resource") {
val mainResource = "local:/main.py"
val sparkConf = new SparkConf(false)
.set(PYSPARK_MAJOR_PYTHON_VERSION, "2")
val spec = applyFeatureStep(
PythonMainAppResource(mainResource),
conf = sparkConf,
Expand All @@ -58,7 +57,7 @@ class DriverCommandFeatureStepSuite extends SparkFunSuite {
val envs = spec.pod.container.getEnv.asScala
.map { env => (env.getName, env.getValue) }
.toMap
val expected = Map(ENV_PYSPARK_MAJOR_PYTHON_VERSION -> "2")
val expected = Map(ENV_PYSPARK_MAJOR_PYTHON_VERSION -> "3")
assert(envs === expected)
}

Expand Down Expand Up @@ -93,7 +92,6 @@ class DriverCommandFeatureStepSuite extends SparkFunSuite {
test("SPARK-25355: python resource args with proxy-user") {
val mainResource = "local:/main.py"
val sparkConf = new SparkConf(false)
.set(PYSPARK_MAJOR_PYTHON_VERSION, "2")
val spec = applyFeatureStep(
PythonMainAppResource(mainResource),
conf = sparkConf,
Expand Down
7 changes: 1 addition & 6 deletions resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,7 @@ if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
fi

if [ "$PYSPARK_MAJOR_PYTHON_VERSION" == "2" ]; then
pyv="$(python -V 2>&1)"
export PYTHON_VERSION="${pyv:7}"
export PYSPARK_PYTHON="python"
export PYSPARK_DRIVER_PYTHON="python"
elif [ "$PYSPARK_MAJOR_PYTHON_VERSION" == "3" ]; then
if [ "$PYSPARK_MAJOR_PYTHON_VERSION" == "3" ]; then
pyv3="$(python3 -V 2>&1)"
export PYTHON_VERSION="${pyv3:7}"
export PYSPARK_PYTHON="python3"
Expand Down
1 change: 0 additions & 1 deletion ...-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ private[spark] trait DecommissionSuite { k8sSuite: KubernetesSuite =>
test("Test basic decommissioning", k8sTestTag) {
sparkAppConf
.set(config.DECOMMISSION_ENABLED.key, "true")
.set("spark.kubernetes.pyspark.pythonVersion", "3")
.set("spark.kubernetes.container.image", pyImage)
.set(config.STORAGE_DECOMMISSION_ENABLED.key, "true")
.set(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED.key, "true")
Expand Down
2 changes: 0 additions & 2 deletions ...n-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PythonTestsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ private[spark] trait PythonTestsSuite { k8sSuite: KubernetesSuite =>
test("Run PySpark with Python3 to test a pyfiles example", k8sTestTag) {
sparkAppConf
.set("spark.kubernetes.container.image", pyImage)
.set("spark.kubernetes.pyspark.pythonVersion", "3")
Comment thread
HyukjinKwon marked this conversation as resolved.
runSparkApplicationAndVerifyCompletion(
appResource = PYSPARK_FILES,
mainClass = "",
Expand All @@ -57,7 +56,6 @@ private[spark] trait PythonTestsSuite { k8sSuite: KubernetesSuite =>
test("Run PySpark with memory customization", k8sTestTag) {
sparkAppConf
.set("spark.kubernetes.container.image", pyImage)
.set("spark.kubernetes.pyspark.pythonVersion", "3")
.set("spark.kubernetes.memoryOverheadFactor", s"$memOverheadConstant")
.set("spark.executor.pyspark.memory", s"${additionalMemory}m")
runSparkApplicationAndVerifyCompletion(
Expand Down