ZEPPELIN-270 added env varibles for pyspark's correct function in yarn.

[riverajo]

Simple doc fix for pyspark in yarn.

[felixcheung]

It should build this automatically when SPARK_HOME is defined right?

[Leemoonsoo]

I think PYTHONPATH supposed to build automatically when SPARK_HOME is defined by bin/interpreter.sh. But SPARK_YARN_USER_ENV is not taken care of.

[felixcheung]

Hmm, should we use SparkConf instead then:

https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

    // Keep this for backwards compatibility but users should move to the config
    sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
      YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
    }

I believe all spark.* in the interpreter settings are passed to SparkConf.

https://github.com/apache/incubator-zeppelin/blob/master/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java

Is this a bug:

      if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
        logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
        conf.set(key, val);
      }

Shouldn't it say if (key.startsWith("spark.")?

[riverajo]

It should build this automatically when SPARK_HOME is defined right?

That seems right, I didn't need to set PYTHONPATH if SPARK_HOME is set correctly, but I do still need SPARK_YARN_USER_ENV.

However it seems as thought the PYTHONPATH gets created after the zeppelin-env.sh gets run, If I don't set both PYTHONPATH and SPARK_YARN_USER_ENV in that file, It complains that SPARK_YARN_USER_ENV is empty with an array out of bounds exception.

[Leemoonsoo]

@felixcheung

Is this a bug:

if (!key.startsWith("spark.") || !val.trim().isEmpty()) {
        logger.debug(String.format("SparkConf: key = [%s], value = [%s]", key, val));
        conf.set(key, val);
      }

Shouldn't it say if (key.startsWith("spark.")?

I think it's not a bug. Designed to not pass empty value when the key starts with "spark."
This unittest might helpful to understand.
https://github.com/apache/incubator-zeppelin/blob/master/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java#L168

[Leemoonsoo]

Hmm, should we use SparkConf instead then:

https://github.com/apache/spark/blob/69c9c177160e32a2fbc9b36ecc52156077fca6fc/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala

// Keep this for backwards compatibility but users should move to the config
    sys.env.get("SPARK_YARN_USER_ENV").foreach { userEnvs =>
      YarnSparkHadoopUtil.setEnvFromInputString(env, userEnvs)
    }

I believe all spark.* in the interpreter settings are passed to SparkConf.

How about put the same comment Keep this for backwards compatibility but users should move to the config in the document and proceed?

How about take care zeppelin-env.sh.template, too?

[felixcheung]

@Leemoonsoo good idea.

[corneadoug]

@riverajo @felixcheung @Leemoonsoo
Any changes to be done here?
Is this documentation change still needed?

[felixcheung]

I'm not sure if we do but the py4j version changes for the last few Spark releases, so if we do, we would need a way to set the right version as per spark version