ZEPPELIN-383 CLASSPATH_OVERRIDES environment variable

[echarles]

Prepend CLASSPATH_OVERRIDES environment variable when building CLASSPATH in the shell scripts.

[Leemoonsoo]

Thanks for the contribution.
But I think ZEPPELIN_CLASSPATH already does the job. isn't it?

[radekg]

@Leemoonsoo: I think not exactly. ZEPPELIN_CLASSPATH is built by prepending elements to the classpath, meaning, whatever the user puts in the class path is, at the time of complete class path, placed at the very end of the class path. Consider a scenario:

Somebody wants to put a patched spark-core jar into the class path. He will set ZEPPELIN_CLASSPATH to his /opt/whatever/spark-core_2.10-1.5.1.jar and proceed with the standard procedure for zeppelin.sh. His jar will not be the effective one. The reason is that the class path built by Zeppelin is the following:

ZEPPELIN_CLASSPATH=<addJar>:$ZEPPELIN_CLASSPATH
...
ZEPPELIN_CLASSPATH=<addJar>:$ZEPPELIN_CLASSPATH

I believe what @echarles introduces here, is the mechanism for the user to be able to override any arbitrary functionality no matter what.

Very nice feature, @echarles.

[Leemoonsoo]

@radekg Thanks for the explanation. But i think @echarles proposed codes prepend CLASSPATH_OVERRIDES before CLASSPATH, like

-cp ${CLASSPATH_OVERRIDES}:${CLASSPATH}

That's why i think it is the same to how ZEPPELIN_CLASSPATH is being populated.
please correct me if my understanding is wrong.

[Leemoonsoo]

Oh, i misunderstood how ZEPPELIN_CLASSPATH is being populated @radekg you're right.

Then ZEPPELIN_CLASSPATH will append and CLASSPATH_OVERRIDES will prepend classpath to CLASSPATH. Do you think we need both?

[echarles]

@Leemoonsoo I changed the bin/zeppelin.sh script to print the build CLASSPATH just after the execution of the java command with

echo CLASSPATH=$CLASSPATH

$(exec $ZEPPELIN_RUNNER $JAVA_OPTS -cp $CLASSPATH_OVERRIDES:$CLASSPATH $ZEPPELIN_SERVER "$@")

When I run exporting, I get

$ ZEPPELIN_CLASSPATH=MYJAR.JAR ./bin/zeppelin.sh
CLASSPATH=/opt/apache-ivy-2.3.0-rc1/ivy-2.3.0-rc1.jar:/opt/clojure-1.6.0/clojure-1.6.0.jar::/opt/zeppelin-incubator/zeppelin-server/target/lib/*:/opt/zeppelin-incubator/zeppelin-zengine/target/lib/*:/opt/zeppelin-incubator/zeppelin-interpreter/target/lib/*:/opt/zeppelin-incubator/*:MYJAR.JAR:/opt/zeppelin-incubator/conf:/opt/zeppelin-incubator/zeppelin-interpreter/target/classes:/opt/zeppelin-incubator/zeppelin-zengine/target/classes:/opt/zeppelin-incubator/zeppelin-server/target/classes
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-server/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-zengine/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/zeppelin-incubator/zeppelin-interpreter/target/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
...

which shows the MYJAR.JAR is not taken into account, hence the need for CLASSPATH_OVERRIDES.

This is why we need a way to override by prepending, not appending.

This is useful when we deploy with e.g. a patched jar.

[radekg]

@Leemoonsoo that is exactly the point. JVM will use first classes found on the class path. The user can do the following:

export CLASSPATH_OVERRIDES=/opt/whatever/spark-core_2.10-1.5.1-patched-and-fixed.jar

And then it does not matter what org.apache.spark classes (as long as class version matches) are further down the class path: http://stackoverflow.com/a/6644467/56250

The current mechanism does not allow hot patching anything. The only patching mechanism is recompiling Zeppelin with modified JARs, which maybe difficult / not doable.

Zeppelin prepends new entries but there is no way to tell it "after all default prepends, add this at the start". And replying to your 2 comments - yes, indeed :)

[echarles]

I rebased here, but should have done for #385

[Leemoonsoo]

I think you have merged other branches. Could you manage this PR includes only your commits?

[echarles]

#398 replaces this one.