Skip to content

Conversation

@Leemoonsoo
Copy link
Member

When Zeppelin constructs JVM CLASSPATH, it adds path of every single jar. That creates really long CLASSPATH and it may cause some problem like https://issues.apache.org/jira/browse/ZEPPELIN-68.
This PR solves it by constructing CLASSPATH with wildcard.

Another problem is, Zeppelin constructs CLASSPATH with duplicated entry or not related entry.
This PR solves it by constructing CLASSPATH not in common.sh but in each script zeppelin-daemon.sh, interpreter.sh, zeppelin.sh while they need different entry in CLASSPATH.

@bzz
Copy link
Member

bzz commented May 10, 2015

I tried spark-local and md interpreters both from project root (dev mode) and from packaged distribution artchive - both worked well for me.

@Leemoonsoo do you know why on CI we have RAT check failure? Do not see any changes in licences in this PR...

[INFO] Zeppelin ........................................... FAILURE [  6.111 s]
[INFO] Zeppelin: Interpreter .............................. SKIPPED
....
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (verify.rat) on project zeppelin: Too many files with unapproved license: 35 See RAT report in: /home/travis/build/apache/incubator-zeppelin/target/rat.txt -> [Help 1]

@khalidhuseynov can you please also try these changes with the latest code for thift-tajo-server?
May be I missed you PR as can not find that code anywhere.

@Leemoonsoo
Copy link
Member Author

@bzz it's strange. I'm trying to figure out why RAT check is failing.

@bzz
Copy link
Member

bzz commented May 10, 2015

same happened with me on #30 ... but #67 is OK, on top of the same master

can RAT check be flaky?

@khalidhuseynov
Copy link
Member

Yes, it solved the problem with CLASSPATH length and path duplication. Tajo thrift interpreter works fine in my local build now. Looks good to me.

@Leemoonsoo
Copy link
Member Author

RAT check looks like flaky. I made CI to print rat.txt when it fails. so next time, we'll know what causes RAT check fail.

Ready to merge!

@bzz
Copy link
Member

bzz commented May 13, 2015

Looks great to me! Thank you for printing RAT, will check it next time
Lets merge

@asfgit asfgit closed this in dcb03a9 May 13, 2015
@RPCMoritz
Copy link
Contributor

While this was a decent effort (it actually made my classpath-related error almost readable, where previously it was three screens worth of text), there is still an issue with classpath's lack of simplicity. In partcular, (in combination with yarn) it pulls in (without any obvious documentation) many different classpaths, some of which may depend on environment variables. I'm currently (Zeppelin master, HDP 2.2w/ spark tech preview on yarn) getting "bad substitution" errors from launch_container.sh:27, an issue similar to Ambari-8028 (https://issues.apache.org/jira/browse/AMBARI-8028) and which I suspect is down to remaining references to ${hdp.version} (and potentially $PWD, $HADOOP_CONF_DIR), which I'm chasing down now.

I think it might be a good idea to move the classpath over to zeppelin-site.xml in any case, since over the long term managing the classpath via shell scripts is going to be painful and not user-friendly. Especially the CLASSPATH+= makes identifying the issue I encountered very difficult.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants