-
Notifications
You must be signed in to change notification settings - Fork 114
Run interpreter on separate JVM #364
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
I've failed to launch Zeppelin in this branch. Attaching log. ==> ./logs/zeppelin-swkim-SangWoos-MacBook-Pro.local.out <== ==> ./logs/zeppelin-swkim-SangWoos-MacBook-Pro.local.log <== INFO [2015-02-27 15:22:48,484]({main} StdSchedulerFactory.java[instantiate]:1339) - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties' ==> ./logs/zeppelin-swkim-SangWoos-MacBook-Pro.local.out <== ==> ./logs/zeppelin-swkim-SangWoos-MacBook-Pro.local.log <== |
|
@swkimme I've pushed a fix. to , to test this branch. (it results all interpreter run in separate process. After GUI implemented, it'll be able to selectively turn on/off) |
|
Now it works! |
Conflicts: zeppelin-web/app/scripts/controllers/interpreter.js
|
I got exception while running %dep |
|
Also got similar exception while running a simple paragraph in spark local mode. |
|
I tested this branch, |
|
@swkimme I think i fixed exception related to getFormType(). |
|
The issue still exists in Spark 1.3. |
|
Created an issue for it. |
|
Ready to be merged. Please review and test. This PR resolves classloader issue by separating JVM process. The issues was blocking support for Spark 1.2.x. I'm running this branch with some spark 1.2.1 clusters and everything looks fine to me. Changes
|
|
@Leemoonsoo |
|
@swkimme |
|
@Leemoonsoo |
|
@swkimme No problem. I'll update the code to not show 'fork' option. |
|
@swkimme I've removed fork option. |
|
LGTM and tested with complex codes, works great. |
|
I'm merging this PR if there're no more discussions! |
|
@Leemoonsoo This PR doesn't pass ZEPPELIN_JAVA_OPTS. I have a ZEPPELIN_JAVA_OPTS="-Dspark.io.compress.codec=lzf". This is configured on ZeppelinServer but spark doesn't read this. I'm debugging now. |
|
@Leemoonsoo Unlikely from zeppelin-daemon.sh, interpreter.sh launches interpreter without JAVA_OPTS. Please add this when Z executes remoteInterpreter. |
|
@jongyoul Thanks for the feedback. I just pushed a fix. The problem was because of ZEPPELIN_JAVA_OPTS is not used in interpreter JVM process. conf/zeppelin-env.sh.templates have short description. |
|
@Leemoonsoo Good! |
|
@jongyoul |
Run interpreter on separate JVM
Update documentation for separate interpreter process #364
This PR came from ZEPL/zeppelin#397 After ZEPL/zeppelin#364, order of submitted job to the remote interpreter is not preserved. Result is, when user run a lot of paragraphs in a short time, they run in random order(like click run 'Run all paragraphs). https://github.com/NFLabs/zeppelin/issues/395 is one of the possible problem. this PR adds some unittests and fix the problem. Ready to merge. Author: Lee moon soo <leemoonsoo@gmail.com> Author: Lee moon soo <moon@apache.org> Closes #8 from Leemoonsoo/fix/run_order and squashes the following commits: 4a6a230 [Lee moon soo] Prevent status remaining RUNNING after job executed 1fb3715 [Lee moon soo] Fix daedlock 6405abe [Lee moon soo] Update RemoteSchedulerTest c31a807 [Lee moon soo] Make RemoteScheduler keep order of job submitted
This PR addresses https://issues.apache.org/jira/browse/ZEPPELIN-360 Removing "Copyright 2014, NFLabs inc." from LICENSE. Author: Lee moon soo <moon@apache.org> Closes ZEPL#364 from Leemoonsoo/ZEPPELIN-360 and squashes the following commits: cfb814b [Lee moon soo] Remove "Copyright (c) 2010-2015 NFLabs Inc." from NOTICE fb75e86 [Lee moon soo] ZEPPELIN-360 Remove "Copyright 2014, NFLabs inc." from LICENSE

Running interpreter in separate process discussed in #278 and mailinglist. @bzz made PoC of it https://github.com/bzz/zeppelin-multiprocess-interpreter-poc
This PR implements #278 using thrift.