sync update #4
Merged
sync update #4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
### What is this PR for? The root cause of this issue is that we didn't redirect java output to interpreter output. This PR fix it via redirect java output before interpreting python code in both PySparkInterpreter & IPySparkInterpreter. Unit test is also added to verify this feature. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4692 ### How should this be tested? * Unit test is added, also manually tested it ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? NO Author: Jeff Zhang <[email protected]> Closes #3696 from zjffdu/ZEPPELIN-4692 and squashes the following commits: e1a5ead [Jeff Zhang] [ZEPPELIN-4692]. zeppelin pyspark doesn't print java output
… interpreter ### What is this PR for? Yarn client mode and local mode will run driver in the same machine with zeppelin server, this would be dangerous for production. Because it may run out of memory when there's many spark interpreters running at the same time. So in this PR, I introduce one configuration `zeppelin.spark.only_yarn_cluster`, by default it is turned off. When it is turned on, only yarn-cluster mode can be used. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4673 ### How should this be tested? * Manually tested ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3697 from zjffdu/ZEPPELIN-4673 and squashes the following commits: d51e423 [Jeff Zhang] [ZEPPELIN-4673]. Add option to only allow yarn-cluster mode for spark interpreter
### What is this PR for? This PR just filter the empty env in SparkInterpreterLauncher, only pass non-empty env to spark interpreter. Otherwise PySparkInterpreter will fail to launch due to invalid `SPARK_HOME` ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4698 ### How should this be tested? * Unit test is added ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3701 from zjffdu/ZEPPELIN-4698 and squashes the following commits: 41aa620 [Jeff Zhang] [ZEPPELIN-4698]. empty env is passed to spark interpreter
### What is this PR for? it adds missing license header to some files in the zeppelin-web-angular project ### What type of PR is it? Bug Fix ### What is the Jira issue? * ZEPPELIN-4701 ### How should this be tested? * Doesn't require testing Author: Alex Ott <[email protected]> Closes #3703 from alexott/ZEPPELIN-4701 and squashes the following commits: ece5870 [Alex Ott] [ZEPPELIN-4701] some zeppelin-web-angular files don't have license
… message ### What is this PR for? Trivial PR to fix the NPE in SparkSqlInterpreter. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4704 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3706 from zjffdu/ZEPPELIN-4704 and squashes the following commits: 8d68c5f [Jeff Zhang] [ZEPPELIN-4704]. NPE in SparkSqlInterpreter cause no meaningful error message
maybe misspelling(`intake-parquet=0.2.2 altair=3.2.0` should be `intake-parquet==0.2.2 altair==3.2.0`) Author: Hchen <[email protected]> Closes #3705 from pzghost/typo and squashes the following commits: f8ea50c [Hchen] typo (cherry picked from commit 1df4adf) Signed-off-by: Lee moon soo <[email protected]>
…external endpoint ### What is this PR for? Zeppelin uses the Quartz scheduler which has a built in update checker. This reaches out to a Terracotta update server which leaks some information about the running server. This is recommended to be disabled in the docs. http://www.quartz-scheduler.org/documentation/2.4.0-SNAPSHOT/best-practices.html#skip-update-check ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4697 ### How should this be tested? * CI tests * Manually confirmed this removes the update check call to the Terracotta server ### Questions: * Does the licenses files need update? - No * Is there breaking changes for older versions? - No * Does this needs documentation? - No Author: Kevin Risden <[email protected]> Closes #3700 from risdenk/ZEPPELIN-4697 and squashes the following commits: 511d868 [Kevin Risden] [ZEPPELIN-4697] Zeppelin Quartz scheduler checks for updates against external endpoint
spelling mistake Author: Trevor-zhang <[email protected]> Closes #3708 from Trevor-zhang/master and squashes the following commits: 91b2092 [Trevor-zhang] spelling mistake
…onnect presto via JDBC generic interpreter ### What is this PR for? This PR is to fix a issue of connect presto via jdbc interpreter when more than one element needs to be removed from the jdbc properties. In this PR, I only change a method which iterate keys from jdbc properties. I tested it manully with presto-0.215. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4721 ### How should this be tested? Compiles locally, no unit test performed. There should has more than one element needs to be removed in the properties. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: qiguoxiaosheng <[email protected]> Closes #3712 from qiguoxiaosheng/master and squashes the following commits: 1ecb53e [qiguoxiaosheng] modify the presto properties
### What is this PR for? Updates travis file to new semantic ### What type of PR is it? * Improvement ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4719 ### How should this be tested? * Checkout _Build config validation_ in travis job * **Travis-Link**: https://travis-ci.org/github/Reamer/zeppelin/builds/670088229 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3714 from Reamer/update_travis_config and squashes the following commits: 462c9a8 [Philipp Dallig] Use new travis semantic
### What is this PR for? Fixed a regression in #3668. With this fix zeppelin can save the notebook in HDFS. ### What type of PR is it? Regression Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4718 ### How should this be tested? * **Travis-Link**: https://travis-ci.org/github/Reamer/zeppelin/builds/670126594 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3713 from Reamer/notebook_dir_hdfs and squashes the following commits: 0117609 [Philipp Dallig] Check for filesystem with a scheme
### What is this PR for? Zeppelin when installed with default configuration options doesn't enable the common web application security headers, e.g. zeppelin.server.xframe.options, zeppelin.server.xxss.protection, zeppelin.server.jetty.name, zeppelin.server.xcontent.type.options. This leaves the Zeppelin installation vulnerable. ### What type of PR is it? Improvement ### Todos * Discuss HSTS header config (zeppelin.server.strict.transport) which if enabled requires TLS to be configured for Zeppelin to work ### What is the Jira issue? * [ZEPPELIN-4723](https://issues.apache.org/jira/browse/ZEPPELIN-4723) ### How should this be tested? * Below headers can be verified with received HTTP response Server: X-Content-Type-Options: nosniff X-FRAME-OPTIONS: SAMEORIGIN X-XSS-Protection: 1; mode=block Here is Travis test run link which passed: https://travis-ci.org/github/krishna-pandey/zeppelin/builds/670946421 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Krishna Pandey <[email protected]> Closes #3716 from krishna-pandey/ZEPPELIN-4723 and squashes the following commits: afde17f [Krishna Pandey] Added default config values 01d3040 [Krishna Pandey] Added documentation for enabled headers with default values f7578ad [Krishna Pandey] removed HSTS header, requires TLS config for Zeppelin to start 6fbd30e [Krishna Pandey] Enable HTTP security headers by default
### What is this PR for? This PR is to fix the issue of FLINK-16936 by a workaround, already creating tableenv before execution scala or python code. Building tablenv is pretty light which won't cost much time. So it is acceptable for this workaround. Another this PR try to fix is the ClassLoader issue for PyFlinkInterpreter. This PR will always set classloader before executing python code so that pyflink api can call udf defined in scala. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4714 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3715 from zjffdu/ZEPPELIN-4714 and squashes the following commits: 55d6135 [Jeff Zhang] [ZEPPELIN-4714]. Flink table api doesn't work in multiple threads
### What is this PR for? Minor PR too change default value of flink.yarn.queue should be `default` which is usually the default queue of yarn. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4729 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? NO Author: Jeff Zhang <[email protected]> Closes #3720 from zjffdu/ZEPPELIN-4729 and squashes the following commits: 84690a1 [Jeff Zhang] [ZEPPELIN-4729]. Default value of flink.yarn.queue should be `default`
…refused ### What is this PR for? - an interpreter process refused to connect due to it abnormal stop , which eventually caused the remove paragraph operation to fail to complete when i try remove a paragraph. ### What type of PR is it? - Bug Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4712 ### Questions: Does the licenses files need update? NO Is there breaking changes for older versions? NO Does this needs documentation? NO Author: xiejiajun <[email protected]> Author: xie-jia-jun <[email protected]> Closes #3710 from xiejiajun/branch-0.9 and squashes the following commits: 9bb7341 [xiejiajun] Merge remote-tracking branch 'origin/branch-0.9' into branch-0.9 9bc5605 [xiejiajun] bug fix: when we removing a paragraph , an interpreter process refused to connect due to it abnormal stop , which eventually caused the remove paragraph operation to fail to complete. 97d2714 [xie-jia-jun] Merge pull request #2 from apache/branch-0.9 9b3c744 [xiejiajun] added timeout for getting Thrift client to avoid situations where the interpreter may not be restarted when the interpreter process exits unexpectedly (cherry picked from commit 3fdd8bb) Signed-off-by: Jeff Zhang <[email protected]>
### What is this PR for?
This is for injecting appId into custom spark url. Currently we allow user to set `zeppelin.spark.uiWebUrl` for a custom spark ui link. But we didn't inject appId into it. this make it less flexible for some cases. So this PR is to allow inject application id into custom spark url. e.g.
`url_prefix/{{applicationId}}`
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4710
### How should this be tested?
* Unit test added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <[email protected]>
Closes #3711 from zjffdu/ZEPPELIN-4710 and squashes the following commits:
a887199 [Jeff Zhang] remove code duplicate
68fceec [Jeff Zhang] [ZEPPELIN-4710]. Allow to inject application id into custom spark url
### What is this PR for? Remove deprecation for SslContextFactory. https://www.eclipse.org/jetty/javadoc/9.4.26.v20200117/org/eclipse/jetty/util/ssl/SslContextFactory.html#%3Cinit%3E() Sorry for this little change. My IDE alerts me to this deprecation every time. ### What type of PR is it? * Improvement ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4736 ### How should this be tested? * **Travis-CI**: https://travis-ci.org/github/Reamer/zeppelin/builds/672439814 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3727 from Reamer/direct_SslContextFactory_deprecation and squashes the following commits: 2fdb39d [Philipp Dallig] The direct use of SslContextFactory is deprecated
### What is this PR for? jetty improvement jetty/jetty.project#3892 ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? * ZEPPELIN-4733 ### How should this be tested? * https://travis-ci.org/github/ChowXu/zeppelin/builds/672386044 ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation?No Author: xuzhou <[email protected]> Closes #3725 from ChowXu/ZEPPELIN-4733 and squashes the following commits: e645878 [xuzhou] [ZEPPELIN-4733]Upgrade jetty 9.4.18.v20190429 to 9.4.27.v20200227
…terpreterLauncher ### What is this PR for? This PR is to just merge process env into interpreter process env in InterpreterLauncher so that env defined in zeppelin-env.sh also apply to interpreters. e.g. `FLINK_HOME` specified in `zeppelin-env.sh` will apply on flink interpreter. But flink interpreter can override it. ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4725 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3719 from zjffdu/ZEPPELIN-4725 and squashes the following commits: 33ecff4 [Jeff Zhang] address comment 41f1f87 [Jeff Zhang] [ZEPPELIN-4725]. Merge process env into interpreter process env in InterpreterLauncher
### What is this PR for? Remove system property for quartz scheduler update check and some cleanup in Quartz-Scheduler. Update check was removed in quartz-scheduler/quartz@e1f7846 ### What type of PR is it? Bug Fix ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4772 ### How should this be tested? * Travis-CI: https://travis-ci.org/github/Reamer/zeppelin/builds/680159756 ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3751 from Reamer/quartz_scheduler and squashes the following commits: bc79d97 [Philipp Dallig] Remove system property for skipping quartz scheduler update check 90d6fc8 [Philipp Dallig] Some cleanup in QuartzScheduler
### What is this PR for? Solves https://issues.apache.org/jira/browse/ZEPPELIN-4783 ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-4783 ### How should this be tested? Run the docker build and see no warning with the new Key. Checks that from an inherited docker image we can now apt-get update without warnings and apt-get install succeed also, so the build of he inherited file is OK. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Yann PETIT <[email protected]> Closes #3755 from finalspy/branch-0.9 and squashes the following commits: 1cde033 [Yann PETIT] [ZEPPELIN-4783] Avoid warning on PUBKEY when apt update cran.rstudio (cherry picked from commit f256c4d) Signed-off-by: Lee moon soo <[email protected]>
### What is this PR for? With this PR, we render the K8s-Service name into the Zeppelin interpreter start command for the initial connection. The K8s-Service name should be variable, because the name can vary from environment to environment. ### What type of PR is it? - Bug Fix ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4782 ### How should this be tested? * Travis-CI: https://travis-ci.org/github/Reamer/zeppelin/builds/680915013 ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3753 from Reamer/k8s_service and squashes the following commits: 74be24f [Philipp Dallig] Use K8s service to connect Zeppelin interpreter with Zeppelin server 7ecee5d [Philipp Dallig] Some cleanup 84e4e33 [Philipp Dallig] Add ".factorypath" to .gitignore it's created by eclipse
### What is this PR for? This PR refactors Cassandra interpreter to use new DataStax Java driver (4.x) that has new features & new architecture, that caused significant changes in the code base. But as result, we're getting: * We get supported version of the driver (3.x line will only receive critical bug fixes) * We get support for additional functionality of the DSE (DataStax Enterprise) - security, additional data types, etc. * Improvements in the load balancing, etc. ### What type of PR is it? Refactoring ### What is the Jira issue? * ZEPPELIN-4378 ### How should this be tested? * Travis CI build: https://travis-ci.org/github/alexott/zeppelin/builds/681878444 * Was tested manually in addition to the unit tests ### Questions: * Does the licenses files need update? Done, minor updates * Is there breaking changes for older versions? There are renames & deletion of the configuration properties, but this doesn't break existing setups. The only breaking change is removal of "standard" retry policies from syntax - but it was necessary to do, as they not shipped with driver anymore. * Does this needs documentation? Documentation was updated Author: Alex Ott <[email protected]> Closes #3699 from alexott/ZEPPELIN-4378 and squashes the following commits: 1c91453 [Alex Ott] [ZEPPELIN-4378] porting Cassandra interpreter to new driver
### What is this PR for? The goal is to add support for querying InfluxDB 2.x using Flux language in Zeppelin notebook. InfluxDB 2.0 (beta) docs https://v2.docs.influxdata.com/v2.0/ Flux language docs: https://docs.influxdata.com/flux/ ### What type of PR is it? Feature ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN-4602 ### How should this be tested? * First time? Setup Travis CI as described on https://zeppelin.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. ### Screenshots (if appropriate) linked in docs/interpreter/influxdb.md Author: Robert Hajek <[email protected]> Closes #3640 from rhajek/master and squashes the following commits: 206848e [Robert Hajek] [ZEPPELIN-4602] updated influxdb client libraries to 1.7.0, added support for InfluxDB 1.8, interpreter code cleanup 34cd9ed [Robert Hajek] [ZEPPELIN-4602] BaseZeppelinContext replaced 8955205 [Robert Hajek] [ZEPPELIN-4602] Updated README.md, removed specific maven-checkstyle-plugin configuration 04b7e28 [Robert Hajek] [ZEPPELIN-4602] Added licences b6b9a48 [Robert Hajek] [ZEPPELIN-4602] Added initial version of InfluxDB interpreter
### What is this PR for? Right now, if you merge PR into 2 branches, like, `master`, and `branch-0.9`, script generates suggestion for "Fix Version" as `0.9.0,0.9.0`, because both master & 0.9 branch are using the same base version. This small fix leaves only unique values for suggested version(s). ### What type of PR is it? Development Improvement Author: Alex Ott <[email protected]> Closes #3757 from alexott/make-version-list-unique and squashes the following commits: 53337e8 [Alex Ott] [dev fix] make fix_version unique
### What is this PR for? This PR is to update and add more flink tutorial notes. After this PR, there will be 8 flink tutorials notes. I also add number for each note so that beginner users can know which note should read first. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4767 ### How should this be tested? * CI pass ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3756 from zjffdu/ZEPPELIN-4767 and squashes the following commits: ddda367 [Jeff Zhang] [ZEPPELIN-4767]. Update and add more flink tutorial notes
### What is this PR for? This PR is to resolve the paragraph shaking issue further. This PR use angular instead of html, so that each time we only update the angular object instead of the whole result. This can totally resolve the paragraph shaking issue. ### What type of PR is it? [ Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4780 ### How should this be tested? * CI ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3754 from zjffdu/ZEPPELIN-4780 and squashes the following commits: a901df5 [Jeff Zhang] [ZEPPELIN-4780]. Use angular instead of html for flink streaming output
### What is this PR for? on MacOS, `[0-9]\+` doesn't work, so it was replaced with `[0-9][0-9]*` ### What type of PR is it? Bug Fix ### What is the Jira issue? * ZEPPELIN-4791 ### How should this be tested? * Tested manually on Mac & Linux * https://travis-ci.org/github/alexott/zeppelin/builds/682342684 Author: Alex Ott <[email protected]> Closes #3758 from alexott/fix-java-version-check-on-mac and squashes the following commits: 9a77874 [Alex Ott] [HotFix] Fix Java version check on MacOS
### What is this PR for? Position literals first in comparisons, if the second argument is null then NullPointerExceptions can be avoided, they will just return false. Then I will move to annotations and other issues that my static analyzer shows (My previous branch was removed because of the multiple problems, so I decided to start over) ### What type of PR is it? Refactoring ### What is the Jira issue? This task is not on jira, but I'll move to this https://issues.apache.org/jira/browse/ZEPPELIN-3225?filter=12348928 ### How should this be tested? https://travis-ci.org/github/KirillTurchinskii/zeppelin/builds/682716706 ### Screenshots (if appropriate) ### Questions: * What else besides annotations (that I will add) I can fix/refactor using static analysis? Author: Kirill Turchinskii <[email protected]> Closes #3760 from KirillTurchinskii/fix2 and squashes the following commits: 6b4ce16 [Kirill Turchinskii] Position literals first in comparisons, if the second argument is null then NullPointerExceptions can be avoided, they will just return false.
### What is this PR for? This PR is to support `z.show(table)` in flink interpreter. It could not only be used in scala api but also pyflink api. See the screenshot below. ### What type of PR is it? [ Feature ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4800 ### How should this be tested? * CI pass ### Screenshots (if appropriate)   ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3762 from zjffdu/ZEPPELIN-4800 and squashes the following commits: 713b999 [Jeff Zhang] [ZEPPELIN-4800]. Support z.show(table) in flink interpreter
### What is this PR for? This PR will throw exception when using flink for scala 2.12 which is not supported now. See the screenshot below. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4790 ### How should this be tested? * Manually tested. ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3763 from zjffdu/ZEPPELIN-4790 and squashes the following commits: eac7b9a [Jeff Zhang] address comment 57b0737 [Jeff Zhang] [ZEPPELIN-4790]. Throw exception when using flink for scala 2.12
### What is this PR for? Before this PR, user have to set property `default.splitQueries` to enable multiple queries in one paragraph, this PR remove this property, this PR remove this property, and make multiple sql query in support by default. ### What type of PR is it? [Feature | Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4797 ### How should this be tested? * Unit test is updated ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3764 from zjffdu/ZEPPELIN-4797 and squashes the following commits: 2323470 [Jeff Zhang] address comment fcb9e61 [Jeff Zhang] [ZEPPELIN-4797]. Enable multiple sql in jdbc interpreter by default
### What is this PR for? This PR is to use isolated interpreter for cron note job. Cron job is for production job, so we should use dedicated interpreter for it instead of using shared interpreter with other notes or users. Currently interpreter binding mode is determined by note or users (scoped or isolated), This PR introduce `ExecutionContext` which include other factors (isCronMode) to determine how we create interpreters. ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4763 ### How should this be tested? * Unit test is updated, and manually tested ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3765 from zjffdu/ZEPPELIN-4763 and squashes the following commits: 39147c3 [Jeff Zhang] address comment 7bc690d [Jeff Zhang] [ZEPPELIN-4763]. Use isolated interpreter for cron note job
### What is this PR for? This PR is to just reorgnize zeppelin tutorial notes, so that user know which tutorial to read first. ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4795 ### How should this be tested? * No test needed ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3766 from zjffdu/ZEPPELIN-4795 and squashes the following commits: 801d0fd [Jeff Zhang] [ZEPPELIN-4795]. Update zeppelin tutorial notes
…in cluster mode ### What is this PR for? Flink doesn't work in cluster mode because we didn't use `FlinkInterpreterLauncher` in `DockerInterpreterLauncher` which is used in cluster mode. It is a straightforward change to just use `FlinkInterpreterLauncher` in `DockerInterpreterLauncher` ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4809 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3768 from zjffdu/ZEPPELIN-4809 and squashes the following commits: 1733a71 [Jeff Zhang] [ZEPPELIN-4809]. Zeppelin FlinkInterpreterLauncher not work in Zeppelin cluster mode
… yarn mode for flink interpreter ### What is this PR for? This PR actually solve 2 things in flink yarn mode. 1. You don't need flink-hadoop-shaded jar when flink is not in yarn mode (before this PR, it is still needed even when it is not in yarn mode) 2. Throw proper error message when hadoop jar is missing in yarn mode. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4806 ### How should this be tested? * CI pass ### Screenshots (if appropriate)  ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? NO Author: Jeff Zhang <[email protected]> Closes #3767 from zjffdu/ZEPPELIN-4806 and squashes the following commits: 5b75838 [Jeff Zhang] [ZEPPELIN-4806]. Throw proper exception when hadoop jar is missing in yarn mode for flink interpreter
…and on 300/400 pages ### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html ### What type of PR is it? [Improvement] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4586 ### How should this be tested? * Travis should pass With default`zeppelin.server.send.jetty.name = true` ``` $ curl -i localhost:8080/nonexisting HTTP/1.1 404 Not Found Date: Fri, 31 Jan 2020 19:16:33 GMT Access-Control-Allow-Credentials: true Access-Control-Allow-Headers: authorization,Content-Type Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, HEAD, DELETE X-FRAME-OPTIONS: SAMEORIGIN X-XSS-Protection: 1 X-Content-Type-Options: nosniff Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Content-Length: 328 Server: Jetty(9.4.18.v20190429) <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /nonexisting. Reason: <pre> Not Found</pre></p><hr><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.4.18.v20190429</a><hr/> </body> </html> ``` By changing to false `zeppelin.server.send.jetty.name = false` ``` $ curl -i localhost:8080/nonexisting HTTP/1.1 404 Not Found Date: Fri, 31 Jan 2020 19:15:57 GMT Access-Control-Allow-Credentials: true Access-Control-Allow-Headers: authorization,Content-Type Access-Control-Allow-Methods: POST, GET, OPTIONS, PUT, HEAD, DELETE X-FRAME-OPTIONS: SAMEORIGIN X-XSS-Protection: 1 X-Content-Type-Options: nosniff Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=iso-8859-1 Content-Length: 243 <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /nonexisting. Reason: <pre> Not Found</pre></p> </body> </html> ``` ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes Author: Javier Fuentes <[email protected]> Author: Javier <[email protected]> Closes #3622 from javierivanov/ZEPPELIN-SENDVERSION and squashes the following commits: 44e0474 [Javier Fuentes] fixed template b720757 [Javier Fuentes] fixes 430b92d [Javier] added parameter and doc
### What is this PR for? With this PR, we use spark configuration values for K8s Pod resources. A memory limit is not set because of a potential OOM-Killer. https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits > If you set a memory limit of 4GiB for that Container, the kubelet (and container runtime ) enforce the limit. The runtime prevents the container from using more than the configured resource limit. For example: when a process in the container tries to consume more than the allowed amount of memory, the system kernel terminates the process that attempted the allocation, with an out of memory (OOM) error. zjffdu Are using a YARN cluster to schedule your Interpreters? Maybe we should change the location of the calculation class. ### What type of PR is it? - Improvement ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4799 ### How should this be tested? * **Travis-CI**: https://travis-ci.org/github/Reamer/zeppelin/builds/683269227 ### Questions: * **Maybe a higher default memory overhead?** Edit: No, because we doesn't need it in the past. * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Philipp Dallig <[email protected]> Closes #3761 from Reamer/spark_k8s_resources and squashes the following commits: 64bd912 [Philipp Dallig] Add short comment for limits.memory 45e565f [Philipp Dallig] Use Spark config values for K8s Interpreter Pod resources 3218309 [Philipp Dallig] Some cleanup
…ntial file ### What is this PR for? This is a trivial PR which remove `loadCredentials` in `getUserCredentials`. It is not necessary and it would cause each `getInterpreterContext` call will load user credential file. ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4778 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3770 from zjffdu/ZEPPELIN-4778 and squashes the following commits: 47cba07 [Jeff Zhang] [ZEPPELIN-4778] Use entry.user & entry.password instead of user.entry and password.entry for credential injection 8453460 [Jeff Zhang] [ZEPPELIN-4778]. Each getInterpreterContext call will load user credential file
…er credential file" This reverts commit f92327a.
… and password.entry for credential injection ### What is this PR for? This is a refactoring PR which just use entry.user & entry.password instead of user.entry and password.entry for credential injection . ### What type of PR is it? [Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4726 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3771 from zjffdu/ZEPPELIN-4726 and squashes the following commits: fb3a8cb [Jeff Zhang] [ZEPPELIN-4726] Use entry.user & entry.password instead of user.entry and password.entry for credential injection
### What is this PR for? This PR is to fix the issue that no error message returned of running paragraph. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4805 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Jeff Zhang <[email protected]> Closes #3772 from zjffdu/ZEPPELIN-4805 and squashes the following commits: e8ed2f0 [Jeff Zhang] [ZEPPELIN-4805]. Error Stack Trace not received on rest API
### What is this PR for?
Before this PR, the exported `ipynb` is always `undefined.zepl.ipynb` because note name is not passed correctly, this PR fix this issue and the exported ipynb will be `{note_name}.ipynb`
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4823
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang <[email protected]>
Closes #3774 from zjffdu/ZEPPELIN-4823 and squashes the following commits:
b5af8aa [Jeff Zhang] [ZEPPELIN-4823]. exported ipynb file name is undefined
… paragraph execution order ### What is this PR for? Before this PR, if user run the note in non-blocking way via rest api, the execution order of paragraphs is not determined, paragraph 2 will start to run after paragraph 1 enter running state. But we should only start paragraph 2 when paragraph 1 is finished. This PR fix this issue by minor change the `RemoteScheduler` and introduce `.execution.mode` to tell `RemoteScheduler` that whether the job is running as individual paragraph or as part of note execution. ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4832 ### How should this be tested? * Unit test is added ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang <[email protected]> Closes #3775 from zjffdu/ZEPPELIN-4832 and squashes the following commits: 36338a8 [Jeff Zhang] [ZEPPELIN-4832]. run note in non-blocking way could not guarantee the paragraph execution order
xiejiajun
added a commit
that referenced
this pull request
Nov 15, 2020
…Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ### What is this PR for? - fix the [CI failure](https://travis-ci.org/github/apache/zeppelin/builds/709913046) due to [PR-3852](apache#3852) ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * [ZEPPELIN-4962](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-4962) ### How should this be tested? * CI test ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? Yes Author: xiejiajun <[email protected]> Author: xie-jia-jun <[email protected]> Author: JakeXie <[email protected]> Closes apache#3860 from xiejiajun/ZEPPELIN-4962 and squashes the following commits: 9128c9b [JakeXie] spark.repl.target docs update ad4c0e3 [xiejiajun] Clear irrelevant code a12d3a9 [xiejiajun] Support for manually specifying the Java version of Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ab2b191 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 5569788 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 0a9af6c [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master be36b37 [xiejiajun] 合并Apache Master分支冲突解决 1335d55 [xiejiajun] Merge remote-tracking branch 'origin/master' fc59f57 [JakeXie] Merge pull request #4 from apache/master 9cc70fe [xiejiajun] Merge remote-tracking branch 'origin/master' 6ef9b23 [xie-jia-jun] Merge pull request #3 from apache/master 45af87a [xiejiajun] added timeout for getting Thrift client to avoid situations where the interpreter may not be restarted when the interpreter process exits unexpectedly f149c3b [xie-jia-jun] Merge pull request #1 from apache/master 5d4b645 [xie-jia-jun] Support OSSConfigStorage of Aliyun dbb6639 [xie-jia-jun] Add Aliyun OSS SDK bb47849 [xie-jia-jun] Support S3ConfigStorage of AWS
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
Todos
What is the Jira issue?
How should this be tested?
Screenshots (if appropriate)
Questions: