Improve spark.md #1
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
|
|
||
| > Note that scala / python environment shares the same SparkContext, SQLContext, ZeppelinContext instance. | ||
| > Note that Scala / Python environment shares the same SparkContext, SQLContext and ZeppelinContext instance. | ||
Owner
There was a problem hiding this comment.
I think R also share the same SparkContext and etc.
Owner
|
Thanks @AhyoungRyu , 2 comments, otherwise LGTM. After fixing these 2 places, I think can commit it. |
AhyoungRyu
force-pushed
the
spark_doc_fix/ahyoung
branch
from
fd8bfdd to
2336900
Compare
Author
|
@zjffdu Thanks! I addressed your feedback to my last commit . Can you check again? |
| It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs | ||
| Apache Spark is supported in Zeppelin with | ||
| Spark Interpreter group, which consists of five interpreters. | ||
| It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. |
Author
|
Typos that you mentioned are fixed. Regarding this,
I think it would be better this can be confirmed in the main PR not in here. Then not only Mina but also other ppl can verify that :) |
Owner
|
Make sense, other LGTM. 😄 |
Author
|
@zjffdu Yeah thanks! |
zjffdu
pushed a commit
that referenced
this pull request
Sep 12, 2016
### What is this PR for? There are 2 issues and their proposed fixes: 1. On a paragraph run, for every line of output, there is a broadcast of the new line from zeppelin. In case of thousands of lines of output, the browser/s would hang because of the volume of these append-output events. 2. In the above case, besides the browser-hang, another bug observed is that result data is will repeated twice (coming from append-output calls + finish-event calls). The proposed solution for #1 is: - Buffer the append-output event into a queue instead of sending the event immediately. - In a separate thread, read from the queue periodically and send the append-output event. Solution for #2 is: - Donot append output to result if the paragraph is not runnig. ### What type of PR is it? Improvement + Bug Fix ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1292 ### How should this be tested? The test could be to run a simple paragraph with large result. Eg: ``` %sh for i in {1..10000} do echo $i done ``` PS: One will need to clear browser cache between running with and without this code patch since there are javascript changes as well. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? It could need for the design. Otherwise I have added code comments explaining behaviour. Author: Beria <[email protected]> Closes apache#1283 from beriaanirudh/ZEPPELIN-1292 and squashes the following commits: 17f0524 [Beria] Use diamond operator 7852368 [Beria] nit 4b68c86 [Beria] fix checkstyle d168614 [Beria] Remove un-necessary class CheckAppendOutputRunner 2eae38e [Beria] Make AppendOutputRunner non-static 72c316d [Beria] Scheduler service to replace while loop in AppendOutputRunner 599281f [Beria] fix unit tests that run after dd24816 [Beria] Add license in test file 3984ef8 [Beria] fix tests when ran with other tests 1c893c0 [Beria] Add licensing 1bdd669 [Beria] fix javadoc comment 27790e4 [Beria] Avoid infinite loop in tests 5057bb3 [Beria] Incorporate feedback 1. Synchronize on AppendOutputRunner creation 2. Use ScheduledExecutorService instead of while loop 3. Remove Thread.sleep() from tests 82e9c4a [Beria] Fix comment 7020f0c [Beria] Buffer append output results + fix extra incorrect results
zjffdu
pushed a commit
that referenced
this pull request
Sep 22, 2016
### What is this PR for? Several changes on doc of spark interpreter. * %spark, %sql, %pyspark only works when spark is the default interpreter group of note. So I update the doc to use the full interpreter name. * Add SparkSession for 2.0 * Also add comments inline with other changes to explain the reason. ### What type of PR is it? [Documentation] ### Todos * [ ] - Task ### What is the Jira issue? * No jira created. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: AhyoungRyu <[email protected]> Author: Jeff Zhang <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#1398 from zjffdu/spark_doc_fix and squashes the following commits: ac01f2b [Jeff Zhang] Merge pull request #1 from AhyoungRyu/spark_doc_fix/ahyoung 5fa523f [AhyoungRyu] Fix typos 3c0f678 [AhyoungRyu] Add 'R' and refine a sentence 2336900 [AhyoungRyu] Improve spark.md 40d4b11 [Jeff Zhang] [MINOR] Doc fix for spark interpreter
zjffdu
pushed a commit
that referenced
this pull request
Oct 7, 2016
### What is this PR for? This is some refactoring of zeppelin-web to apply the IIFE as stated in the Good Practice Guide #1 (https://zeppelin.apache.org/contribution/zeppelinweb/goodPracticeGuide01.html) ### What type of PR is it? Refactoring ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1496 ### How should this be tested? Run The webapp, and go around the page, things should work like usual. But overall, just need to check that there is no type or parameter mismatch during the refactoring process (controller to controller, params in $inject etc...) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Damien CORNEAU <[email protected]> Closes apache#1469 from corneadoug/ZEPPELIN-1496 and squashes the following commits: 966cb27 [Damien CORNEAU] refactor components controllers 1807cf6 [Damien CORNEAU] Refactor components directives 0055191 [Damien CORNEAU] Refactor components service and factory f86adb4 [Damien CORNEAU] Finish refactoring of App 409c65c [Damien CORNEAU] IIFE refactoring of job page 4c45e08 [Damien CORNEAU] Refactor for IIFE credential, home and interpreter controller 7e88eef [Damien CORNEAU] Add congiguration.controller.js 6adb522 [Damien CORNEAU] Move use strict outside IFFE ffba0f3 [Damien CORNEAU] Add IIFE to app.controller.js
zjffdu
pushed a commit
that referenced
this pull request
Oct 27, 2016
### What is this PR for? Fix minor typos in docs ### What type of PR is it? Documentation ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: k <[email protected]> Author: Mina Lee <[email protected]> Author: laesunk <[email protected]> Closes apache#1542 from laesunk/fix_typo and squashes the following commits: 7b703a7 [laesunk] Merge pull request #1 from minahlee/fix_typo 28fc9ce [Mina Lee] Fix more typos a55ae4a [Mina Lee] Restore wrong changes during rebase 190ac05 [k] fix words 01ce71b [k] fix likly
zjffdu
pushed a commit
that referenced
this pull request
Oct 27, 2016
### What is this PR for? This PR fixes wrong written NotebookID to NoteID. ### What type of PR is it? [Improvement] ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1549 ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? No Author: hyonzin <[email protected]> Author: 정현진 <[email protected]> Author: Mina Lee <[email protected]> Closes apache#1518 from hyonzin/ZEPPELIN-1549 and squashes the following commits: 2c5d461 [hyonzin] fix pullNoteID to pullNoteId f843abd [hyonzin] Fix missed line 22aecb3 [hyonzin] Merge branch 'master' of https://github.com/apache/zeppelin into ZEPPELIN-1549 ac03666 [정현진] Merge pull request #1 from minahlee/ZEPPELIN-1549 8b3fffd [Mina Lee] Change notebook to note and fix indentation 000605f [hyonzin] Change clonedNotebookId to clonedNoteId 496695c [hyonzin] Change noteID to noteId 1e87463 [hyonzin] Remove tab indent 5647d37 [hyonzin] Rebase and solve conflicts 09bacd8 [hyonzin] Fix more lines unchanged 070bc2d [hyonzin] fix more in ZeppelinRestApiTest.java 24822a3 [hyonzin] Fix more code not changed (notebookIndex to noteSearchService) 4b4e1e8 [hyonzin] Fix detail (function's name) & Change some placeholder 429203d [hyonzin] Fix details & convention to camel 5fa270d [hyonzin] pull upstream master & fix some details 294bea5 [hyonzin] Fix some wrong written term: Notebook -> Note cc0d315 [hyonzin] Change NotebookID variable name to NoteID
zjffdu
pushed a commit
that referenced
this pull request
Oct 27, 2016
### What is this PR for? A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html ### What type of PR is it? [ Documentation ] ### Todos * [x] - Task ### What is the Jira issue? * Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/ * Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533] ### How should this be tested? Outline the steps to test the PR here. ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? * Is there breaking changes for older versions? * Does this needs documentation? spelling error responce => response seprated => separated Author: terrylee <[email protected]> Author: Mina Lee <[email protected]> Closes apache#1544 from hunylee/patch-1 and squashes the following commits: 1095a19 [terrylee] Merge pull request #1 from minahlee/fix_typo 9c26998 [terrylee] correct separated 1a3e187 [Mina Lee] fix typo 0490435 [terrylee] spelling error
zjffdu
pushed a commit
that referenced
this pull request
Nov 23, 2016
…Provides LdapRealm Functionality similar to Apache Knox
### What is this PR for?
ZEPPELIN-1472 - Create new LdapRealm based on Apache Knox LdapRealm: Provides LdapRealm Functionality similar to what Apache Knox provides. This is critical as in large enterprise environments Active Directory Global Catalogs are used for lookup with samAccountName and using a DN Template is not an option as their are multiple OUs. Also searching on "userPrincipalName" is risky in an AD environment since the explicit UPN vs Implicit UPN can be different this is definitely the case with environments using Office 365. And the LDAP userPrincipalName attribute is the explicit UPN which can be defined by the directory administrator to any value and it can be duplicated. SamAccountName is unique per domain and Microsoft states best practice is to not allow duplicate samAccountName's across the forest.
In addition to the above changes I have adjusted and moved the LdapGroupRealm and ActiveDirectoryGroupRealm into the org.apache.zeppelin.realm package structure to make all Realm's consistent.
The LdapRealm class also works with role to group mapping for usage within Zeppelin for notebook authorization.
I have adjusted SecurityUtils to use ClassName vs realmName in determining what to use as you may have companies that decide to use their own custom realmname in shiro.ini and may not realize you cannot so using className is much safer.
Example - SecurityUtils
String name = realm.getClass().getName();
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
allRoles = ((IniRealm) realm).getIni().get("roles");
break;
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
allRoles = ((LdapRealm) realm).getListRoles();
break;
}
Example - SecurityRestApi:
String name = realm.getClass().getName();
if (LOG.isDebugEnabled()) {
LOG.debug("RealmClass.getName: " + name);
}
if (name.equals("org.apache.shiro.realm.text.IniRealm")) {
usersList.addAll(getUserListObj.getUserList((IniRealm) realm));
rolesList.addAll(getUserListObj.getRolesList((IniRealm) realm));
} else if (name.equals("org.apache.zeppelin.realm.LdapGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((JndiLdapRealm) realm, searchText));
} else if (name.equals("org.apache.zeppelin.realm.LdapRealm")) {
usersList.addAll(getUserListObj.getUserList((LdapRealm) realm, searchText));
rolesList.addAll(getUserListObj.getRolesList((LdapRealm) realm));
} else if (name.equals("org.apache.zeppelin.realm.ActiveDirectoryGroupRealm")) {
usersList.addAll(getUserListObj.getUserList((ActiveDirectoryGroupRealm) realm,
searchText));
} else if (name.equals("org.apache.shiro.realm.jdbc.JdbcRealm")) {
usersList.addAll(getUserListObj.getUserList((JdbcRealm) realm));
}
Please see feedback from previous PRs related to this JIRA:
apache#1513
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1472
### How should this be tested?
Update shiro.ini to use configuration similar to below:
# Sample LDAP configuration, for user Authentication, currently tested for single Realm
[main]
ldapADGCRealm = org.apache.zeppelin.realm.LdapRealm
ldapADGCRealm.contextFactory.systemUsername = CN=hdplookup,OU=hadoop,DC=hdpusr,DC=senia,DC=org
ldapADGCRealm.contextFactory.systemPassword = ldapBindPassword
ldapADGCRealm.searchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.userSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.groupSearchBase = dc=hdpusr,dc=senia,dc=org
ldapADGCRealm.authorizationEnabled = true
ldapADGCRealm.contextFactory.url = ldap://seniadc1.hdpusr.senia.org:3268
ldapADGCRealm.userSearchAttributeName = sAMAccountName
ldapADGCRealm.contextFactory.authenticationMechanism = simple
ldapADGCRealm.groupObjectClass = group
ldapADGCRealm.memberAttribute = member
ldapADGCRealm.rolesByGroup = hdpeng: admin, \
hadoopusers: user
securityManager.realms = $ldapADGCRealm
sessionManager = org.apache.shiro.web.session.mgt.DefaultWebSessionManager
### If caching of user is required then uncomment below lines
#cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
#securityManager.cacheManager = $cacheManager
securityManager.sessionManager = $sessionManager
# 86,400,000 milliseconds = 24 hour
securityManager.sessionManager.globalSessionTimeout = 86400000
shiro.loginUrl = /api/login
[roles]
# 'admin' role has all permissions, indicated by the wildcard '*'
admin = *
user = *
[urls]
# anon means the access is anonymous.
# authcBasic means Basic Auth Security
# authc means Form based Auth Security
# To enfore security, comment the line below and uncomment the next one
#/api/version = anon
#/** = anon
/api/interpreter/** = authc, roles[admin]
/api/configurations/** = authc, roles[admin]
/api/credential/** = authc, roles[admin]
/api/login = authc
/api/login/logout = authc
/api/security/ticket = authc
/** = authc, roles[admin, user]
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? n
* Is there breaking changes for older versions? n
* Does this needs documentation? y
merge latest commits
Author: gss2002 <[email protected]>
Author: gss2002 <[email protected]>
Closes apache#1614 from gss2002/ZEPPELIN-1472 and squashes the following commits:
d6a7cea [gss2002] ZEPPELIN-1472 - LdapRealm Additions based on Knox LdapRealm and support of using roles with LdapRealms. Also adjusted to use className and not actual name of the realm in shiro.ini. As using realmName in code could cause problems for people who want to use alternate names. Also migrated the LdapGroupRealm and ActiveDirectoryRealm to org.apache.zeppelin.realm packages per a recommendation.
1702cc5 [gss2002] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Dec 15, 2016
### What is this PR for? This PR is for making docker images for zeppelin releases. It contains a script for building image for each release. Another script is used for publishing images to zeppelin Dockerhub account. This repo, https://github.com/mfelgamal/zeppelin-dockers, is a demonstration of this PR. It contains zeppelin-base image and an image for each zeppelin release. ### What type of PR is it? [Feature] ### Todos - Review Comments - Documentation ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-1386 ### How should this be tested? - run create_release script or publish_release script. ### Screenshots (if appropriate) ### Questions: - Does the licenses files need update? no - Is there breaking changes for older versions? no - Does this needs documentation? yes Author: mahmoudelgamal <[email protected]> Author: mfelgamal <[email protected]> Author: Mahmoud Elgamal <[email protected]> Author: 1ambda <[email protected]> Closes apache#1538 from mfelgamal/zeppelin-dockers and squashes the following commits: cc8493f [Mahmoud Elgamal] Merge pull request #3 from 1ambda/fix/remove-startzeppelinsh d48ecef [1ambda] fix: Remove start-zeppelin.sh b64c680 [mahmoudelgamal] Remove gcc and g++ for decreasing the size 1f093d4 [mahmoudelgamal] Add script start-zeppelin to zeppelin-base d2c744e [mahmoudelgamal] add scala to zeppelin-base fd23970 [mahmoudelgamal] remove bash erorr message. e1d4b77 [mahmoudelgamal] add R and python to zeppelin-base e731cb4 [mahmoudelgamal] Add java-cacerts to zeppelin-base e642309 [mahmoudelgamal] Add documentation and some modifications 231a414 [mahmoudelgamal] Add zeppelin-base image ac06f3a [mahmoudelgamal] Make docker image for zeppelin release 48d0a01 [mfelgamal] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Jan 9, 2017
### What is this PR for? remove org.apache.spark.sql.hive.HiveSharedState class check because it has been removed since spark 2.1.x ### What type of PR is it? [Bug Fix ] ### What is the Jira issue? [ZEPPELIN-1909](https://issues.apache.org/jira/browse/ZEPPELIN-1909) Author: lichenglin <[email protected]> Closes apache#1856 from lichenglin/master and squashes the following commits: 60c3c04 [lichenglin] Merge pull request #1 from lichenglin/lichenglin-patch-1 3f90485 [lichenglin] make hive enable under spark 2.1.0
zjffdu
pushed a commit
that referenced
this pull request
Aug 17, 2017
…y '.' can not be found in docker environment ### What is this PR for? shell interpreter complained that working directory '.' can not be found in docker environment. I add a line of code to set current working directory to USER`s home, and it works. ### What type of PR is it? Bug Fix ### Todos * tests ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2841 ### How should this be tested? run shell interpreter`s test units ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Shu Jiaming <[email protected]> Author: 束佳明 <[email protected]> Closes apache#2521 from vistep/master and squashes the following commits: 34a0049 [Shu Jiaming] ZEPPELIN-2841 fix a bug where shell interpreter complained that working directory '.' can not be found while zeppelin was running in docker enviroment. d02104a [束佳明] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Aug 24, 2017
###What is this PR for? in save-as.service.js, if we use URI Data scheme, we could only contain 2MB data in chrome. using the createObjectURL and File API's blob feature, i managed to upgrade the capacity to about 900MB. plus this update is better in debugging too. if we exceed the 2MB limit in URI data scheme, the download just failed with no accurate console log originally, so it was kinda hard to know why this happens. But using this technique, if it exceeds the 900MB limit, the console log points directly about what the problem is. like this : Uncaught RangeError: Failed to construct 'Blob': Array length exceeds supported limit. https://github.com/apache/zeppelin/blob/master/zeppelin-web/src/app/notebook/save-as/save-as.service.js ###What type of PR is it? Improvement ###Todos nothing more i guess ###What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2850 ###How should this be tested? open zeppelin using chrome. make a table by select, then download it by csv or tsv. the table should be BIG, like really big, (but not that big for companies, which is my case) to test. in the original version if the whole data exceeds 2MB, you could see that the download fails. but using my script, it doesn't fail until it reaches about 900MB~1GB, which is a tremendous improvement. ###Screenshots (if appropriate) i'll post it later if you really need it. but i'm pretty sure you guys know what i'm talking about :) ###Questions: Does the licenses files need update? no (i guess) Is there breaking changes for older versions? no Does this needs documentation? maybe? Author: imnotkind <[email protected]> Closes apache#2532 from imnotkind/master and squashes the following commits: 075c4ec [imnotkind] Update save-as.service.js db778b1 [imnotkind] Merge pull request #1 from imnotkind/imnotkind-patch-1 e9ad52e [imnotkind] Update save-as.service.js
zjffdu
pushed a commit
that referenced
this pull request
Oct 24, 2017
### What is this PR for? This PR will add tab as auto complete invoker if paragraph is non-md and user has not pressed the tab as a first character in the line ### What type of PR is it? [Improvement] ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-277 * https://issues.apache.org/jira/browse/ZEPPELIN-2736 ### How should this be tested? - Build: mvn clean package -Denforcer.skip -DskipTests -Drat.skip - Open a paragraph - Press tab with following options: first character, after space ### Questions: * Does the licenses files need an update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Malay Majithia <[email protected]> Author: Lee moon soo <[email protected]> Closes apache#2542 from malayhm/ZEPPELIN-277 and squashes the following commits: 436f22d [Malay Majithia] Added Tab auto completion flag for python sql and r b37e084 [Malay Majithia] Fixed lint error 18fc814 [Malay Majithia] Merge branch 'master' into ZEPPELIN-277 b09730e [Malay Majithia] Merge branch 'master' into ZEPPELIN-277 63d69e1 [Malay Majithia] Merge branch 'ZEPPELIN-277' of github.com:malayhm/zeppelin into ZEPPELIN-277 a75f0fe [Malay Majithia] Improved the first character check logic 2ec879d [Malay Majithia] Merge pull request #1 from Leemoonsoo/ZEPPELIN-277-completion-key 77afdba [Lee moon soo] fix style 77b47b6 [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab 46f612a [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs 865c0a6 [Lee moon soo] Set python and spark interpreter completionKey 05d5860 [Lee moon soo] Update doc 973068b [Lee moon soo] apply tab completion based on editor.completionKey 5f4d81c [Malay Majithia] If all the previous line characters are tab, don't show autocomplete on tab 655ba88 [Malay Majithia] ZEPPELIN-277 Add Tab as Autocomplete for Notebook non-md paragraphs
zjffdu
pushed a commit
that referenced
this pull request
Nov 10, 2017
…y '.' can not be found in docker environment ### What is this PR for? shell interpreter complained that working directory '.' can not be found in docker environment. I add a line of code to set current working directory to USER`s home, and it works. ### What type of PR is it? Bug Fix ### Todos * tests ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2841 ### How should this be tested? run shell interpreter`s test units ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Shu Jiaming <[email protected]> Author: 束佳明 <[email protected]> Closes apache#2521 from vistep/master and squashes the following commits: 34a0049 [Shu Jiaming] ZEPPELIN-2841 fix a bug where shell interpreter complained that working directory '.' can not be found while zeppelin was running in docker enviroment. d02104a [束佳明] Merge pull request #1 from apache/master (cherry picked from commit 71d1305) Signed-off-by: Lee moon soo <[email protected]>
zjffdu
pushed a commit
that referenced
this pull request
Jan 19, 2018
### What is this PR for? This PR fixes the issue of newlines and tabs breaking results in the SQL interpreter in Livy. The Livy interpreter will return incorrect results if a row contains \n or \t characters. In the case of the newline, the result will be: Line is missing from results if the \n appears anywhere but the end of a cell `String index out of range: 17` if it appears at the end of a cell In the case of the tab, the result will be misaligned columns if the tab appears in the middle of a cell The output showing these error is attached to the Jira. I have changed the parsing and any newline or tab characters will be escaped ### What type of PR is it? Bug Fix ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3098 ### How should this be tested? Unittests have been added ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Alex Bush <[email protected]> Author: Alex Bush <[email protected]> Closes apache#2701 from bushnoh/zeppelin-3098 and squashes the following commits: 9f2f6e3 [Alex Bush] Merge pull request #3 from apache/master 14e120d [Alex Bush] Merge pull request #2 from apache/master 46981fd [Alex Bush] Merge pull request #1 from apache/master ee5a41b [Alex Bush] Merge remote-tracking branch 'upstream/master' 67a93b5 [Alex Bush] Merge branch 'master' of https://github.com/bushnoh/zeppelin 31cdbdc [Alex Bush] Added another comment explaining the regexp change d054af0 [Alex Bush] Force a dummy change for Travis 64a42be [Alex Bush] Fix for newline and tab in data
zjffdu
pushed a commit
that referenced
this pull request
Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. [Improvement] * [x] - Wait until Spark 2.4.0 is officially released. * https://issues.apache.org/jira/browse/ZEPPELIN-3810 Verified manually against Spark 2.4.0 RC3 * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <[email protected]> Author: Hyukjin Kwon <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4 (cherry picked from commit 4f73272)
zjffdu
pushed a commit
that referenced
this pull request
Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. [Improvement] * [x] - Wait until Spark 2.4.0 is officially released. * https://issues.apache.org/jira/browse/ZEPPELIN-3810 Verified manually against Spark 2.4.0 RC3 * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <[email protected]> Author: Hyukjin Kwon <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4 (cherry picked from commit 4f73272)
zjffdu
pushed a commit
that referenced
this pull request
Nov 13, 2018
Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. [Improvement] * [x] - Wait until Spark 2.4.0 is officially released. * https://issues.apache.org/jira/browse/ZEPPELIN-3810 Verified manually against Spark 2.4.0 RC3 * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <[email protected]> Author: Hyukjin Kwon <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4 (cherry picked from commit 4f73272)
zjffdu
pushed a commit
that referenced
this pull request
Nov 14, 2018
### What is this PR for? Spark 2.4 changed it's Scala version from 2.11.8 to 2.11.12 (see SPARK-24418). There are two problems for this upgrade at Zeppelin side: 1.. Some methods that are used in private by reflection, for instance, `loopPostInit` became inaccessible. See: - https://github.com/scala/scala/blob/v2.11.8/src/repl/scala/tools/nsc/interpreter/ILoop.scala - https://github.com/scala/scala/blob/v2.11.12/src/repl/scala/tools/nsc/interpreter/ILoop.scala To work around this, I manually ported `loopPostInit` at 2.11.8 to retain the behaviour. Some functions that are commonly existing at both Scala 2.11.8 and Scala 2.11.12 are used inside of the new `loopPostInit` by reflection. 2.. Upgrade from 2.11.8 to 2.11.12 requires `jline.version` upgrade. Otherwise, we will hit: ``` Caused by: java.lang.NoSuchMethodError: jline.console.completer.CandidateListCompletionHandler.setPrintSpaceAfterFullCompletion(Z)V at scala.tools.nsc.interpreter.jline.JLineConsoleReader.initCompletion(JLineReader.scala:139) ``` To work around this, I tweaked this by upgrading jline from `2.12.1` to `2.14.3`. ### What type of PR is it? [Improvement] ### Todos * [x] - Wait until Spark 2.4.0 is officially released. ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3810 ### How should this be tested? Verified manually against Spark 2.4.0 RC3 ### Questions: * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No Author: hyukjinkwon <[email protected]> Author: Hyukjin Kwon <[email protected]> Author: Jeff Zhang <[email protected]> Closes apache#3206 from HyukjinKwon/ZEPPELIN-3810 and squashes the following commits: c2456c9 [Hyukjin Kwon] Py4J 0.10.6 to 0.10.7 573f07d [Jeff Zhang] add test for spark 2.4 (#1) 9ac1797 [hyukjinkwon] Support Spark 2.4
zjffdu
pushed a commit
that referenced
this pull request
Jan 31, 2020
…ookRepo.java to allow it to work on windows
### What is this PR for?
The rootNotebookFolder variable in VFSNotebookRepo.java does not contain the root directory in windows. For e.g. if notebook directory is "C:/Users/zeppelin-notes", the value of rootNotebookFolder set in VFSNotebookRepo.java:91 is "/Users/zeppelin-notes", which is not found in windows and throws an exception at launch.
Using `this.rootNotebookFolder = rootNotebookFileObject.getName().getURI().replace("file:///", "/");` instead allows us to set the correct value for rootNotebookFolder (i.e. /C:/Users/zeppelin-notes).
Similarly, the noteFileName (VFSNotebookRepo.java:110) variable is missing root directory in windows and using `String noteFileName = fileObject.getName().getURI().replace("file:///", "/");` fixes this.
### What type of PR is it?
Bug Fix
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4574
THIS SOFTWARE IS CONTRIBUTED SUBJECT TO THE TERMS OF THE APACHE SOFTWARE FOUNDATION SOFTWARE GRANT AND CORPORATE CONTRIBUTOR LICENSE AGREEMENT VERSION R190612.
THIS SOFTWARE IS LICENSED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OF NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THIS SOFTWARE MAY BE REDISTRIBUTED TO OTHERS ONLY BY EFFECTIVELY USING THIS OR ANOTHER EQUIVALENT DISCLAIMER IN ADDITION TO ANY OTHER REQUIRED LICENSE TERMS.
Author: Muhammad Taufiq <[email protected]>
Author: Muhammad-ms <[email protected]>
Closes apache#3615 from Muhammad-ms/zeppelin9_vfs and squashes the following commits:
b0a65c0 [Muhammad Taufiq] [ZEPPELIN-4574] Added comments explaining the change
81b054e [Muhammad Taufiq] [ZEPPELIN-4574] Fixed rootNotebookFolder and noteFileName in VFSNotebookRepo.java to allow it to work on windows
4f532aa [Muhammad-ms] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Jan 31, 2020
…otePath variable for note title ### What is this PR for? The variable noteName does not contain leading "/", unlike notePath. Therefore, using former prevents the leading "/" from appearing in the note name displayed on Zeppelin page. ### What type of PR is it? Bug Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4575 THIS SOFTWARE IS CONTRIBUTED SUBJECT TO THE TERMS OF THE APACHE SOFTWARE FOUNDATION SOFTWARE GRANT AND CORPORATE CONTRIBUTOR LICENSE AGREEMENT VERSION R190612. THIS SOFTWARE IS LICENSED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OF NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THIS SOFTWARE MAY BE REDISTRIBUTED TO OTHERS ONLY BY EFFECTIVELY USING THIS OR ANOTHER EQUIVALENT DISCLAIMER IN ADDITION TO ANY OTHER REQUIRED LICENSE TERMS. Author: Muhammad-ms <[email protected]> Author: Muhammad Taufiq <[email protected]> Closes apache#3616 from Muhammad-ms/zeppelin9_noteTitle and squashes the following commits: a3b79be [Muhammad Taufiq] [ZEPPELIN-4575] Switched to the use of noteName variable instead of notePath variable as it does not contain leading / 4f532aa [Muhammad-ms] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Jan 31, 2020
…a .war file ### What is this PR for? This PR disables the local extraction of zeppelin-web.war if ZEPPELIN_WAR is pointing to a .war file. This has 2 advantages: 1. Skipping WAR extraction improves Zeppelin launch time 2. It prevents any problems arising from the corruption of local zeppelin-web extraction ### What type of PR is it? Improvement ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4573 THIS SOFTWARE IS CONTRIBUTED SUBJECT TO THE TERMS OF THE APACHE SOFTWARE FOUNDATION SOFTWARE GRANT AND CORPORATE CONTRIBUTOR LICENSE AGREEMENT VERSION R190612. THIS SOFTWARE IS LICENSED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY OF NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THIS SOFTWARE MAY BE REDISTRIBUTED TO OTHERS ONLY BY EFFECTIVELY USING THIS OR ANOTHER EQUIVALENT DISCLAIMER IN ADDITION TO ANY OTHER REQUIRED LICENSE TERMS. Author: Muhammad-ms <[email protected]> Author: Muhammad Taufiq <[email protected]> Closes apache#3614 from Muhammad-ms/zeppelin9_war_extraction and squashes the following commits: d01bdd2 [Muhammad Taufiq] [ZEPPELIN-4564] Disable zeppelin-web.war extraction when pointing to a .war file 4f532aa [Muhammad-ms] Merge pull request #1 from apache/master
zjffdu
pushed a commit
that referenced
this pull request
Oct 16, 2020
…Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ### What is this PR for? - fix the [CI failure](https://travis-ci.org/github/apache/zeppelin/builds/709913046) due to [PR-3852](apache#3852) ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * [ZEPPELIN-4962](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-4962) ### How should this be tested? * CI test ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? Yes Author: xiejiajun <[email protected]> Author: xie-jia-jun <[email protected]> Author: JakeXie <[email protected]> Closes apache#3860 from xiejiajun/ZEPPELIN-4962 and squashes the following commits: 9128c9b [JakeXie] spark.repl.target docs update ad4c0e3 [xiejiajun] Clear irrelevant code a12d3a9 [xiejiajun] Support for manually specifying the Java version of Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ab2b191 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 5569788 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 0a9af6c [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master be36b37 [xiejiajun] 合并Apache Master分支冲突解决 1335d55 [xiejiajun] Merge remote-tracking branch 'origin/master' fc59f57 [JakeXie] Merge pull request #4 from apache/master 9cc70fe [xiejiajun] Merge remote-tracking branch 'origin/master' 6ef9b23 [xie-jia-jun] Merge pull request #3 from apache/master 45af87a [xiejiajun] added timeout for getting Thrift client to avoid situations where the interpreter may not be restarted when the interpreter process exits unexpectedly f149c3b [xie-jia-jun] Merge pull request #1 from apache/master 5d4b645 [xie-jia-jun] Support OSSConfigStorage of Aliyun dbb6639 [xie-jia-jun] Add Aliyun OSS SDK bb47849 [xie-jia-jun] Support S3ConfigStorage of AWS
zjffdu
pushed a commit
that referenced
this pull request
Oct 18, 2020
…Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ### What is this PR for? - fix the [CI failure](https://travis-ci.org/github/apache/zeppelin/builds/709913046) due to [PR-3852](apache#3852) ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * [ZEPPELIN-4962](https://issues.apache.org/jira/projects/ZEPPELIN/issues/ZEPPELIN-4962) ### How should this be tested? * CI test ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? NO * Is there breaking changes for older versions? NO * Does this needs documentation? Yes Author: xiejiajun <[email protected]> Author: xie-jia-jun <[email protected]> Author: JakeXie <[email protected]> Closes apache#3860 from xiejiajun/ZEPPELIN-4962 and squashes the following commits: 9128c9b [JakeXie] spark.repl.target docs update ad4c0e3 [xiejiajun] Clear irrelevant code a12d3a9 [xiejiajun] Support for manually specifying the Java version of Spark Interpreter Scala REPL and fix the CI failure due to low Scala version ab2b191 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 5569788 [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master 0a9af6c [xiejiajun] Merge branch 'master' of https://github.com/apache/zeppelin into apache-master be36b37 [xiejiajun] 合并Apache Master分支冲突解决 1335d55 [xiejiajun] Merge remote-tracking branch 'origin/master' fc59f57 [JakeXie] Merge pull request #4 from apache/master 9cc70fe [xiejiajun] Merge remote-tracking branch 'origin/master' 6ef9b23 [xie-jia-jun] Merge pull request #3 from apache/master 45af87a [xiejiajun] added timeout for getting Thrift client to avoid situations where the interpreter may not be restarted when the interpreter process exits unexpectedly f149c3b [xie-jia-jun] Merge pull request #1 from apache/master 5d4b645 [xie-jia-jun] Support OSSConfigStorage of Aliyun dbb6639 [xie-jia-jun] Add Aliyun OSS SDK bb47849 [xie-jia-jun] Support S3ConfigStorage of AWS (cherry picked from commit 185ffd4) Signed-off-by: Jeff Zhang <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I fixed some typos, grammar errors and removed useless markdown escaping in
spark.mdbased on yourspark_doc_fixbranch. Please feel free to fix/add more if it's needed :D