-
Notifications
You must be signed in to change notification settings - Fork 2.8k
R Interpreter for Zeppelin #208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| --- | ||
| layout: page | ||
| title: "R Interpreter" | ||
| description: "" | ||
| group: manual | ||
| --- | ||
| {% include JB/setup %} | ||
|
|
||
| ## R Interpreter | ||
|
|
||
| This is a the Apache (incubating) Zeppelin project, with the addition of support for the R programming language and R-spark integration. | ||
|
|
||
| ### Requirements | ||
|
|
||
| Additional requirements for the R interpreter are: | ||
|
|
||
| * R 3.1 or later (earlier versions may work, but have not been tested) | ||
| * The `evaluate` R package. | ||
|
|
||
| For full R support, you will also need the following R packages: | ||
|
|
||
| * `knitr` | ||
| * `repr` -- available with `devtools::install_github("IRkernel/repr")` | ||
| * `htmltools` -- required for some interactive plotting | ||
| * `base64enc` -- required to view R base plots | ||
|
|
||
| ### Configuration | ||
|
|
||
| To run Zeppelin with the R Interpreter, the SPARK_HOME environment variable must be set. The best way to do this is by editing `conf/zeppelin-env.sh`. | ||
|
|
||
| If it is not set, the R Interpreter will not be able to interface with Spark. | ||
|
|
||
| You should also copy `conf/zeppelin-site.xml.template` to `conf/zeppelin-site.xml`. That will ensure that Zeppelin sees the R Interpreter the first time it starts up. | ||
|
|
||
| ### Using the R Interpreter | ||
|
|
||
| By default, the R Interpreter appears as two Zeppelin Interpreters, `%r` and `%knitr`. | ||
|
|
||
| `%r` will behave like an ordinary REPL. You can execute commands as in the CLI. | ||
|
|
||
| [](screenshots/repl2plus2.png) | ||
|
|
||
| R base plotting is fully supported | ||
|
|
||
| [](screenshots/replhist.png) | ||
|
|
||
| If you return a data.frame, Zeppelin will attempt to display it using Zeppelin's built-in visualizations. | ||
|
|
||
| [](screenshots/replhead.png) | ||
|
|
||
| `%knitr` interfaces directly against `knitr`, with chunk options on the first line: | ||
|
|
||
| [](screenshots/knitgeo.png) | ||
| [](screenshots/knitstock.png) | ||
| [](screenshots/knitmotion.png) | ||
|
|
||
| The two interpreters share the same environment. If you define a variable from `%r`, it will be within-scope if you then make a call using `knitr`. | ||
|
|
||
| ### Using SparkR & Moving Between Languages | ||
|
|
||
| If `SPARK_HOME` is set, the `SparkR` package will be loaded automatically: | ||
|
|
||
| [](screenshots/sparkrfaithful.png) | ||
|
|
||
| The Spark Context and SQL Context are created and injected into the local environment automatically as `sc` and `sql`. | ||
|
|
||
| The same context are shared with the `%spark`, `%sql` and `%pyspark` interpreters: | ||
|
|
||
| [](screenshots/backtoscala.png) | ||
|
|
||
| You can also make an ordinary R variable accessible in scala and Python: | ||
|
|
||
| [](screenshots/varr1.png) | ||
|
|
||
| And vice versa: | ||
|
|
||
| [](screenshots/varscala.png) | ||
| [](screenshots/varr2.png) | ||
|
|
||
| ### Caveats & Troubleshooting | ||
|
|
||
| * Almost all issues with the R interpreter turned out to be caused by an incorrectly set `SPARK_HOME`. The R interpreter must load a version of the `SparkR` package that matches the running version of Spark, and it does this by searching `SPARK_HOME`. If Zeppelin isn't configured to interface with Spark in `SPARK_HOME`, the R interpreter will not be able to connect to Spark. | ||
|
|
||
| * The `knitr` environment is persistent. If you run a chunk from Zeppelin that changes a variable, then run the same chunk again, the variable has already been changed. Use immutable variables. | ||
|
|
||
| * (Note that `%spark.r` and `$r` are two different ways of calling the same interpreter, as are `%spark.knitr` and `%knitr`. By default, Zeppelin puts the R interpreters in the `%spark.` Interpreter Group. | ||
|
|
||
| * Using the `%r` interpreter, if you return a data.frame, HTML, or an image, it will dominate the result. So if you execute three commands, and one is `hist()`, all you will see is the histogram, not the results of the other commands. This is a Zeppelin limitation. | ||
|
|
||
| * If you return a data.frame (for instance, from calling `head()`) from the `%spark.r` interpreter, it will be parsed by Zeppelin's built-in data visualization system. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This help doc makes reference to both %r and %spark.r, can we pick one for consistency, or explain the difference (if there is a difference). |
||
|
|
||
| * Why `knitr` Instead of `rmarkdown`? Why no `htmlwidgets`? In order to support `htmlwidgets`, which has indirect dependencies, `rmarkdown` uses `pandoc`, which requires writing to and reading from disc. This makes it many times slower than `knitr`, which can operate entirely in RAM. | ||
|
|
||
| * Why no `ggvis` or `shiny`? Supporting `shiny` would require integrating a reverse-proxy into Zeppelin, which is a task. | ||
|
|
||
| * Max OS X & case-insensitive filesystem. If you try to install on a case-insensitive filesystem, which is the Mac OS X default, maven can unintentionally delete the install directory because `r` and `R` become the same subdirectory. | ||
|
|
||
| * Error `unable to start device X11` with the repl interpreter. Check your shell login scripts to see if they are adjusting the `DISPLAY` environment variable. This is common on some operating systems as a workaround for ssh issues, but can interfere with R plotting. | ||
|
|
||
| * akka Library Version or `TTransport` errors. This can happen if you try to run Zeppelin with a SPARK_HOME that has a version of Spark other than the one specified with `-Pspark-1.x` when Zeppelin was compiled. | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| Copyright (c) 2013-2015, David B. Dahl, Brigham Young University | ||
|
|
||
| Redistribution and use in source and binary forms, with or without | ||
| modification, are permitted provided that the following conditions are | ||
| met: | ||
|
|
||
| Redistributions of source code must retain the above copyright | ||
| notice, this list of conditions and the following disclaimer. | ||
|
|
||
| Redistributions in binary form must reproduce the above copyright | ||
| notice, this list of conditions and the following disclaimer in | ||
| the documentation and/or other materials provided with the | ||
| distribution. | ||
|
|
||
| Neither the name of the <ORGANIZATION> nor the names of its | ||
| contributors may be used to endorse or promote products derived | ||
| from this software without specific prior written permission. | ||
|
|
||
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
| "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
| LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
| A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
| HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
| SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
| LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
| DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
| THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
| (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
| OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| Copyright (c) 2002-2016 EPFL | ||
| Copyright (c) 2011-2016 Lightbend, Inc. (formerly Typesafe, Inc.) | ||
|
|
||
| All rights reserved. | ||
|
|
||
| Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: | ||
|
|
||
| Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. | ||
| Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. | ||
| Neither the name of the EPFL nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. | ||
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Found leading ( with no matching )
you could drop the (, and change it to read:
Note:
%spark.rand$rare two different ways of calling the same interpreter,