ZEPPELIN-25 Ability to create rich gui inside of Notebook

[Leemoonsoo]

This PR implements https://issues.apache.org/jira/browse/ZEPPELIN-25

Here's a short video demo of this feature.

for someone who want to try, here's api

// bind 'varName' variable with 'v' value
z.angularBind(varName: String, v: Object)

// unbind 'varName'
z.angularUnbind(varName: String)

// get value of 'varName'
z.angular(varName:String)

// add watcher to 'varName' variable.
// that is monitoring change and run 'func' when it is changed.
z.angularWatch(varName:String, func: (Object, Object) => Unit))

// remove watcher from 'varName'
z.angularUnwatch(varName:String)

Any paragraph's output starting with '%angular' will be considered as angular code. %angular as a interpreter also available.

Any feedback is welcome!

[corneadoug]

I don't think that binding from angular to scala (angularWatch) is really necessary, the binding would be most of the time the other way around, from data to visualization.

Also saving the scope values on the Zeppelin context, is having multiple unwanted effects (If I'm right): All of the Zeppelin context is sent back with the angular interpreter result. Considering this also works cross-notebooks, we can imagine the effect of doing a binding of a query result per notebook on all angular interpreter response, also that result would be saved in the notebook.

This is a cool demo playing with scope and allowing data sharing between interpreters, but without really working on allowing user to build angular code, it is not doing much more than what could be achieved with a println that would insert its data inside of html tags and add scripts tags for some javascript code.

However it's nice to be able to show people what could be possible with the idea of rich gui through angular

[Leemoonsoo]

@corneadoug Thanks for the feedback.

I think binding from angular to scala is also really necessary. For example, want to run analytics routine in scala by clicking button in angular. I expect two way (angular -> scala, scala -> angular) binding is used for most cases.

Here's some rule implemented for scope value.

• scope value change is broadcasted to all notes that is using the interpreter that creates that variable'
• scope value is saved inside of individual note.
• updating scope variable does not trigger note save. when note is saved (like by run paragraph), related scope variable is saving in that particular note.
• when zeppelin restarted, saved scope variable is restored to angular. but it is not restored to scala side.
• defining new scope variable in angular side does not create bind in scala side.

Here's implementation goal of this PR.

• print angular template from interpreter
• implement controller in scala(or any interpreter).

So interpreter can interact with GUI, fully. I don't this it is not allowing user to build angular code. Can you be more specific about it?

[corneadoug]

Binding to scala could be done using a simple service in angularJS. For example: zeppelinInterpreterService.call(interpreterType, functionName, params[])

While having cross-notebooks scope is interesting, rewriting every angular function in scala and saving all scope on the backend seems too complicated, performance risky and heavy.

We can write angular code and it will work in some cases, but you will have some problems related to the scope as soon as you start using controller and modules.

[Leemoonsoo]

I'm not sure why you think you need rewrite every angular function in scala and saving all scope on the backend. You'll write the function only and if only you want to run some code in scala.

Scope variable is saved only when it is defined by you. not all scope variable Zeppelin uses.
Otherwise, Zeppelin will not able to restore exactly the same visual output after restart.
Performance is only depends on how many scope variables user create

One great thing of AngularJS is, it is watching scope variable's change. I wanted to leverage this advantage. So API is designed to automatically watch scope variable change in scala side. Not js side to manually call some function to notify scope variable change.

[eranwitkon]

I really like the way the display system is growing but I think it is much more interesting if this can be done by separate users. e.g. someone who can code SQL or Scala or Angular and someone who can interact with the data and dive better insights.
How would you see such interaction take place? can one edit the notebook and other one use it?
How are other people see this happening?

[Leemoonsoo]

@goi
Thanks for the interest. Because of angular scope is shared across notebook when they use same interpreter, one can do scala and generate some value from spark and the other in different notebook can draw some nice graphic using angular.

[Leemoonsoo]

I have added another example video, constructing simple dashboard by leveraging Angular display system.

Here's code used in this example

paragraph 0

val bankText = sc.textFile("/tmp/bank-full.csv")
case class Bank(age:Integer, job:String, marital : String, education : String, balance : Integer)
val bank = bankText.map(s=>s.split(";")).filter(s=>s(0)!="\"age\"").map(
    s=>Bank(s(0).toInt, 
            s(1).replaceAll("\"", ""),
            s(2).replaceAll("\"", ""),
            s(3).replaceAll("\"", ""),
            s(5).replaceAll("\"", "").toInt
        )
)
bank.toDF.registerTempTable("bank");
bank.filter(s=>s.marital == "married").toDF.registerTempTable("married");

val tables = sqlContext.sql("show tables").collect.map(s=>s(0))
z.angularBind("tables", tables)
z.angularBind("selectedTable", "bank")

val selectedTableContext = z.getInterpreterContext()
z.angularUnwatch("selectedTable")
z.angularWatch("selectedTable", (before:Object, after:Object) => {
    z.run(2, selectedTableContext)
    z.run(3, selectedTableContext)
    z.run(4, selectedTableContext)
})

paragraph 1

%angular 
Please select table <select ng-model="selectedTable" ng-options="o as o for o in tables"></select>

paragraph 2

{
    val stat = sqlContext.sql("select count(*) as cnt, min(balance), cast(avg(balance) as INT), max(balance) from " + z.angular("selectedTable")).collect

    val count = stat(0)(0)
    val minBalance = stat(0)(1)
    val maxBalance = stat(0)(2)
    val avgBalance = stat(0)(3)

    print(s"""%angular
    <h2>Table {{selectedTable}}</h2>
    <hr />
    <div class="row">
        <div class="col-md-6"><center><h3>$count</h3>Total customers</center></div>
        <div class="col-md-6"><center><h3>$minBalance</h3>Minimum balance</center></div>
    </div>
    <br />
    <div>
        <div class="col-md-6"><center><h3>$maxBalance</h3>Average balance</center></div>
        <div class="col-md-6"><center><h3>$avgBalance</h3>Maximum balance</center></div>
    </div>
    <br /><br /><br /><br /><br /><br /><br />

    """)
}

paragraph 3

z.show(sqlContext.sql("select marital, count(1) as cnt from " + z.angular("selectedTable") + " group by marital order by cnt desc limit 5"))

paragraph 4

z.show(sqlContext.sql("select * from " + z.angular("selectedTable").toString))

[Leemoonsoo]

Here's another fun example, that monitors spark worker's cpu and display them on Zeppelin, in realtime.

The code is not perfect. but you'll see the idea and capabilities of this Angular display system.

Here's scala part

import sys.process._

// get cluster cpu info using spark
def getClusterCpu = {
    sc.setLocalProperty("spark.scheduler.pool", "fair");
    val clusterCpu = sc.parallelize(1 to 10).map{s=>
        (("hostname" !!).replace("\n", ""), "mpstat 1 1" !!)
    }.map{ s =>
        (s._1, ((100.0 - s._2.split("\n")(3).split("\\s+")(10).toDouble)*100).toInt.toDouble / 100)
    }.reduceByKey((a,b) => scala.math.max(a,b)).collect.sortWith(_._1 < _._1)
    z.angularBind("cpuUsage", clusterCpu)
}

// start monitoring. once a second.
val t = new java.util.Timer()
val task = new java.util.TimerTask {
  def run() = getClusterCpu
}
t.schedule(task, 1000L, 1000L)

here's angular part

%angular
Cluster CPU Usage

<table class="table">
    <tr>
        <th style="width:200px">Host</th>
        <th style="width:100px">Cpu</th>
        <th></th>
    </tr>
    <tr ng-repeat="info in cpuUsage">
        <td>{{info._1}}</td>
        <td>{{info._2}}</td>
        <td>
            <div class="progress">
              <div class="progress-bar" role="progressbar" aria-valuenow="{{info._2}}" aria-valuemin="0" aria-valuemax="100" style="width: {{info._2}}%;">
                <span class="sr-only">{{info._2}}%</span>
              </div>
            </div>
        </td>
    </tr>
</table>

[minahlee]

It's awesome!!

[Leemoonsoo]

Ready to merge. Documentation will follow.

I have made some test and some examples and all works fine.
We can discuss more about improvements in separate issue, like controller in Js side @corneadoug suggested.

[corneadoug]

Please replace the jquery in the controller by angular if you plan on merging

Ex:
$('#p'+$scope.paragraph.id+'_angular') -> angular.element('#p'+$scope.paragraph.id+'_angular')

[Leemoonsoo]

@corneadoug Thanks for pointing that out.

I think replacing jquery to angular is out of this PR's scope.
Because, not only the code introduced by this PR but also many existing code uses jquery instead of angular.

So i think it's better to create a separate issue for that and handle them there. Cool?

[corneadoug]

There is a lot of them here and there in the code, and these needs to be done in a different PR,
but it doesn't mean we should introduce more of them with this one.

Otherwise we would have a PR every 15days to clean it which is not acceptable.

[corneadoug]

Also there is only 3 of them to change in this PR, in paragraph.js between line 82 and 88

[Leemoonsoo]

@corneadoug Changed those 3

[Leemoonsoo]

Merging it if there is no more discussions :-)

[eranwitkon]

Hi,

I tried running this sample on the latest build but got an error the the
angular interpreter was not found. This is also does not appear on the
interpreter setting page.
Did I miss something?
Wasn't this merged to the main branch?
Eran

On Sun, Apr 12, 2015 at 8:58 AM, asfgit [email protected] wrote:

Closed #27 #27 via
58b70e3
58b70e3
.

—
Reply to this email directly or view it on GitHub
#27 (comment).

Eran | CTO

[Leemoonsoo]

@goi
Please make sure your conf/zeppelin-site.xml has org.apache.zeppelin.angular.AngularInterpreter in your zeppelin.interpreters property.

If you restart Zeppelin, it'll be there.

[eranwitkon]

Hi,
Since it just was a fresh build from master I did not have
zeppelin-site.xml (only the .xml.template) and as expected the
org.apache.zeppelin.angular.AngularInterpreter by default is in it.
Still it does not work.
Any idea?

On Mon, Apr 13, 2015 at 3:49 PM, Lee moon soo [email protected]
wrote:

@goi https://github.com/goi
Please make sure your conf/zeppelin-site.xml has
org.apache.zeppelin.angular.AngularInterpreter in your
zeppelin.interpreters property.

If you restart Zeppelin, it'll be there.

—
Reply to this email directly or view it on GitHub
#27 (comment)
.

Eran | CTO

[Leemoonsoo]

Have you created interpreter for angular in 'Interpreter' menu?

[eranwitkon]

No, I did not.
Should I?
WHat are the parameters needed?
Where is that documented?
Why is Angular interpreter different from other interpreters? e.g. needs to
be configured manually
Eran

On Tue, Apr 14, 2015 at 3:33 PM, Lee moon soo [email protected]
wrote:

Have you created interpreter for angular in 'Interpreter' menu?

—
Reply to this email directly or view it on GitHub
#27 (comment)
.

Eran | CTO

[Leemoonsoo]

Yes you should.

When Zeppelin starts, if there's no conf/interpreter.json found, Zeppelin creates one with all available interpreters. But you seems you already have conf/interpreter.json created the version without Angular interpreter. So Zeppelin uses existing file and you need manually create it.
Angular interpreter does not require any special parameter.
I'm working on documentation for Angular Interpreter.
Thanks!

[ejono]

Ah, that must be why I couldn't get the new HiveInterpreter working last week. Would it be possible for Zeppelin to add missing interpreter configs to the interpreter.json file rather than only adding them when interpreter.json doesn't exist yet? Otherwise this will always be a problem for users as new interpreters are added.

[Leemoonsoo]

@ejono
ZeppelinConfiguration.java misses HiveInterpreter in it's default configuration. I'll make a patch.
And I agree. it really make sense to make something happening when new interpreters are added.
Appreciate for the feedback.

[soubhik-c]

With introduction of interpreterGroup following code from the above example doesn't work anymore. I could track upto different InterpreterContextRunnerPool getting created for multiple interpreters and only one have interpreterContextRunners populated with all my paragraphs. So, sparkInterpreter when sends event to another RemoteInterpreter, it finds the list empty there.

z.angularWatch("selectedTable", (before:Object, after:Object) => {
z.run(2, selectedTableContext)
z.run(3, selectedTableContext)
z.run(4, selectedTableContext)
})

It will be great if somebody can fix it asap (because we have a demo pending in another day or two).

[Leemoonsoo]

@soubhik-c
Thanks for reporting a problem. The problem was there for a while. Let me take a look and fix it.

[Leemoonsoo]

@soubhik-c
I created PR with fix. Please try #124

[soubhik-c]

@Leemoonsoo works like a charm... thanks a lot Moon. I've few suggestions/requests that I suppose will make this product better. How can I get started ?

[Leemoonsoo]

I think mailing list is good place to start sharing suggestions/requests. Creating issue on JIRA works, too. Any suggestions/requests is very much appreciated. so please don't hesitate!

[Benzhu10222015]

Great demo!

I am new with AngularJS. Just wondering how to use it to show network topology (tree-like) in Zeppelin

[joku86]

Looks realy cool but i have an problem maybe one have an quick fix for me
this:
println(z.angular("selectedTable")) works fine and will be updated when change the selection inside the selectbox.
this:
println("%angular {{selectedTable}}") is always "bank" like inside example (z.angularBind("selectedTable", "bank"))

[corneadoug]

@joku86 You would need to do an z.angularBind for the selectedTable value beforehand.
As you can see in the documentation: https://zeppelin.incubator.apache.org/docs/0.6.0-incubating-SNAPSHOT/displaysystem/angular.html