[ZEPPELIN-732] Helium Application

[Leemoonsoo]

What is this PR for?

This PR implements pluggable Application Framework.

Here's summary of the change.

Applicaiton

Base class of user application. User application should extends org.apache.zeppelin.helium.Application.

It looks like

public abstract class Application {

  /**
   * When application loaded
   * @params context Application context
   */
  public Application(ApplicationContext context) {
    this.context = context;
  }

  /**
   * This method can be invoked multiple times before unload(),
   * Invoked just after application loaded or when paragraph output is updated
   * @param args Required resources from paragraph
   */
  public abstract void run(ResourceSet args)
      throws ApplicationException, IOException;


  /**
   * this method is invoked just before application is removed
   */
  public abstract void unload() throws ApplicationException;
}

HeliumPackage

org.apache.zeppelin.helium.HeliumPackage keeps information of application package.
Package informations includes

{
  type : "APPLICATION",                            // type of this package
  name : "zeppelin.app",                           // application name. [organizationName].[appName] format
  description : "App description",                 // description
  artifact : "groupId:artifactId:version",         // artifact name for maven repository
  className : "org.apache.zeppelin.helium.MyApp",  // application class that extends org.apache.zeppelin.helium.Application
  resources : [[]],                                // required resource
  icon : '<i class="fa fa-clock-o"></i>'           // icon
}

resources field defines what kind of data this application requires, from ResourcePool.
"resourceName" and ":className" works in the array.
inner array combines each resourceName, :className with 'AND'
outer array combines inner array with 'OR'.

For example,

resources : [
    [ "name1", ":java.util.Date"],
    [ "name1", "name2"]
]

Then Zeppelin searches ResourcePool, first for resource name "name1" and resource type "java.util.Date" from resourcePool then resource name "name1" and "name2".

Once resources are found, they'll be passed to Application.run() method

Package Registry

org.apache.zeppelin.helium.HeliumRegistry provides list of available HeliumPackages.
Currently HeliumLocalRegistry is implemented and it provides list by reading HeliumPackage json files under ./helium directory in the file system. Later there will be some class like `HeliumOnlineRegistry' which reads available package from a community managed online central registry.

Development mode
org.apache.zeppelin.interpreter.dev.* package is provided for development support.
Developer can run their app inside of their IDE, that connects to running Zeppelin instance and display output.

Application Suggestion
Once paragraph becomes FINISHED status after run, Zeppelin searches resources from ResourcesPool that belongs to current paragraph. And compare all HeliumPackage if these resources satisfies their requirements ('resources' field). If there're available app, Helium button will be displayed.

User can see and select available application for this paragraph.

Application selection
After application is loaded, application icon will be displayed. So user can select and switch between apps.

if built-in visualization is available, application icon will be displayed next to built-in visualizations

Experimental

While this is new feature and API and specification may change, i'd suggest mark this feature as a experimental for a while.

What type of PR is it?

Feature

Todos

• - Helium Application framework
• - App launcher
• - App display and selectors
• - Package registry
• - Development mode
• - Make CI green
• - Improve comment in source codes
• - Documentation
• - Examples
• - Mark experimental
• - Review

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-732

How should this be tested?

Will be updated with examples

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? yes

[jongyoul]

@Leemoonsoo I've thought Helium is a kind of backend system. But as I saw your screenshot, it looks like a frontend system. Do you have any change? or does Helium support both of them?

[Leemoonsoo]

@jongyoul Thanks for the interest.

Helium just runs jar library (which extends Application). Which may include some output and the output is displayed in Zeppelin notebook. And the output is always handled by Angular display system. So displayed output can interact with the code that runs on backend.

Does this approach makes sense? what do you think?

[bzz]

Looks great, do you plan to rebase on master?

I think there was also @anish18sun using it under ZEPPELIN-684 so \cc @anish18sun for feedback as well

[anish18sun]

read all the code changes, I think the usage of print() and println() around context().out.write() in Application.java is really intuitive for users!

[Leemoonsoo]

Addressed comment from @bzz. @anish18sun Thanks for taking a look.

Now CI is green and i'll keep this PR open in this weekend for further reviews. if there're no further discussions, then i'll merge it into the master.

[AhyoungRyu]

@Leemoonsoo How about changing this title to "What is Apache Zeppelin Application? (Experimental)" with using <h1> instead of <h3>?

[Leemoonsoo]

Thanks @AhyoungRyu for the review. Comments are addressed.

[AhyoungRyu]

@Leemoonsoo Thanks. Awesome work indeed!

[dalong25]

Hi @Leemoonsoo , I have a question about Helium Application.

How to install the application for Zeppelin? And I do not find any application in Zeppelin.
The application format is Jar Format. When we want to deploy the application in production. we should put the jar package into which folder.

Thanks

[Leemoonsoo]

jar package needs to be in the classpath of the interpreter process. I think you can place jar under ZEPPELIN_HOME/interpreter/<interpreter_name> directory if you're deploying jar without publishing to a package manager like maven.

I think the Helium application is not very well adopted among Zeppelin users. Any feedback (improvement point, potential use case, etc, etc) would be really appreciated.