ZEPPELIN-374: Adding result API

[fireboy1919]

Nondestructive addition of a unified API for handling separate result sets and an implementation of it for Hive and the Virtual File system notebook repo.

[bzz]

Would you be willing to remove @author tag from JavaDoc here and below please?
Although not clearly documented we, as many other ASF projects, do not use them to keep track of contributions, but use git logs and JIRA issues instead.

[fireboy1919]

Absolutely!

[bzz]

@fireboy1919 thank you for contribution!

The approach you take looks interesting, please help me to understand how much would that affect other existing notebook storages\interpreters implementations?

As review takes some time and effort, there are few things you can do to seed things us, by simplifying life of the reviewer a lot (in order of simplicity)

• us the template for PR description from the docs
• minimize the re-formatting changes like this or this, etc, as it makes hard to distinguish important changes
• make sure that ALL your code style is consistent with the project's style guide's code conventions and the other code that you have changed. This looks like a nit, but it speeds the process up, removing the need to scatter reviewers attention between important changes and just minor style one's
• keep your branch rebased on latest master
• make sure the CI is green
• ping somebody (using Gihutb mentions in this PR) as soon as you feel that it is ready for a review\merge

[Leemoonsoo]

Thank you @bzz for organizing the check list very well.

Overall concept and approach of this PR make sense a lot.
How about think little bit more before we move on?

Implementation side, I'd like to generalize little bit more and align more with other components that already exists or planned in the future.

I think it make sense to introduce some common data type (like RDD in Spark) for the Zeppelin, the common data type abstracts underlaying physical data. In addition to current InterpreterResult.

And this common data type can be exported to ResourcePool from any interpreter.

And then we can create proxy object of this common data type in Zeppelin server side, so Zeppelin server can bring data from interpreter process.

This data can be provided by REST API or any other API.
So basically approach is the same but little bit more generalized and use existing component.

[fireboy1919]

In order to minimize changes, the code that I wrote applies when the data is serialized rather than being written into the interpreters. So it's probably enough different to warrant scrapping the request, even if I would take some of the ideas from it.

In lieu of making another pull request, and in light of Leemoonsoo's comments, might I suggest splitting this into the issues that he mentioned:

1. Create a serializable common tabular data type (e.g., TabularResultType or JDBCResultType).
2. Make the base interpreter capable of sending its results to data pools.
3. Make the front-end capable of using this data (for that part, I'd probably refactor quite a lot of what I've written in this pull request).
4. Make a REST API that can download tabular data that comes from the resource pool (this looks to have a JIRA ticket for it already: ZEPPELIN-201)

I can't find any tickets for any of the first three things, but I'm happy to adapt the code that I've written to cover things one at a time. The second item could potentially have more thought.

Does that cover what you're thinking, @Leemoonsoo? And if so, are there any tickets or existing work being done that I am unaware of that relates to these issues, or should I write some?

[Leemoonsoo]

@fireboy1919 It sounds like a good plan. 1) and 2) are also required for ZEPPELIN-533. To do 2), current data pool needs some upgrade. it'll need capability of removing data automatically on paragraph/notebook deletion.
I have created issue for it https://issues.apache.org/jira/browse/ZEPPELIN-713

[Leemoonsoo]

Regarding 3), how i tried to handle it in https://cwiki.apache.org/confluence/display/ZEPPELIN/Helium+proposal,

is that, depends on the result data available in ResourcePool, Zeppelin suggest different type of pluggable visualization. That's maybe too long way to get 'TabularResult' visualized, but i believe it's more extensible and generalized way. So it'll be a framework for not only 'TabularResult' data, but any other type of resource can be exposed to ResourcePool and get displayed in front-end.