[ZEPPELIN-554] Streaming interpreter output to front-end

[Leemoonsoo]

What is this PR for?

Output from interpreter is displayed after completion of paragraph execution.
It'll be useful if output can be streamed to front-end during execution.

Previous work #593 injects InterpreterOutput stream object to Interpreter.
This PR is based on #593 and stream the data from InterpreterOutput to front-end.

This implementation only streams output is %text. Other output type (%html, %angular, %table) is not streamed to the front end.

While this PR keeps backward compatibility, Interpreter who want to use this feature will need to modify code to write output into InterpreterOutput instead of return with InterpreterResult.

This PR includes modification of SparkInterpreter to use InterpreterOutput.

What type of PR is it?

Feature

Todos

Is there a relevant Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-554

How should this be tested?

Run such code using Spark interpreter

(1 to 10).foreach{ i=>
    Thread.sleep(1000)
    println("Hello " + i)
}

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? no

[prabhjyotsingh]

I tried the default example it works awesomely well.

I might not have understood what all this PR is supposed to cover;

1. I tried pyspark, which does not stream.

%pyspark
import time
print("1")
time.sleep(1)
print("2")
time.sleep(1)
print("3")
time.sleep(1)
print("4")

2. also when I run following

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset
(1 to 2).foreach{ i=>
    Thread.sleep(1000)
    println("Hello " + i)
}

before this PR the output was

import org.apache.commons.io.IOUtils
import java.net.URL
import java.nio.charset.Charset
Hello 1
Hello 2

but now its just

Hello 1
Hello 2

[corneadoug]

I don't have any output for sc.version for example

[Leemoonsoo]

@prabhjyotsingh @corneadoug Thanks for the feedback. Let me push more commits for the output issue as well as pyspark.

[swkimme]

This is something what I REALLY needed. AWESOME!!

[corneadoug]

Made a PR to refactor the JS code :)

[Leemoonsoo]

@prabhjyotsingh @corneadoug Addressed the comments

[prabhjyotsingh]

@Leemoonsoo this is so awesome and useful feature. Thanks a lot of this.

[randerzander]

I wonder if this can be used in combination with #152/Zeppelin-157 ? I'm thinking a streaming job emits GPS points which are dynamically populated on a map. @Madhuka @corneadoug

[Leemoonsoo]

This PR stream output only when it's %text. It keeps the same behavior for %table, %html, %angular and all the other display system.

For dynamically populated data on a map, here's great example. https://gist.github.com/granturing/a09aed4a302a7367be92. Which leverage %angular display system

[corneadoug]

Found another issue:
Clear paragraph output is not working for me anymore

[felixcheung]

very cool. should SparkOutputStream be generalized - i think maybe other interpreters could stream output too (and maybe %table could use some partial result too?)

[Leemoonsoo]

@corneadoug trying to fix clear paragraph output. no success for now
@felixcheung InterpreterOutput is generalized output stream and any other interpreter can use. Yes, handling %table and use partial result would be useful. It can be done as a future work i think.

[Leemoonsoo]

@corneadoug Fixed the clear paragraph output.

[minahlee]

Tested and LGTM

[Leemoonsoo]

Thanks @minahlee for the review. I'm merging it into master.