[ZEPPELIN-1880] Fix shell interpreter output streaming result

[AhyoungRyu]

What is this PR for?

Shell interpreter streaming output had been available by #683, but currently it's broken after #1087 merged. This patch is for putting it back.

What type of PR is it?

Bug Fix

TODO

• Fix test

What is the Jira issue?

ZEPPELIN-1880

How should this be tested?

%sh

date && sleep 3 && date

the each timestamp must be printed as streaming output

Screenshots (if appropriate)

• before
  

• after
  

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? no

[astroshim]

Hi.
In my case it seems doesn`t work properly.. What am i missing?

[Leemoonsoo]

Tested but i also see the same result @astroshim has.

[AhyoungRyu]

Seems the streaming output works only once right after I start Zeppelin (or restart shell intp). Let me dig into more. Thanks for the review @astroshim @Leemoonsoo!

[AhyoungRyu]

@astroshim @Leemoonsoo I guess the issue that I told in the above is not a matter of Shell intp itself. I found wired behaviour of current streaming output feature. This occurs in master branch.

If I use only Spark, it's streaming output works properly like below.

• Run only Spark (tried 3 times in a row)

%spark

(1 to 5).foreach{ i=>
    Thread.sleep(500)
    println("Hello " + i)
}

But if I use Spark -> Shell -> Spark intp, then Spark's streaming output also doesn't work.

• Run Spark -> Shell -> Spark
  

@Leemoonsoo Are there any suspicious points in this problem? I tested also in 0.6.2, but the above issue is not in 0.6.2.

[soralee]

I tested and I also faced the problem in this branch. (It is same problem with above @AhyoungRyu's comment.)

At first time, It was worked very well.

But when I rerun same paragraph, it was not worked.

[AhyoungRyu]

@soralee Thanks for testing it out. Right I think the rerunning issue is due to that I mentioned in this comment.

@astroshim @Leemoonsoo BTW just to be making sure, did you build this branch including shell interpreter like mvn clean package -DskipTests -pl 'shell, zeppelin-interpreter, zeppelin-server, zeppelin-zengine' to test this patch? After build and restart Zeppelin, the result should be same with @soralee and me.

[AhyoungRyu]

I rebased from master after #1922 and #1927 merged. Both are fixing the problem that I mentioned in this comment. Here is the screenshot.

@astroshim Thanks again for the fix!!

[AhyoungRyu]

@astroshim @Leemoonsoo @soralee If you don't mind, can someone help review this again? :)

[astroshim]

LGTM it's working well.

[soralee]

Tested and it works very well!

[Leemoonsoo]

Tested. LGTM

[AhyoungRyu]

Thanks all you guys for testing it out!
Will merge if there are no more comments.