ZEPPELIN-1197. Should print output directly without invoking function print in pyspark interpreter

[zjffdu]

What is this PR for?

For now, user need to invoke print to make the output displayed on the notebook. This behavior is not natural and consistent with other notebooks. This PR is to make the pyspark interpreter in zeppelin behave the same as other notebook. 2 main changes

• use single mode to compile the last statement, so that the evaluation result of the last statement will be printed to stdout, this is consistent with other notebooks (like jupyter)
• Make SparkOutputStream extends LogOutputStream so that we can see the output of inner process (Python/R), it is helpful for diagnosing.

What type of PR is it?

[Bug Fix]

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-1197

How should this be tested?

Tested it manually. Input the following text in pyspark paragraph,

1+1
sc.version

And get the following output

u'1.6.1'

Questions:

• Does the licenses files need update? No
• Is there breaking changes for older versions? User don't need to call print explicitly.
• Does this needs documentation? Yes

[Leemoonsoo]

Thanks @zjffdu for great improvement.

I have tested bit and i could see some inconsistent behavior.

Is it expected result?

[zjffdu]

@Leemoonsoo Thanks for the careful checking. I compare it with jupyter, only the second case is different. Let me investigate how to fix it.

[zjffdu]

I also compare it with native python repl, the second case is consistent. So I think this behvior is fine, although it is different from jyputer.

[Leemoonsoo]

Thanks you for explanation. LGTM

[Leemoonsoo]

Merge into master if there're no more discussions.

[Leemoonsoo]

@zjffdu @bzz How about bring this change to python interpreter as well?

[zjffdu]

Sure, let me do it for python interpreter as well.

[zjffdu]

Just take a look at python interpreter, it uses a different way with pyspark interpreter. Might need to take more time to investigate that.