Update to Spark 1.6.3 and 2.0.2

[felixcheung]

What is this PR for?

Update Spark versions

What type of PR is it?

Improvement

What is the Jira issue?

N/A - minor version updates only

How should this be tested?

CI

Questions:

• Does the licenses files need update? Nope
• Is there breaking changes for older versions? Nope
• Does this needs documentation? Nope?

[felixcheung]

failed, to be fixed with #1683

[zjffdu]

https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L466 &
https://github.com/apache/zeppelin/blob/master/spark/pom.xml#L480 also needs to be updated.

[zjffdu]

BTW, have you checked the spark's dependencies version needs to be updated, like py4j & protobuf ?

[felixcheung]

I did check - I don't think they change in patch version. They have changed in upcoming Spark 2.1.0 though.

[felixcheung]

not sure why, there is no error in the log. I think travis terminates it only because the log is too long

The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over).

The job has been terminated

[felixcheung]

nice, CI passed. ready for review (I will likely undo the --quiet switch...?), and rebase, after review.

[zjffdu]

LGTM, but looks like you need to rebase

[felixcheung]

What do people think about the --quiet switch here? Seems like it help reduce the Travis "log too long" build error

[bzz]

Great update @felixcheung !

On the --quite switch - before fixing the symptoms I would rather prefer finding the reason, if possible. Do you have any idea why does it start to bite us only recently? I.e one thing is - our tests produce A LOT of garbage output..

[felixcheung]

@bzz from what I can see here, downloading new dependencies that are not in cache generates more than half of the log spew.

As for other cases where CI were failing, perhaps we were seeing increasing occurrences of getting "uncached"? that we have to re-download more frequently?

Since this has passed I can undo the --quiet to see if it should pick up from the cache without having to redownload everything. I'll try that once this CI run is completed.

[bzz]

That makes sense, let's try!

In case of download logs --quite might help, but I think we might be better move it to common MAVEN_OPTS rather than adding it to each profile.

@1ambda and me are already working on downoad Spark src stability if not cached and going to nail it in #1696 and #1689

[bzz]

There is really strange test failure in markdown tests on first CI profile (flaky-test?)

Tests run: 13, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 101.166 sec <<< FAILURE! - in org.apache.zeppelin.markdown.PegdownParserTest
testWebsequencePlugin(org.apache.zeppelin.markdown.PegdownParserTest)  Time elapsed: 99.979 sec  <<< FAILURE!
java.lang.AssertionError: 
Expected: a string containing "<img src=\"http://www.websequencediagrams.com/?png="
     but: was "Error while parsing action 'Root/Sequence/ZeroOrMore/Sequence/Block/FirstOf/Table/Sequence/Optional/Sequence/NodeSequence/Sequence/TableRow/Sequence/OneOrMore/Sequence/TableCell/NodeSequence/Sequence/OneOrMore/Sequence/Inline/Inline_Action1' at input position (line 10, pos 1):

^

org.pegdown.ParsingTimeoutException"
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:865)
	at org.junit.Assert.assertThat(Assert.java:832)
	at org.apache.zeppelin.markdown.PegdownParserTest.testWebsequencePlugin(PegdownParserTest.java:320)


Results :

Failed tests: 
  PegdownParserTest.testWebsequencePlugin:320 
Expected: a string containing "<img src=\"http://www.websequencediagrams.com/?png="
     but: was "Error while parsing action 'Root/Sequence/ZeroOrMore/Sequence/Block/FirstOf/Table/Sequence/Optional/Sequence/NodeSequence/Sequence/TableRow/Sequence/OneOrMore/Sequence/TableCell/NodeSequence/Sequence/OneOrMore/Sequence/Inline/Inline_Action1' at input position (line 10, pos 1):

^

org.pegdown.ParsingTimeoutException"

@1ambda have you seen that before?

[zjffdu]

I also see this error in #1612

[1ambda]

The test above is non-deterministic actually because calling websequence API (network operation) is required to get the rendered result. We might @ignore the test if it is not necessary. (i think this is better than having non-deterministic tests)

This test passes usually, but sometime having trouble with calling API if websequencediagrams.com is not available.

[bzz]

@zjffdu thank you for pointing it out!

@1ambda can you please create a new JIRA issue with flaky-test label and link those 2 CI failures there? It would be great if we could improve this to be deterministic and avoid network call by mocking, in subsequent PRs.

As for useless console output - /testing/install_external_dependencies.sh with it's native dependencies build logs seems like a very good candidate to redirect output to /dev/null

@felixcheung other 2 CI profile failures are not relevant and will be taken care of in different PR (#1689)

Looks great to me.

[1ambda]

@bzz

It is definitely better to do. I will create new JIRA issue for it.

[bzz]

@felixcheung I think after #1709 there might be no need for mvn -q option any more

[bzz]

Looks great to me, thank you @felixcheung let's merge if there is no further discussion!

CI failure of tests in zeppelin-zengine on 1 profile looks not relevant and will be taken care under ZEPPELIN-1737

Failed tests: 
  NotebookTest.testAbortParagraphStatusOnInterpreterRestart:746 expected:<ABORT> but was:<RUNNING>

Tests run: 122, Failures: 1, Errors: 0, Skipped: 0

@1ambda just for sake of history, please, feel free to post a link to JIRA issue on flaky markdown tests here

[bzz]

Merging to master, if there is no further discussion.