Spark 3754 spark streaming file system api callable from java by holdenk · Pull Request #2703 · apache/spark

holdenk · 2014-10-08T00:58:11Z

This is a WIP of making the filesystem api callable for the Java version of Spark Streaming.

AmplabJenkins · 2014-10-08T01:12:17Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21433/Test FAILed.

SparkQA · 2014-10-09T00:04:56Z

QA tests have started for PR 2703 at commit 47cfdd5.

This patch merges cleanly.

holdenk · 2014-10-09T00:06:56Z

@tdas this fixes the issue with the Java API I found while working on the book.

SparkQA · 2014-10-09T01:22:18Z

QA tests have finished for PR 2703 at commit 47cfdd5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-09T01:22:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21496/Test FAILed.

SparkQA · 2014-10-09T01:44:38Z

QA tests have started for PR 2703 at commit 80704c5.

This patch merges cleanly.

SparkQA · 2014-10-09T02:47:34Z

QA tests have finished for PR 2703 at commit 80704c5.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-09T02:47:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21507/Test FAILed.

AmplabJenkins · 2014-10-09T20:32:17Z

Can one of the admins verify this patch?

holdenk · 2014-10-22T16:27:26Z

jenkins, retest this please

…allable-from-java

tdas · 2014-10-23T21:59:26Z

Jenkins, this is ok to test.

SparkQA · 2014-10-23T22:05:05Z

QA tests have started for PR 2703 at commit 2bc3687.

This patch merges cleanly.

SparkQA · 2014-10-23T23:16:40Z

QA tests have finished for PR 2703 at commit 2bc3687.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-23T23:16:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22090/
Test FAILed.

tdas · 2014-10-23T23:48:55Z

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

This does not seem to be related to SPARK 3754

I am not sure whether there is any use case for exposing this.

the use case of exposing this was so that I could expose equivelent java functionality to what was available in Scala

Hey @holdenk, although the underlying fileinputdstream supported it, it was not exposed in the API. I am not sure its worth exposing this kind of functionality. So lets not change this.

tdas · 2014-10-23T23:54:03Z

Thanks @holdenk for doing this. This is a great fix.
But since this is an API change which breaks binary compatibility (I am okay with it, as no one is probably using this method), you have to add exclusions in the mima.

Run the mima test with dev/mima, see the error and use that output to add to the exclusions in project/MimaExcludes

holdenk · 2014-11-05T23:01:17Z

@tdas: Ok will do. No one should be using the old API because it ins't callable from Java.

…anyways

SparkQA · 2014-11-05T23:40:32Z

QA tests have started for PR 2703 at commit 3a2cd21.

This patch does not merge cleanly.

SparkQA · 2014-11-06T00:55:46Z

QA tests have finished for PR 2703 at commit 3a2cd21.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-06T00:55:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22958/
Test FAILed.

tdas · 2014-11-07T22:29:17Z

Jenkins, test this again

SparkQA · 2014-11-10T08:05:27Z

Test build #23146 has started for PR 2703 at commit efac1fc.

This patch merges cleanly.

SparkQA · 2014-11-10T09:32:15Z

Test build #23146 has finished for PR 2703 at commit efac1fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-11-10T09:32:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23146/
Test PASSed.

tdas · 2014-11-11T01:09:06Z

streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala

Same reason as above, no need to expose this functionality of newFilesOnly

Correction on this comment. newFilesOnly should be exposed as it is exposed in the Scala api.

tdas · 2014-12-31T00:56:20Z

Ping?

holdenk · 2014-12-31T01:05:57Z

I'll take a look this weekend when I have some time.

tdas · 2014-12-31T01:42:03Z

Thanks!

tdas · 2015-02-10T21:04:44Z

This has already been fixed by #4101 Can you please close this PR?

holdenk added 2 commits October 7, 2014 17:39

Draft of adding filestream support to Java API with tests

2632a1e

hmm

95f2bfe

Ok looks like it was mostly just a test issue with the clock.

47cfdd5

holdenk changed the title ~~WIP: Spark 3754 spark streaming file system api callable from java~~ Spark 3754 spark streaming file system api callable from java Oct 9, 2014

Fix the test

80704c5

Merge branch 'master' into SPARK-3754-spark-streaming-fileSystemAPI-c…

2bc3687

…allable-from-java

tdas reviewed Oct 23, 2014
View reviewed changes

Add the excludes because of API changes. The old API wasn't callable …

3a2cd21

…anyways

Merge in master

efac1fc

tdas reviewed Nov 11, 2014
View reviewed changes

holdenk closed this Feb 11, 2015

Conversation

holdenk commented Oct 8, 2014

Uh oh!

AmplabJenkins commented Oct 8, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

holdenk commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

SparkQA commented Oct 9, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

AmplabJenkins commented Oct 9, 2014

Uh oh!

holdenk commented Oct 22, 2014

Uh oh!

tdas commented Oct 23, 2014

Uh oh!

SparkQA commented Oct 23, 2014

Uh oh!

SparkQA commented Oct 23, 2014

Uh oh!

AmplabJenkins commented Oct 23, 2014

Uh oh!

tdas Oct 23, 2014

Choose a reason for hiding this comment

Uh oh!

tdas Oct 23, 2014

Choose a reason for hiding this comment

Uh oh!

holdenk Oct 27, 2014

Choose a reason for hiding this comment

Uh oh!

tdas Nov 11, 2014

Choose a reason for hiding this comment

Uh oh!

tdas commented Oct 23, 2014

Uh oh!

holdenk commented Nov 5, 2014

Uh oh!

SparkQA commented Nov 5, 2014

Uh oh!

SparkQA commented Nov 6, 2014

Uh oh!

AmplabJenkins commented Nov 6, 2014

Uh oh!

tdas commented Nov 7, 2014

Uh oh!

SparkQA commented Nov 10, 2014

Uh oh!

SparkQA commented Nov 10, 2014

Uh oh!

AmplabJenkins commented Nov 10, 2014

Uh oh!

tdas Nov 11, 2014

Choose a reason for hiding this comment

Uh oh!

tdas Dec 27, 2014

Choose a reason for hiding this comment

Uh oh!

tdas commented Dec 31, 2014

Uh oh!

holdenk commented Dec 31, 2014

Uh oh!

tdas commented Dec 31, 2014

Uh oh!

tdas commented Feb 10, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone