Skip to content

[SPARK-8200] [MLlib] Check for empty RDDs in StreamingLinearAlgorithm #6713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pparkkin wants to merge 5 commits into from
Closed

[SPARK-8200] [MLlib] Check for empty RDDs in StreamingLinearAlgorithm #6713

pparkkin wants to merge 5 commits into from

Conversation

@pparkkin
Copy link
Contributor

@pparkkin pparkkin commented Jun 9, 2015

Test cases for both StreamingLinearRegression and StreamingLogisticRegression, and code fix.

Edit:
This contribution is my original work and I license the work to the project under the project's open source license.

@srowen
Copy link
Member

srowen commented Jun 9, 2015

@pparkkin please review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark Among other things this needs an actual title.

srowen
srowen reviewed Jun 9, 2015
View reviewed changes
mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala Outdated
Copy link
Member

@srowen srowen Jun 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use !rdd.isEmpty as it will be notably faster.

@pparkkin pparkkin changed the title Fix SPARK-8200 [SPARK-8200] [MLlib] Check for empty RDDs in StreamingLinearAlgorithm Jun 9, 2015
@pparkkin pparkkin changed the title [SPARK-8200] [MLlib] Check for empty RDDs in StreamingLinearAlgorithm [SPARK-8200] [MLlib] [WIP] Check for empty RDDs in StreamingLinearAlgorithm Jun 9, 2015
@pparkkin
Copy link
Contributor Author

pparkkin commented Jun 9, 2015

Thanks for the quick review and comments! Sorry about not following the guide completely.

I updated the PR title and description.

I'll make the code changes when I get back to work tomorrow morning.

@srowen
Copy link
Member

srowen commented Jun 9, 2015

Sorry, I merged another PR to fix the superfluous match construct here. You may need a rebase too now.

@pparkkin pparkkin force-pushed the streamingmodel-empty-rdd branch from e3e358f to 54ad89e Compare June 9, 2015 22:49
@pparkkin
Copy link
Contributor Author

pparkkin commented Jun 9, 2015

Done.

@srowen
Copy link
Member

srowen commented Jun 10, 2015

OK to test

@srowen
Copy link
Member

srowen commented Jun 10, 2015

Jenkins, test this please

@SparkQA
Copy link

SparkQA commented Jun 10, 2015

Test build #34587 has finished for PR 6713 at commit ff5cd78.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 10, 2015

Test build #894 timed out for PR 6713 at commit ff5cd78 after a configured wait of 175m.

srowen
srowen reviewed Jun 10, 2015
View reviewed changes
mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearAlgorithm.scala
Copy link
Member

@srowen srowen Jun 10, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(no need for toString here but don't worry about it)

@srowen
Copy link
Member

srowen commented Jun 10, 2015

Is this ready -- just checking since there is [WIP] in the title.

@pparkkin pparkkin changed the title [SPARK-8200] [MLlib] [WIP] Check for empty RDDs in StreamingLinearAlgorithm [SPARK-8200] [MLlib] Check for empty RDDs in StreamingLinearAlgorithm Jun 10, 2015
@pparkkin
Copy link
Contributor Author

pparkkin commented Jun 10, 2015

Changed title.

@asfgit asfgit closed this in b928f54 Jun 10, 2015
asfgit pushed a commit that referenced this pull request Jun 10, 2015
@pparkkin @srowen
[SPARK-8200] [MLLIB] Check for empty RDDs in StreamingLinearAlgorithm
59fc3f1 
Test cases for both StreamingLinearRegression and StreamingLogisticRegression, and code fix.

Edit:
This contribution is my original work and I license the work to the project under the project's open source license.

Author: Paavo <[email protected]>

Closes #6713 from pparkkin/streamingmodel-empty-rdd and squashes the following commits:

ff5cd78 [Paavo] Update strings to use interpolation.
db234cf [Paavo] Use !rdd.isEmpty.
54ad89e [Paavo] Test case for empty stream.
393e36f [Paavo] Ignore empty RDDs.
0bfc365 [Paavo] Test case for empty stream.

(cherry picked from commit b928f54)
Signed-off-by: Sean Owen <[email protected]>
@pparkkin pparkkin deleted the streamingmodel-empty-rdd branch June 10, 2015 22:38
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@pparkkin @nemccarthy
[SPARK-8200] [MLLIB] Check for empty RDDs in StreamingLinearAlgorithm
e96247c 
Test cases for both StreamingLinearRegression and StreamingLogisticRegression, and code fix.

Edit:
This contribution is my original work and I license the work to the project under the project's open source license.

Author: Paavo <[email protected]>

Closes apache#6713 from pparkkin/streamingmodel-empty-rdd and squashes the following commits:

ff5cd78 [Paavo] Update strings to use interpolation.
db234cf [Paavo] Use !rdd.isEmpty.
54ad89e [Paavo] Test case for empty stream.
393e36f [Paavo] Ignore empty RDDs.
0bfc365 [Paavo] Test case for empty stream.
@liancheng liancheng mentioned this pull request Dec 17, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pparkkin @srowen @SparkQA