Skip to content

SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions #3690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions #3690

wants to merge 1 commit into from

Conversation

@srowen
Copy link
Member

@srowen srowen commented Dec 14, 2014

This looked like perhaps a simple and important one. combineByKey looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in PairRDDFunctions which delegate to it.

@SparkQA
Copy link

SparkQA commented Dec 14, 2014

Test build #24438 has started for PR 3690 at commit 8df68fe.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 14, 2014

Test build #24438 has finished for PR 3690 at commit 8df68fe.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 14, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24438/
Test FAILed.

@srowen
Copy link
Member Author

srowen commented Dec 14, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Dec 14, 2014

Test build #24442 has started for PR 3690 at commit 8df68fe.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 14, 2014

Test build #24442 has finished for PR 3690 at commit 8df68fe.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 14, 2014

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24442/
Test FAILed.

@srowen
Copy link
Member Author

srowen commented Dec 15, 2014

Jenkins, retest this please.

@SparkQA
Copy link

SparkQA commented Dec 15, 2014

Test build #24469 has started for PR 3690 at commit 8df68fe.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Dec 15, 2014

Test build #24469 has finished for PR 3690 at commit 8df68fe.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented Dec 15, 2014

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24469/
Test PASSed.

@JoshRosen
Copy link
Contributor

JoshRosen commented Dec 16, 2014

This looks good to me; I'm surprised that this bug existed for so long. I'm going to merge this into all of the maintenance branches (except for branch-1.2) and tag this in JIRA for a 1.2.1 backport.

@asfgit asfgit closed this in 2a28bc6 Dec 16, 2014
asfgit pushed a commit that referenced this pull request Dec 16, 2014
@srowen @JoshRosen
SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions
fa3b3e3 
This looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.

Author: Sean Owen <[email protected]>

Closes #3690 from srowen/SPARK-785 and squashes the following commits:

8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey

(cherry picked from commit 2a28bc6)
Signed-off-by: Josh Rosen <[email protected]>
asfgit pushed a commit that referenced this pull request Dec 16, 2014
@srowen @JoshRosen
SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions
b9b6762 
This looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.

Author: Sean Owen <[email protected]>

Closes #3690 from srowen/SPARK-785 and squashes the following commits:

8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey

(cherry picked from commit 2a28bc6)
Signed-off-by: Josh Rosen <[email protected]>
asfgit pushed a commit that referenced this pull request Dec 16, 2014
@srowen @JoshRosen
SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions
63c0ff9 
This looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.

Author: Sean Owen <[email protected]>

Closes #3690 from srowen/SPARK-785 and squashes the following commits:

8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey

(cherry picked from commit 2a28bc6)
Signed-off-by: Josh Rosen <[email protected]>
@JoshRosen
Copy link
Contributor

JoshRosen commented Dec 17, 2014

I've merged this into branch-1.2, so this fix will be included in Spark 1.2.1.

asfgit pushed a commit that referenced this pull request Dec 17, 2014
@srowen @JoshRosen
SPARK-785 [CORE] ClosureCleaner not invoked on most PairRDDFunctions
76c88c6 
This looked like perhaps a simple and important one. `combineByKey` looks like it should clean its arguments' closures, and that in turn covers apparently all remaining functions in `PairRDDFunctions` which delegate to it.

Author: Sean Owen <[email protected]>

Closes #3690 from srowen/SPARK-785 and squashes the following commits:

8df68fe [Sean Owen] Clean context of most remaining functions in PairRDDFunctions, which ultimately call combineByKey

(cherry picked from commit 2a28bc6)
Signed-off-by: Josh Rosen <[email protected]>
@srowen srowen deleted the SPARK-785 branch December 18, 2014 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@srowen @SparkQA @AmplabJenkins @JoshRosen