Skip to content

[SPARK-7120][SPARK-7121] Closure cleaner nesting + documentation + tests #5685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 27 commits into from
Closed

[SPARK-7120][SPARK-7121] Closure cleaner nesting + documentation + tests #5685

Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
86f7823
Implement transitive cleaning + add missing documentation
Apr 24, 2015
2390a60
Feature flag this new behavior
Apr 24, 2015
438c68f
Minor changes
Apr 24, 2015
a4866e3
Add tests (still WIP)
Apr 24, 2015
06fd668
Make closure cleaning idempotent
Apr 24, 2015
263593d
Finalize tests
Apr 24, 2015
2106f12
Merge branch 'master' of github.com:apache/spark into closure-cleaner
Apr 25, 2015
6d36f38
Fix closure cleaner visibility
Apr 25, 2015
e672170
Guard against potential infinite cycles in method visitor
Apr 25, 2015
a3aa465
Add more tests for individual closure cleaner operations
Apr 25, 2015
eb127e5
Use private method tester for a few things
Apr 25, 2015
8b71cdb
Update a few comments
Apr 25, 2015
e45e904
More minor updates (wording, renaming etc.)
Apr 25, 2015
4aab379
Merge branch 'master' of github.com:apache/spark into closure-cleaner
Apr 25, 2015
6d4d3f1
Fix scala style?
Apr 26, 2015
9419efe
Bypass SerializationDebugger for now (SPARK-7180)
Apr 28, 2015
d889950
Revert "Bypass SerializationDebugger for now (SPARK-7180)"
Apr 28, 2015
9187066
Merge branch 'master' of github.com:apache/spark into closure-cleaner
Apr 28, 2015
3998168
In sc.runJob, actually clean the inner closure
Apr 29, 2015
e909a42
Guard against NPE if CC is used outside of an application
Apr 29, 2015
6f75784
Revert "Guard against NPE if CC is used outside of an application"
Apr 29, 2015
26c7aba
Revert "In sc.runJob, actually clean the inner closure"
Apr 29, 2015
16fbcfd
Merge branch 'master' of github.com:apache/spark into closure-cleaner
May 2, 2015
26c5072
Address comments
May 2, 2015
ea874bc
Fix tests
May 2, 2015
0bbe77f
Fix style
May 2, 2015
cd46230
Revert a small change that affected streaming
May 2, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1752,7 +1752,8 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
* serializable
*/
private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean = true): F = {
ClosureCleaner.clean(f, checkSerializable)
val cleanTransitively = conf.getBoolean("spark.closureCleaner.transitive", true)

@pwendell pwendell Apr 24, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that it makes sense to flag this since (provided we add scopes) then every user program will break if this is toggled. I think this feature will need to be removed wholesale if there are issues.

ClosureCleaner.clean(f, checkSerializable, cleanTransitively)
f
}

Expand Down
Loading