[SPARK-3885] Provide mechanism to remove accumulators once they are no longer used#4021

Closed

ilganeli wants to merge 20 commits intoapache:masterfrom

ilganeli:SPARK-3885

Commits on Jan 7, 2015

Updated Accumulators class to store weak references instead of strong references to allow garbage collection of old accumulators
Ilya Ganelin
committed
Merge remote-tracking branch 'upstream/master' into SPARK-3885
Ilya Ganelin
committed

Commits on Jan 9, 2015

Merge remote-tracking branch 'upstream/master' into SPARK-3885
Ilya Ganelin
committed
Updated DAGScheduler and Suite to correctly use new implementation of WeakRef Accumulator storage
Ilya Ganelin
committed
Updated DAGSchedulerSUite to fix bug
Ilya Ganelin
committed

Commits on Jan 10, 2015

Removed obsolete comment
Ilya Ganelin
committed

Commits on Jan 12, 2015

reset
Ilya Ganelin
committed

Commits on Jan 13, 2015

Merge remote-tracking branch 'upstream/master' into SPARK-3885
Ilya Ganelin
committed

Commits on Jan 15, 2015

Reverting permissions changes
Ilya Ganelin
committed
Updated to throw Exception when accessing a GCd accumulator
Ilya Ganelin
committed

Commits on Jan 16, 2015

Added test for accumulator garbage collection
Ilya Ganelin
committed
Fixing gc test
Ilya Ganelin
committed
Still not being properly garbage collected
Ilya Ganelin
committed

Commits on Feb 7, 2015

Added working test for accumulator garbage collection
Ilya Ganelin
committed

Commits on Feb 8, 2015

Fixed build error
Ilya Ganelin
committed

Commits on Feb 21, 2015

Merge remote-tracking branch 'upstream/master' into SPARK-3885
Ilya Ganelin
committed
Added cleanup method for accumulators to remove stale references within Accumulators.original to accumulators that are now out of scope
Ilya Ganelin
committed
File deleted somehow
Ilya Ganelin
committed
Extra code
Ilya Ganelin
committed
Fixed error in test suite
Ilya Ganelin
committed