[SPARK-10620][WIP] Migrate TaskMetrics to accumulators#10717

Closed

andrewor14 wants to merge 72 commits intoapache:masterfrom

andrewor14:task-metrics-to-accums

Commits on Jan 4, 2016

Remove unused methods or replace them
Andrew Or
committed
Implement initial framework to migrate metrics to accums
Andrew Or
committed
Migrate a few more easy metrics
Andrew Or
committed
ShuffleReadMetrics + namespacing accumulators
Andrew Or
committed

Commits on Jan 5, 2016

General code cleanup
Andrew Or
committed
ShuffleWriteMetrics
Andrew Or
committed
OutputMetrics
Andrew Or
committed

Commits on Jan 6, 2016

InputMetrics
Andrew Or
committed
Fix JsonProtocol + JsonProtocolSuite
Andrew Or
committed
Fix tests where TaskMetrics had no accumulators
Andrew Or
committed
Rename a few shuffle write metrics for consistency
Andrew Or
committed
Fix metrics in local mode (tests)
Andrew Or
committed
Fix "harmless" exception in peak memory tests
Andrew Or
committed
Simplify internal accumulator update mechanism
Andrew Or
committed
Fix tests
Andrew Or
committed
Clean up
Andrew Or
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed

Commits on Jan 7, 2016

Do not send TaskMetrics from executors to driver
Andrew Or
committed
Restore accumulator serialization semantics
Andrew Or
committed
Fix semantics of accumulators when tasks fail
Andrew Or
committed
Fix a few more tests
Andrew Or
committed

Commits on Jan 8, 2016

Fix SQL UI
Andrew Or
committed
Clean up: lift odd unique name requirement
Andrew Or
committed
Move smaller metrics classes to their own files
Andrew Or
committed
Fix SQLQuerySuite
Andrew Or
committed
Remove unused hostname from TaskMetrics
Andrew Or
committed
Reinitialize OutputMetrics et al during reconstruction
Andrew Or
committed
Fix *ShuffleSuite
Andrew Or
committed

Commits on Jan 9, 2016

Fix DAGSchedulerSuite
Andrew Or
committed
Simplify accumulator update code a little
Andrew Or
committed
Track updated blocks as an accumulator
Andrew Or
committed
Minor clean ups + docs
Andrew Or
committed

Commits on Jan 11, 2016

Replace all accumulator updates with Seq[AccumulableInfo]
Andrew Or
committed
Always send Seq[AccumulableInfo] instead of TaskMetrics
Andrew Or
committed

Commits on Jan 12, 2016

Fix JsonProtocolSuite + cleanup
Andrew Or
committed
More cleanups
Andrew Or
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed
Fix style
Andrew Or
committed
Fix MiMa
Andrew Or
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed
Fix MiMa
Andrew Or
committed
Rename updatedBlocks -> updatedBlockStatuses
Andrew Or
committed
Fix wrong comment by removing it :)
Andrew Or
committed
Fix StringAccumulatorParam semantics
Andrew Or
committed
Relax scheduler dependency order + simplify some code
Andrew Or
committed

Commits on Jan 14, 2016

Merge github.com:apache/spark into task-metrics-to-accums
andrewor14
committed
Fix HistoryServerSuite

andrewor14
authored and
andrewor14
committed
Fix TaskMetricsSuite
andrewor14
committed
Fix JsonProtocolSuite
andrewor14
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed
Fix TaskContextSuite + add some param docs
Andrew Or
committed
Fix StagePageSuite
Andrew Or
committed
Fix ExternalSorterSuite
Andrew Or
committed
Fix SQLMetricsSuite
Andrew Or
committed
Fix PartitionBatchPruningSuite
Andrew Or
committed
Clean up Accumulators object a little + some docs
Andrew Or
committed
Fix DAGSchedulerSuite, trivial omission
Andrew Or
committed

Commits on Jan 15, 2016

Fix style + default
Andrew Or
committed
Split Accumulators.scala into 3 files
Andrew Or
committed
Fix MiMa
Andrew Or
committed
Minor cleanups
Andrew Or
committed
Fix style
Andrew Or
committed
Add InternalAccumulatorsSuite
Andrew Or
committed
Fix JsonProtocolSuite
Andrew Or
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed
Add more tests to AccumulatorSuite
Andrew Or
committed
Fix style
Andrew Or
committed
Add test for accum updates during failures
Andrew Or
committed
Merge branch 'master' of github.com:apache/spark into task-metrics-to-accums
Andrew Or
committed

Commits on Jan 16, 2016

Add tests for TaskMetrics
Andrew Or
committed
Add fine-grained test for collecting accumulators during failures
Andrew Or
committed

Commits on Jan 18, 2016

Fix style
Andrew Or
committed