[SPARK-5945] Spark should not retry a stage infinitely on a FetchFailedException#5636

Closed

ilganeli wants to merge 50 commits intoapache:masterfrom

ilganeli:SPARK-5945

Commits on Apr 22, 2015

[SPARK-5945] Added map to track reasons for stage failures and supporting function to check whether to abort a stage when it fails for a single reason more than N times.
Ilya Ganelin
committed
Fixed method scoping error
Ilya Ganelin
committed
Made StageFailure private to spark scheduler
Ilya Ganelin
committed
Made fail() method public
Ilya Ganelin
committed
Added config option for stageFailure count and documentation
Ilya Ganelin
committed

Commits on Apr 24, 2015

Moved failure tracking to Stage class. Added clear of failre count upon Stage success
Ilya Ganelin
committed
Removed stale documentation and fixed some erroneous spacing
Ilya Ganelin
committed

Commits on Apr 28, 2015

Added test case for stage abort after N failures
Ilya Ganelin
committed
Nit
Ilya Ganelin
committed

Commits on Apr 29, 2015

Added test that also validates that the listenerBus sees the JobFailed error
Ilya Ganelin
committed

Commits on Apr 30, 2015

updated to fix last few items
Ilya Ganelin
committed
Removed extraneous assert
Ilya Ganelin
committed
Fixed a couple minor issues. Removed unecessary asserT
Ilya Ganelin
committed
Fixed a couple minor issues. Removed unecessary asserT
Ilya Ganelin
committed
Fixed missing maxStageFailures
Ilya Ganelin
committed

Commits on May 4, 2015

Updated to add test cases for multiple task failures within a Stage. Updated to only increment fetch failures once per attempt ID for a stage.
Ilya Ganelin
committed
Typo fix
Ilya Ganelin
committed
Merge remote-tracking branch 'upstream/master' into SPARK-5945
Ilya Ganelin
committed

Commits on Jul 23, 2015

Resolved merge conflicts. Now simply count the nubmer stage failures for a particular stage (don't need to maintain a map since concurrent failures are no longer possible.
Ilya Ganelin
committed
Style
Ilya Ganelin
committed
restoring lost files
Ilya Ganelin
committed

Commits on Jul 24, 2015

Updated test suite to properly create task sets and force stage failures
Ilya Ganelin
committed
got rid of println
Ilya Ganelin
committed

Commits on Jul 27, 2015

Updated tests for stage failures.
Ilya Ganelin
committed
Merge remote-tracking branch 'upstream/master' into SPARK-5945
Ilya Ganelin
committed

Commits on Jul 28, 2015

Updated test for fetch failures. Added validation of successfully generated output
Ilya Ganelin
committed
Added a test for multiple fetch failures inside a single stage
Ilya Ganelin
committed
Added a test for multiple fetch failures inside a single stage
Ilya Ganelin
committed
Added test to ensure that stage failure only triggers with successive fetch failures. NOTE: This test does not presently work
Ilya Ganelin
committed
Test updates. Some tests still failing, unsure why.
Ilya Ganelin
committed

Commits on Jul 29, 2015

Added stage ID checks in most places and fixed naming for attempts to be 0-indexed
Ilya Ganelin
committed
All tests passing. Still need to refactor multiple fetch failures per stage into a three task test
Ilya Ganelin
committed
Updated remaining test for sequential test failures to have three stages to avoid introducing a fetch failure in stage 0.
Ilya Ganelin
committed
Nit fixes
Ilya Ganelin
committed
Style
Ilya Ganelin
committed
Naming
Ilya Ganelin
committed

Commits on Aug 4, 2015

refactored tests to eliminate reused code
Ilya Ganelin
committed
Style
Ilya Ganelin
committed

Commits on Aug 5, 2015

Nit fixes.
Ilya Ganelin
committed
Style
Ilya Ganelin
committed
Style
Ilya Ganelin
committed

Commits on Aug 6, 2015

Nits
Ilya Ganelin
committed

Commits on Aug 19, 2015

Fixed merge conflict
Ilya Ganelin
committed
Style fix
Ilya Ganelin
committed

Commits on Sep 1, 2015

Merge remote-tracking branch 'upstream/master' into SPARK-5945
Ilya Ganelin
committed
Updated PR description and minor nits
Ilya Ganelin
committed

Commits on Sep 2, 2015

Updated with feedback from PR
Ilya Ganelin
committed
Style fix
Ilya Ganelin
committed
Test case updates and nit fixes
Ilya Ganelin
committed
Added more comments to clarify tricky test case
Ilya Ganelin
committed