[SPARK-2677][SPARK-2717]BasicBlockFetchIterator#next can wait forever

[witgo]

No description provided.

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17280/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17280/consoleFull

[pwendell]

Can you create a test for this? I'm not sure what happens here if the timeout is encountered.

[sarutak]

@witgo @pwendell I have already noticed there is not a configuration for timeout for ConnectionManager, but the timeout for ConnectionManager does not resolve this issue because the channel used by receiving ack is implemented as non blocking I.O and SO_TIMEOUT is effects read after establishing connection. So, if remote executor hangs, it cannot establish connections with fetching executors.

Additionally, BasicBlockFetcherIterator is wait on LinkedBlockingQueue#take (result.take) so we should set FetchResult object which size is -1 to result queue of BasicBlockFetcherIterator.
(FetchResult which size is -1 means fetch failed)

I think remote errors can be classified following 2 cases.

1. Remote Executor hang
   In this case, we need timeout for Fetch Request (Not read timeout)
   I'm trying to resolve this case in [SPARK-2677] BasicBlockFetchIterator#next can wait forever #1632

2. Remote Executor not hang but error occurred
   In this case, remote executor should send message which means error occurred in remote Executor.
   I'm trying to resolve this case in [SPARK-2583] ConnectionManager cannot distinguish whether error occurred or not #1490
   This is ongoing.
   Can anyone review this too?

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17350/consoleFull

[witgo]

@sarutak I think add a heartbeat detection mechanism is a good solution

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17350/consoleFull

[witgo]

@sarutak ConnectionManager.scala#L259 to deal with the situation of connection cannot be established.

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17356/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17356/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17449/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17450/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17451/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17449/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17450/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17451/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17580/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17580/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17583/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17583/consoleFull

[SparkQA]

QA tests have started for PR 1619. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18097/consoleFull

[SparkQA]

QA results for PR 1619:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18097/consoleFull