[SPARK-6640][Core] Fix the race condition of creating HeartbeatReceiver and retrieving HeartbeatReceiver

[zsxwing]

This PR moved the code of creating HeartbeatReceiver above the code of creating schedulerBackend to resolve the race condition.

[SparkQA]

Test build #29525 has started for PR 5306 at commit d7c250d.

[SparkQA]

Test build #29525 has finished for PR 5306 at commit d7c250d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29525/
Test PASSed.

[viirya]

Looks weird here. I think it should be if (scheduler != null)?

[zsxwing]

Right. It's a typo. Thank you for pointing it out.

[SparkQA]

Test build #29546 has started for PR 5306 at commit dd202c7.

[SparkQA]

Test build #29546 has finished for PR 5306 at commit dd202c7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29546/
Test PASSed.

[sryza]

Why the need for reregistering vs. just ignoring the heartbeat?

[zsxwing]

For reregistering: Because Executor uses askWithReply to send Heartbeat, I want to send something back to Executor to avoid timeout.

[andrewor14]

@zsxwing this looks fine, though a little funky because we send TaskScheduler and all of its pointers through Akka, though I don't see another simpler alternative. In the past I've seen Akka silently drop messages that contain objects that are too big. Here we log a warning if that happens so I suppose it will be fine. I will merge this once you address the minor comments.

[zsxwing]

@zsxwing this looks fine, though a little funky because we send TaskScheduler and all of its pointers through Akka, though I don't see another simpler alternative.

We can get TaskScheduler from SparkContext since HeartbeatReceiver has a pointer to SparkContext. I updated the codes to eliminate TaskScheduler from the message.

[SparkQA]

Test build #29677 has started for PR 5306 at commit 840399d.

[SparkQA]

Test build #29677 has finished for PR 5306 at commit 840399d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29677/
Test PASSed.

[andrewor14]

LGTM merging this into master thanks. The new solution looks much better!

[rxin]

Should this be in 1.3.2? Seems like people are hitting it?