[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty. #3625

rxin · 2014-12-05T21:22:07Z

It's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2.

…nodes in Netty.

SparkQA · 2014-12-05T21:30:16Z

Test build #24193 has started for PR 3625 at commit 3e1306c.

This patch merges cleanly.

SparkQA · 2014-12-05T22:36:22Z

Test build #24193 has finished for PR 3625 at commit 3e1306c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-05T22:36:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24193/
Test FAILed.

aarondav · 2014-12-05T22:43:38Z

network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java

We can make this a private static class if we make this a constructor parameter.

aarondav · 2014-12-05T22:59:55Z

network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java

2 spaces for indent

SparkQA · 2014-12-05T23:05:59Z

Test build #24198 has started for PR 3625 at commit 9076b4a.

This patch merges cleanly.

aarondav · 2014-12-05T23:08:20Z

Looks mostly good to me, a few remaining synchronization issues. Will take another long look after you address all comments. I'd really appreciate a test, though, if we can get one in -- we really don't want to be regressing at this point, and we also really want to make sure we're fixing the issue.

SparkQA · 2014-12-06T00:05:16Z

Test build #24199 has started for PR 3625 at commit 41dfcb2.

This patch merges cleanly.

SparkQA · 2014-12-06T00:06:20Z

Test build #24198 has finished for PR 3625 at commit 9076b4a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-06T00:06:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24198/
Test FAILed.

SparkQA · 2014-12-06T00:20:00Z

Test build #24200 has started for PR 3625 at commit 0fefabb.

This patch merges cleanly.

SparkQA · 2014-12-06T01:31:20Z

Test build #24199 has finished for PR 3625 at commit 41dfcb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-06T01:31:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24199/
Test PASSed.

SparkQA · 2014-12-06T01:45:53Z

Test build #24200 has finished for PR 3625 at commit 0fefabb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-06T01:45:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24200/
Test PASSed.

aarondav · 2014-12-06T03:04:15Z

network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java

Q: Can we use JavaUtils.closeQuietly(client) here?

SparkQA · 2014-12-06T03:25:08Z

Test build #24206 has started for PR 3625 at commit ad4241a.

This patch merges cleanly.

AmplabJenkins · 2014-12-06T03:32:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24205/
Test FAILed.

aarondav · 2014-12-06T04:00:44Z

LGTM

rxin · 2014-12-06T04:24:42Z

Jenkins, retest this please.

SparkQA · 2014-12-06T04:27:55Z

Test build #24207 has started for PR 3625 at commit ad4241a.

This patch merges cleanly.

SparkQA · 2014-12-06T04:45:46Z

Test build #24206 has finished for PR 3625 at commit ad4241a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-06T04:45:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24206/
Test PASSed.

SparkQA · 2014-12-06T05:48:37Z

Test build #24207 has finished for PR 3625 at commit ad4241a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-06T05:48:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24207/
Test PASSed.

rxin · 2014-12-10T01:49:40Z

I'm merging this one in master & branch-1.2.

…nodes in Netty. It's been reported that when the number of disks is large and the number of nodes is small, Netty network throughput is low compared with NIO. We suspect the problem is that only a small number of disks are utilized to serve shuffle files at any given point, due to connection reuse. This patch adds a new config parameter to specify the number of concurrent connections between two peer nodes, default to 2. Author: Reynold Xin <[email protected]> Closes #3625 from rxin/SPARK-4740 and squashes the following commits: ad4241a [Reynold Xin] Updated javadoc. f33c72b [Reynold Xin] Code review feedback. 0fefabb [Reynold Xin] Use double check in synchronization. 41dfcb2 [Reynold Xin] Added test case. 9076b4a [Reynold Xin] Fixed two NPEs. 3e1306c [Reynold Xin] Minor style fix. 4f21673 [Reynold Xin] [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty. (cherry picked from commit 2b9b726) Signed-off-by: Reynold Xin <[email protected]>

rxin added 2 commits December 5, 2014 13:20

[SPARK-4740] Create multiple concurrent connections between two peer …

4f21673

…nodes in Netty.

Minor style fix.

3e1306c

aarondav reviewed Dec 5, 2014
View reviewed changes

Fixed two NPEs.

9076b4a

aarondav reviewed Dec 5, 2014
View reviewed changes

network/common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java Outdated

Copy link

Contributor

aarondav Dec 5, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 spaces for indent

Added test case.

41dfcb2

rxin changed the title ~~[SPARK-4740] [WIP] Create multiple concurrent connections between two peer nodes in Netty.~~ [SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty. Dec 5, 2014

Use double check in synchronization.

0fefabb

aarondav reviewed Dec 6, 2014
View reviewed changes

rxin added 2 commits December 5, 2014 19:14

Code review feedback.

f33c72b

Updated javadoc.

ad4241a

asfgit closed this in 2b9b726 Dec 10, 2014

[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty. #3625

[SPARK-4740] Create multiple concurrent connections between two peer nodes in Netty. #3625

Uh oh!

Conversation

rxin commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

AmplabJenkins commented Dec 5, 2014

Uh oh!

aarondav Dec 5, 2014

Choose a reason for hiding this comment

Uh oh!

aarondav Dec 5, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 5, 2014

Uh oh!

aarondav commented Dec 5, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

aarondav Dec 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

aarondav commented Dec 6, 2014

Uh oh!

rxin commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

SparkQA commented Dec 6, 2014

Uh oh!

AmplabJenkins commented Dec 6, 2014

Uh oh!

rxin commented Dec 10, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants