Skip to content

[WIP][SPARK-30694][SHUFFLE]If exception occured while fetching blocks by ExternalBlockClient, fail early when External Shuffle Service is not alive #27419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AngersZhuuuu wants to merge 5 commits into from
Closed

[WIP][SPARK-30694][SHUFFLE]If exception occured while fetching blocks by ExternalBlockClient, fail early when External Shuffle Service is not alive #27419

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b9c3075
Update ExternalBlockStoreClient.java
AngersZhuuuu Jan 31, 2020
32d2be8
follow comment
AngersZhuuuu Feb 3, 2020
40dd076
Update RetryingBlockFetcherSuite.java
AngersZhuuuu Feb 3, 2020
c9061c5
Update ExternalBlockStoreClient.java
AngersZhuuuu Feb 4, 2020
2c4995e
check host rechable and port connectable
AngersZhuuuu Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions ...work-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockStoreClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,14 +103,22 @@ public void fetchBlocks(
try {
RetryingBlockFetcher.BlockFetchStarter blockFetchStarter =
(blockIds1, listener1) -> {
// Unless this client is closed.
if (clientFactory != null) {
TransportClient client = clientFactory.createClient(host, port);

@cloud-fan cloud-fan Feb 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we only add try-catch to wrap this line?

@AngersZhuuuu AngersZhuuuu Feb 3, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we only add try-catch to wrap this line?

It's OK to only wrap this line.

@AngersZhuuuu AngersZhuuuu Feb 3, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we only add try-catch to wrap this line?

Updated, hope for review current change

new OneForOneBlockFetcher(client, appId, execId,
blockIds1, listener1, conf, downloadFileManager).start();
} else {
logger.info("This clientFactory was closed. Skipping further block fetch retries.");
}
try{
// Unless this client is closed.
if (clientFactory != null) {
TransportClient client = clientFactory.createClient(host, port);
new OneForOneBlockFetcher(client, appId, execId,
blockIds1, listener1, conf, downloadFileManager).start();
} else {
logger.info("This clientFactory was closed. Skipping further block fetch retries.");
}
} catch (IOException e) {
logger.info("The relative remote external shuffle service (host: " + host + "," +
"port: "+ port + "), which maintains the block data can't been connected.");

@cloud-fan cloud-fan Feb 3, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what you do is just printing a log when error happens?

@AngersZhuuuu AngersZhuuuu Feb 3, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so what you do is just printing a log when error happens?

Sorry, I miss the way to stop retry.

May be we should add some way to check external shuffle service alive?

throw e;

@AngersZhuuuu AngersZhuuuu Feb 3, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @cloud-fan
I want to throw exception like https://github.com/AngersZhuuuu/spark/blob/e2dbe4bca387542e2043abc0801190531e805684/core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala#L130

But this module don't dependency spark-core, Add deps and throw exception like

/**
 * Exception throw when can't connect to External Shuffle Service
 */
private[spark] case class ExternalShuffleServiceLostException(message: String)
  extends SparkException(message)

Is what I want to do.

And seem we don't have way to check external shuffle service is alive.

} catch (Exception e) {
throw e;
}
};

int maxRetries = conf.maxIORetries();
Expand Down