[Issue 8093]Fix client lookup hangs when broker restarts #8101
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
codelipenghui
requested review from
merlimat,
codelipenghui,
jiazhai and
sijie
| @@ -400,6 +400,11 @@ public boolean registerNamespace(String namespace, boolean ensureOwned) throws P | |||
| private void searchForCandidateBroker(NamespaceBundle bundle, | |||
| CompletableFuture<Optional<LookupResult>> lookupFuture, | |||
| LookupOptions options) { | |||
| if( null == pulsar.getLeaderElectionService() || ! pulsar.getLeaderElectionService().isElected()) { | |||
| LOG.warn("The leader election has not yet been completed! NamespaceBundle[{}]", bundle); | |||
| lookupFuture.completeExceptionally(new IllegalStateException("The leader election has not yet been completed!")); | |||
There was a problem hiding this comment.
Thanks for the great contribution, I think it's better to return a retryable exception to the client? So that the client can reconnect later. Does this make sense?
There was a problem hiding this comment.
As discussed with @4onni, this will resultin a client lookup timeout, and the client will relookup later, so it's not a problem here.
codelipenghui
approved these changes
Sep 22, 2020
|
/pulsarbot run-failure-checks |
4 similar comments
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
jiazhai
approved these changes
Sep 22, 2020
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
4 similar comments
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
jiazhai
pushed a commit
to jiazhai/pulsar
that referenced
this pull request
Sep 24, 2020
Fixes apache#8093 ### Motivation Client hangs forever when all brokers stop and then restart. There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below: ``` PulsarService#start(): broker.start(); // Step 1 webService.start(); // Step 2 leaderElectionService.start(); //Step 3 ``` If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly. ### Modifications Client can only connect to the broker after the election service started successfully ### Verifying this change This change added tests and can be verified as follows: - * Added 2 test cases under `LeaderElectionServiceTest` (cherry picked from commit 65cf9c0)
jiazhai
pushed a commit
to jiazhai/pulsar
that referenced
this pull request
Sep 24, 2020
Fixes apache#8093 Client hangs forever when all brokers stop and then restart. There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below: ``` PulsarService#start(): broker.start(); // Step 1 webService.start(); // Step 2 leaderElectionService.start(); //Step 3 ``` If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly. Client can only connect to the broker after the election service started successfully This change added tests and can be verified as follows: - * Added 2 test cases under `LeaderElectionServiceTest` (cherry picked from commit 65cf9c0)
lbenc135
pushed a commit
to lbenc135/pulsar
that referenced
this pull request
Oct 3, 2020
Fixes apache#8093 ### Motivation Client hangs forever when all brokers stop and then restart. There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below: ``` PulsarService#start(): broker.start(); // Step 1 webService.start(); // Step 2 leaderElectionService.start(); //Step 3 ``` If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly. ### Modifications Client can only connect to the broker after the election service started successfully ### Verifying this change This change added tests and can be verified as follows: - * Added 2 test cases under `LeaderElectionServiceTest`
|
already merged into branch-2.6 for 2.6.2 release |
wolfstudy
added a commit
that referenced
this pull request
Oct 30, 2020
…)" This reverts commit a4a363c.
wolfstudy
pushed a commit
that referenced
this pull request
Oct 30, 2020
Fixes #8093 ### Motivation Client hangs forever when all brokers stop and then restart. There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below: ``` PulsarService#start(): broker.start(); // Step 1 webService.start(); // Step 2 leaderElectionService.start(); //Step 3 ``` If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly. ### Modifications Client can only connect to the broker after the election service started successfully ### Verifying this change This change added tests and can be verified as follows: - * Added 2 test cases under `LeaderElectionServiceTest` (cherry picked from commit 65cf9c0)
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Nov 13, 2020
Fixes apache#8093 ### Motivation Client hangs forever when all brokers stop and then restart. There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below: ``` PulsarService#start(): broker.start(); // Step 1 webService.start(); // Step 2 leaderElectionService.start(); //Step 3 ``` If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly. ### Modifications Client can only connect to the broker after the election service started successfully ### Verifying this change This change added tests and can be verified as follows: - * Added 2 test cases under `LeaderElectionServiceTest`
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #8093
Motivation
Client hangs forever when all brokers stop and then restart.
There are several steps need to be finished before the broker can be fully started, as illustrated in the pseudo code below:
If a lookup request gets in between Step 2 and Step 3, a NPE would be thrown, which will block all other coming requests from getting processed properly.
Modifications
Client can only connect to the broker after the election service started successfully
Verifying this change
This change added tests and can be verified as follows:
LeaderElectionServiceTest