[fix][meta] Bookie Info lost by notification race condition.

[mattisonchao]

Motivation

This fixes #20643.

Investigating

Why is the exception Bookie handle is not available?

We found this problem in the broker's continued to report Bookie handle is not available exception longer than 6 hours. The call chain should be as follows:

1. Bk client is trying to obtain a new channel for connection.
2. The channel was disconnected, So it is trying to connect again.
3. Bk Client is trying to get the bookie info by bookie id.
4. Bk client uses pulsar implemented PulsarRegistrationClient to get bookie info, but the local cache doesn't have it, and then throws the BKBookieHandleNotAvailableException

Why did cache lose Bookie's info?

The following is an analysis that has yet to be rigorously tested. (Since this will consume a relatively long time)

1. Check out our implementation. We will update the local cache async/sync(Depending on the internal metadata cache) and invalidate the local cache synchronously.
2. In Bookie ZKRegistrationManager's implementation, transit bookie mode from writable to read-only should create a read-only bookie node and then delete the writable bookie node.
3. writable -> read-only notification should be as follow:

• CREATED [read-only path]:bookieId
• DELETE [writable path]: bookieId

4. Consider that metadata has a cache that invokes the CREATEED notification logic synchronously. So, we may create this bookie info first and then delete it. (bookie info lost here)

Modifications

• Make the notification process sequentially.
• Support bookie mode in the local cache.

Verifying this change

• Make sure that the change passes the CI checks.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[github-actions]

@mattisonchao Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[hangc0276]

Nice catch!

[codecov-commenter]

Codecov Report

Merging #20642 (40d326a) into master (43b3622) will increase coverage by 1.06%.
The diff coverage is 81.11%.

@@             Coverage Diff              @@
##             master   #20642      +/-   ##
============================================
+ Coverage     72.05%   73.12%   +1.06%     
- Complexity    31718    32061     +343     
============================================
  Files          1855     1867      +12     
  Lines        138376   138782     +406     
  Branches      15198    15266      +68     
============================================
+ Hits          99703   101479    +1776     
+ Misses        30656    29262    -1394     
- Partials       8017     8041      +24     

Flag	Coverage Δ
inttests	24.11% <37.77%> (+0.01%)	⬆️
systests	24.97% <25.55%> (?)
unittests	72.41% <81.11%> (+0.60%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../metadata/bookkeeper/PulsarRegistrationClient.java	80.85% <81.11%> (-4.11%)	⬇️

... and 243 files with indirect coverage changes

[tisonkun]

Great.

@mattisonchao one suggestion is that you can link to a permalink instead of blob/main where the line number will quickly be invalid.

[liangyepianzhou]

@hangc0276 @mattisonchao The zookeeper version quoted in this P person is 3.8.1(master) instead of 3.6.4 (branch-2.10).
We should not upgrade the version in branch 2.10, according to #19425.
Could you please help take a look at whether we need to cherry-pick this PR to branch 2.10?