[fix][broker] Broker failed to load v1 namespace resources cache

[rdhabalia]

Motivation

Right now, Pulsar service loads namespace policies cache on startup or namespace policy update. However, it doesn't consider v1 namespaces, don't load v1 namespace policy cache and also try to deserialize incorrect znode of v1 namespace policy path which causes below exception during server startup if you have v1 namespace: /admin/policies/prop/us-east1/nsv1

00:57:11.633 [ForkJoinPool.commonPool-worker-23] ERROR org.apache.pulsar.broker.resourcegroup.ResourceGroupNamespaceConfigListener - Exception when getting namespace weather/global
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException$ContentDeserializationException: Failed to deserialize payload for key '/admin/policies/prop/us-east1'
        at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?]
        at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?]
        at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1074) ~[?:?]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
        at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleGetResult(ZKMetadataStore.java:244) ~[pulsar-metadata.jar]
        at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:188) ~[pulsar-metadata.jar]
        at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$3$1.processResult(PulsarZooKeeperClient.java:490) ~[pulsar-metadata.jar]
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:721) ~[zookeeper-3.6.2.jar:3.6.2]
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:562) ~[zookeeper-3.6.2.jar:3.6.2]

Therefore, broker should traverse v1 namespace policies correctly to load v1 namespace policy cache and avoid deserialization failure for v1 namespace policy path.

Modifications

• introduce metadata-store recursive traversal till leaf nodes
• namespace resources traverse and retrieve v1 namespace policy nodes and caches v1 namespace policy

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[codecov-commenter]

Codecov Report

Merging #20783 (2979881) into master (4e2ba4b) will increase coverage by 0.02%.
The diff coverage is 80.00%.

@@             Coverage Diff              @@
##             master   #20783      +/-   ##
============================================
+ Coverage     73.10%   73.12%   +0.02%     
- Complexity    32108    32137      +29     
============================================
  Files          1866     1866              
  Lines        139028   139055      +27     
  Branches      15294    15299       +5     
============================================
+ Hits         101634   101689      +55     
+ Misses        29333    29310      -23     
+ Partials       8061     8056       -5     

Flag	Coverage Δ
inttests	24.17% <66.66%> (+<0.01%)	⬆️
systests	24.95% <66.66%> (-0.07%)	⬇️
unittests	72.41% <80.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...va/org/apache/pulsar/broker/service/ServerCnx.java	71.39% <ø> (ø)
...lsar/common/policies/data/OffloadPoliciesImpl.java	83.43% <37.50%> (-1.28%)	⬇️
.../apache/pulsar/broker/resources/BaseResources.java	72.72% <95.23%> (+5.31%)	⬆️
...he/pulsar/broker/resources/NamespaceResources.java	81.42% <100.00%> (ø)

... and 77 files with indirect coverage changes