Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Broker] Refine topic level backlog quota policies warning log #11863

Conversation

gaoran10
Copy link
Contributor

Motivation

Currently, if the TopicPolicies isn't initialized completely, the topic gets backlog quota will print the error log stack, this is not necessary, because after the TopicPolicies initialized, the topic level backlog quota will come into effect.

15:24:01.962 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN  org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read topic policies data, will apply the namespace backlog quota: topicName=persistent://public/ns3/t1
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
	at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:148) ~[org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:85) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2172) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.isBacklogQuotaExceeded(PersistentTopic.java:2181) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$20(ServerCnx.java:1100) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.lambda$openLedgerComplete$2(BrokerService.java:1125) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815) [?:?]
	at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799) [?:?]
	at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2121) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:1111) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:425) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:397) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:476) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:466) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getCursors$8(MetaStoreImpl.java:122) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
	at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) [org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]

Modifications

Remove the error log stack.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

@gaoran10 gaoran10 self-assigned this Aug 31, 2021
@Anonymitaet Anonymitaet added the doc-not-needed Your PR changes do not impact docs label Sep 1, 2021
@codelipenghui codelipenghui added this to the 2.9.0 milestone Sep 1, 2021
…acklogQuotaManager.java

Co-authored-by: Michael Marshall <[email protected]>
@sijie sijie merged commit d370d00 into apache:master Sep 2, 2021
@gaoran10 gaoran10 deleted the gaoran/refine-backlog-quota-topic-policies-warn-log branch September 2, 2021 02:06
codelipenghui pushed a commit that referenced this pull request Sep 2, 2021
### Motivation

Currently, if the `TopicPolicies` isn't initialized completely, the topic gets backlog quota will print the error log stack, this is not necessary, because after the TopicPolicies initialized, the topic level backlog quota will come into effect.

```
15:24:01.962 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN  org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read topic policies data, will apply the namespace backlog quota: topicName=persistent://public/ns3/t1
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
	at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:148) ~[org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:85) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2172) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.isBacklogQuotaExceeded(PersistentTopic.java:2181) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$20(ServerCnx.java:1100) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.lambda$openLedgerComplete$2(BrokerService.java:1125) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815) [?:?]
	at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799) [?:?]
	at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2121) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:1111) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:425) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:397) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:476) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:466) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getCursors$8(MetaStoreImpl.java:122) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
	at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) [org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
```

### Modifications

Remove the error log stack.

(cherry picked from commit d370d00)
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Sep 2, 2021
codelipenghui pushed a commit that referenced this pull request Sep 6, 2021
…licies (#11897)

### Motivation

There is a bug that using the partitioned topic name to get topic policies. The PR #11294 fix this issue, but it's hard to cherry-pick the PR to `branch-2.7`, so create this PR to fix the issue in `branch-2.7`.

This PR contains [PR-11294](#11294) and [PR-11863](#11863).

### Modifications

1. Fix using the partitioned topic name to get topic policies.
2. Change some warning logs to debug level for the getting topic policies operation.

### Verifying this change

The test method `TopicPoliciesTest#testBacklogQuotaWithPartitionedTopic` is used to verify getting topic policies by the partitioned topic name.
@codelipenghui codelipenghui added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Sep 6, 2021
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
…e#11863)

### Motivation

Currently, if the `TopicPolicies` isn't initialized completely, the topic gets backlog quota will print the error log stack, this is not necessary, because after the TopicPolicies initialized, the topic level backlog quota will come into effect.

```
15:24:01.962 [bookkeeper-ml-workers-OrderedExecutor-2-0] WARN  org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read topic policies data, will apply the namespace backlog quota: topicName=persistent://public/ns3/t1
org.apache.pulsar.broker.service.BrokerServiceException$TopicPoliciesCacheNotInitException: Topic policies cache have not init.
	at org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService.getTopicPolicies(SystemTopicBasedTopicPoliciesService.java:148) ~[org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.BacklogQuotaManager.getBacklogQuota(BacklogQuotaManager.java:85) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.getBacklogQuota(PersistentTopic.java:2172) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.isBacklogQuotaExceeded(PersistentTopic.java:2181) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleProducer$20(ServerCnx.java:1100) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.lambda$openLedgerComplete$2(BrokerService.java:1125) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815) [?:?]
	at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799) [?:?]
	at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2121) [?:?]
	at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:1111) [org.apache.pulsar-pulsar-broker-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$8(ManagedLedgerFactoryImpl.java:425) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) [?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) [?:?]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:397) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:476) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3.operationComplete(ManagedLedgerImpl.java:466) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getCursors$8(MetaStoreImpl.java:122) [org.apache.pulsar-managed-ledger-2.7.3.jar:2.7.3]
	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
	at org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203) [org.apache.bookkeeper-bookkeeper-common-4.12.0.jar:4.12.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
	at java.lang.Thread.run(Thread.java:834) [?:?]
```

### Modifications

Remove the error log stack.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/broker cherry-picked/branch-2.7 Archived: 2.7 is end of life cherry-picked/branch-2.8 Archived: 2.8 is end of life doc-not-needed Your PR changes do not impact docs release/2.7.4 release/2.8.1 release/2.9.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants