v2.11.4

What's Changed

Broker

• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix lookupRequestSemaphore leak when topic not found (#21646)
• [fix][broker] Fix the issue of topics possibly being deleted. (#21704)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Update topic policies as much as possible when some ex was thrown (#21810)
• [fix][broker] fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher (#21765)
• [fix][broker]Delete compacted ledger when topic is deleted (#21745) (#21850)
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException (#21856)
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter (#22011)
• [fix][ml] Fix retry mechanism of deleting ledgers to invalidate (#21869)
• [improve][admin] internalGetMessageById shouldn't be allowed on partitioned topic (#19013)
• [improve][broker] Add fine-grain authorization to retention admin API (#22163)
• [improve][broker] Avoid print redirect exception log when get list from bundle (#20846)
• [improve][broker] Consistently add fine-grain authorization to REST API (#22202)
• [improve][broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. (#21995)
• [improve][broker] Make get list from bundle Admin API async (#20652)
• [refactor][broker] Suppress error logging when message expiration fails (#19778)

Misc

• [improve][build] Upgrade Apache ZooKeeper to 3.9.1 (#20933)
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE (#21057)
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 (#21341)
• [fix][sec] Exclude avro from hadoop-client (#21719)
• [fix][sec] Upgrade Jetty to 9.4.53 to address CVE-2023-44487 (#21395)
• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 (#22144)
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 (#21397)
• [fix][sec] Upgrade commons-compress to 1.26.0 (#22086)
• [fix][sec] cve: exclude ch.qos.logback in canal.protocol * resolve CVE-2023-6378 (95e1de7)
• [fix][test] Fix PerformanceProducer send count error (#21706)
• [fix][test] Fix test testTransactionBufferMetrics (#22117)
• [fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation (#21976)
• [fix][test] testModularLoadManagerRemoveBundleAndLoad (#19710)

Proxy

• [improve][proxy] Fix comment about enableProxyStatsEndpoints (#21757)

Functions

• [fix][fn] Fix Deadlock in Functions Worker LeaderService (#21711)
• [improve][fn] Add configuration for connector & functions package url sources (#22184)
• [improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access (#22122)

Clients

• [fix][client] fix huge permits if acked a half batched message (#22091)

Full Changelog: v2.11.3...v2.11.4

v2.10.6

Broker

Improvements

• [improve][broker] Consistently add fine-grain authorization to REST API 22202
• [improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995
• [improve][proxy] Fix comment about enableProxyStatsEndpoints 21757
• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578
• [improve] [client] Merge lookup requests for the same topic 21232
• [improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220
• [improve] [proxy] Not close the socket if lookup failed caused by too many requests 21216
• [improve] [broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex 21211
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger 21201
• [improve] [broker] improve read entry error log for troubleshooting 21169
• [improve][proxy] Support disabling metrics endpoint 21031
• [improve][broker] Improve cache handling for partitioned topic metadata when doing lookup 21063
• [improve][broker] Avoid print redirect exception log when get list from bundle 20846
• [improve][sql] Fix the wrong format of the logs 20907
• [improve][broker] Add consumer-id into the log when doing subscribe. 20568
• [improve] [broker] Print warn log if compaction failure 19405
• [improve][admin]internalGetMessageById shouldn't be allowed on partitioned topic 19013

Fixes

• [fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 22144
• [fix][sec] Upgrade commons-compress to 1.26.0 22086
• [fix][broker] Support running docker container with gid != 0 22081
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044
• [fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011
• [fix] [broker] Replication stopped due to unload topic failed 21947
• [fix] [broker] fix write all compacted out entry into compacted topic 21917
• [fix] [ml] Fix retry mechanism of deleting ledgers to invalidate 21869
• [fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856
• [fix][broker] Fix String wrong format 21829
• [fix] [broker] Update topic policies as much as possible when some ex was thrown 21810
• [fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765
• [fix][sec] Exclude avro from hadoop-client 21719
• [fix][test] Fix PerformanceProducer send count error 21706
• [fix][broker] Fix the issue of topics possibly being deleted. 21704
• [fix][broker] Fix typo in the config key 21690
• [fix] [broker] network package lost if enable haProxyProtocolEnabled 21684
• [fix][broker] Fix memory leak during topic compaction 21647
• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591
• [fix][broker] Correct schema deletion for partitioned topic 21574
• [fix][broker] Fix namespace bundle stuck in unloading status (#21445) 21567
• [fix][broker] Fix create topic with different auto creation strategies causes race condition 21545
• [fix] [ml] Fix orphan scheduled task for ledger create timeout check 21542
• [fix] [broker] Fix thousands orphan PersistentTopic caused OOM 21540
• [fix][ml] Fix unfinished callback when deleting managed ledger 21530
• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510
• [fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495
• [branch-2.10][fix][proxy] Move status endpoint out of auth coverage 21494
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider 21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy 21481
• [fix][broker] Fix namespace bundle stuck in unloading status 21445
• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429
• [fix][proxy] Move status endpoint out of auth coverage 21428
• [fix][sec] Upgrade Netty to 4.1.100 to address CVE-2023-44487 21397
• [fix][sec] Bump avro version to 1.11.3 for CVE-2023-39410 21341
• [fix][test] Fix flaky test NarUnpackerTest 21328
• [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. 21327
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear 21302
• [fix][sec] Upgrade snappy-java to 1.1.10.5 21280
• [fix][txn] Ack all message ids when ack chunk messages with transaction. 21268
• [fix][broker][branch-2.10] Fix inconsistent topic policy 21258
• [fix] [ml] fix wrong msg backlog of non-durable cursor after trim ledgers 21250
• [fix] [ml] Reader can set read-pos to a deleted ledger 21248
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled 21237
• [fix][broker] Fix inconsistent topic policy 21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats 21225
• [fix][broker] Make the new exclusive consumer instead the inactive one faster 21183
• [fix][broker][branch-2.10] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker 21182
• [Branch-2.10] [imporve] [bookie] Upgrade BookKeeper dependency to 4.14.8 for branch 2.10 21148
• [fix][client] Fix repeat consume when using n-ack and batched messages 21116
• [fix][client] Avoid ack hole for chunk message 21101
• [branch-2.10] [fix] [broker] Fix isolated group not work problem. 21098
• [fix] [broker] Fix isolated group not work problem. 21096
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects 21081
• [fix][client] Fix consumer can't consume resent chunked messages 21070
• [fix][broker] Make sure all inflight writes have finished before completion of compaction 21067
• [fix][broker] Use MessageDigest.isEqual when comparing digests 21061
• [fix][misc] Bump GRPC version to 1.55.3 to fix CVE 21057
• [fix] [bk] Correctct the bookie info after ZK client is reconnected 21035
• [fix][io] Update test certs for Elasticsearch 21001
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId 20988
• [fix][broker] Fix message loss during topic compaction [20980](https://github.com/apache/pulsar/p...

v3.2.0

PIPs

• PIP-323: Complete Backlog Quota Telemetry #21709
• PIP-322: Pulsar Rate Limiting Refactoring #21680
• PIP-320: OpenTelemetry Scaffolding #21635
• PIP-318: Don't retain null-key messages during topic compaction #21541
• PIP-315: Configurable max delay limit for delayed delivery #21490
• PIP-313: Support force unsubscribe using consumer api #21452
• PIP-312:
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21597
  • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21438
• PIP-307:
  • Support subscribing multi topics for WebSocket #21390
  • Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager #20748
• PIP-305: Customize DNS servers to use for Pulsar Client #21352
• PIP-303: Add optional parameters for getPartitionedStats #21228
• PIP-302: Introduce refreshAsync API for TableView #21271
• PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD #21129
• PIP-300: Add custom dynamic configuration for plugins #21127
• PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage #21118
• PIP-297: Support terminating Function & Connector with the fatal exception #21079
• PIP-293: Delete config disableBrokerInterceptors #20993
• PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy #20923
• PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index #20867
• PIP-277: Add current option in the Clusters list cmd #20614
• PIP-264: Enhanced OTel-based metric system #21080

Broker

• Replication stopped due to unload topic failed #21947
• Fix getMessageById throws 500 #21919
• Fix write all compacted out entry into compacted topic #21917
• Fix deadlock while skip non-recoverable ledgers #21915
• Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894
• Fix DLQ producer name conflicts when multiples consumers send messages to DLQ #21890
• Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885
• Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842
• Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765
• Fix the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764
• Fix closing of AbstractDispatcherSingleActiveConsumer and reduce flakiness of test #21736
• Skip topic auto-creation for ExtensibleLoadManager internal topics #21729
• Fix ServiceUnitStateChannel monitor to tombstone only inactive bundle states #21721
• Avoid compaction task stuck when the last message to compact is a marker #21718
• Fix the issue of topics possibly being deleted #21704
• CheckTopicExists supports checking partitioned topic without index #21701
• Remove internalGetListAsync and validatePartitionTopicUpdateAsync #21699
• Cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696
• Fix typo in the config key #21690
• network package lost if enable haProxyProtocolEnabled #21684
• Pulsar Rate Limiting Refactoring changes (PIP-322) #21681
• Record GeoPersistentReplicator.msgOut before producersendAsync #21673
• Print recoverBucketSnapshot log if cursorProperties are empty #21651
• Fix memory leak during topic compaction #21647
• Fix lookupRequestSemaphore leak when topic not found #21646
• Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633
• Fix getting incorrect KeyValue schema version #21632
• Add optional parameters for getPartitionedStats #21611
• PIP-301 Part-3: Add QuotaResources #21596
• Delete topic timeout due to NPE #21595
• Fix incorrect unack count when using shared subscription on non-persistent topic #21592
• Duplicate LedgerOffloader creation when namespace/topic… #21591
• Support not retaining null-key message during topic compaction #21578
• Correct schema deletion for partitioned topic #21574
• Fix create topic with different auto creation strategies causes race condition #21545
• Fix thousands orphan PersistentTopic caused OOM #21540
• Fix setReplicatedSubscriptionStatus incorrect behavior #21510
• Fix failure while creating non-durable cursor with inactive managed-ledger #21508
• Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
• Do not write replicated snapshot marker when the topic which is not enable replication #21495
• Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• Fix issue with consumer read uncommitted messages from compacted topic #21465
• Delete methods marked Lombok @Getter #21462
• Fix resource_quota_zpath #21461
• Fix namespace bundle stuck in unloading status #21445
• Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered #21434
• Fix MultiRoles token provider NPE when using anonymous clients #21429
• Allow broker deployment in heterogeneous hw config cluster without restricting nic speed detection #21409
• Fix avoid creating new topic after migration is started #21368
• Fix race condition of replication cluster connection during migration topic #21364
• Optimize and clean up aggregation of topic stats #21361
• Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• PIP-188 Fix cluster migration state store into local metadatastore #21359
• PIP-188 Add support to auto create topic resources into green cluster before migration [part-3] #21354
• Fix heartbeat namespace create transaction internal topic #21348
• Fix unload operation stuck when use ExtensibleLoadManager #21332
• Fix typos in Subscription interface #21321
• Use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
• Cleanup deprecated method of metadata store etcd...

v2.11.3

Broker

• [fix][broker] Fix typo in the config key #21690
• [fix] [broker] network package lost if enable haProxyProtocolEnabled #21684
• [fix][broker] Fix memory leak during topic compaction #21647
• [fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic #21592
• [fix][broker] Duplicate LedgerOffloader creation when namespace/topic… #21591
• [improve][broker][PIP-318] Support not retaining null-key message during topic compaction #21578
• [fix][broker] Correct schema deletion for partitioned topic #21574
• [fix][broker] Fix namespace bundle stuck in unloading status (#21445) #21566
• [fix][broker] Fix create topic with different auto creation strategies causes race condition #21545
• [fix] [ml] Fix orphan scheduled task for ledger create timeout check #21542
• [fix] [broker] Fix thousands orphan PersistentTopic caused OOM #21540
• [fix][ml] Fix unfinished callback when deleting managed ledger #21530
• [fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior #21510
• [fix][txn] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• [fix][broker] Fix issue with consumer read uncommitted messages from compacted topic #21465
• [fix][broker] Fix resource_quota_zpath #21461
• [fix][broker] Fix namespace bundle stuck in unloading status #21445
• [fix][broker] Fix MultiRoles token provider NPE when using anonymous clients #21429
• [fix][ci] Fix docker image building by releasing more disk space before building #21365
• [fix][broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• [fix][broker] Fix heartbeat namespace create transaction internal topic #21348
• [fix] [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem. #21327
• [fix] [auto-recovery] [branch-2.11] Fix pulsar ledger auditor dead lock problem. #21319
• [fix] [auto-recovery] [branch-2.11] Fix PulsarLedgerUnderreplicationManager notify problem. #21318
• [fix][auto-recovery] [branch-2.11] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21315
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
• [fix][txn] Ack all message ids when ack chunk messages with transaction. #21268
• [fix] [ml] fix wrong msg backlog of non-durable cursor after trim ledgers #21250
• [fix] [ml] Reader can set read-pos to a deleted ledger #21248
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled #21237
• [fix][broker] Fix inconsistent topic policy #21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats #21225
• [improve] [broker] Let the producer request success at the first time if the previous one is inactive #21220
• [improve] [proxy] Not close the socket if lookup failed caused by too many requests #21216
• [improve] [broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
• [fix][broker] replicator leak when removeReplicator in NonPersistentTopic #21205
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger #21201
• [fix] [broker] Make the new exclusive consumer instead the inactive one faster #21183
• [fix] [auto-recovery] Fix pulsar ledger auditor dead lock problem. #21181
• [fix][broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21178
• [improve] [broker] improve read entry error log for troubleshooting #21169
• [branch-2.11] Fix compatibility issues #21163
• [fix] [auto-recovery] Fix PulsarLedgerUnderreplicationManager notify problem. #21161
• [fix][auto-recovery] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21159
• [fix] [bookie] Fix RocksDB configuration #21157
• [fix] [broker] Make specified producer could override the previous one #21155
• [improve] [broker] Improve logs for troubleshooting #21141
• [fix][broker] Fix unack count when mixing non batch index and batch index acks #21126
• [fix] [broker] Fix isolated group not work problem. #21096
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
• [fix] [broker] remove bundle-data in local metadata store. #21078
• [fix][proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called #21077
• [fix][broker] Make sure all inflight writes have finished before completion of compaction #21067
• [improve][broker] Make read compacted entries support maxReadSizeBytes limitation #21065
• [improve] [broker] Improve cache handling for partitioned topic metadata when doing lookup #21063
• [fix][broker] Use MessageDigest.isEqual when comparing digests #21061
• [fix] [auto-recovery] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes #21059
• [fix][broker] Fix potential case cause retention policy not working on topic level #21041
• [fix][fn] Fix ProducerConfig cannot update error #21037
• [fix] [bk] Correctct the bookie info after ZK client is reconnected #21035
• [fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
• [fix][fn] Fix the --batch-builder not working error for functions #21023
• [fix][meta] Fix deadlock in AutoRecovery. #21010
• [fix][broker] Fix incorrect unack msk count when dup ack a message #20990
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
• [fix][broker] Fix compaction subscription delete by inactive subscription check. #20983
• [fix][broker] Fix message loss during topic compaction #20980
• [fix][broker] Fix incorrect number of read compacted entries #20978
• [fix] [admin] Fix get topic stats fail if a subscription catch up concurrently #20971
• [branch-2.11] [fix] [broker] fix multiple versions of bouncy-castle #20960
• [fix] [broker] Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
• [fix][broker]Fix chunked messages will be filtered by duplicating #20948
• [fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
• [fix][broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
• [fix][broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication. #20905
• [fix] [ml] fix discontinuous ledger deletion #20898
• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent #20888
• [improve][txn]Add log when can't send TcClientConnectRequest to broker. #20880
• [fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached #20855
• [fix][broker] Inconsistent behaviour for topic auto_creation #20843
• [improve] [ws] add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
• [fix][build] Upgrade PyYaml version to 6.0.1 #20831
• [fix][broker] Avoid infinite bundle unloading #20822
• [improve][offload] Extend the offload policies to allow specifying more conf #20804
• [improve][offload] Create offload resources lazily #20775
• [fix] [broker] Can not receive any messages after switch to standby cluster #20767
• [fix][broker] Fix get topic policies as null during clean cache #20763
• [fix][broker] Avoid throwing RestException in BrokerService #20761
• [improve][offload] Support to configure more offload driver #20736
• [fix][broker] Gracefully shutdown does not work with admin cli in standalone #20709
• [fix][broker] Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
• [improve] [broker] Add consumer-id into the log when doing subscribe. #20568
• [fix][broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
• [improve][broker] Replace ScheduledExecutorService to ExecutorService in ModularLoadManagerImpl #19656
• [fix][broker] Fix namespace deletion if __change_events topic has not been created yet #18804
• [improve] Introduce the sync() API to ensure consistency on reads during critical metadata operation paths #18518
• [fix][broker] Fix namespace not found will cause request timeout #18512
• [fix][broker] fix bug caused by optimistic locking #18390
• [fix][broker] fix broker unackmessages become a negative number #17003

Clients

• [fix][client] Fix print error log 'Auto getting partitions failed' when expend partition. #21485
• [fix] [client] fix reader.hasMessageAvailable return false when incoming queue is not empty #21259
• [improve] [client] Merge lookup requests for the same topic #21232
• [fix] [client] fix same producer/consumer use more than one connection per broker #21144
• [fix][client] Fix repeat consume when using n-ack and batched messages #21116
• [fix][client] Fix logging problem in pulsar client #21094
• [fix][client] Fix subscribing pattern topics through Proxy not working #20739
• [fix][client] Fix producer could send timeout when enable batching #19191
• [fix][client] Fix perf-producer get OOM with high publish latency #20914
• [fix][client] Fix RawReader hasMessageAvailable returns true when no messages #21032
• [fix][client] Fix cannot retry chunk messages and send to DLQ #21048
• [fix][client] Fix consumer can't consume resent chunked messages #21070

Pulsar IO and Pulsar Functions

• [fix] [io] elastic-search sink connector not support JSON.String schema. #20741
• [improve][sql] Fix the wrong format of the logs #20907
• [fix][io] Fix --retain[-key]-ordering not working error for sink #21060
• [fix][io] Update test certs for Elasticsearch #21001
• [fix][io][branch-2.11] Not restart instance when kafka source pol...

v3.1.2

PIPs

• PIP-318: Support not retaining null-key message during topic compaction by @coderzc in #21578

Broker

• Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states by @heesung-sn in #21721
• Fix typo in the config key by @coderzc in #21690
• Record GeoPersistentReplicator.msgOut before producer#sendAsync by @nodece in #21673
• Print recoverBucketSnapshot log if cursorProperties are empty by @coderzc in #21651
• Support not retaining null-key message during topic compaction by @coderzc in #21601
• Delete topic timeout due to NPE by @poorbarcode in #21595
• Duplicate LedgerOffloader creation when namespace/topic policies are updated by @shibd in #21591
• Fix issue with consumer read uncommitted messages from compacted topic by @coderzc in #21571
• Fix create topic with different auto creation strategies causes race condition by @mattisonchao in #21545
• Fix orphan scheduled task for ledger create timeout check by @poorbarcode in #21542
• Fix thousands orphan PersistentTopic caused OOM by @poorbarcode in #21540
• Fix unfinished callback when deleting managed ledger by @mattisonchao in #21530
• Fix setReplicatedSubscriptionStatus incorrect behavior by @liudezhi2098 in #21510
• Fix failure while creating non-durable cursor with inactive managed-ledger by @rdhabalia in #21508
• Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 by @lhotari in #21496
• Do not write replicated snapshot marker when the topic which is not enable replication by @liangyepianzhou in #21495
• Avoid pass null role in MultiRolesTokenAuthorizationProvider by @mattisonchao in #21486
• Fix the deadlock when using BookieRackAffinityMapping with rackaware policy by @erobot in #21481
• Fix issue with consumer read uncommitted messages from compacted topic by @coderzc in #21465
• Fix resource_quota_zpath by @AnonHxy in #21461
• Fix namespace bundle stuck in unloading status by @mattisonchao in #21445
• Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered. by @coderzc in #21434
• Fix MultiRoles token provider NPE when using anonymous clients by @Technoboy- in #21429
• Fix heartbeat namespace create event topic and cannot delete heartbeat topic by @TakaHiR07 in #21360
• Fix heartbeat namespace create transaction internal topic by @TakaHiR07 in #21348
• Fix unload operation stuck when use ExtensibleLoadManager by @Demogorgon314 in #21332
• Fix lookup heartbeat and sla namespace bundle when using extensible load manager(#21213) by @Demogorgon314 in #21314
• Fix thread safe issue with RangeCache.put and RangeCache.clear by @lhotari in #21302
• use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error by @heesung-sn in #21282
• Fix inconsistent topic policy by @mattisonchao in #21255
• Let the producer request success at the first time if the previous one is inactive by @poorbarcode in #21220
• Fix lookup heartbeat and sla namespace bundle when using extensible load manager by @Demogorgon314 in #21213
• Avoid record inactiveproducers when deduplication is disable by @lifepuzzlefun in #21193
• Make specified producer could override the previous one by @poorbarcode in #21155
• Rackaware policy is ineffective when delete zk rack info after bkclient initialize by @TakaHiR07 in #20944
• Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin by @tuteng in #20939

Client

• Fix inconsistent API annotations of getTopicName by @liangyepianzhou in #21620
• Fix print error log 'Auto getting partitions failed' when expend partition by @hanmz in #21485
• Avert extensive time consumption during table view construction by @liangyepianzhou in #21270

Transaction

• OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly by @TakaHiR07 in #21505
• Ack all message ids when ack chunk messages with transaction by @liangyepianzhou in #21268

CI and Test

• Fix LocalBookkeeperEnsemble resource leak in tests by @lhotari in #21407
• Add new CI unit test group "Broker Group 4" with cluster migration tests by @lhotari in #21391
• Fix resource leaks with Pulsar Functions tests by @lhotari in #21378
• Fix some resource leaks in compaction tests by @lhotari in #21374
• Fix a resource leak in ClusterMigrationTest by @lhotari in #21366
• Fix docker image building by releasing more disk space before building by @lhotari in #21365
• Fix flaky CompactionTest.testDispatcherMaxReadSizeBytes by @lhotari in #21329
• Fix flaky test NarUnpackerTest by @lhotari in #21328
• Fix zookeeper related flacky test by @horizonzy in #21310
• fix flaky test PatternTopicsConsumerImplTest by @poorbarcode in #21222

Security

• Upgrade rabbitmq client to address CVE-2023-46120 by @liangyepianzhou in #21619
• Upgrade Zookeeper to 3.8.3 to address CVE-2023-44981 by @lhotari in #21398
• Upgrade Netty to 4.1.100 to address CVE-2023-44487 by @lhotari in #21397
• Upgrade Jetty to 9.4.53 to address CVE-2023-44487 by @lhotari in #21395
• Fix MultiRoles token provider when using anonymous clients by @merlimat in #21338
• Add OWASP Dependency Check suppressions by @lhotari in #21281
• Upgrade snappy-java to 1.1.10.5 by @lhotari in #21280

Build

• Fix Stage Docker images fail on M1 Mac by @AnonHxy in #21659
• Fix apt download issue in building the docker image by @lhotari in #21489
• Rename schema_example.conf to schema_example.json by @poorbarcode in #21447
• Upgrade Lombok to 1.18.30 to support compiling with JDK21 by @lhotari in #21278

Others

• [offload] Don't cleanup data when offload met MetaStore exception by @zymap in #21686
• [proxy] Move status endpoint out of auth coverage by @mattisonchao in #21428
• [auto-recovery] Migrate the replication testing from BookKeeper to Pulsar by @horizonzy in #21340
• [bk-client] Fix bk client MinNumRacksPerWriteQuorum and EnforceMinNumRacksPerWriteQuorum not work problem by @horizonzy in #21327
• [sql] Support UUID for json and avro by @liangyepianzhou in #21267
• [log] fix the vague response if topic not found by @poorbarcode in #20932

v3.0.2

Broker

• [fix][broker] Fix namespace bundle stuck in unloading status (21445) #21565
• [fix][broker] Fix create topic with different auto creation strategies causes race condition #21545
• [fix][broker] Fix thousands orphan PersistentTopic caused OOM #21540
• [fix][broker] Fix failure while creating non-durable cursor with inactive managed-ledger #21508
• [fix][broker] Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
• [fix][broker] Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
• [fix][broker] Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
• [fix][broker] Fix namespace bundle stuck in unloading status #21445
• [fix][broker] Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered. #21434
• [fix][broker] Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
• [fix][broker] Fix heartbeat namespace create transaction internal topic #21348
• [fix][broker] Fix unload operation stuck when use ExtensibleLoadManager #21332
• [fix][broker][branch-3.0] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (21213) #21313
• [improve][broker] use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
• [fix][broker][branch-3.0] Fix inconsistent topic policy #21256
• [fix][broker]Fixed produce and consume when anonymousUserRole enabled #21237
• [fix][broker] Fix inconsistent topic policy #21231
• [fix][broker] Fixed reset for AggregatedNamespaceStats #21225
• [fix][broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager #21213
• [improve][broker] Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
• [fix][broker] replicator leak when removeReplicator in NonPersistentTopic #21205
• [improve] [broker] Print warn log if ssl handshake error & print ledger id when switch ledger #21201
• [fix][broker] Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21179
• [improve] [broker] improve read entry error log for troubleshooting #21169
• [fix][broker][branch-3.0] Fix write duplicate entries into the compacted ledger after RawReader reconnects (21081) #21165
• [improve][broker][branch-3.0] Make read compacted entries support maxReadSizeBytes limitation (21065) #21164
• [fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor #21156
• [fix][broker] Make specified producer could override the previous one #21155
• [improve][broker] Upgrade bookkeeper to 4.16.3 #21146
• [improve][broker] Improve logs for troubleshooting #21141
• [fix][broker] Fix web tls url null cause NPE #21137
• [fix][broker] Fix unack count when mixing non batch index and batch index acks #21126
• [fix][broker] Fix unsubscribe non-durable subscription error #21099
• [fix][broker] Fix isolated group not work problem. #21096
• [fix][broker] Fix deleting topic not delete the related topic policy and schema. #21093
• [fix][broker] Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor #21091
• [fix][broker] revert remove duplicate topics name when deleteNamespace #21087
• [fix][broker] Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
• [fix][broker] Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
• [fix][broker] remove bundle-data in local metadata store. #21078
• [fix][broker] Make sure all inflight writes have finished before completion of compaction #21067
• [improve][broker] Make read compacted entries support maxReadSizeBytes limitation #21065
• [improve][broker] Improve cache handling for partitioned topic metadata when doing lookup #21063
• [fix][broker] Use MessageDigest.isEqual when comparing digests #21061
• [fix][broker] Fix potential case cause retention policy not working on topic level #21041
• [fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
• [fix][broker] Fix incorrect unack msk count when dup ack a message #20990
• [fix][broker] Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
• [fix][broker] Fix compaction subscription delete by inactive subscription check. #20983
• [fix][broker] Fix message loss during topic compaction #20980
• [fix][broker] Fix incorrect number of read compacted entries #20978
• [fix][broker] Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
• [fix][broker]Fix chunked messages will be filtered by duplicating #20948
• [fix][broker] rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
• [fix][broker]Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
• [fix][broker] fix MessageDeduplication throw NPE when enable broker dedup and set namespace disable deduplication. #20905
• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent #20888
• [fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached #20855
• [improve][broker] Avoid print redirect exception log when get list from bundle #20846
• [fix][broker] Inconsistent behaviour for topic auto_creation #20843
• [fix][broker] Can not receive any messages after switch to standby cluster #20767
• [fix][broker] Fix get topic policies as null during clean cache #20763
• [fix][broker] Avoid throwing RestException in BrokerService #20761
• [fix][broker] Gracefully shutdown does not work with admin cli in standalone #20709
• [fix][broker] Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
• [improve][broker] Add consumer-id into the log when doing subscribe. #20568
• [fix][broker] consider iowait as idle. #19110
• [fix][broker] fix bug caused by optimistic locking #18390
• [fix][ml] Fix orphan scheduled task for ledger create timeout check #21542
• [fix][ml] Fix unfinished callback when deleting managed ledger #21530
• [fix][ml] fix discontinuous ledger deletion #20898
• [fix][ml] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
• [fix][ml] fix wrong msg backlog of non-durable cursor after trim ledgers #21250
• [fix][ml] Reader can set read-pos to a deleted ledger #21248
• [improve][ml] Persist mark deleted ops to Z...

v3.1.1

Broker

• Fix incorrect number of read compacted entries #20978
• Fix message loss during topic compaction #20980
• Fix get outdated compactedTopicContext after compactionHorizon has been updated #20984
• Fix compaction subscription delete by inactive subscription check #20983
• Fix incorrect unack msk count when dup ack a message #20990
• Fix ModularLoadManagerImpl always delete active bundle-data #20620
• Use MessageDigest.isEqual when comparing digests #21061
• Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
• Make sure all inflight writes have finished before completion of compaction #21067
• Fix chunked messages will be filtered by duplicating #20948
• Fix potential case cause retention policy not working on topic level #21041
• Make read compacted entries support maxReadSizeBytes limitation #21065
• Remove bundle-data in local metadata store #21078
• Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
• Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
• Avoid splitting one batch message into two entries in StrategicTwoPhaseCompactor #21091
• Fix unsubscribe non-durable subscription error #21099
• Fix isolated group not work problem #21096
• Consider iowait as idle #19110
• Fix deleting topic not delete the related topic policy and schema #21093
• Revert remove duplicate topics name when deleteNamespace #21087
• Improve logs for troubleshooting #21141
• Fix web tls url null cause NPE #21137
• Fix unack count when mixing non batch index and batch index acks #21126
• Backport fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21180
• Fix bug caused by optimistic locking #18390
• Disable balancing based on DirectMemory #21168
• Fix replicator leak when removeReplicator in NonPersistentTopic #21205
• Fixed reset for AggregatedNamespaceStats #21225
• Fixed produce and consume when anonymousUserRole enabled #21237

Pulsar IO and Functions

• [FN] Fix ProducerConfig cannot update error #21037
• [FN] Fix the --batch-builder not working error for functions #21023
• [FN] Fix functions_log4j2.xml delete strategy config #21215
• [IO] Fix --retain[-key]-ordering not working error for sink #21060

Client

• Fix perf-producer get OOM with high publish latency #20914
• Fix RawReader hasMessageAvailable returns true when no messages #21032
• Fix consumer can't consume resent chunked messages #21070
• Fix cannot retry chunk messages and send to DLQ #21048
• Fix logging problem in pulsar client #21094
• Avoid ack hole for chunk message #21101
• Fix repeat consume when using n-ack and batched messages #21116
• Fix same producer/consumer use more than one connection per broker #21144
• Merge lookup requests for the same topic #21232

CLI

• [Admin] Fix get topic stats fail if a subscription catch up concurrently #20971

Others

• [META] Fix deadlock in AutoRecovery #21010
• [META] Improve fault tolerance of blocking calls by supporting timeout #21028
• [META] Improve to the ReplicaitonWorker performance by deleting invalid underreplication nodes #21059
• [META] Fix metadata store deadlock due to BookkeeperInternalCallbacks.Processor #21159
• [META] Fix PulsarLedgerUnderreplicationManager notify problem #21161
• [META] Fix pulsar ledger auditor dead lock problem #21181
• [ML] Persist mark deleted ops to ZK if create cursor ledger was failed #20935
• [BK] Improve getIsolationGroup by avoid creating arrayList #20952
• [BK] Fix RocksDB configuration #21157
• [Proxy] Support disabling metrics endpoint #21031
• [Proxy] Fix Proxy 502 gateway error when it is configured with Keystore TLS and admin API is called #21077
• [TXN] Fix the consumer stuck due to deduplicated messages in pending ack state #21177

Library updates

• Bump GRPC version to 1.55.3 to fix CVE #21057
• Bump broker okio version to 3.4.0 #21064
• Upgrade bookkeeper to 4.16.3 #21146

v3.1.0

Important notice

• Fix the reason label of authentication metrics #20030

PIPs

• PIP-275: Introduce topicOrderedExecutorThreadNum to deprecate numWorkerThreadsForNonPersistentTopic #20504
• PIP-255: Make the partition assignment strategy pluggable #20537
• PIP-278: Support pluggable topic compaction service #20624

Broker

• Fix MessageDeduplication throw NPE when enabling broker deduplication and disabling namespace deduplication #20905
• In replication scenarios, remote consumers could not be registered if there were no message sent #20888
• Fix IndexOutOfBoundsException in the CompactedTopicUtils #20887
• Add annotation for topic compaction strategy #20858
• Fix inconsensus namespace policies by getPoliciesIfCached #20855
• Pass bytesToRead when reading compacted entries #20850
• Fix the built-in admin failed to delete a topic with a custom authz provider #20848
• Avoid print redirect exception log when getting a list from bundle #20846
• Inconsistent behaviour for topic auto_creation #20843
• Follow up 4196 use PulsarByteBufAllocator handle OOM #20837
• Add broker filter sync method back to guarantee the API compatibility #20826
• Fix direct memory leak by delayed index OutOfDirectMemory #20823
• Avoid infinite bundle unloading #20822
• Add the MessageExpirer interface to make code clear #20800
• Broker failed to load v1 namespace resources cache #20783
• Cannot receive any messages after switching to a standby cluster #20767
• Fix get topic policies as null during clean cache #20763
• Fix typo in ServerCnx.java #20762
• Avoid throwing RestException in BrokerService #20761
• Call ServerCnxcloseProducer from correct thread #20747
• Ignore openIDTokenIssuerTrustCertsFilePath conf when blank #20745
• Support pluggable topic compaction service - part2 #20718
• Revert "Skip loading broker interceptor when disableBrokerInterceptors is true 20422" #20710
• Gracefully shutdown does not work with admin cli in standalone #20709
• Improve CompactedTopicImpl lock #20697
• Added the skipped message handler for ServiceUnitStateChannel #20677
• Make ExtensibleLoadManagerImpl's broker filter pure async #20666
• Add logs for topics or subscriptions when enabling rate limit #20660
• Make get list from bundle Admin API async #20652
• Fix the publish latency spike from the contention of MessageDeduplication #20647
• Support pluggable topic compaction service - part1 #20645
• Fix get owned service units NPE #20625
• Topic policy cannot work well if the replay policy message has any exceptions #20613
• Fix the publish latency spike issue with a large number of producers #20607
• AuthenticationFilter.doFilter #20598
• Fix NPE when resetting Replicator's cursor by position. #20597
• Add consumer-id into the log when doing subscribe #20568
• Release orphan replicator after topic closed #20567
• New load balancer system topic should not be auto-created now #20566
• Remove unused cache executor in PulsarService #20563
• Handle get owned namespaces admin API in ExtensibleLoadManager #20552
• Handle heartbeat namespace in ExtensibleLoadManager #20551
• Validate authz earlier in delete subscription logic #20549
• Should not throw NotFoundException when metadata already exists #20539
• REST Client Producer fails with TLS only #20535
• Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API #20528
• Restore solution for certain topic unloading race conditions #20527
• Emit the namespace bundle listener event on extensible load manager #20525
• Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer #20522
• Disable EntryFilters for system topics #20514
• Support get/remove permissions for AuthorizationProvider #20496
• Improve the efficiency of checking message deletion #20490
• Reformat property in generateResponseWithEntry #20481
• Add metric prefix for topic_load_times #20472
• Fix return the earliest position when query position by timestamp. #20457
• Support revoking permission for AuthorizationProvider #20456
• Replace String.intern() with Guava Interner #20432
• Remove ineffective solution for reducing GC pressure #20428
• Skip loading broker interceptor when disableBrokerInterceptors is true #20422
• Avoid PersistentSubscription.expireMessages logic check backlog twice #20416
• Change limitStatsLogging config default value to true #20409
• PIP-255 Part-1: Add listener interface for namespace service #20406
• Use immutable base cursor properties #20400
• Fix partitioned __change_events topic is policy topic #20392
• Do not expose bucketDelayedIndexStats #20383
• Fix flaky test - testClusterMigrationWithReplica… #20379
• Change some static fields referencing mutable objects to ordinary instance fields in BrokerService #20372
• Fix pre-create non-partitioned system topics for load balance extension #20370
• Fix memory leak when rebatchMessage #20369
• Invalidate metadata children cache after key deleted #20363
• Fix managedLedger.getConfig().getProperties().putAll(properties) NPE #20361
• Update rest endpoint method names to avoid swagger conflict #20359
• Fix broker load manager class filter NPE #20350
• Copy BrokerEntryMetadata when rebatchMessage #20337
• In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck #20335
• When creating a namespace anti-affinity group with the default local policies, use the bundle size supplied by the user. #20327
• Fix skip message API when hole messages exist #20326
• Gracefully shut down load balancer extension #20315
• Fix class name typo PrecisPublishLimiter to "Precise" #20310
• Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
• Fix NPE cause by topic publish rate limiter. #20302
• Remove Streaming Dispatcher Code #20279
• Fix UnsupportedOperationException when updating topic...

v3.0.1

Important notice

• Fix the reason label of authentication metrics in #20030

Broker

• [Broker] Add annotation for topic compaction strategy by @Demogorgon314 in #20858
• [Broker] Add broker filter sync method back to guarantee the API compatibility by @Demogorgon314 in #20826
• [Broker] Avoid infinite bundle unloading by @Demogorgon314 in #20822
• [Broker] Call ServerCnx#closeProducer from correct thread by @michaeljmarshall in #20747
• [Broker] Added the skipped message handler for ServiceUnitStateChannel by @heesung-sn in #20677
• [Broker] Make ExtensibleLoadManagerImpl's broker filter pure async by @Demogorgon314 in #20666
• [Broker] Fix the publish latency spike from the contention of MessageDeduplication by @codelipenghui in #20647
• [Broker] Fix get owned service units NPE by @Demogorgon314 in #20625
• [Broker] Topic policy can not work well if replay policy message has any exception. by @zhanghaou in #20613
• [Broker] Fix the publish latency spike issue with a large number of producers by @codelipenghui in #20607
• [Broker] New load balancer system topic should not be auto-created now by @heesung-sn in #20566
• [Broker] Handle get owned namespaces admin API in ExtensibleLoadManager by @Demogorgon314 in #20552
• [Broker] Handle heartbeat namespace in ExtensibleLoadManager by @Demogorgon314 in #20551
• [Broker] Validate authz earlier in delete subscription logic by @michaeljmarshall in #20549
• [Broker] REST Client Producer fails with TLS only by @maanders-tibco in #20535
• [Broker] Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API by @Demogorgon314 in #20528
• [Broker] Restore solution for certain topic unloading race conditions by @lhotari in #20527
• [Broker] Emit the namespace bundle listener event on extensible load manager by @Demogorgon314 in #20525
• [Broker] Disable EntryFilters for system topics by @lhotari in #20514
• [Broker] Replace test call to Auth0 with call to WireMock by @michaeljmarshall in #20465
• [Broker] Fix return the earliest position when query position by timestamp. by @hanmz in #20457
• [Broker] Avoid PersistentSubscription.expireMessages logic check backlog twice. by @lifepuzzlefun in #20416
• [Broker] PIP-255 Part-1: Add listener interface for namespace service by @hleecs in #20406
• [Broker] Partitioned __change_events topic is policy topic by @michaeljmarshall in #20392
• [Broker] Do not expose bucketDelayedIndexStats by @coderzc in #20383
• [Broker] Pre-create non-partitioned system topics for load balance extension by @heesung-sn in #20370
• [Broker] ManagedLedger.getConfig().getProperties().putAll(properties) NPE by @heesung-sn in #20361
• [Broker] Fix broker load manager class filter NPE by @Demogorgon314 in #20350
• [Broker] In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck by @poorbarcode in #20335
• [Broker] When creating a namespace anti-affinity group with the default local policies, use the bundle size supplied by the user. by @syk-coder in #20327
• [Broker] Fix skip message API when hole messages exist by @crossoverJie in #20326
• [Broker] Gracefully shut download balancer extension by @heesung-sn in #20315
• [Broker] Fix class name typo PrecisPublishLimiter to "Precise" by @JooHyukKim in #20310
• [Broker] Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion by @liangyepianzhou in #20304
• [Broker] Fix NPE caused by topic publish rate limiter. by @dragonls in #20302
• [Broker] Fix UnsupportedOperationException when updating topic properties. by @liangyepianzhou in #20261
• [Broker] Fix ledger cachemiss size metric by @AnonHxy in #20257
• [Broker] Fix default bundle size used while setting bookie affinity by @syk-coder in #20250
• [Broker] Fix the behavior of delayed message in Key_Shared mode by @codelipenghui in #20233
• [Broker] Fix Return value of getPartitionedStats doesn't contain subscription type by @1Jack2 in #20210
• [Broker] Skip split bundle if only one broker by @poorbarcode in #20190
• [Broker] Fix RoaringBitmap.contains can't check value 65535 by @coderzc in #20176
• [Broker] Release EntryBuffer after parsing proto object by @coderzc in #20170
• [Broker] Remove useless field in the DelayedMessageIndexBucketSegment.proto by @coderzc in #20166
• [Broker] Optimization protobuf code in the bucket delayed tracker by @coderzc in #20158
• [Broker] Move bitmap from lastMutableBucket to ImmutableBucket by @coderzc in #20156
• [Broker] Fix tenant admin authorization bug. (#20068)" by @michaeljmarshall in #20143
• [Broker] Producer/Consumer should call allowTopicOperationAsync by @michaeljmarshall in #20142
• [Broker] Fix entry filter feature for the non-persistent topic by @gaoran10 in #20141
• [Broker] Optimize delayed metadata index bitmap by @mattisonchao in #20136
• [Broker] Fast fix infinite HTTP call getSubscriptions caused by wrong topicName by @poorbarcode in #20131
• [Broker] Make timer execute immediately after load index by @coderzc in #20126
• [Broker] Cache LedgerHandle in BookkeeperBucketSnapshotStorage by @coderzc in #20117
• [Broker] Close temporary open ledger in BookkeeperBucketSnapshotStorage by @lifepuzzlefun in #20111
• [Broker] Fix use error TimeUnit to record publish latency by @TakaHiR07 in #20074
• [Broker] Fix tenant admin authorization bug. by @dragonls in #20068
• [Broker] Fix getPartitionedStats miss subscription's messageAckRate by @wangjialing218 in #19870
• [Broker] Do not filter system topic while shedding. by @thetumbled in #18949
• [Broker] There are two same-named managed ledgers in the one broker by @poorbarcode in #18688
• [Broker] If ledger lost, cursor mark delete position can not forward by @poorbarcode in #18620
• [Broker] Support cgroup v2 by using jdk.internal.platform.Metrics in Pulsar Loadbalancer by @coderzc in #16832

Clients

• [Client] Fix the deadlock issue of consumers while using multiple IO threads by @codelipenghui in #20669
• [Client] Make the whole grabCnx() progress atomic by @BewareMyPower in #20595
• [Client] Thread-safe seek by @tisonkun in #20321
• [Client] Fix where the function getMsgNumInReceiverQueue always returns 0 when using message listener by @crossoverJie in #20245
• [Client] Java Client's Seek Logic Not Threadsafe #1 by @JooHyukKim in #20242

Pulsar IO and Pulsar Functions

• [Functions] Exit JVM when the main thread throws an exception by @michaeljmarshall in #20689
• [Functions] TLS args admin download command use zero arity by @michaeljmarshall in #20513
• [Functions] Support customizing TLS config for function download command by @michaeljmarshall in [#20482](https://github.com/apache/pulsa...

v2.10.5

Broker

• [fix][broker] In replication scenario, remote consumer could not be registered if there has no message was sent (#20888)
• [improve] [broker] Print warn log if compaction failure (#19405)
• [branch-2.10][fix][broker] Fix inconsensus namespace policies by getPoliciesIfCached (#20873)
• [branch-2.10][fix][broker] Inconsistent behaviour for topic auto_creation (#20872)
• [fix] [broker] Can not receive any messages after switching to standby cluster (#20767)
• [improve] [broker] Add consumer-id into the log when doing subscribe. (#20568)
• [fix][broker][branch-2.10] Fix NPE when resetting Replicator's cursor by position. (#20597) (#20781)
• [fix][broker] Fix namespace deletion if __change_events topic has not been created yet (#18804)
• [fix][schema] Only handle exception when there has (#20730)
• [fix][broker] Topic policy can not work well if replay policy message has any exception. (#20613)
• [fix][broker] Fix return the earliest position when query position by timestamp. (#20457)
• [fix][broker] Return if AbstractDispatcherSingleActiveConsumer closed (#19934)
• [broker] clean inactive bundle from bundleData in loadData and bundlesCache (#13974)
• [fix][branch-2.10] Fix compilation issue introduced by fixing Repeated messages of shared dispatcher (#16812)
• Issue 16802: fix Repeated messages of shared dispatcher (#16812)
• [fix][branch-2.10]Fix compilation issue introduced by Save createIfMissing in TopicLoadingContext (#19993)
• [improve][broker] Save createIfMissing in TopicLoadingContext (#19993)
• [fix][broker] Invalidate metadata children cache after key deleted (#20363)
• [improve] [broker] Avoid PersistentSubscription.expireMessages logic check backlog twice. (#20416)
• [fix][broker]fix the publish latency spike issue with large number of producers (#20607)
• [fix][branch-2.10] Fix duplicated deleting topics (#20685)
• [improve][broker][branch-2.10] Backport Linux metrics changes from master branch (#20659)
• [cleanup][broker] Validate authz earlier in delete subscription logic (#20549)
• [fix][broker] release orphan replicator after topic closed (#20567)
• [fix][broker] REST Client Producer fails with TLS only (#20535)
• [fix][broker] Restore solution for certain topic unloading race conditions (#20527)
• [fix][ml] There are two same-named managed ledgers in the one broker (#18688)
• [fix][broker] Fix skip message API when hole messages exists (#20326)
• [fix][broker] If ledger lost, cursor mark delete position can not forward (#18620)
• [fix][broker] partitioned __change_events topic is policy topic (#20392)
• [fix][ml] Fix ledger left in OPEN state when enabling inactiveLedgerRollOverTimeMs (#20276)
• [fix][broker] Fix default bundle size used while setting bookie affinity (#20250)
• [fix][broker] Fix the behavior of delayed message in Key_Shared mode (#20233)
• [improve][broker] Get lowest PositionImpl from NavigableSet (#18278)
• [fix] [broker] error TimeUnit to record publish latency (#20074)
• [fix] [broker] In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck (#20335)
• [fix][broker]Fix deadlock of metadata store (#20189)
• [fix] [broker] [branch-2.10] Upgrade rocksDB version to 6.16.4 to keep sync with BookKeeper 4.14.7 (#20312)
• [improve] [broker] Skip split bundle if only one broker (#20190)
• [fix][monitor] topic with double quote breaks the prometheus format (#20230)
• [fix][broker] Fix RoaringBitmap.contains can't check value 65535 (#20176)
• [fix][broker] Fix the reason label of authentication metrics (#20030)
• [fix] [broker] Fix infinite ack of Replicator after topic is closed (#20232)
• [fix] [broker] Producer created by replicator is not displayed in topic stats (#20229)
• [fix] [broker] delete topic failed if disabled system topic (#19735)
• [fix] [ml] make the result of delete cursor is success if cursor is deleted (#19825)
• [fix] [broker] Fast fix infinite HTTP call getSubscriptions caused by wrong topicName (#20131)
• [fix][broker] Fix issue where msgRateExpired may not refresh forever (#19759)
• [fix][broker] Fix can't send ErrorCommand when message is null value (#19899)
• [fix][broker] Fix Return value of getPartitionedStats doesn't contain subscription type (#20210)
• [branch-2.10][fix][build] Upgrade swagger version to fix CVE-2022-1471 (#20172)
• [fix][broker] Fix getPartitionedStats miss subscription's messageAckRate (#19870)
• [fix] Use scheduled executor in BinaryProtoLookupService (#20043)
• [improve] [broker] Fix broker restart logic (#20113)

Transaction

• [fix] [txn] fix consumer can receive aborted txn message when readType is replay (#19815)

Pulsar IO and Pulsar Functions

• [fix][io][branch-2.10] Not restart instance when kafka source poll exception. (#20816)
• [fix][offload] Filesystem offloader class not found hadoop-hdfs-client (#20365)
• [fix][fn] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors (#19584)
• [fix][fn] Fix JavaInstanceStarter inferring type class name error (#19896)
• [fix][io] Close the kafka source connector got stuck (#20698)
• [fix][fn] Exit JVM when main thread throws exception (#20689)
• Optimize conusmer pause (#14566)
• [fix][broker] Fix the publish latency spike from the contention of MessageDeduplication (#20647)
• [fix][fn] Configure pulsar admin for TLS (#20533)
• [fix][fn] Go functions must retrieve consumers by non-particioned topic ID (#20413)
• [fix][io] Close the kafka source connector if there is uncaught exception (#20479)
• [fix][fn] Go functions need to use static grpcPort in k8s runtime (#20404)
• [fix][fn] Make pulsar-admin support update py/go with package url (#19897)
• [fix][txn] Fix transaction is not aborted when send or ACK failed (#20240)

CLI

• [fix] [cli] the variable producerName of BatchMsgContainer is null (#20819)
• [fix][ci] Update nar maven plugin version to fix excessive downloads (#20410)

Admin

• [improve][admin] Return BAD_REQUEST on cluster data is null for createCluster (#20346)

Security

• [fix][sec] Upgrade snappy-java to address multiple CVEs (#20604)
• [fix][sec] Upgrade Guava to 32.0.0 to address CVE-2023-2976 (#20459)
• [improve][misc] Upgrade Netty to 4.1.93.Final (#20423)
• [improve][misc] Upgrade Netty to 4.1.89.Final (#19649)
• [fix][sec] Upgrade sqlite-jdbc to resolve CVE-2023-32697 (#20411)

CI & Test

• [fix][test] Replace test call to Auth0 with call to WireMock (#20465)
• [fix][test] Fix flaky testCreateTopicWithZombieReplicatorCursor (#20037)
  *[fix][flaky-test]NamespaceServiceTest.flaky/testModularLoadManagerRemoveBundleAndLoad (#17487)
• [improve][test][branch-2.10] Backport disabling disk usage threshold for Elastic Testcontainers (#20676)

Others

• [fix][build]Fix compatibility issue caused by #20819 (#20834)
• [fix][build] Mongo is fixed for 2.10.5 (#20810)
• [fix][meta] Bookie Info lost by notification race condition. (#20642)
• [fix] [meta]Switch to the metadata store thread after zk operation (#20303)
• [fix][ws] Remove unnecessary ping/pong implementation (#20733)
• fix: bundle-data metadata leak because of bundlestats was not clean (#17095)
• [fix] [Perf] PerformanceProducer do not produce expected number of messages. (#19775)
• [fix][meta] Adding the missed bookie id in the registration manager. (#20641)
• [fix][build] Don't publish docker image with "latest" tag to docker repository
• [fix][build][branch-2.10] Fix ci-license check (#20505)
• [improve][monitor] Add JVM start time metric (#20381)
• [fix][build] update the zookeeper version to 3.6.4 (#20333)
• [fix][monitor] Fix the partitioned publisher topic stat aggregation bug (#18807)
• [fix][meta] deadlock of zkSessionWatcher when zkConnection loss (#20122)
• [improve][build] Upgrade dependencies to reduce CVE. (#20162)
• [branch-2.10][improve][build] Upgrade snakeyaml version to 2.0 (#20118)

Client

• [fix] [client] Messages lost when consumer reconnect (#20695)
• [fix][client] Make the whole grabCnx() progress atomic (#20595)
• [fix][client] Fix race condition that leads to caching failed CompletableFutures in ConnectionPool (#19661)
• [fix][client] Fix deadlock issue of consumer while using multiple IO threads (#20669)
• [fix][client] Cache empty schema version in ProducerImpl schemaCache. (#19929)
• [fix][fn] Reset idle timer correctly (#20450)
• [fix][client] Release the orphan producers after the primary consumer is closed (#19858)
• [fix][client] Fix DeadLetterProducer creation callback blocking client io thread. (#19930)