[discussion] Pulsar release 2.6.0

[sijie]

This issue is used for collecting the features planned for release 2.6.0.

Features

• [PIP-37] Large message size support PIP 37: [pulsar-client] support large message size #4400 approved doc-added
• [PIP-39] Namespace change events(System Topic) [PIP-39] Introduce system topic and topic policies service #4955 merged no-need-doc
• [PIP-45] Switch ManagedLedger to use MetadataStore interface PIP-45: Switch ManagedLedger to use MetadataStore interface #5358 merged
• [PIP 53] Contribute DotPulsar to Apache Pulsar
• [PIP-54] Support acknowledgment at batch index level [PIP-54] Support acknowledgment at batch index level #6052 merged doc-added
• [PIP 57] Improve Zookeeper Session Timeout Handling Refactor zookeeper session timeout handling into an interface #6347 review-required
• [PIP-60] Support SNI routing to support various proxy-server [PIP-60] [Proxy-Server] Support SNI routing to support various proxy-server in pulsar #6566 approved
• [PIP-61] Advertise multiple addresses PIP-61: Advertise multiple addresses #6903 review-required doc-added
• [PIP-65] Adapting Pulsar IO Sources to support Batch Sources Extend PulsarIO to support Batch Sources. This pr implements PIP-65 #7090 review-required
• [Load balancer] Add ThresholdShedder Strategy for loadbalancer Add ThresholdShedder Strategy for loadbalancer and expose loadbalance metric to prometheus #6772 merged
• [Load balancer] Expose loadbalance metric to Prometheus Add ThresholdShedder Strategy for loadbalancer and expose loadbalance metric to prometheus #6772 merged
• • [Key Shared] Ordering dispatch when adding consumer Fixed ordering issue in KeyShared dispatcher when adding consumer #7106 Message replays on Key shared subscription are breaking ordering #7108 Allow to configure KeyShared consumer with relaxed ordering #7188
• [Key Shared] Add support for key hash range reading [java-client] Add support for key hash range reading. #5928 merged
• [Schema] Allow for schema reader and writer registration on SchemaDefinition [Issue #6842] feat(schema): allow for schema reader and writer registration on SchemaDefinition #6905 merged
• Use pure-java Air-Compressor instead of JNI based libraries Use pure-java Air-Compressor instead of JNI based libraries #5390 approved
• Support multiple pulsar clusters to use the same bk cluster [Issue 5935] Support multi pulsar clusters to use the same bk cluster #5985 merged
• Support delete inactive topic when subscriptions caught up Support delete inactive topic when subscriptions caught up #6077 merged
• Support consumers set custom message retry delay Support Consumers Set Custom Retry Delay #6449 merged doc-added
• Add configuration to disable auto-creation of subscriptions [Issue 6394] Add configuration to disable auto creation of subscriptions #6456 merged
• Add a flag to skip broker shutdown on transient OOM [pulsar-broker] add flag to skip broker shutdown on transient OOM #6634 merged
• Make zookeeper cache expiry time configurable [pulsar-broker] support configurable zk-cache expiry time #6668 merged doc-added
• Check replicator periodically to avoid issue due to zookeeper missing watch [pulsar-broker] check replicator periodically to avoid issue due to zk missing watch #6674 merged
• Expose managedLedgerCache, managedLedger, loadBalance metrics to Prometheus expose managedLedgerCache, managedLedger, loadBalance metrics to prometheus #6705 merged
• Optimize consumer fetch messages in case of batch message fix consumer fetch message number maps to read entry number bug and expose avgMessagesPerEntry metric #6719 approved
• Add config to limit max partitions for a partiitoned topic [Issue 6793][broker] Add config of maxNumPartitionsPerPartitionedTopic #6794 merged
• Add options for the GetBookieInfo check [broker] Expose BookKeeper GetBookieInfo settings #6800 merged
• Change default FlushEntryLogBytes to 256MB to improve bookie io throughput Suggest adjust bookkeekper.conf default FlushEntryLogBytes to 256MB to improve bookie io throughput #6915 merged
• Make Nar Extraction Directory configurable Make Nar Extraction Directory configurable #6933 merged
• Custom functions runtime
• Introduce precise topic publish rate limiting introduce precise topic publish rate limiting #7078 merged doc-required
• • Expose new entries check delay in the broker.conf Expose new entries check delay in the broker.conf #7154
• • Add broker interceptor for Intercepting all Pulsar command and REST API requests Add broker interceptor for Intercepting all Pulsar command and REST API requests #7143
• • Only close active consumer for Failover subscription when seek() Only close active consumer for Failover subscription when seek(). #7141
• • [Schema] Support use null key and null value in KeyValue Schema Support KeyValue Schema Use Null Key And Null Value #7139
• • Allow to delete topics that are failing to recover Allow to delete topics that are failing to recover #7131
• • Support set netty max frame size in bookkeeper.conf fix support set nettyMaxFrameSizeBytes on bookeeper config #7116
• • Trigger rollover when meeting maxLedgerRolloverTimeMinutes Trigger rollover when meeting maxLedgerRolloverTimeMinutes #7111
• • Cap the dispatcher batch size in bytes to fixed max Cap the dispatcher batch size in bytes to fixed max #7097
• • Support specify managedLedgerMaxSizePerLedgerMbytes in broker.conf add feature: enable maxSizePerLedger settings in configuration #7085
• • Allow to grant permissions when the authorization is disabled Allow to grant permissions when the authorization is disabled #7074
• • Add messages and bytes counter stats to broker-stats/topics Add msg and bytes counter stats to broker-stats/topics #7045

Pulsar Proxy

• Add REST API to get connection and topic stats [pulsar-proxy] add rest-api to get connection and topic stats #6473 merged doc-required-it is generated automatically
• Add advertised address option [pulsar-proxy] add advertised address option for proxy #6942 merged doc-added
• Add proxyLogLevel into config [pulsar-proxy] add proxyLogLevel into config file #6948 merged

Admin

• Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies Add maxUnackedMessagesPerSubscription and maxUnackedMessagesPerConsumer on namespaces policies #5936 merged
• Support get a message by message ID in pulsar-admin [Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331 merged
• Support delete subscription forcefully [Issue 6043] Support force deleting subscription #6383 merged
• Add subscribe initial position for consumer CLI Add subscribe initial position for consumer cli #6442 merged
• Support to get managed ledger info of a partitioned topic [Issue 5902] Support to get managed ledger info of a partitioned topic #6532 merged
• Support compact all partitions of a partitioned topic [Issue 5903] Support compact all partitions of a partitioned topic #6537 merged
• Support multi-hosts in PulsarAdmin Handle multi-hosts in PulsarAdmin for both sync and async requests. #6547 merged
• Support to get internal stats for a partitioned topic [pulsar-client-tools] Add topics CLI support to get partitioned internal stats #6624 merged
• Support enable or disable subscription auto-creation at namespace level [pulsar-broker] Implement AutoSubscriptionCreation by namespace override #6637 merged
• Enable to set the subscription expiration time for each namespace [broker] Enable users to set subscription expiration time for each namespace #6851 merged doc-added

Functions

• Built-in Functions [#6003][pulsar-functions] Possibility to add builtin Functions #6895 approved
• [Function-Go] Add Go Function heartbeat (and gRPC service) for production usage [Issue 4175] [pulsar-function-go] Add Go Function heartbeat (and gRPC service) for production usage #6031 merged
• Add custom property option to functions [Issue 6170][functions] add custom property option to functions #6348 merged
• Separate TLS configuration of function worker and broker separate function worker and broker client TLS configuration #6602 merged
• Added ability to build consumers in functions and sources Added ability to build consumers in functions and sources #6954 merged
• • Support DLQ on sources and sinks Pulsar supporting DLQ for sources/sinks #7032

Pulsar SQL

• KeyValue schema support KeyValue schema support for pulsar sql #6325 merged
• Multiple Avro schema version Support multiple Avro schema version in Pulsar SQL #4847 merged
• • Fix presto SQL does not start metrics service before queue execute [Issue: 7029] Fix presto SQL does not start prometheus service by default #7030

Pulsar IO

• [RabbitMQ] Allow routing key per message to RabbitMQ sink connector [Pulsar IO] Allow routing key per message to RabbitMQ sink connector #5890 merged
• [RabbitMQ] Add passive config options [][io-rabbitmq] Add passive to io-rabbitmq config options #6679 merged
• [debezium] Upgrad from v0.10.0-Final to v1.0.0-Final [Issue 5941][Pulsar IO]Upgrade debezium from v0.10.0-Final to v1.0.0-Final #5972 merged
• [debezium] Support avro schema for debezium connector [Pulsar IO][Issue 5633]Support avro schema for debezium connector #6034 merged
• [influxdb2] Add support for influxdb2 in pulsar-influxdb-sink ISSUE-6514 Add support for influxdb2 in pulsar-influxdb-sink(#6514) #6601 merged
• [jdbc] Add jdbc sinks: postgres, mariadb, clickhouse [Issue 5215,4601,5215][pulsar-io] Add jdbc sinks: postgres, mariadb, clickhouse #6835 merged
• Added ability for sources to publish messages on their own Added ability for sources to publish messages on their own #6941 merged

Java Client

• Change the time unit of patternAutoDiscoveryPeriod to seconds [Issue 5927][Pulsar Client] Change the time unit of `patternAutoDiscoveryPeriod to seconds #5950 merged
• Support waiting for inflight messages while closing producer [pulsar-client] add api to wait for inflight messages while closing producer #6648 merged
• Add support to load TLS certs/key dynamically from input stream [pulsar-client] Add support to load tls certs/key dynamically from inputstream #6760 merged
• Support return sequence ID when throwing Exception for async send message support async send msg return sequenceId when throw Exception #6825 merged
• • Change the default value of maxLookupRedirects of Java client to 20 [client] Change the default value of maxLookupRedirects of Java client #7126
• • Limit the number of times lookup requests are redirected [client] Limit the number of times lookup requests are redirected #7096
• • Fix duplicated messages sent to dead letter topic fix Duplicated messages are sent to dead letter topic #6960 #7021

Python Client

• Add deliver_at and deliver_after for the producer [pulsar-client] Add deliver_at and deliver_after to python client #6737 merged

CPP Client

• • Support seek by time on partitioned topic [pulsar-client-cpp] Support Seek on Partitioned Topic by Time #7198
• • Refresh authentication credentials [C++] PIP-55: Refresh Authentication Credentials #7070
• • Fix Result can't be serialized to string inside the library [C++] Fix Result can't be serialized to string inside the library #7034
• • Supo zstd and Snappy compression to C API [C++] Expose zstd and Snappy compression to C API #7014

Bug Fixes

• [Websocket] Fix incorrect topic URL parse Issue #6619: WebSocket Topic URL Parser is Incorrect #6630 merged
• [Broker] Make ZkBookieRackAffinityMapping work as expected Make ZkBookieRackAffinityMapping work as expected #6917 merged
• [Broker] Fix backlog and backlog size stats keeps growing Reset msg backlog and backlog size #7082
• [Java Client] Fix connection leak [Issue 6319][Pulsar client] connection leak fix #6524 merged
• [Java Client] Fix message id compare between MessageId and BatchMessageId [Issue 6433][Pulsar-client]Fix message id compare between MessageId and BatchMessageId #6621 merged
• [Pulsar SQL] Fix problem with multiple zookeeper address [Pulsar SQL] Fix pulsar sql use multi zk address failed #6947 merged
• • [CPP Client] Fix deadlock of consumer for topics auto discovery [Issue 7168][C++] Fix pattern consumer's deadlock for topics auto discovery #7206
• • [Managed Ledger] Fix NPE on opening non-durable cursors on an empty managed ledger Fix NPE on opening non-durable cursors on an empty managed ledger #7133
• • [Java Client] Fix memory leak when create producer with not exsits topic Close producer without state check while the topic does not exists. #7120 Ensure the timer tasks in the producer and consumer are cleanup after closed #7124
• • [Broker] Fixed increasing number of partitions with attached readers Fixed increasing number of partitions with attached readers #7077

Others

• Work on JDK 11

[sijie]

Please link or comment the issues here.

[frank-dkvan]

hopefully this bug #4976 can be fixed in release 2.6.0 :-D

[murong00]

Both pulsar and bk should support ipv6, such as #5713

[Jennifer88huang-zz]

Some users are expecting Pulsar transaction and KOP in the new release.

[Lanayx]

Custom functions runtime

[sijie]

@codelipenghui we need to make sure Pulsar 2.6.0 is released with JDK 11.

[Anonymitaet]

@codelipenghui

I've replaced some doc-required tags w/ doc-added since docs have been added by @Huanli-Meng and me.

#4955, #6977: @Huanli-Meng has confirmed no doc needed for these PRs

#6473: Confirmed w/ @tuteng : REST API doc is generated from code automatically, no need to update it manually.

[codelipenghui]

Thanks @Anonymitaet

[Anonymitaet]

@codelipenghui
The item Custom functions runtime does not have a PR link.

I've searched Custom functions runtime w/ the label milestone 2.6.0 and got the result.

Is [Pulsar Functions] Support config admin CLI in funtions runtime #6681 the one I'm looking for?

[Lanayx]

@Anonymitaet Looks like this item made it way to the list because of my proposal (which actually meant custom language runtime like .net) which doesn't seem to even be planned for implementation, so it should be removed.

[Anonymitaet]

@Lanayx OK, thanks.

[Anonymitaet]

@codelipenghui Does the last item Work on JDK 11 has a PR link?

[lhotari]

I wish that the bug in negative acknowledgements for batch messages (#6869) would also be fixed in 2.6.0 . It would make the improvement for batch message acknowledgements PIP-54 / #6052 more complete.

[codelipenghui]

@lhotari I have started the 2.6.0 candidate, we can plan #6869 in 2.6.1.

[Anonymitaet]

@codelipenghui shall we add the following item to 2.6.0 release notes as @sijie points here?

Feature

Admin

• Change the permission level of managing subscription from super-user to tenant admin [pulsar-admin] allow tenant admin to manage subscription permission #6122

or

Feature

2.6.0-SNAPSHOT (under development)

• Change the permission level of managing subscription from super-user to tenant admin [pulsar-admin] allow tenant admin to manage subscription permission #6122

?

[feeblefakie]

@codelipenghui @sijie @merlimat

I retied the issue I faced with 2.5.0 with pulsar-2.6.0-candidate-1.
http://mail-archives.apache.org/mod_mbox/pulsar-users/202003.mbox/%3CCAPDOW74LN3WtdhpG_cgCCSg9MuMmkNV6giCGD5p%3DW1wWji0W7w%40mail.gmail.com%3E

But I'm still facing inconsistent order in between consumers with the default way (splitting range).
I didn't face it if I switch it to use consistent hashing.

So, this seems not resolved yet.
#6554
#6977

If 2.6.0 will be released without fixing the above, consistent hashing should be the default ?

I created a how to reproduce doc, so please take a look.
https://github.com/feeblefakie/misc/blob/master/pulsar/HOW-TO-REPRODUCE.md

[codelipenghui]

@feeblefakie Thanks for your update. #7106 is to fix the order problem in the Key_Shared subscription, It looks there are some problems with the auto-split mechanism. I will take a look later.

[codelipenghui]

close this issue since 2.6.0 was released.