v0.1.1-beta

What's Changed

• fix(kafka_issues563): check record continuous by @superhx in #833
• feat(s3stream): add strict check mode by @superhx in #834
• feat(s3stream): add read block cache context to trace status by @SCNieh in #835
• fix(s3stream): catch storage callback exception by @superhx in #836
• fix(s3stream): fix sanity check when stream is trimmed in the middle by @SCNieh in #837
• fix(kafka_issues562): fix duplicated release by @superhx in #838
• feat(s3stream): show s3 configure in DefaultS3Operator; add S3Utils by @mooc9988 in #840
• fix(s3stream): find in data block with real record size by @SCNieh in #839
• refactor(broker): refactor opentelemertry exporter by @ShadowySpirits in #842
• fix(kafka_issues581): fix over capacity deadlock by @superhx in #843
• feat(proxy): add metrics and trace for remoting send and pull RPC by @ShadowySpirits in #844
• refactor(s3stream): show advices in error msg by @mooc9988 in #841
• feat(s3stream): add more metrics by @SCNieh in #845
• fix(ci): temporarily disable the remoting e2e test. by @ShadowySpirits in #846
• feat(s3stream): add metrics for read ahead latency and r/w queued time by @SCNieh in #848
• refactor(store): refactor MessageStateMachine by @ShadowySpirits in #849
• refactor(s3stream): only lock the deltaWAL.append by @Chillax-0v0 in #847
• feat(s3stream): set explicit boundaries for network queue time hist by @SCNieh in #852
• perf(S3Storage): speed up forceUpload by @Chillax-0v0 in #851
• feat(s3stream): support filter operation status by label by @SCNieh in #853
• feat(stream): implement MemoryMetadataManager by @ShadowySpirits in #854
• chore(s3stream): metrics of upload wal by @Chillax-0v0 in #856
• feat(kafka_issues601): optimize memory usage when upload too much blocks by @superhx in #857
• perf(s3storage): speedup forceUpload during shutdown by @Chillax-0v0 in #858
• feat(stream): add stream range continuous check for MemoryMetadataManager by @ShadowySpirits in #860
• fix(s3stream): fix infinite loop in s3 check by @ShadowySpirits in #862
• feat(s3stream): make AbstractOrderedCollection could be extented by others by @superhx in #863
• test(s3stream/util): test FutureTicker by @Chillax-0v0 in #859
• feat(s3stream): introduce tracing to s3stream by @SCNieh in #866
• perf(s3stream): thenAcceptAsync without executor -> thenAccept by @Chillax-0v0 in #867
• refactor(s3stream): standardize the code style by @ShadowySpirits in #864
• chore(s3stream): update the version not updated in commit f4e37d1 by @Chillax-0v0 in #868
• feat(s3stream): reuse attributes to reduce cost during recording metrics by @SCNieh in #869
• test(s3stream/util): fix tests by @Chillax-0v0 in #865
• feat(s3stream): support record metrics by level for s3stream by @SCNieh in #870
• refactor(stream): format and cleanup code by @ShadowySpirits in #874
• feat(s3stream): change read ahead size metrics level to INFO by @SCNieh in #871
• fix(s3stream/wal): fix checks before scaleOutWindow by @Chillax-0v0 in #873
• refactor(stream): make CompactionManager#compact public by @ShadowySpirits in #877
• ci(stream): add e2e ci by @ShadowySpirits in #878
• fix(stream): use getBucketACL instead of headBucket to check if bucket exist by @ShadowySpirits in #879
• feat(s3stream): remove the logic that checks if the bucket exists by @ShadowySpirits in #880
• feat(s3stream): wait for the background executor to close when shutting down the S3 storage by @ShadowySpirits in #876
• feat(kafka_issues618): stream object compact support drop expired data by @superhx in #881
• feat(s3stream): copy write based on max part size by @superhx in #883
• perf(s3stream): use read lock rather than write lock in append by @Chillax-0v0 in #886
• fix(s3Stream): remove list-bucket checking by @mooc9988 in #885
• feat(kafka_issues642): object format support data block group by @superhx in #887
• feat(s3stream): optimize metrics performance by @SCNieh in #889
• feat(kafka_issues642): stream object compact group small data blocks by @superhx in #890
• feat(s3stream): merge small data blocks during compaction by @SCNieh in #891
• fix(s3stream): fix operation latency unit by @SCNieh in #892
• perf(s3stream): use for loop rather than stream in critical paths by @Chillax-0v0 in #894
• chore: optimize S3Utils output by @KaimingWan in #895
• fix(s3stream): delete out-dated object directly during compaction by @SCNieh in #896
• feat(s3stream): atomic failover feature by @superhx in #899
• refactor(s3stream): create S3Operator with credential providers by @Chillax-0v0 in #898
• chore: add gradle build system for s3stream by @daniel-y in #900

New Contributors

• @KaimingWan made their first contribution in #895

Full Changelog: v0.1.0-beta...v0.1.1-beta

v0.1.0-beta

Major improvements

In this release, we added transaction support and the ability to reassign queues. Moreover, this release includes numerous performance improvements.

What's Changed

• feat(issues694): add sub stream size to request by @superhx in #703
• fix: make reserve-time accurate by @lizhanhui in #704
• feat(issues690): concurrent read log cache by @superhx in #701
• fix(s3stream): fix potential dangling read ahead future by @SCNieh in #705
• feat(kafka_isssues447): add command util and update version to 0.5.2 by @superhx in #709
• refactor(s3stream/wal): init the sliding window during start by @Chillax-0v0 in #708
• feat(s3stream): command utils make run to static method by @superhx in #710
• feat: add default serverless by @superhx in #711
• feat(s3stream): 0.5.4 by @superhx in #712
• feat(s3stream): dynamically adjust read ahead speed by @SCNieh in #713
• fix: #716 invoke data-store to potentially delete S3 Object prior to rollback expired prepare-records by @lizhanhui in #717
• fix(s3stream): use lazy initialization to prevent metrics missing by @SCNieh in #724
• fix: revert ZGC option as it causes OOM due to multi-mapping of heap VMA by @lizhanhui in #723
• fix: escape dollar symbol in docker-compose.xml for linux by @lizhanhui in #721
• feat: add contributing guideline by @lizhanhui in #719
• perf(s3stream/storage): remove the single main write thread by @Chillax-0v0 in #728
• fix(s3stream): update to 0.5.6 and fix serveless cmd by @superhx in #730
• feat(s3stream): optimize cache performance by adding offset index by @superhx in #731
• fix(store): fix end offset of fetch result by @ShadowySpirits in #725
• chore: add maven-source-plugin for s3stream lib by @leizhiyuan in #707
• perf(s3stream/storage): shard callback locks by stream id by @Chillax-0v0 in #729
• feat(s3stream_kafka471): support fast read option by @superhx in #732
• fix: #733 resolve class clashes by @lizhanhui in #734
• fix: handle error when open streams by @lizhanhui in #736
• chore: release helm chart when the tag starts with helm by @wangxye in #726
• fix: include node-id in close-stream-request during forwarding requests by @lizhanhui in #738
• chore: rename release action about docker build by @wangxye in #739
• feat(s3stream): adapt to new serverless cmd by @superhx in #742
• refactor: refactor cmd line in cli by @wangxye in #744
• fix(s3stream/storage): calculate the confirm offset from unordered requests by @Chillax-0v0 in #743
• fix(s3stream): serverless cmd by @superhx in #750
• fix(kafka_issues475): do not log when fast fail by @superhx in #751
• refactor: rename wal object to stream set object by @daniel-y in #749
• perf(s3storage/wal): use PriorityQueue rather than TreeSet by @Chillax-0v0 in #753
• fix: unify controller error handling while forwarding requests to leader node by @lizhanhui in #758
• chore: support nightly build in three different versions by @wangxye in #727
• chore: update helm chart to avoid invalid value by @wangxye in #741
• chore(s3stream/wal): record metrics on block polled by @Chillax-0v0 in #760
• fix(kafka_issues447): serverless cmd by @superhx in #761
• feat(distribution): update image tag in docker-compose.yaml by @ShadowySpirits in #762
• fix(proxy): remove shaded grpc dependency by @ShadowySpirits in #759
• feat(s3stream): blockCache performance optimization by @SCNieh in #766
• chore: support for e2e run on commits by @wangxye in #755
• feat(s3stream): add boringssl dependence by @superhx in #767
• feat(s3stream/wal): pre-check the real capacity of the block device by @Chillax-0v0 in #771
• fix(kafka_issues491): ignore await after scheduled task cancel by @superhx in #770
• chore: support helm chart defaults to deploy full clusters by @wangxye in #769
• feat(s3stream): block cache memory optimization by @SCNieh in #774
• feat(s3stream): optimize compaction throttle by @SCNieh in #776
• fix(s3stream): guard BlockCache OOM handle with lock by @SCNieh in #775
• fix(s3stream): fix inflight read throttle size by @SCNieh in #778
• feat(distribution): refactor helm chart by @ShadowySpirits in #777
• chore: upgrade chart version to 0.0.4 by @wangxye in #780
• feat(kafka_issues500): add pooledbuf read options by @superhx in #782
• feat(s3stream): add inflight pooled fetch request record by @superhx in #783
• fix(s3stream): add clean up on future exception for BlockCache by @SCNieh in #781
• fix(s3stream): release records that won't be returned on exception by @SCNieh in #785
• feat(s3stream): avoid copy when operator read by @superhx in #788
• chore: update helm chart deploy for e2e test by @wangxye in #789
• feat(store): refactor TimerService and implement TransactionService by @ShadowySpirits in #790
• feat(s3stream): detect fetch timeout by @superhx in #791
• fix(s3stream): fix timeout detect by @superhx in #793
• fix(s3stream): fix potential dangling cf when read not aligned with data block by @SCNieh in #795
• fix: fix non-unique instance id in the node by @wangxye in #798
• feat(cli): add ResetConsumeOffset command by @ShadowySpirits in #800
• feat(s3stream): clean up buffer on compaction exception by @SCNieh in #799
• feat(proxy): support transaction message by @ShadowySpirits in #796
• feat(proxy): add topicStats rpc by @ShadowySpirits in #801
• feat(cli): add PrintTopicStats command by @ShadowySpirits in #802
• fix(s3Stream): support aliyun oss by @mooc9988 in #804
• feat(cli): add ProducerClientConnection command by @ShadowySpirits in #806
• feat(s3stream): smooth upload wal by @superhx in #809
• chore: fix helm chart install eof by @wangxye in #794
• feat(cli): add ConsumerClientConnection command by @ShadowySpirits in #807
• feat(proxy): add relay rpc by @ShadowySpirits in #810
• refactor(proxy): refactor DeadLetterService by @ShadowySpirits in #811
• fix(s3stream/wal): check whether the file system supports direct IO by @Chillax-0v0 in #808
• fix(proxy): add PROPERTY_PRODUCER_GROUP property for transaction message by @ShadowySpirits in #803
• test(proxy): add unit...

v0.0.3-alpha

Major improvements

This release focuses on stable improvement:

1. Improve the CI process and implement end-to-end (E2E) testing for every pull request (PR).
2. Introduce tracing and profiling and gather more metrics.
3. Improve performance for s3stream.
4. Fix some bugs as follows.

What's Changed

• fix(proxy): fix flaky test MessageStoreTest#pop_normal by @ShadowySpirits in #525
• feat: revamp substreams serde by @lizhanhui in #528
• chore(s3stream): log wal upload time cost by @superhx in #527
• test(store): add unit test for store module by @ShadowySpirits in #529
• feat: use json type for topic.accept_message_types and s3walobject.sub_streams columns by @lizhanhui in #530
• feat(store): support trimming retry stream by @TheR1sing3un in #509
• fix: make Group#subMode required during creation by @lizhanhui in #531
• chore(ci): ignore metrics package in test coverage calculation by @ShadowySpirits in #534
• fix(s3stream): try fix issue413 stream compaction hang io thread by @superhx in #536
• fix(controller): fix illegal basedatetTimestamp in commit wal by @wangxye in #535
• test(proxy): add unit test for MessageServiceImpl by @ShadowySpirits in #533
• perf(proxy): improve metrics performance by @ShadowySpirits in #526
• feat: cache S3StreamObject metadata by @lizhanhui in #538
• fix: separate database accessor to metadata-jdbc module by @lizhanhui in #540
• fix: inherit maven properties from parent module for metadata-jdbc by @lizhanhui in #541
• feat(s3stream): refine s3stream config name by @SCNieh in #542
• feat: localize s3 metadata access by @lizhanhui in #543
• chore(proxy): add global exception handler for proxy by @ShadowySpirits in #537
• chore(log): separate stream log into dedicated file by @ShadowySpirits in #544
• feat(s3stream): optimize s3stream metrics name by @SCNieh in #545
• chore(stream): use the separate S3 operator for compaction purposes by @ShadowySpirits in #546
• feat(s3stream): change s3stream metrics tags key by @SCNieh in #547
• fix: commit wal implementation by @lizhanhui in #548
• fix: set retention period when create topic and add test case for top… by @lizhanhui in #550
• feat: cluster node list now includes advertise address by @lizhanhui in #551
• fix(s3stream): fix potential NPE in compaction by @SCNieh in #552
• chore(stream): add network limit config for stream module by @ShadowySpirits in #549
• feat: expose controller config by @lizhanhui in #554
• feat(s3stream): simplify operation counter metrics by @SCNieh in #553
• chore(doc): update README.md by @ShadowySpirits in #555
• feat: delete S3 objects when streams are deleted by @lizhanhui in #560
• chore(controller): add some logs by @ShadowySpirits in #559
• feat: move log4j2.xml to conf by @lizhanhui in #561
• fix(distribution): change gc log directory by @ShadowySpirits in #562
• fix(distribution): fix typo by @ShadowySpirits in #563
• fix(s3stream): fix compaction read throttle by @SCNieh in #564
• feat(store): add s3 stream compaction config by @ShadowySpirits in #565
• fix: avoid unmodifiable container by @lizhanhui in #569
• fix(store): fix bug in decoding reset consume offset operation by @ShadowySpirits in #572
• fix(store): fix bug network limiter by @ShadowySpirits in #574
• fix(store): skip initializing StreamMetricsManager when exporter disabled by @TheR1sing3un in #557
• feat(store): add config streamSplitSizeThreshold by @ShadowySpirits in #576
• fix(controller): fix bug in recycling stream object by @ShadowySpirits in #577
• fix(s3stream): measure time elapse with higher precision by @SCNieh in #568
• fix(store): ignore corrupt operation when recovering by @ShadowySpirits in #579
• refactor(issues575-issue575): rename WAL object to SST object by @superhx in #580
• fix: trim stream before recycle its S3 assets by @lizhanhui in #581
• feat: trim stream prior to deleting objects by @lizhanhui in #578
• feat: finalize trim stream by @lizhanhui in #582
• chore: project structure maintenance by @lizhanhui in #583
• fix: default config should have defaults that make sense by @lizhanhui in #584
• feat(proxy): optimize SuspendRequestService by @ShadowySpirits in #585
• feat: log SQL statements to a dedicated appender by @lizhanhui in #586
• feat(broker): introduce continuous profiling by @ShadowySpirits in #592
• fix: set default database connection pool size by @lizhanhui in #593
• feat: Specify log4j2 configuration file using Java option log4j2.conf… by @lizhanhui in #588
• feat(s3stream): add compaction throttle to flatten load peak by @SCNieh in #595
• feat(s3stream): optimization on crc calculation by @SCNieh in #596
• fix: fix the distribution of run server by @wangxye in #597
• feat: retry completable future on exception by @lizhanhui in #599
• feat: retry invocations with metadata that are caused by internal error or mocked by @lizhanhui in #600
• feat(s3stream): support network throttle with priority by @SCNieh in #598
• feat(store): forbidden negative metrics by @ShadowySpirits in #567
• feat(broker): introduce tracing by @ShadowySpirits in #602
• feat: prepend CRC32 checksum header to ops stream record by @lizhanhui in #601
• chore: opt quick start for docker compose by @leizhiyuan in #605
• refactor(s3stream): rename sst to stream set object by @superhx in #609
• feat(s3stream): check read permits before merge read task by @SCNieh in #610
• chore: optimize helm chart deployment by @wangxye in #611
• fix: wal recovery by @lizhanhui in #604
• feat: dump heap on fatal error for diagnosis by @lizhanhui in #612
• feat(common): add s3Namespace configuration item by @TheR1sing3un in #558
• feat(broker): introduce tracing to both proxy and store modules by @ShadowySpirits in #606
• feat: sync config files to helm deployments by @wangxye in #616
• feat(s3stream): add s3 operation timeout by @SCNieh in #618
• feat(broker): make trace and profiler exporter configurable by @ShadowySpirits in #617
• feat: implement describe stream command by @lizhanhui in #620
• feat(store): notify message arriving by @ShadowySpirits in https://github.com/...

v0.0.2-alpha

Major improvements

Feature

1. Support RocketMQ 4.x remoting protocol
2. Support delayed message
3. Implement topic-level TTL
4. Cluster mode

Tools

1. reset consume offset
2. cluster related operation
3. list group or topic

Observability

1. Introduce Metrics for stream module

What's Changed

• feat(s3stream/wal): group commit by @Chillax-0v0 in #422
• feat(s3stream): use put object when data size is less than 32MB by @superhx in #431
• fix(store): ignore operation with illegal argument in recovery process by @ShadowySpirits in #429
• feat: support pull for remoting protocol in the early stage by @daniel-y in #424
• feat: improve cache to avoid request collapsing by @lizhanhui in #430
• feat: implement topic retention period by @lizhanhui in #436
• feat: lookup topic/queue owner node id by @lizhanhui in #437
• feat(s3stream): support network inbound/outbound bandwidth throttle by @SCNieh in #438
• perf(s3stream/wal): use the memory pool by @Chillax-0v0 in #435
• feat: support getStreams for StreamManager by @wangxye in #432
• feat: filter out trimmed data on compaction by @SCNieh in #379
• feat: use protobuf-util to convert between protobuf message and json by @lizhanhui in #441
• feat: expose node config to StoreMetadataService by @lizhanhui in #442
• fix(s3stream/wal): no need to release recovered records internally by @Chillax-0v0 in #443
• fix(s3stream): fix memory leak by @SCNieh in #446
• fix(s3stream/wal): flush the file after write by @Chillax-0v0 in #447
• fix(s3stream/wal): add block into the writing list; update the start offset correctly by @Chillax-0v0 in #448
• feat(s3stream): cache evict by @superhx in #452
• feat(cli): support topic ttl in CreateTopic command by @ShadowySpirits in #439
• feat(s3stream/wal): merge write by @Chillax-0v0 in #454
• feat(proxy): support pull message request for remoting protocol by @ShadowySpirits in #449
• feat: controller handles planned termination by @lizhanhui in #457
• fix(s3stream): fix duplicated release by @superhx in #459
• feat: add protobuf rpc to request a node to terminate gracefully by @lizhanhui in #460
• test: specify test cases for s3wal and s3stream by @wangxye in #461
• feat: support shutdown broker gracefully by @daniel-y in #462
• feat: add CLI and partially implement graceful shutdown by @lizhanhui in #463
• perf(s3stream/wal): marshal records lazily by @Chillax-0v0 in #464
• fix: persist assignment status by @lizhanhui in #465
• fix: fix the empty of streamIds in getStreams by @wangxye in #466
• feat: implement leader transfer by @lizhanhui in #467
• feat(proxy): check client is supported for remoting protocol by @ShadowySpirits in #468
• chore: update test access key/secret by @lizhanhui in #451
• feat(store): support resetting consume offset by @TheR1sing3un in #434
• feat(store): support stream related metrics by @TheR1sing3un in #453
• fix(s3stream): prevent sort on unmodifiable list; delete unused code by @SCNieh in #472
• fix: expose port shifted by 1 by @lizhanhui in #473
• fix: stream domain object should use boxed type by @lizhanhui in #476
• feat(proxy): support pull orderly by @ShadowySpirits in #474
• fix: fix stream state when schedule queue/streams among nodes by @lizhanhui in #478
• fix(s3stream): force evict cache when OOM by @superhx in #479
• fix(s3stream): fix data block memory footprint by @superhx in #480
• feat(proxy): refactor long polling by @ShadowySpirits in #477
• feat: implement command to describe cluster by @lizhanhui in #485
• feat: pretty describe cluster output by @lizhanhui in #487
• feat: Cluster description shall include a summary section by @lizhanhui in #488
• feat: support for unified date management of metadata by @wangxye in #484
• fix: refine api of metadata store by @lizhanhui in #490
• fix: simplify stream mapper by @lizhanhui in #491
• fix(s3stream): fix kos issue389 index ouf of bound by @superhx in #494
• fix(s3stream): fix kos issue390 record out of order by @superhx in #495
• feat: support deploy in helm and kubeblock by @wangxye in #492
• feat(s3stream): stats bytebuf alloc by @superhx in #496
• feat(s3stream): add copyOnWrite interface to Writer by @SCNieh in #497
• fix(s3stream): fix kos issue403 repeated add backoff record by @superhx in #498
• feat(s3stream): isolate s3 readwrite by @superhx in #499
• feat(s3stream): limit s3 operator inflight request by @superhx in #500
• feat(s3stream): force split one wal at a time, limit memory consumpti… by @SCNieh in #501
• fix(s3stream): operator release by @superhx in #502
• feat(proxy): introduce TimerService by @ShadowySpirits in #493
• feat: expose create topic API to ProxyMetadataService by @lizhanhui in #503
• fix(s3stream): acquire deadlock by @superhx in #504
• feat: list topics by @lizhanhui in #505
• feat: implement list-group command by @lizhanhui in #507
• fix(s3stream): deadlock by @superhx in #508
• feat: implement update-group command by @lizhanhui in #512
• fix(s3stream/wal): fix memory leak during recovery by @Chillax-0v0 in #513
• feat: add command to delete{topic,group} by @lizhanhui in #514
• chore: register cli commands by @lizhanhui in #516
• feat: support addressOfNode in async by @wangxye in #517
• feat(s3stream): force exit when commit wal fatal error by @superhx in #520
• feat(store): support delayed message by @ShadowySpirits in #521
• refactor(store): optimize stream metrics' name by @TheR1sing3un in #518
• feat: support subscription mode for group by @lizhanhui in #522
• fix(s3stream): compaction fix by @SCNieh in #506
• chore: display queue-id prior to node-id by @lizhanhui in #523
• feat(proxy): limit client consumption mode by @ShadowySpirits in #524

Full Changelog: v0.0.1-alpha...v0.0.2-alpha

v0.0.1-alpha

The first release! 🎉

In this version, we provide:

1. Full compatibility with RocketMQ gRPC protocol
2. Basic admin command line tools to create or modify the resource
3. Metrics for proxy and store module

Known issue:

1. Sporadic errors when recovering from the crash: #416