[RIP-66] Support KV(Rocksdb) Storage

[fujian-zfj]

Before Creating the Enhancement Request

• I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary

We will introduce kv(rocksdb) storage to solve the possible Full GC caused by large object memory allocation when metadata such as topic is serialized, and the performance degradation caused by random writing of a large number of small ConsumeQueue files in a million-topic scenario

Motivation

we have encountered and been plagued by some probelms in the actual production environment. In some actual production scenarios, millions of topics and subsriptions will be created, and both of them may be deleted and created frequently. In current architecture, both topics and subcriptions are persisted in real time, which means each request of topics and subsciptions updating will trigger the persist interface, and the persistence of such metadata is written in full rather than append-only.

In the scenario of millions of topics, frequent persistence generates the large memory object jsonString of the topicConfigTable. When the memory is tight, the large memory object jsonString will be directly allocated to the old generation, resulting in frequent Full GC.

In addition, millions of topics will inevitably bring millions of indexed ConsumeQueue small files, a large number of indexed small files will destroy the advantages of rocketmq sequential writing, and the random writing of ConsumeQueue will make the performance drop sharply.

Describe the Solution You'd Like

This proposal mainly optimizes and solves the existing perfermance problems in the million-topic scenario from two levels:

1. Metadata like topic, subscription, consumerOffset implement kv storage
2. ConsumeQueue index file implements kv storage

Describe Alternatives You've Considered

no

Additional Context

No response

[RongtongJin]

https://github.com/apache/rocketmq/wiki/RIP-66-Support-KV(Rocksdb)-Storage