[server] Create separate of RocksDBMemoryStats for RMD CF

[majisourav99]

Create separate of RocksDBMemoryStats for RMD CF

Today block-cache usage are emitting combined value for value and RMD familties. This PR isolates the block-cache part of the metric with a separate name limiter.

How was this PR tested?

CI

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change. This PR will separate the blockcache metric of rocksdb to a separate metric.

[ZacAttack]

Would this block rebalance triggered state transitions? Could we just use a thread safe iterator?

[gaojieliu]

These changes don't look correct..

There are two sets of metrics: block cache metrics and others.
Block cache metrics should only be collected once and other metrics will aggregate among all the instance of storage partitions.

RMD is a column family and ReplicationMetadataRocksDBStoragePartition contains two CFs: defaultand timestamp_metadata, and I think the correct logic should as follows:

for (String metric: PARTITION_METRIC_DOMAINS) {
      // Skip the block cache related metrics when using PlainTable format is enabled.
      if (plainTableEnabled && BLOCK_CACHE_METRICS.contains(metric)) {
        continue;
      }
      registerSensor(new AsyncGauge((ignored, ignored2) -> {
        Long total = 0L;
        // Lock down the list of RocksDB interfaces while the collection is ongoing
        synchronized (hostedRocksDBPartitions) {
          for (RocksDBStoragePartition dbPartition: hostedRocksDBPartitions.values()) {
            try {
              total += dbPartition.getRocksDBStatValue(metric, REPLICATION_METADATA_COLUMN_FAMILY);
            } catch (VeniceException e) {
              LOGGER.warn("Could not get rocksDB metric {} with error:", metric, e);
              continue;
            }
            if (INSTANCE_METRIC_DOMAINS.contains(metric)) {
              // Collect this metric once from any available partition and move on
              break;
            }
          }
        }
        return total;
      }, metric));
//==============New Code ===================
registerSensor(new AsyncGauge((ignored, ignored2) -> {
        Long total = 0L;
        // Lock down the list of RocksDB interfaces while the collection is ongoing
        synchronized (hostedRocksDBPartitions) {
          for (RocksDBStoragePartition dbPartition: hostedRocksDBPartitions.values()) {
            try {
             if (dbPartition instanceof ReplicationMetadataRocksDBStoragePartition) {
             // =========New code to add a new function to extract stat value for non-default CF.
                total += dbPartition.getRocksDBStatValue(metric, REPLICATION_METADATA_COLUMN_FAMILY);
              }
            } catch (VeniceException e) {
              LOGGER.warn("Could not get rocksDB metric {} with error:", metric, e);
              continue;
            }
            if (INSTANCE_METRIC_DOMAINS.contains(metric)) {
              // Collect this metric once from any available partition and move on
              break;
            }
          }
        }
        return total;
      }, "rmd-" + metric));
    }

[majisourav99]

Thanks, I did not know they have a separate metric for CF
dbPartition.getRocksDBStatValue(metric, REPLICATION_METADATA_COLUMN_FAMILY);

[gaojieliu]

Why skipping?
IIUC, the instance of ReplicationMetadataRocksDBStoragePartition contains two CFs: default and RMD and here we should continue to call dbPartition.getRocksDBStatValue(metric); to measure the default CF usage, right?

That means we should remove this if statement.

[gaojieliu]

Shall we check whether separate block cache is enabled or not before emitting another set of block cache metrics for rmd CF?