[improve][broker] Support cgroup v2 by using jdk.internal.platform.Metrics in Pulsar Loadbalancer

[coderzc]

Master Issue: #16601

Motivation

The Pulsar load balancer detects CPU limits using cgroup v1 API, and the jdk.internal.platform.Metrics already support cgroup (V1, v2) so we should use jdk.internal.platform.Metrics to get the cgroup metrics.

Reference: https://code.yawk.at/java/17/java.base/jdk/internal/platform/

Modifications

Use jdk.internal.platform.Metrics to get the cgroup metrics in the LinuxInfoUtils.

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as testCGroupMetrics.

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API: (yes / no)
• The schema: (yes / no / don't know)
• The default values of configurations: (yes / no)
• The wire protocol: (yes / no)
• The rest endpoints: (yes / no)
• The admin cli options: (yes / no)
• Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

• doc-required
  (Your PR needs to update docs and you will update later)

• doc-not-needed
  (Please explain why)

• doc
  (Your PR contains doc changes)

• doc-complete
  (Docs have been already added)

[coderzc]

/pulsarbot run-failure-checks

[nodece]

LGTM

[coderzc]

/pulsarbot run-failure-checks

[codelipenghui]

We should check if the test really uses the jdk.internal.platform.Metrics not the old way?

[coderzc]

Good idea, we can assert metrics != null.

[coderzc]

/pulsarbot run-failure-checks

[github-actions]

The pr had no activity for 30 days, mark with Stale label.

[nodece]

Any updates? I think this PR makes sense to use the cgroup v2.

[heesung-sn]

For backward compatibility, don't we need to multiply the limit in calculateBrokerHostUsage?

public void calculateBrokerHostUsage() {
... 
       double totalCpuLimit = getTotalCpuLimit(isCGroupsEnabled);
        if (isCGroupsEnabled && metrics != null && getCpuUsageMethod != null) {
            // cgroup cpuUsage is already scaled, [0.0, 1.0]
            usage.setCpu(new ResourceUsage(cpuUsage * totalCpuLimit, totalCpuLimit));
        } else {
            usage.setCpu(new ResourceUsage(cpuUsage, totalCpuLimit));
        }
        

[coderzc]

The Metrics.getCpuUsage will return the aggregate time, so I think we don't need to multiply the limit.

    /**
     * Returns the aggregate time, in nanoseconds, consumed by all
     * tasks in the Isolation Group.
     *
     * @return Time in nanoseconds, -1 if unknown or
     *         -2 if the metric is not supported.
     *
     */
    public long getCpuUsage();

[heesung-sn]

I see. Thanks for the check.

Ref:
https://github.com/openjdk/jdk17/blob/master/src/java.base/share/classes/jdk/internal/platform/Metrics.java

https://github.com/openjdk/jdk17/tree/master/src/java.base/linux/classes/jdk/internal/platform

[coderzc]

I will update this PR as soon.

[coderzc]

I guess that requires a separate PR. Maybe we can push that PR in but add an issue to refactor (delete it out) the dependency on limit and only use percentage (which should be enough) or refactor to avoid using limit and be more specific like ResourceUsage.CpuUsage, MemoryUsage, NetworkUsage, each providing it's own method, where the CPU would only provide percentage.

Yes, we need a separate PR to refactor it.

[lhotari]

We need this to be included in branch-3.0 and branch-2.11 asap. I backported the changes to branch-2.10 with all dependencies in #20659.

This is becoming urgent.

AKS Kubernetes 1.25+ switches to use cgroup v2:
https://github.com/Azure/AKS/releases/tag/2023-03-05

AKS Kubernetes 1.24 goes End-of-life on July 30, 2023.

GKE contains to have a way to select between cgroup v1 & cgroup v2:
https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#cgroup-mode-options
GKE will default to cgroup v2 in new Kubernetes 1.26 clusters or node pools.
AWS EKS v1.26 nodes will default to cgroup v2.