[Bug]: Cruise control is not honoring hard goals

[sreelu27]

Bug Description

Cruise control is not honoring hard goals. RackAwareGoal only provided as hard goal for cruise Control, but KafkaRebalance is failing due to NetworkInboundCapacityGoal not statisfied, even though NetworkInboundCapacityGoal is not part of hard goals.

kafka cluster spec :

cruiseControl:
    config:
      hard.goals:
        com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal

KafkaRebalance failing with error "com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [NetworkInboundCapacityGoal] Insufficient capacity for networkInbound"

Steps to reproduce

1. Add com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal only as hard.goals for cruise Control Configuration.
2. Deployed the system.
3. Produce events with more than inbound Network capacity configured per node in cruise Control Configuration (Default is 10000 Kib/s), so if you have 3 node and planning to scale out for 6 nodes, then events to be produce more than inboutNetworkCapatiy*no.of node, in this case it should more than 70000 kib/s.
4. Keep produce events and scale out Kafka pods from 3 to 6, and wait till all pods to come running state.
5. Apply KafkaRebalance (below is for reference)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: rebalance
  labels:
    strimzi.io/cluster: bug
# no goals specified, using the default goals from the Cruise Control configuration
spec:
  mode: add-brokers 
  brokers: [3, 4, 5]

6. KafkaRebalance failing with error "com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException:
   [NetworkInboundCapacityGoal] Insufficient capacity for networkInbound".

Expected behavior

If hard goals are statisfied, then other goals can ignore even if it not satisfies and KafkaRebalance should come to PROPOSALREADY as true.

Strimzi version

0.36.1

Kubernetes version

1.27.6

Installation method

Helm chart

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[scholzj]

You should probably start by sharing the full Kafka custom resource and make sure it is correctly formatted. You should also compare it with the API reference and make sure what you have there corresponds to it. I don't think there are any enableAutomation, enable or autoApprovalInterval fields.

[sreelu27]

[scholzj]

While there are some similarities, that does not look like Strimzi. Strimzi custom resource would look for example like this: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/kafka/kafka-persistent.yaml

[ppatierno]

It's a MessageBus custom resource ... for sure not part of Strimzi :-)

[sreelu27]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: bug
  namespace: tr
spec:
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 10KB/s
      outboundNetwork: 10KB/s
    config:
      hard.goals: com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal
    resources:
      limits:
        cpu: "3"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 1Gi
  entityOperator:
    topicOperator:
      resources:
        limits:
          cpu: "3"
          memory: 2Gi
        requests:
          cpu: "2"
          memory: 1Gi
    userOperator:
      resources:
        limits:
          cpu: "3"
          memory: 2Gi
        requests:
          cpu: "2"
          memory: 1Gi
  kafka:
    config:
      default.replication.factor: 3
      min.insync.replicas: 2
    rack:
      topologyKey: kubernetes.io/hostname
    replicas: 6
    resources:
      limits:
        cpu: "3"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 1Gi
    storage:
      class: network-block
      size: 25Gi
      type: persistent-claim
    version: 3.5.1
  zookeeper:
    replicas: 1
    resources:
      limits:
        cpu: "3"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 1Gi
    storage:
      class: network-block
      size: 15Gi
      type: persistent-claim``

[sreelu27]

Hi all, Is there any update on this issue?

[kyguy]

Not yet @sreelu27, will create an issue for it for upstream Cruise Control today and link it here

[kyguy]

Hey @sreelu27, opened an issue for this upstream [1] I'll have a closer look at it after discussing the issue with the Cruise Control maintainer

[1] linkedin/cruise-control#2111

[kyguy]

Hey @sreelu27 I have just left a comment upstream with more details on the issue after an investigation [1] In summary: We cannot configure which goals are "hard" and which goals are "soft" using the "hard.goals" config. The hard.goals config is NOT a list of goals "that must be satisfied" but rather list of goals "that must be executed".

To fix the issue posted above we must exclude com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal from the default.goals list. com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal is hard coded as a hard goal in upstream Cruise Control. Therefore, if we included it in our CC default.goals config, it must be satisfied for an optimization proposal to be generated.

The purpose and explanation of the hard.goals config in upstream Cruise Control to be misleading and unintuitive. I am going to suggest that we either enhance Cruise Control config to work in the way the config implies or update the CC documentation.

I also need to update the following Strimzi docs which are NOT correct based on my findings [2]

You configure hard goals in the Cruise Control deployment configuration, by editing the hard.goals property in Kafka.spec.cruiseControl.config.
- To inherit the preset hard goals from Cruise Control, do not specify the hard.goals property in Kafka.spec.cruiseControl.config
- To change the preset hard goals, specify the desired goals in the hard.goals property, using their fully-qualified domain names.

[1] linkedin/cruise-control#2111
[2] https://github.com/strimzi/strimzi-kafka-operator/blob/6dee155a32e8b1e782e2e33f39449e6d82eedc4b/documentation/modules/cruise-control/con-optimization-goals.adoc#hard-and-soft-optimization-goals

[kyguy]

Just opened a PR to update the Strimzi documentation to reflect this information here [1]

[1]#9678