[Bug]: Failed to aquire Lock on KafkaUser

[strowi]

Bug Description

Hi,

when creating a new KafkaUser and pre-configured secret i am getting the following warning for each user:

[kafka1-entity-operator-5c4fcb9bcb-5d4cx user-operator] 2024-02-14 20:31:26 WARN  AbstractControllerLoop:125 - KafkaUser-ControllerLoop-4: Failed to acquire lock KafkaUser::kafka::userXYZ. The resource will be re-queued for later.
...

[kafka1-entity-operator-5c4fcb9bcb-bbzb9 user-operator] 2024-02-14 20:31:28 INFO  UserControllerLoop:123 - Reconciliation #73(watch) KafkaUser(kafka/userXYZ): reconciled

The user is still being created and ready, But the log-line appears every couple of minutes.

It looks like the warning is being printed too early? This happens all within a few seconds. Or am i missing a setting to configure the delayed warning?

Steps to reproduce

Apply the following yaml to strimzi-operator 0.39 on kubernetes 1.27:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  labels:
    strimzi.io/cluster: kafka01
    tanka.dev/environment: dcc934623f6855899e19b4d9624ee96372107d4b2e30a392
  name: testuser-user1
  namespace: kafka
spec:
  authentication:
    type: scram-sha-512
  authorization:
    acls:
    - host: '*'
      operations:
      - Create
      - Describe
      - Write
      resource:
        name: test.prefix.topic1
        patternType: prefix
        type: topic
    type: simple
---
apiVersion: v1
data:
  password: '***'
kind: Secret
metadata:
  labels:
    strimzi.io/cluster: kafka01
    strimzi.io/kind: KafkaUser
    tanka.dev/environment: dcc934623f6855899e19b4d9624ee96372107d4b2e30a392
  name: testuser-user1
  namespace: kafka
type: Opaque

Expected behavior

User being created and no recurring log-warning.

Strimzi version

0.39

Kubernetes version

1.27.9-k3s

Installation method

Helm

Infrastructure

Bare-Metal/K3s

Configuration files and logs

[kafka1-entity-operator-5c4fcb9bcb-5d4cx user-operator] 2024-02-14 20:31:26 WARN AbstractControllerLoop:125 - KafkaUser-ControllerLoop-4: Failed to acquire lock KafkaUser::kafka::userXYZ. The resource will be re-queued for later.
...

[kafka1-entity-operator-5c4fcb9bcb-bbzb9 user-operator] 2024-02-14 20:31:28 INFO UserControllerLoop:123 - Reconciliation #73(watch) KafkaUser(kafka/userXYZ): reconciled

Additional context

No response

[scholzj]

The Failed to acquire lock message means that there is already one reconciliation running for this user. You see this likely more often because you create the KafkaUser resource and the Secret belonging to the user at the same time, but it can happen also in other cases. You should consider creating your own Secret with the password and linking it from the KafkaUser .spec.authentication section: https://strimzi.io/docs/operators/latest/full/configuring.html#type-KafkaUserScramSha512ClientAuthentication-reference ... that is a safer way that might help to avoid race conditions. But the way you do it should in general work in most cases as well, just ignore the warnings when the show up from time to time.

[strowi]

@scholzj Sorry for the late reply, i seem to not have gotten any notifications. Because of some race-condition between the automatic secret creation (via strimzi) and our custom secret we've since switched to the approach you mentioned:

kind: KafkaUser
metadata:
spec:
  authentication:
    password:
      valueFrom:
        secretKeyRef:
          key: password
          name: user-xy-secret
...

We have 2 identical clusters running, but the warnings only appeared on one of them when deploying via tanka. But nvm, we'll ignore those. Thx again!