[Bug]: unprotected JMX port failing to start

[dermasmid]

Bug Description

when trying to enable JMX without a passwork, kafka is still trying to load a password file, that doesnt exists

Steps to reproduce

create a kafka cluster with a open jmx port:

spec:
  kafka:
    # ...
    jmxOptions: {}

Expected behavior

the server should start with an unprotected jmx port

Strimzi version

0.40.0

Kubernetes version

1.27.8-gke.1067004

Installation method

yaml files

Infrastructure

No response

Configuration files and logs

2024-04-11 10:53:49.788
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)
2024-04-11 10:53:49.788
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447)
2024-04-11 10:53:49.788
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437)
2024-04-11 10:53:49.788
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595)
2024-04-11 10:53:49.787
jdk.internal.agent.AgentConfigurationError
2024-04-11 10:53:49.787
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password
2024-04-11 10:53:49.078
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
2024-04-11 10:53:49.068
Using KRaft [false]
2024-04-11 10:53:49.060
Kafka metadata config state [0]
2024-04-11 10:53:49.057
Configuring Java heap: -Xms5368709120 -Xmx5368709120
2024-04-11 10:53:48.994
zookeeper.ssl.truststore.type=PKCS12
2024-04-11 10:53:48.994
zookeeper.ssl.truststore.password=[hidden]
2024-04-11 10:53:48.994
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
2024-04-11 10:53:48.994
zookeeper.ssl.keystore.type=PKCS12
2024-04-11 10:53:48.994
zookeeper.ssl.keystore.password=[hidden]
2024-04-11 10:53:48.994
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
2024-04-11 10:53:48.994
zookeeper.ssl.client.enable=true
2024-04-11 10:53:48.994
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
2024-04-11 10:53:48.994
zookeeper.connect=sonic-cluster-zookeeper-client:2181
2024-04-11 10:53:48.994
##########
2024-04-11 10:53:48.994
# Zookeeper
2024-04-11 10:53:48.994
##########
2024-04-11 10:53:48.994
{}
2024-04-11 10:53:48.994
{}
2024-04-11 10:53:48.994
log.message.format.version=3.6
2024-04-11 10:53:48.994
transaction.state.log.replication.factor=1
2024-04-11 10:53:48.994
transaction.state.log.min.isr=1
2024-04-11 10:53:48.994
offsets.topic.replication.factor=1
2024-04-11 10:53:48.994
min.insync.replicas=1
2024-04-11 10:53:48.994
inter.broker.protocol.version=3.6
2024-04-11 10:53:48.994
default.replication.factor=1
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# User provided configuration
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
ssl.endpoint.identification.algorithm=HTTPS
2024-04-11 10:53:48.993
sasl.enabled.mechanisms=
2024-04-11 10:53:48.993
control.plane.listener.name=CONTROLPLANE-9090
2024-04-11 10:53:48.993
inter.broker.listener.name=REPLICATION-9091
2024-04-11 10:53:48.993
advertised.listeners=CONTROLPLANE-9090://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9090,REPLICATION-9091://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io:9092,TLS-9093://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9093
2024-04-11 10:53:48.993
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093
2024-04-11 10:53:48.993
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Common listener configuration
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
listener.name.tls-9093.ssl.keystore.type=PKCS12
2024-04-11 10:53:48.993
listener.name.tls-9093.ssl.keystore.password=[hidden]
2024-04-11 10:53:48.993
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Listener configuration: TLS-9093
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Listener configuration: PLAIN-9092
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.client.auth=required
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.truststore.type=PKCS12
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.truststore.password=[hidden]
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.keystore.type=PKCS12
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.keystore.password=[hidden]
2024-04-11 10:53:48.993
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Replication listener
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.client.auth=required
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
2024-04-11 10:53:48.993
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Control Plane listener
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
log.dirs=/var/lib/kafka/data-0/kafka-log1
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
# Kafka message logs configuration
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
broker.id=1
2024-04-11 10:53:48.993
node.id=1
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
Node / Broker ID
2024-04-11 10:53:48.993
##########
2024-04-11 10:53:48.993
{}
2024-04-11 10:53:48.993
##############################
2024-04-11 10:53:48.993
##############################
2024-04-11 10:53:48.993
# Any changes to this file will be ignored and overwritten!
2024-04-11 10:53:48.993
# This file is automatically generated by the Strimzi Cluster Operator
2024-04-11 10:53:48.993
##############################
2024-04-11 10:53:48.993
##############################
2024-04-11 10:53:48.984
Starting Kafka with configuration:
2024-04-11 10:53:48.978
Preparing truststore for client authentication is complete
2024-04-11 10:53:48.791
Certificate was added to keystore
2024-04-11 10:53:48.553
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
2024-04-11 10:53:48.547
Preparing truststore for client authentication
2024-04-11 10:53:48.547
Preparing keystore for replication and clienttls listener is complete
2024-04-11 10:53:48.536
Preparing keystore for replication and clienttls listener
2024-04-11 10:53:48.535
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
2024-04-11 10:53:48.519
Looking for the right CA
2024-04-11 10:53:48.519
Preparing truststore for replication listener is complete
2024-04-11 10:53:48.320
Certificate was added to keystore
2024-04-11 10:53:48.018
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
2024-04-11 10:53:48.013
Preparing truststore for replication listener
2024-04-11 10:53:48.002
Enabling JMX access -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false
2024-04-11 10:53:47.998
STRIMZI_BROKER_ID=1
2024-04-11 10:53:47.994
removed '/tmp/strimzi.properties'
2024-04-11 10:53:47.994
removed directory '/tmp/kafka'
2024-04-11 10:53:47.994
removed '/tmp/kafka/cluster.truststore.p12'
2024-04-11 10:53:47.994
removed '/tmp/kafka/cluster.keystore.p12'
2024-04-11 10:53:47.994
removed '/tmp/kafka/clients.truststore.p12'
2024-04-11 10:53:47.994
removed '/tmp/kafka/strimzi.kafka.metadata.config.state'
2024-04-11 10:53:47.993
removed directory '/tmp/hsperfdata_kafka'

Additional context

No response

[scholzj]

Seems to work fine for me locally as well as pass in the tests. You should probably:

• Share the full Kafka CR file, maybe something you have there causes the problem
• Share the full log from the pod from the start and make sure it is properly formatted to make it readable

[dermasmid]

fixed the logs format, and here's the kafka cluster spec:

spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 1
      inter.broker.protocol.version: '3.6'
      min.insync.replicas: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
    jmxOptions: {}
    listeners:
      - configuration:
          brokers:
            - advertisedHost: stg.sonic-cluster-kafka-broker-0.internal.sonicwhale.io
              broker: 0
            - advertisedHost: stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io
              broker: 1
            - advertisedHost: stg.sonic-cluster-kafka-broker-2.internal.sonicwhale.io
              broker: 2
        name: plain
        port: 9092
        tls: false
        type: internal
      - name: tls
        port: 9093
        tls: true
        type: internal
    replicas: 3
    resources:
      limits:
        cpu: 16
        memory: 128Gi
      requests:
        cpu: 8
        memory: 32Gi
    storage:
      type: jbod
      volumes:
        - deleteClaim: false
          id: 0
          size: 1000Gi
          type: persistent-claim
    version: 3.7.0
  zookeeper:
    replicas: 3
    storage:
      deleteClaim: false
      size: 100Gi
      type: persistent-claim

[scholzj]

I don't think the log is complete - it normally starts by printing configurations and explaining what it is doing.

[scholzj]

That said, the Kafka resource (with the only modification of removing the resources) seems to work fine for me:

NAME                                          READY   STATUS    RESTARTS   AGE     IP              NODE                  NOMINATED NODE   READINESS GATES
my-cluster-entity-operator-6d8bfbb475-ks6fd   2/2     Running   0          32s     172.16.41.164   192.168.1.86.nip.io   <none>           <none>
my-cluster-kafka-0                            1/1     Running   0          58s     172.16.41.183   192.168.1.86.nip.io   <none>           <none>
my-cluster-kafka-1                            1/1     Running   0          58s     172.16.41.171   192.168.1.86.nip.io   <none>           <none>
my-cluster-kafka-2                            1/1     Running   0          58s     172.16.41.155   192.168.1.86.nip.io   <none>           <none>
my-cluster-zookeeper-0                        1/1     Running   0          83s     172.16.41.191   192.168.1.86.nip.io   <none>           <none>
my-cluster-zookeeper-1                        1/1     Running   0          83s     172.16.41.160   192.168.1.86.nip.io   <none>           <none>
my-cluster-zookeeper-2                        1/1     Running   0          83s     172.16.41.168   192.168.1.86.nip.io   <none>           <none>
strimzi-cluster-operator-86df856f74-wx5vq     1/1     Running   0          2m40s   172.16.41.154   192.168.1.86.nip.io   <none>           <none>

[scholzj]

Wait ... I see now that your log is mixed up and is in the wrong order. I think there is some issue with how it works on your networking stack. I guess you have some dual stack networking that causes this:

-Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6

Maybe that causes the problem?

[dermasmid]

the pod is in a crashloopbackoff heres all the logs between the crash error (with a bit of overlap):

	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595) 
jdk.internal.agent.AgentConfigurationError 
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password 
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties 
Using KRaft [false] 
Kafka metadata config state [0] 
Configuring Java heap: -Xms5368709120 -Xmx5368709120 
zookeeper.ssl.truststore.type=PKCS12 
zookeeper.ssl.truststore.password=[hidden] 
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
zookeeper.ssl.keystore.type=PKCS12 
zookeeper.ssl.keystore.password=[hidden] 
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
zookeeper.ssl.client.enable=true 
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
zookeeper.connect=sonic-cluster-zookeeper-client:2181 
########## 
# Zookeeper 
########## 
log.message.format.version=3.6 
transaction.state.log.replication.factor=1 
transaction.state.log.min.isr=1 
offsets.topic.replication.factor=1 
min.insync.replicas=1 
inter.broker.protocol.version=3.6 
default.replication.factor=1 
########## 
# User provided configuration 
########## 
ssl.endpoint.identification.algorithm=HTTPS 
sasl.enabled.mechanisms= 
control.plane.listener.name=CONTROLPLANE-9090 
inter.broker.listener.name=REPLICATION-9091 
advertised.listeners=CONTROLPLANE-9090://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9090,REPLICATION-9091://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io:9092,TLS-9093://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9093 
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093 
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL 
########## 
# Common listener configuration 
########## 
listener.name.tls-9093.ssl.keystore.type=PKCS12 
listener.name.tls-9093.ssl.keystore.password=[hidden] 
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Listener configuration: TLS-9093 
########## 
########## 
# Listener configuration: PLAIN-9092 
########## 
listener.name.replication-9091.ssl.client.auth=required 
listener.name.replication-9091.ssl.truststore.type=PKCS12 
listener.name.replication-9091.ssl.truststore.password=[hidden] 
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.replication-9091.ssl.keystore.type=PKCS12 
listener.name.replication-9091.ssl.keystore.password=[hidden] 
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Replication listener 
########## 
listener.name.controlplane-9090.ssl.client.auth=required 
listener.name.controlplane-9090.ssl.truststore.type=PKCS12 
listener.name.controlplane-9090.ssl.truststore.password=[hidden] 
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.controlplane-9090.ssl.keystore.type=PKCS12 
listener.name.controlplane-9090.ssl.keystore.password=[hidden] 
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Control Plane listener 
########## 
log.dirs=/var/lib/kafka/data-0/kafka-log1 
########## 
# Kafka message logs configuration 
########## 
broker.id=1 
node.id=1 
########## 
# Node / Broker ID 
########## 
############################## 
############################## 
# Any changes to this file will be ignored and overwritten! 
# This file is automatically generated by the Strimzi Cluster Operator 
############################## 
############################## 
Starting Kafka with configuration: 
Preparing truststore for client authentication is complete 
Certificate was added to keystore 
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca 
Preparing truststore for client authentication 
Preparing keystore for replication and clienttls listener is complete 
Preparing keystore for replication and clienttls listener 
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt 
Looking for the right CA 
Preparing truststore for replication listener is complete 
Certificate was added to keystore 
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca 
Preparing truststore for replication listener 
Enabling JMX access -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false 
STRIMZI_BROKER_ID=1 
removed '/tmp/strimzi.properties' 
removed directory '/tmp/kafka' 
removed '/tmp/kafka/cluster.truststore.p12' 
removed '/tmp/kafka/cluster.keystore.p12' 
removed '/tmp/kafka/clients.truststore.p12' 
removed '/tmp/kafka/strimzi.kafka.metadata.config.state' 
removed directory '/tmp/hsperfdata_kafka' 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595) 
jdk.internal.agent.AgentConfigurationError 
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password 
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties 
Using KRaft [false] 
Kafka metadata config state [0] 
Configuring Java heap: -Xms5368709120 -Xmx5368709120 
zookeeper.ssl.truststore.type=PKCS12 
zookeeper.ssl.truststore.password=[hidden] 
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
zookeeper.ssl.keystore.type=PKCS12 
zookeeper.ssl.keystore.password=[hidden] 
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
zookeeper.ssl.client.enable=true 
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
zookeeper.connect=sonic-cluster-zookeeper-client:2181 
########## 
# Zookeeper 
########## 
log.message.format.version=3.6 
transaction.state.log.replication.factor=1 
transaction.state.log.min.isr=1 
offsets.topic.replication.factor=1 
min.insync.replicas=1 
inter.broker.protocol.version=3.6 
default.replication.factor=1 
########## 
# User provided configuration 
########## 
ssl.endpoint.identification.algorithm=HTTPS 
sasl.enabled.mechanisms= 
control.plane.listener.name=CONTROLPLANE-9090 
inter.broker.listener.name=REPLICATION-9091 
advertised.listeners=CONTROLPLANE-9090://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9090,REPLICATION-9091://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io:9092,TLS-9093://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9093 
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093 
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL 
########## 
# Common listener configuration 
########## 
listener.name.tls-9093.ssl.keystore.type=PKCS12 
listener.name.tls-9093.ssl.keystore.password=[hidden] 
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Listener configuration: TLS-9093 
########## 
########## 
# Listener configuration: PLAIN-9092 
########## 
listener.name.replication-9091.ssl.client.auth=required 
listener.name.replication-9091.ssl.truststore.type=PKCS12 
listener.name.replication-9091.ssl.truststore.password=[hidden] 
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.replication-9091.ssl.keystore.type=PKCS12 
listener.name.replication-9091.ssl.keystore.password=[hidden] 
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Replication listener 
########## 
listener.name.controlplane-9090.ssl.client.auth=required 
listener.name.controlplane-9090.ssl.truststore.type=PKCS12 
listener.name.controlplane-9090.ssl.truststore.password=[hidden] 
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.controlplane-9090.ssl.keystore.type=PKCS12 
listener.name.controlplane-9090.ssl.keystore.password=[hidden] 
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Control Plane listener 
########## 
log.dirs=/var/lib/kafka/data-0/kafka-log1 
########## 
# Kafka message logs configuration 
########## 
broker.id=1 
node.id=1 
########## 
# Node / Broker ID 
########## 
############################## 
############################## 
# Any changes to this file will be ignored and overwritten! 
# This file is automatically generated by the Strimzi Cluster Operator 
############################## 
############################## 
Starting Kafka with configuration: 
Preparing truststore for client authentication is complete 
Certificate was added to keystore 
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca 
Preparing truststore for client authentication 
Preparing keystore for replication and clienttls listener is complete 
Preparing keystore for replication and clienttls listener 
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt 
Looking for the right CA 
Preparing truststore for replication listener is complete 
Certificate was added to keystore 
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca 
Preparing truststore for replication listener 
Enabling JMX access -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false 
STRIMZI_BROKER_ID=1 
removed '/tmp/strimzi.properties' 
removed directory '/tmp/kafka' 
removed '/tmp/kafka/cluster.truststore.p12' 
removed '/tmp/kafka/cluster.keystore.p12' 
removed '/tmp/kafka/clients.truststore.p12' 
removed '/tmp/kafka/strimzi.kafka.metadata.config.state' 
removed directory '/tmp/hsperfdata_kafka' 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595) 
jdk.internal.agent.AgentConfigurationError 
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password 
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties 
Using KRaft [false] 
Kafka metadata config state [0] 
Configuring Java heap: -Xms5368709120 -Xmx5368709120 
zookeeper.ssl.truststore.type=PKCS12 
zookeeper.ssl.truststore.password=[hidden] 
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
zookeeper.ssl.keystore.type=PKCS12 
zookeeper.ssl.keystore.password=[hidden] 
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
zookeeper.ssl.client.enable=true 
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
zookeeper.connect=sonic-cluster-zookeeper-client:2181 
########## 
# Zookeeper 
########## 
log.message.format.version=3.6 
transaction.state.log.replication.factor=1 
transaction.state.log.min.isr=1 
offsets.topic.replication.factor=1 
min.insync.replicas=1 
inter.broker.protocol.version=3.6 
default.replication.factor=1 
########## 
# User provided configuration 
########## 
ssl.endpoint.identification.algorithm=HTTPS 
sasl.enabled.mechanisms= 
control.plane.listener.name=CONTROLPLANE-9090 
inter.broker.listener.name=REPLICATION-9091 
advertised.listeners=CONTROLPLANE-9090://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9090,REPLICATION-9091://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io:9092,TLS-9093://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9093 
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093 
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL 
########## 
# Common listener configuration 
########## 
listener.name.tls-9093.ssl.keystore.type=PKCS12 
listener.name.tls-9093.ssl.keystore.password=[hidden] 
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Listener configuration: TLS-9093 
########## 
########## 
# Listener configuration: PLAIN-9092 
########## 
listener.name.replication-9091.ssl.client.auth=required 
listener.name.replication-9091.ssl.truststore.type=PKCS12 
listener.name.replication-9091.ssl.truststore.password=[hidden] 
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.replication-9091.ssl.keystore.type=PKCS12 
listener.name.replication-9091.ssl.keystore.password=[hidden] 
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Replication listener 
########## 
listener.name.controlplane-9090.ssl.client.auth=required 
listener.name.controlplane-9090.ssl.truststore.type=PKCS12 
listener.name.controlplane-9090.ssl.truststore.password=[hidden] 
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.controlplane-9090.ssl.keystore.type=PKCS12 
listener.name.controlplane-9090.ssl.keystore.password=[hidden] 
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Control Plane listener 
########## 
log.dirs=/var/lib/kafka/data-0/kafka-log1 
########## 
# Kafka message logs configuration 
########## 
broker.id=1 
node.id=1 
########## 
# Node / Broker ID 
########## 
############################## 
############################## 
# Any changes to this file will be ignored and overwritten! 
# This file is automatically generated by the Strimzi Cluster Operator 
############################## 
############################## 
Starting Kafka with configuration: 
Preparing truststore for client authentication is complete 
Certificate was added to keystore 
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca 
Preparing truststore for client authentication 
Preparing keystore for replication and clienttls listener is complete 
Preparing keystore for replication and clienttls listener 
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt 
Looking for the right CA 
Preparing truststore for replication listener is complete 
Certificate was added to keystore 
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca 
Preparing truststore for replication listener 
Enabling JMX access -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false 
STRIMZI_BROKER_ID=1 
removed '/tmp/strimzi.properties' 
removed directory '/tmp/kafka' 
removed '/tmp/kafka/cluster.truststore.p12' 
removed '/tmp/kafka/cluster.keystore.p12' 
removed '/tmp/kafka/clients.truststore.p12' 
removed '/tmp/kafka/strimzi.kafka.metadata.config.state' 
removed directory '/tmp/hsperfdata_kafka' 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595) 
jdk.internal.agent.AgentConfigurationError 
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password 
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties 
Using KRaft [false] 
Kafka metadata config state [0] 
Configuring Java heap: -Xms5368709120 -Xmx5368709120 
zookeeper.ssl.truststore.type=PKCS12 
zookeeper.ssl.truststore.password=[hidden] 
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
zookeeper.ssl.keystore.type=PKCS12 
zookeeper.ssl.keystore.password=[hidden] 
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
zookeeper.ssl.client.enable=true 
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
zookeeper.connect=sonic-cluster-zookeeper-client:2181 
########## 
# Zookeeper 
########## 
log.message.format.version=3.6 
transaction.state.log.replication.factor=1 
transaction.state.log.min.isr=1 
offsets.topic.replication.factor=1 
min.insync.replicas=1 
inter.broker.protocol.version=3.6 
default.replication.factor=1 
########## 
# User provided configuration 
########## 
ssl.endpoint.identification.algorithm=HTTPS 
sasl.enabled.mechanisms= 
control.plane.listener.name=CONTROLPLANE-9090 
inter.broker.listener.name=REPLICATION-9091 
advertised.listeners=CONTROLPLANE-9090://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9090,REPLICATION-9091://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://stg.sonic-cluster-kafka-broker-1.internal.sonicwhale.io:9092,TLS-9093://sonic-cluster-kafka-1.sonic-cluster-kafka-brokers.kafka.svc:9093 
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093 
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL 
########## 
# Common listener configuration 
########## 
listener.name.tls-9093.ssl.keystore.type=PKCS12 
listener.name.tls-9093.ssl.keystore.password=[hidden] 
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Listener configuration: TLS-9093 
########## 
########## 
# Listener configuration: PLAIN-9092 
########## 
listener.name.replication-9091.ssl.client.auth=required 
listener.name.replication-9091.ssl.truststore.type=PKCS12 
listener.name.replication-9091.ssl.truststore.password=[hidden] 
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.replication-9091.ssl.keystore.type=PKCS12 
listener.name.replication-9091.ssl.keystore.password=[hidden] 
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Replication listener 
########## 
listener.name.controlplane-9090.ssl.client.auth=required 
listener.name.controlplane-9090.ssl.truststore.type=PKCS12 
listener.name.controlplane-9090.ssl.truststore.password=[hidden] 
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
listener.name.controlplane-9090.ssl.keystore.type=PKCS12 
listener.name.controlplane-9090.ssl.keystore.password=[hidden] 
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
########## 
# Control Plane listener 
########## 
log.dirs=/var/lib/kafka/data-0/kafka-log1 
########## 
# Kafka message logs configuration 
########## 
broker.id=1 
node.id=1 
########## 
# Node / Broker ID 
########## 
############################## 
############################## 
# Any changes to this file will be ignored and overwritten! 
# This file is automatically generated by the Strimzi Cluster Operator 
############################## 
############################## 
Starting Kafka with configuration: 
Preparing truststore for client authentication is complete 
Certificate was added to keystore 
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca 
Preparing truststore for client authentication 
Preparing keystore for replication and clienttls listener is complete 
Preparing keystore for replication and clienttls listener 
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt 
Looking for the right CA 
Preparing truststore for replication listener is complete 
Certificate was added to keystore 
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca 
Preparing truststore for replication listener 
Enabling JMX access -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=2600:1900:4000:deee:0:1e:0:6 10.216.4.6 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.authenticate=false 
STRIMZI_BROKER_ID=1 
removed '/tmp/strimzi.properties' 
removed directory '/tmp/kafka' 
removed '/tmp/kafka/cluster.truststore.p12' 
removed '/tmp/kafka/cluster.keystore.p12' 
removed '/tmp/kafka/clients.truststore.p12' 
removed '/tmp/kafka/strimzi.kafka.metadata.config.state' 
removed directory '/tmp/hsperfdata_kafka' 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599) 
	at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:437) 
	at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:595) 
jdk.internal.agent.AgentConfigurationError 
Error: Password file not found: /usr/lib/jvm/java-17-openjdk-17.0.10.0.7-2.el8.x86_64/conf/management/jmxremote.password 
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties 
Using KRaft [false] 
Kafka metadata config state [0] 
Configuring Java heap: -Xms5368709120 -Xmx5368709120 
zookeeper.ssl.truststore.type=PKCS12 
zookeeper.ssl.truststore.password=[hidden] 
zookeeper.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12 
zookeeper.ssl.keystore.type=PKCS12 
zookeeper.ssl.keystore.password=[hidden] 
zookeeper.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12 
zookeeper.ssl.client.enable=true 
zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty 
zookeeper.connect=sonic-cluster-zookeeper-client:2181 
########## 
# Zookeeper 
##########

[dermasmid]

is there anyway i can overwrite the startup script so i can better debug it?

[dermasmid]

yes, its deployed to a gke cluster with dual stack enabled

[dermasmid]

trying a single stack cluster now

[dermasmid]

yup, it does seem to work fine on single stack, but i really prefer to have it on the dual stack cluster

[dermasmid]

im going to try to force ipv4 here

strimzi-kafka-operator/docker-images/kafka-based/kafka/scripts/set_kafka_jmx_options.sh

Line 9 in 97d4659

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.rmi.port=9999 -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=$(hostname -i) -Djava.net.preferIPv4Stack=true"

by using hostname -I | awk '{print $1}'

[dermasmid]

this worked, thanks for your help.
here's a pr #9951