Broker fails to start with NFS

[lng-sys]

Hi, I hope this is a good place to ask the following.

This is the first time I try to run Kafka and since I know K8s a bit, I am trying to do so with your operator, but so far I wasn't able to run a cluster of Kafka brokers. I have the following YAML:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: nfs-pod
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka-nfs-cluster
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    deleteClaim: true
    size: 10Gi
    type: persistent-claim
    class: nfs-storage-class
    deleteClaim: false
---

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-nfs-cluster
  namespace: kafka
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.7.0
    metadataVersion: 3.7-IV4
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
  entityOperator:
    topicOperator: {}
    userOperator: {}

kubectl get pods shows that Kafka pods are not ready, so I checked what the readiness script does and I see the broker fails to start. So started a shell inside a pod and manually run /opt/kafka/kafka_run.sh but I get the following error. Am I doing something wrong? Thank you in advance for any tip.

removed '/tmp/hsperfdata_kafka/695'
removed directory '/tmp/hsperfdata_kafka'
removed '/tmp/kafka/strimzi.kafka.metadata.config.state'
removed '/tmp/kafka/clients.truststore.p12'
removed '/tmp/kafka/cluster.keystore.p12'
removed '/tmp/kafka/cluster.truststore.p12'
removed directory '/tmp/kafka'
removed '/tmp/strimzi.properties'
STRIMZI_BROKER_ID=0
Preparing truststore for replication listener
Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for replication listener is complete
Looking for the right CA
Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt
Preparing keystore for replication and clienttls listener
Preparing keystore for replication and clienttls listener is complete
Preparing truststore for client authentication
Adding /opt/kafka/client-ca-certs/ca.crt to truststore /tmp/kafka/clients.truststore.p12 with alias ca
Certificate was added to keystore
Preparing truststore for client authentication is complete
Starting Kafka with configuration:
##############################
##############################
# This file is automatically generated by the Strimzi Cluster Operator
# Any changes to this file will be ignored and overwritten!
##############################
##############################

##########
# Node / Broker ID
##########
node.id=0

##########
# Kafka message logs configuration
##########
log.dirs=/var/lib/kafka/data/kafka-log0

##########
# Control Plane listener
##########
listener.name.controlplane-9090.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.controlplane-9090.ssl.keystore.password=[hidden]
listener.name.controlplane-9090.ssl.keystore.type=PKCS12
listener.name.controlplane-9090.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.controlplane-9090.ssl.truststore.password=[hidden]
listener.name.controlplane-9090.ssl.truststore.type=PKCS12
listener.name.controlplane-9090.ssl.client.auth=required

##########
# Replication listener
##########
listener.name.replication-9091.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.replication-9091.ssl.keystore.password=[hidden]
listener.name.replication-9091.ssl.keystore.type=PKCS12
listener.name.replication-9091.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
listener.name.replication-9091.ssl.truststore.password=[hidden]
listener.name.replication-9091.ssl.truststore.type=PKCS12
listener.name.replication-9091.ssl.client.auth=required

##########
# Listener configuration: PLAIN-9092
##########

##########
# Listener configuration: TLS-9093
##########
listener.name.tls-9093.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.tls-9093.ssl.keystore.password=[hidden]
listener.name.tls-9093.ssl.keystore.type=PKCS12


##########
# Common listener configuration
##########
listener.security.protocol.map=CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL
listeners=CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093
advertised.listeners=REPLICATION-9091://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9092,TLS-9093://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9093
inter.broker.listener.name=REPLICATION-9091
sasl.enabled.mechanisms=
ssl.endpoint.identification.algorithm=HTTPS

##########
# User provided configuration
##########
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.min.isr=2
transaction.state.log.replication.factor=3


##########
# KRaft configuration
##########
process.roles=broker,controller
controller.listener.names=CONTROLPLANE-9090
controller.quorum.voters=0@kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090,1@kafka-nfs-cluster-nfs-pod-1.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090,2@kafka-nfs-cluster-nfs-pod-2.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090

##########
# KRaft metadata log dir configuration
##########
metadata.log.dir=/var/lib/kafka/data/kafka-log0
Kafka metadata config state [4]
Using KRaft [true]
Making sure the Kraft storage is formatted with cluster ID Vpquqx0FSJWXEEV8EoVdtg and metadata version 3.7-IV4
2024-07-05 07:26:52,583 INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [main]
2024-07-05 07:26:53,042 INFO KafkaConfig values: 
        advertised.listeners = REPLICATION-9091://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9091,PLAIN-9092://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9092,TLS-9093://kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc:9093
        alter.config.policy.class.name = null
        alter.log.dirs.replication.quota.window.num = 11
        alter.log.dirs.replication.quota.window.size.seconds = 1
        authorizer.class.name = 
        auto.create.topics.enable = true
        auto.include.jmx.reporter = true
        auto.leader.rebalance.enable = true
        background.threads = 10
        broker.heartbeat.interval.ms = 2000
        broker.id = 0
        broker.id.generation.enable = true
        broker.rack = null
        broker.session.timeout.ms = 9000
        client.quota.callback.class = null
        compression.type = producer
        connection.failed.authentication.delay.ms = 100
        connections.max.idle.ms = 600000
        connections.max.reauth.ms = 0
        control.plane.listener.name = null
        controlled.shutdown.enable = true
        controlled.shutdown.max.retries = 3
        controlled.shutdown.retry.backoff.ms = 5000
        controller.listener.names = CONTROLPLANE-9090
        controller.quorum.append.linger.ms = 25
        controller.quorum.election.backoff.max.ms = 1000
        controller.quorum.election.timeout.ms = 1000
        controller.quorum.fetch.timeout.ms = 2000
        controller.quorum.request.timeout.ms = 2000
        controller.quorum.retry.backoff.ms = 20
        controller.quorum.voters = [0@kafka-nfs-cluster-nfs-pod-0.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090, 1@kafka-nfs-cluster-nfs-pod-1.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090, 2@kafka-nfs-cluster-nfs-pod-2.kafka-nfs-cluster-kafka-brokers.kafka.svc.cluster.local:9090]
        controller.quota.window.num = 11
        controller.quota.window.size.seconds = 1
        controller.socket.timeout.ms = 30000
        create.topic.policy.class.name = null
        default.replication.factor = 3
        delegation.token.expiry.check.interval.ms = 3600000
        delegation.token.expiry.time.ms = 86400000
        delegation.token.master.key = null
        delegation.token.max.lifetime.ms = 604800000
        delegation.token.secret.key = null
        delete.records.purgatory.purge.interval.requests = 1
        delete.topic.enable = true
        early.start.listeners = null
        eligible.leader.replicas.enable = false
        fetch.max.bytes = 57671680
        fetch.purgatory.purge.interval.requests = 1000
        group.consumer.assignors = [org.apache.kafka.coordinator.group.assignor.UniformAssignor, org.apache.kafka.coordinator.group.assignor.RangeAssignor]
        group.consumer.heartbeat.interval.ms = 5000
        group.consumer.max.heartbeat.interval.ms = 15000
        group.consumer.max.session.timeout.ms = 60000
        group.consumer.max.size = 2147483647
        group.consumer.min.heartbeat.interval.ms = 5000
        group.consumer.min.session.timeout.ms = 45000
        group.consumer.session.timeout.ms = 45000
        group.coordinator.new.enable = false
        group.coordinator.rebalance.protocols = [classic]
        group.coordinator.threads = 1
        group.initial.rebalance.delay.ms = 3000
        group.max.session.timeout.ms = 1800000
        group.max.size = 2147483647
        group.min.session.timeout.ms = 6000
        initial.broker.registration.timeout.ms = 60000
        inter.broker.listener.name = REPLICATION-9091
        inter.broker.protocol.version = 3.7-IV4
        kafka.metrics.polling.interval.secs = 10
        kafka.metrics.reporters = []
        leader.imbalance.check.interval.seconds = 300
        leader.imbalance.per.broker.percentage = 10
        listener.security.protocol.map = CONTROLPLANE-9090:SSL,REPLICATION-9091:SSL,PLAIN-9092:PLAINTEXT,TLS-9093:SSL
        listeners = CONTROLPLANE-9090://0.0.0.0:9090,REPLICATION-9091://0.0.0.0:9091,PLAIN-9092://0.0.0.0:9092,TLS-9093://0.0.0.0:9093
        log.cleaner.backoff.ms = 15000
        log.cleaner.dedupe.buffer.size = 134217728
        log.cleaner.delete.retention.ms = 86400000
        log.cleaner.enable = true
        log.cleaner.io.buffer.load.factor = 0.9
        log.cleaner.io.buffer.size = 524288
        log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
        log.cleaner.max.compaction.lag.ms = 9223372036854775807
        log.cleaner.min.cleanable.ratio = 0.5
        log.cleaner.min.compaction.lag.ms = 0
        log.cleaner.threads = 1
        log.cleanup.policy = [delete]
        log.dir = /tmp/kafka-logs
        log.dirs = /var/lib/kafka/data/kafka-log0
        log.flush.interval.messages = 9223372036854775807
        log.flush.interval.ms = null
        log.flush.offset.checkpoint.interval.ms = 60000
        log.flush.scheduler.interval.ms = 9223372036854775807
        log.flush.start.offset.checkpoint.interval.ms = 60000
        log.index.interval.bytes = 4096
        log.index.size.max.bytes = 10485760
        log.local.retention.bytes = -2
        log.local.retention.ms = -2
        log.message.downconversion.enable = true
        log.message.format.version = 3.0-IV1
        log.message.timestamp.after.max.ms = 9223372036854775807
        log.message.timestamp.before.max.ms = 9223372036854775807
        log.message.timestamp.difference.max.ms = 9223372036854775807
        log.message.timestamp.type = CreateTime
        log.preallocate = false
        log.retention.bytes = -1
        log.retention.check.interval.ms = 300000
        log.retention.hours = 168
        log.retention.minutes = null
        log.retention.ms = null
        log.roll.hours = 168
        log.roll.jitter.hours = 0
        log.roll.jitter.ms = null
        log.roll.ms = null
        log.segment.bytes = 1073741824
        log.segment.delete.delay.ms = 60000
        max.connection.creation.rate = 2147483647
        max.connections = 2147483647
        max.connections.per.ip = 2147483647
        max.connections.per.ip.overrides = 
        max.incremental.fetch.session.cache.slots = 1000
        message.max.bytes = 1048588
        metadata.log.dir = /var/lib/kafka/data/kafka-log0
        metadata.log.max.record.bytes.between.snapshots = 20971520
        metadata.log.max.snapshot.interval.ms = 3600000
        metadata.log.segment.bytes = 1073741824
        metadata.log.segment.min.bytes = 8388608
        metadata.log.segment.ms = 604800000
        metadata.max.idle.interval.ms = 500
        metadata.max.retention.bytes = 104857600
        metadata.max.retention.ms = 604800000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        min.insync.replicas = 2
        node.id = 0
        num.io.threads = 8
        num.network.threads = 3
        num.partitions = 1
        num.recovery.threads.per.data.dir = 1
        num.replica.alter.log.dirs.threads = null
        num.replica.fetchers = 1
        offset.metadata.max.bytes = 4096
        offsets.commit.required.acks = -1
        offsets.commit.timeout.ms = 5000
        offsets.load.buffer.size = 5242880
        offsets.retention.check.interval.ms = 600000
        offsets.retention.minutes = 10080
        offsets.topic.compression.codec = 0
        offsets.topic.num.partitions = 50
        offsets.topic.replication.factor = 3
        offsets.topic.segment.bytes = 104857600
        password.encoder.cipher.algorithm = AES/CBC/PKCS5Padding
        password.encoder.iterations = 4096
        password.encoder.key.length = 128
        password.encoder.keyfactory.algorithm = null
        password.encoder.old.secret = null
        password.encoder.secret = null
        principal.builder.class = class org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilder
        process.roles = [broker, controller]
        producer.id.expiration.check.interval.ms = 600000
        producer.id.expiration.ms = 86400000
        producer.purgatory.purge.interval.requests = 1000
        queued.max.request.bytes = -1
        queued.max.requests = 500
        quota.window.num = 11
        quota.window.size.seconds = 1
        remote.log.index.file.cache.total.size.bytes = 1073741824
        remote.log.manager.task.interval.ms = 30000
        remote.log.manager.task.retry.backoff.max.ms = 30000
        remote.log.manager.task.retry.backoff.ms = 500
        remote.log.manager.task.retry.jitter = 0.2
        remote.log.manager.thread.pool.size = 10
        remote.log.metadata.custom.metadata.max.bytes = 128
        remote.log.metadata.manager.class.name = org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
        remote.log.metadata.manager.class.path = null
        remote.log.metadata.manager.impl.prefix = rlmm.config.
        remote.log.metadata.manager.listener.name = null
        remote.log.reader.max.pending.tasks = 100
        remote.log.reader.threads = 10
        remote.log.storage.manager.class.name = null
        remote.log.storage.manager.class.path = null
        remote.log.storage.manager.impl.prefix = rsm.config.
        remote.log.storage.system.enable = false
        replica.fetch.backoff.ms = 1000
        replica.fetch.max.bytes = 1048576
        replica.fetch.min.bytes = 1
        replica.fetch.response.max.bytes = 10485760
        replica.fetch.wait.max.ms = 500
        replica.high.watermark.checkpoint.interval.ms = 5000
        replica.lag.time.max.ms = 30000
        replica.selector.class = null
        replica.socket.receive.buffer.bytes = 65536
        replica.socket.timeout.ms = 30000
        replication.quota.window.num = 11
        replication.quota.window.size.seconds = 1
        request.timeout.ms = 30000
        reserved.broker.max.id = 1000
        sasl.client.callback.handler.class = null
        sasl.enabled.mechanisms = []
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.principal.to.local.rules = [DEFAULT]
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism.controller.protocol = GSSAPI
        sasl.mechanism.inter.broker.protocol = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        sasl.server.callback.handler.class = null
        sasl.server.max.receive.size = 524288
        security.inter.broker.protocol = PLAINTEXT
        security.providers = null
        server.max.startup.time.ms = 9223372036854775807
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        socket.listen.backlog.size = 50
        socket.receive.buffer.bytes = 102400
        socket.request.max.bytes = 104857600
        socket.send.buffer.bytes = 102400
        ssl.allow.dn.changes = false
        ssl.allow.san.changes = false
        ssl.cipher.suites = []
        ssl.client.auth = none
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = HTTPS
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.principal.mapping.rules = DEFAULT
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS
        telemetry.max.bytes = 1048576
        transaction.abort.timed.out.transaction.cleanup.interval.ms = 10000
        transaction.max.timeout.ms = 900000
        transaction.partition.verification.enable = true
        transaction.remove.expired.transaction.cleanup.interval.ms = 3600000
        transaction.state.log.load.buffer.size = 5242880
        transaction.state.log.min.isr = 2
        transaction.state.log.num.partitions = 50
        transaction.state.log.replication.factor = 3
        transaction.state.log.segment.bytes = 104857600
        transactional.id.expiration.ms = 604800000
        unclean.leader.election.enable = false
        unstable.api.versions.enable = false
        unstable.metadata.versions.enable = false
        zookeeper.clientCnxnSocket = null
        zookeeper.connect = null
        zookeeper.connection.timeout.ms = null
        zookeeper.max.in.flight.requests = 10
        zookeeper.metadata.migration.enable = false
        zookeeper.metadata.migration.min.batch.size = 200
        zookeeper.session.timeout.ms = 18000
        zookeeper.set.acl = false
        zookeeper.ssl.cipher.suites = null
        zookeeper.ssl.client.enable = false
        zookeeper.ssl.crl.enable = false
        zookeeper.ssl.enabled.protocols = null
        zookeeper.ssl.endpoint.identification.algorithm = HTTPS
        zookeeper.ssl.keystore.location = null
        zookeeper.ssl.keystore.password = null
        zookeeper.ssl.keystore.type = null
        zookeeper.ssl.ocsp.enable = false
        zookeeper.ssl.protocol = TLSv1.2
        zookeeper.ssl.truststore.location = null
        zookeeper.ssl.truststore.password = null
        zookeeper.ssl.truststore.type = null
 (kafka.server.KafkaConfig) [main]
2024-07-05 07:26:53,085 INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [main]
metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, dirs={/var/lib/kafka/data/kafka-log0: MetaProperties(version=1, clusterId=Vpquqx0FSJWXEEV8EoVdtg, nodeId=0, directoryId=w8k21GQ8grS8zkJJTF5gUA)})
All of the log directories are already formatted.
KRaft storage formatting is done
Removing quorum-state file
+ exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/kafka-server-start.sh /tmp/strimzi.properties
[WARN  tini (69149)] Tini is not running as PID 1 and isn't registered as a child subreaper.
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
2024-07-05 07:26:54,804 INFO Starting KafkaAgent with brokerReadyFile=null, sessionConnectedFile=null, sslKeyStorePath=/tmp/kafka/cluster.keystore.p12, sslTrustStore=/tmp/kafka/cluster.truststore.p12 (io.strimzi.kafka.agent.KafkaAgent) [main]
2024-07-05 07:26:54,856 INFO Logging initialized @488ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [main]
2024-07-05 07:26:54,981 INFO jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.11+9-LTS (org.eclipse.jetty.server.Server) [main]
2024-07-05 07:26:55,025 INFO Started o.e.j.s.h.ContextHandler@23fe1d71{/v1/broker-state,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2024-07-05 07:26:55,025 INFO Started o.e.j.s.h.ContextHandler@74a10858{/v1/ready,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2024-07-05 07:26:55,025 INFO Started o.e.j.s.h.ContextHandler@28ac3dc3{/v1/kraft-migration,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2024-07-05 07:26:55,042 ERROR Could not start the server for broker state:  (io.strimzi.kafka.agent.KafkaAgent) [main]
MultiException[java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443, java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080]
        at org.eclipse.jetty.util.MultiException.ifExceptionThrow(MultiException.java:122)
        at org.eclipse.jetty.server.Server.doStart(Server.java:413)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at io.strimzi.kafka.agent.KafkaAgent.startHttpServer(KafkaAgent.java:260)
        at io.strimzi.kafka.agent.KafkaAgent.run(KafkaAgent.java:131)
        at io.strimzi.kafka.agent.KafkaAgent.premain(KafkaAgent.java:490)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:493)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
        Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
                at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
                at org.eclipse.jetty.server.Server.doStart(Server.java:401)
                ... 10 more
        Caused by: java.net.BindException: Address already in use
                at java.base/sun.nio.ch.Net.bind0(Native Method)
                at java.base/sun.nio.ch.Net.bind(Net.java:555)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
                at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
                ... 15 more
        Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
                at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
                at org.eclipse.jetty.server.Server.doStart(Server.java:401)
                ... 10 more
        Caused by: java.net.BindException: Address already in use
                at java.base/sun.nio.ch.Net.bind0(Native Method)
                at java.base/sun.nio.ch.Net.bind(Net.java:555)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
                at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
                ... 15 more
Caused by: [CIRCULAR REFERENCE: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443]
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:493)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503)
Caused by: java.lang.RuntimeException: MultiException[java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443, java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080]
        at io.strimzi.kafka.agent.KafkaAgent.run(KafkaAgent.java:134)
        at io.strimzi.kafka.agent.KafkaAgent.premain(KafkaAgent.java:490)
        ... 6 more
Caused by: MultiException[java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443, java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080]
        at org.eclipse.jetty.util.MultiException.ifExceptionThrow(MultiException.java:122)
        at org.eclipse.jetty.server.Server.doStart(Server.java:413)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
        at io.strimzi.kafka.agent.KafkaAgent.startHttpServer(KafkaAgent.java:260)
        at io.strimzi.kafka.agent.KafkaAgent.run(KafkaAgent.java:131)
        ... 7 more
        Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
                at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
                at org.eclipse.jetty.server.Server.doStart(Server.java:401)
                ... 10 more
        Caused by: java.net.BindException: Address already in use
                at java.base/sun.nio.ch.Net.bind0(Native Method)
                at java.base/sun.nio.ch.Net.bind(Net.java:555)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
                at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
                ... 15 more
        Suppressed: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8080
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:349)
                at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:310)
                at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
                at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234)
                at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
                at org.eclipse.jetty.server.Server.doStart(Server.java:401)
                ... 10 more
        Caused by: java.net.BindException: Address already in use
                at java.base/sun.nio.ch.Net.bind0(Native Method)
                at java.base/sun.nio.ch.Net.bind(Net.java:555)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.netBind(ServerSocketChannelImpl.java:337)
                at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:294)
                at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:89)
                at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344)
                ... 15 more
Caused by: [CIRCULAR REFERENCE: java.io.IOException: Failed to bind to 0.0.0.0/0.0.0.0:8443]
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message Outstanding error when calling method in invokeJavaAgentMainMethod at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 619
*** java.lang.instrument ASSERTION FAILED ***: "success" with message invokeJavaAgentMainMethod failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 459
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x8d0b85]  jni_FatalError+0xc5
V  [libjvm.so+0xa366ad]  JvmtiExport::post_vm_initialized()+0x14d
V  [libjvm.so+0xecc157]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x847
V  [libjvm.so+0x8ec1a5]  JNI_CreateJavaVM+0x65
C  [libjli.so+0x8f7d]  JavaMain+0x9d
C  [libjli.so+0xd3dd]  ThreadJavaMain+0xd

[scholzj]

That is strange. ot sure wh would ports 8080 and 8443 be already in use. You should maybe check that you don't have any strange tooling inject it self into your pod as agent or sidecar etc.

[lng-sys]

You can for a start see if ephemeral storage helps or not to eliminate it as a possibility.

With ephemeral storage it works indeed. I didn't know Kafka doesn't play well with NFS. I will look for alternatives that are not public cloud services and eventually check if ephemeral storage is enough for my use case. I will update the title of the thread mentioning NFS and close it. Thank you very much for you help!

[scholzj]

The ephemeral storage is good for some development or CIs only. Not for any production type use-cases I'm afraid. If you run somewhere on-premise, Local persistent volumes might be an option.

[lng-sys]

@scholzj hi, I'm trying to use local persistent volumes but I don't understand how to use those with Strimzi, is there an updated guide? Thank you

[scholzj]

I do not have any guide. In general, you either create a storage class without provisioner, create manually the PVs and have Kubernetes bind them together. I think the Stirmzi blog has an old blog post about this. Or you use some Local Storage provisioner that will do it automatically for you.

[lng-sys]

I finally succeeded to start the Pods with persistent local storage and it gave me the same error I had with NFS (broker didn't start and so the readiness script return error). This happened yesterday while today I just checked the status of the Pods and they seem to work without any further intervention 🤔