[Bug]: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2457 ms, time limit is 2000 ms #9715

udleinati · 2024-02-20T14:34:21Z

udleinati
Feb 20, 2024

Bug Description

I'm installing strimzi using helm on my k8s. At the beggining everything was working fine, after while it just stopped working. It gets a timeout error.

The error starts after creating a KafkaConnect.

Steps to reproduce

Install strimzi using helm. Version 0.39.0 - Just one line of configuration: replica: 1

Expected behavior

No response

Strimzi version

0.39.0

Kubernetes version

1.29

Installation method

Helm Chart

Infrastructure

Amazon EKS - Operator POD

Configuration files and logs

Auto-detected KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
+ exec /usr/bin/tini -w -e 143 -- java -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20 -classpath :lib/io.strimzi.cluster-operator-0.39.0.jar:lib/io.strimzi.api-0.39.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.2.jar:lib/io.strimzi.config-model-0.39.0.jar:lib/io.strimzi.operator-common-0.39.0.jar:lib/io.strimzi.crd-annotations-0.39.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.2.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.2.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.2.jar:lib/io.fabric8.openshift-model-operator-6.9.2.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.2.jar:lib/io.fabric8.openshift-model-machine-6.9.2.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.2.jar:lib/io.fabric8.openshift-model-monitoring-6.9.2.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.2.jar:lib/io.fabric8.openshift-model-tuned-6.9.2.jar:lib/io.fabric8.openshift-model-console-6.9.2.jar:lib/io.fabric8.openshift-model-config-6.9.2.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.2.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.2.jar:lib/io.fabric8.openshift-model-hive-6.9.2.jar:lib/io.fabric8.openshift-model-installer-6.9.2.jar:lib/io.fabric8.kubernetes-client-6.9.2.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:lib/io.fabric8.kubernetes-client-api-6.9.2.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.2.jar:lib/io.fabric8.kubernetes-model-resource-6.9.2.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.2.jar:lib/io.fabric8.kubernetes-model-batch-6.9.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.2.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.2.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.2.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.2.jar:lib/io.fabric8.kubernetes-model-node-6.9.2.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.3.jar:lib/io.fabric8.kubernetes-model-core-6.9.2.jar:lib/io.fabric8.kubernetes-model-common-6.9.2.jar:lib/io.fabric8.kubernetes-model-events-6.9.2.jar:lib/io.fabric8.kubernetes-model-policy-6.9.2.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.2.jar:lib/io.fabric8.kubernetes-model-apps-6.9.2.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.2.jar:lib/io.fabric8.kubernetes-model-networking-6.9.2.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.2.jar:lib/io.fabric8.openshift-model-6.9.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.3.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.3.jar:lib/org.yaml.snakeyaml-2.1.jar:lib/io.vertx.vertx-core-4.5.0.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.39.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.1.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.5.jar:lib/org.apache.kafka.kafka-server-common-3.6.1.jar:lib/com.yammer.metrics.metrics-core-2.2.0.jar:lib/net.sf.jopt-simple.jopt-simple-5.0.4.jar:lib/org.pcollections.pcollections-4.0.1.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.5.0.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar io.strimzi.operator.cluster.Main
2024-02-20 14:13:20 INFO  Main:72 - ClusterOperator 0.39.0 is starting
2024-02-20 14:13:20 INFO  Util:239 - Using config:
	PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
	container: oci
	STRIMZI_DEFAULT_TOPIC_OPERATOR_IMAGE: quay.io/strimzi/operator:0.39.0
	JAVA_OPTS:  -Dlog4j2.configurationFile=file:/opt/strimzi/custom-config/log4j2.properties -Dvertx.cacheDirBase=/tmp/vertx-cache -Djava.security.egd=file:/dev/./urandom  -XX:+ExitOnOutOfMemoryError -XX:MinRAMPercentage=15 -XX:MaxRAMPercentage=20
	STRIMZI_HOME: /opt/strimzi
	PWD: /opt/strimzi
	STRIMZI_LEADER_ELECTION_IDENTITY: strimzi-cluster-operator-7bb5468c59-jmvc9
	KUBERNETES_PORT_443_TCP: tcp://172.20.0.1:443
	JAVA_MAIN: io.strimzi.operator.cluster.Main
	STRIMZI_LEADER_ELECTION_ENABLED: true
	STRIMZI_VERSION: 0.39.0
	STRIMZI_NAMESPACE: strimzi
	STRIMZI_DEFAULT_KANIKO_EXECUTOR_IMAGE: quay.io/strimzi/kaniko-executor:0.39.0
	TINI_SHA256_S390X: 931b70a182af879ca249ae9de87ef68423121b38d235c78997fafc680ceab32d
	STRIMZI_DEFAULT_MAVEN_BUILDER: quay.io/strimzi/maven-builder:0.39.0
	STRIMZI_FEATURE_GATES:
	TINI_SHA256_AMD64: 93dcc18adc78c65a028a84799ecf8ad40c936fdfc5f2a57b1acda5a8117fa82c
	KUBERNETES_SERVICE_DNS_DOMAIN: cluster.local
	STRIMZI_LEADER_ELECTION_LEASE_NAME: strimzi-cluster-operator
	KUBERNETES_SERVICE_PORT_HTTPS: 443
	SHLVL: 0
	STRIMZI_DEFAULT_KAFKA_BRIDGE_IMAGE: quay.io/strimzi/kafka-bridge:0.27.0
	KUBERNETES_PORT: tcp://172.20.0.1:443
	STRIMZI_KAFKA_MIRROR_MAKER_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	JAVA_HOME: /usr/lib/jvm/jre-17
	STRIMZI_FULL_RECONCILIATION_INTERVAL_MS: 120000
	KUBERNETES_SERVICE_HOST: 172.20.0.1
	JAVA_CLASSPATH: :lib/io.strimzi.cluster-operator-0.39.0.jar:lib/io.strimzi.api-0.39.0.jar:lib/io.fabric8.kubernetes-model-apiextensions-6.9.2.jar:lib/io.strimzi.config-model-0.39.0.jar:lib/io.strimzi.operator-common-0.39.0.jar:lib/io.strimzi.crd-annotations-0.39.0.jar:lib/org.quartz-scheduler.quartz-2.3.2.jar:lib/org.eclipse.jetty.jetty-server-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-http-9.4.53.v20231009.jar:lib/org.eclipse.jetty.jetty-io-9.4.53.v20231009.jar:lib/javax.servlet.javax.servlet-api-3.1.0.jar:lib/io.fabric8.openshift-client-6.9.2.jar:lib/com.github.mifmif.generex-1.0.2.jar:lib/dk.brics.automaton.automaton-1.11-8.jar:lib/io.fabric8.openshift-client-api-6.9.2.jar:lib/io.fabric8.openshift-model-clusterautoscaling-6.9.2.jar:lib/io.fabric8.openshift-model-operator-6.9.2.jar:lib/io.fabric8.openshift-model-operatorhub-6.9.2.jar:lib/io.fabric8.openshift-model-machine-6.9.2.jar:lib/io.fabric8.openshift-model-whereabouts-6.9.2.jar:lib/io.fabric8.openshift-model-monitoring-6.9.2.jar:lib/io.fabric8.openshift-model-storageversionmigrator-6.9.2.jar:lib/io.fabric8.openshift-model-tuned-6.9.2.jar:lib/io.fabric8.openshift-model-console-6.9.2.jar:lib/io.fabric8.openshift-model-config-6.9.2.jar:lib/io.fabric8.openshift-model-machineconfig-6.9.2.jar:lib/io.fabric8.openshift-model-miscellaneous-6.9.2.jar:lib/io.fabric8.openshift-model-hive-6.9.2.jar:lib/io.fabric8.openshift-model-installer-6.9.2.jar:lib/io.fabric8.kubernetes-client-6.9.2.jar:lib/io.fabric8.kubernetes-httpclient-jdk-6.9.2.jar:lib/io.fabric8.kubernetes-client-api-6.9.2.jar:lib/io.fabric8.kubernetes-model-gatewayapi-6.9.2.jar:lib/io.fabric8.kubernetes-model-resource-6.9.2.jar:lib/io.fabric8.kubernetes-model-admissionregistration-6.9.2.jar:lib/io.fabric8.kubernetes-model-autoscaling-6.9.2.jar:lib/io.fabric8.kubernetes-model-batch-6.9.2.jar:lib/io.fabric8.kubernetes-model-certificates-6.9.2.jar:lib/io.fabric8.kubernetes-model-discovery-6.9.2.jar:lib/io.fabric8.kubernetes-model-extensions-6.9.2.jar:lib/io.fabric8.kubernetes-model-flowcontrol-6.9.2.jar:lib/io.fabric8.kubernetes-model-metrics-6.9.2.jar:lib/io.fabric8.kubernetes-model-scheduling-6.9.2.jar:lib/io.fabric8.kubernetes-model-node-6.9.2.jar:lib/org.slf4j.slf4j-api-1.7.36.jar:lib/org.snakeyaml.snakeyaml-engine-2.7.jar:lib/com.fasterxml.jackson.datatype.jackson-datatype-jsr310-2.15.3.jar:lib/io.fabric8.kubernetes-model-core-6.9.2.jar:lib/io.fabric8.kubernetes-model-common-6.9.2.jar:lib/io.fabric8.kubernetes-model-events-6.9.2.jar:lib/io.fabric8.kubernetes-model-policy-6.9.2.jar:lib/io.fabric8.kubernetes-model-rbac-6.9.2.jar:lib/io.fabric8.kubernetes-model-apps-6.9.2.jar:lib/io.fabric8.kubernetes-model-storageclass-6.9.2.jar:lib/io.fabric8.kubernetes-model-networking-6.9.2.jar:lib/io.fabric8.kubernetes-model-coordination-6.9.2.jar:lib/io.fabric8.openshift-model-6.9.2.jar:lib/io.fabric8.zjsonpatch-0.3.0.jar:lib/com.fasterxml.jackson.core.jackson-core-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-databind-2.15.3.jar:lib/com.fasterxml.jackson.core.jackson-annotations-2.15.3.jar:lib/com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.15.3.jar:lib/org.yaml.snakeyaml-2.1.jar:lib/io.vertx.vertx-core-4.5.0.jar:lib/io.netty.netty-common-4.1.100.Final.jar:lib/io.netty.netty-buffer-4.1.100.Final.jar:lib/io.netty.netty-handler-4.1.100.Final.jar:lib/io.netty.netty-handler-proxy-4.1.100.Final.jar:lib/io.netty.netty-codec-http-4.1.100.Final.jar:lib/io.netty.netty-codec-http2-4.1.100.Final.jar:lib/io.netty.netty-resolver-4.1.100.Final.jar:lib/io.netty.netty-resolver-dns-4.1.100.Final.jar:lib/io.netty.netty-codec-dns-4.1.100.Final.jar:lib/io.strimzi.certificate-manager-0.39.0.jar:lib/org.apache.logging.log4j.log4j-core-2.17.2.jar:lib/org.apache.logging.log4j.log4j-api-2.17.2.jar:lib/org.apache.logging.log4j.log4j-slf4j-impl-2.17.2.jar:lib/io.strimzi.kafka-oauth-server-0.14.0.jar:lib/io.strimzi.kafka-oauth-server-plain-0.14.0.jar:lib/io.strimzi.kafka-oauth-client-0.14.0.jar:lib/io.strimzi.kafka-oauth-common-0.14.0.jar:lib/com.nimbusds.nimbus-jose-jwt-9.31.jar:lib/com.github.stephenc.jcip.jcip-annotations-1.0-1.jar:lib/com.jayway.jsonpath.json-path-2.8.0.jar:lib/net.minidev.json-smart-2.4.10.jar:lib/net.minidev.accessors-smart-2.4.9.jar:lib/org.eclipse.jetty.jetty-util-9.4.53.v20231009.jar:lib/org.apache.kafka.kafka-clients-3.6.1.jar:lib/com.github.luben.zstd-jni-1.5.5-1.jar:lib/org.lz4.lz4-java-1.8.0.jar:lib/org.xerial.snappy.snappy-java-1.1.10.5.jar:lib/org.apache.kafka.kafka-server-common-3.6.1.jar:lib/com.yammer.metrics.metrics-core-2.2.0.jar:lib/net.sf.jopt-simple.jopt-simple-5.0.4.jar:lib/org.pcollections.pcollections-4.0.1.jar:lib/org.apache.zookeeper.zookeeper-3.8.3.jar:lib/org.apache.yetus.audience-annotations-0.12.0.jar:lib/commons-io.commons-io-2.11.0.jar:lib/org.apache.zookeeper.zookeeper-jute-3.8.3.jar:lib/io.netty.netty-transport-4.1.100.Final.jar:lib/io.netty.netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/io.netty.netty-transport-native-unix-common-4.1.100.Final.jar:lib/io.netty.netty-transport-classes-epoll-4.1.100.Final.jar:lib/io.micrometer.micrometer-core-1.9.5.jar:lib/org.hdrhistogram.HdrHistogram-2.1.12.jar:lib/org.latencyutils.LatencyUtils-2.0.3.jar:lib/io.micrometer.micrometer-registry-prometheus-1.9.5.jar:lib/io.prometheus.simpleclient_common-0.15.0.jar:lib/io.prometheus.simpleclient-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_common-0.15.0.jar:lib/io.prometheus.simpleclient_tracer_otel_agent-0.15.0.jar:lib/io.vertx.vertx-micrometer-metrics-4.5.0.jar:lib/io.netty.netty-codec-4.1.100.Final.jar:lib/io.netty.netty-codec-socks-4.1.100.Final.jar
	STRIMZI_DEFAULT_CRUISE_CONTROL_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	TINI_VERSION: v0.19.0
	STRIMZI_OPERATION_TIMEOUT_MS: 300000
	KUBERNETES_PORT_443_TCP_ADDR: 172.20.0.1
	STRIMZI_LEADER_ELECTION_LEASE_NAMESPACE: strimzi
	STRIMZI_KAFKA_MIRROR_MAKER_2_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	KUBERNETES_PORT_443_TCP_PROTO: tcp
	STRIMZI_DEFAULT_USER_OPERATOR_IMAGE: quay.io/strimzi/operator:0.39.0
	KUBERNETES_SERVICE_PORT: 443
	STRIMZI_DEFAULT_KAFKA_EXPORTER_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	TINI_SHA256_ARM64: 07952557df20bfd2a95f9bef198b445e006171969499a1d361bd9e6f8e5e0e81
	STRIMZI_DEFAULT_KAFKA_INIT_IMAGE: quay.io/strimzi/operator:0.39.0
	TINI_SHA256_PPC64LE: 3f658420974768e40810001a038c29d003728c5fe86da211cff5059e48cfdfde
	HOSTNAME: strimzi-cluster-operator-7bb5468c59-jmvc9
	STRIMZI_KAFKA_CONNECT_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	STRIMZI_KAFKA_IMAGES: 3.5.0=quay.io/strimzi/kafka:0.39.0-kafka-3.5.0
3.5.1=quay.io/strimzi/kafka:0.39.0-kafka-3.5.1
3.5.2=quay.io/strimzi/kafka:0.39.0-kafka-3.5.2
3.6.0=quay.io/strimzi/kafka:0.39.0-kafka-3.6.0
3.6.1=quay.io/strimzi/kafka:0.39.0-kafka-3.6.1

	KUBERNETES_PORT_443_TCP_PORT: 443
	STRIMZI_DEFAULT_TLS_SIDECAR_ENTITY_OPERATOR_IMAGE: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1
	STRIMZI_OPERATOR_NAME: strimzi-cluster-operator-7bb5468c59-jmvc9
	HOME: /home/strimzi
	MALLOC_ARENA_MAX: 2
	STRIMZI_OPERATOR_NAMESPACE: strimzi

2024-02-20 14:13:21 INFO  Main:75 - Cluster Operator configuration is ClusterOperatorConfig{
	namespaces='[strimzi]'
	reconciliationIntervalMs=120000
	operationTimeoutMs=300000
	connectBuildTimeoutMs=300000
	createClusterRoles=false
	networkPolicyGeneration=true
	versions='versions{3.5.0={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.0}, 3.5.1={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.1}, 3.5.2={proto: 3.5 msg: 3.5 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.5.2}, 3.6.0={proto: 3.6 msg: 3.6 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.0}, 3.6.1={proto: 3.6 msg: 3.6 kafka-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 connect-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 mirrormaker-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 mirrormaker2-image: quay.io/strimzi/kafka:0.39.0-kafka-3.6.1}}'
	imagePullPolicy='null'
	imagePullSecrets='null'
	operatorNamespace='strimzi'
	operatorNamespaceLabels='null'
	customResourceSelector='null'
	featureGates='FeatureGates(UseKRaft=false,KafkaNodePools=true,UnidirectionalTopicOperator=true)'
	zkAdminSessionTimeoutMs=10000
	dnsCacheTtlSec=30
	podSetReconciliationOnly=false
	podSetControllerWorkQueueSize=1024
	operatorName='strimzi-cluster-operator-7bb5468c59-jmvc9'
	podSecurityProviderClass='io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider'
	leaderElectionConfig='LeaderElectionConfig{leaseName='strimzi-cluster-operator', namespace='strimzi', identity='strimzi-cluster-operator-7bb5468c59-jmvc9', leaseDuration=PT15S, renewDeadline=PT10S, retryPeriod=PT2S}'}
2024-02-20 14:13:22 INFO  Main:345 - Health and metrics server is ready on port 8080)
2024-02-20 14:13:22 INFO  Main:255 - Waiting to become a leader
2024-02-20 14:13:22 INFO  LeaderElectionManager:69 - Starting the Leader Elector
2024-02-20 14:13:23 INFO  LeaderElectionManager:121 - The new leader is strimzi-cluster-operator-7bb5468c59-jmvc9
2024-02-20 14:13:23 INFO  LeaderElectionManager:103 - Started being a leader
2024-02-20 14:13:23 INFO  Main:237 - I'm the new leader
2024-02-20 14:13:23 WARN  PlatformFeaturesAvailability:156 - API Group route.openshift.io is not supported
2024-02-20 14:13:23 WARN  PlatformFeaturesAvailability:156 - API Group build.openshift.io is not supported
2024-02-20 14:13:23 WARN  PlatformFeaturesAvailability:156 - API Group image.openshift.io is not supported
2024-02-20 14:13:23 INFO  Main:124 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.29,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2024-02-20 14:13:24 INFO  PodSecurityProviderFactory:43 - Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-20 14:13:24 INFO  PodSecurityProviderFactory:62 - Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2024-02-20 14:13:24 INFO  ClusterOperator:86 - Creating ClusterOperator for namespace strimzi
2024-02-20 14:13:24 INFO  ClusterOperator:100 - Starting ClusterOperator for namespace strimzi
2024-02-20 14:13:24 INFO  StrimziPodSetController:583 - Starting the StrimziPodSet controller
2024-02-20 14:13:24 INFO  StrimziPodSetController:552 - Starting StrimziPodSet controller for namespace strimzi
2024-02-20 14:13:24 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2024-02-20 14:13:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2024-02-20 14:13:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2024-02-20 14:13:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2024-02-20 14:13:24 INFO  StrimziPodSetController:555 - Waiting for informers to sync
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for Kafka operator
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker operator
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for KafkaConnect operator
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for KafkaBridge operator
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for KafkaMirrorMaker2 operator
2024-02-20 14:13:24 INFO  ClusterOperator:114 - Opened watch for KafkaRebalance operator
2024-02-20 14:13:25 INFO  ClusterOperator:123 - Opened watch for KafkaNodePool operator
2024-02-20 14:13:25 INFO  ClusterOperator:131 - Opened watch for KafkaConnector operator
2024-02-20 14:13:25 INFO  ClusterOperator:139 - Setting up periodic reconciliation for namespace strimzi
2024-02-20 14:13:25 INFO  Main:206 - Cluster Operator verticle started in namespace strimzi without label selector
2024-02-20 14:13:26 INFO  StrimziPodSetController:560 - Informers are in-sync
2024-02-20 14:14:24 INFO  AbstractOperator:511 - Reconciliation #1(watch) KafkaConnect(strimzi/strimzi-debezium-cluster): KafkaConnect strimzi-debezium-cluster in namespace strimzi was ADDED
2024-02-20 14:14:24 INFO  AbstractOperator:265 - Reconciliation #1(watch) KafkaConnect(strimzi/strimzi-debezium-cluster): KafkaConnect strimzi-debezium-cluster will be checked for creation or modification
2024-02-20 14:14:27 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2457 ms, time limit is 2000 ms
2024-02-20 14:14:28 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 3457 ms, time limit is 2000 ms
2024-02-20 14:14:29 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 4457 ms, time limit is 2000 ms
2024-02-20 14:14:30 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 5457 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
	at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) ~[?:?]
	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864) ~[?:?]
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465) ~[?:?]
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436) ~[?:?]
	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072) ~[?:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:491) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:753) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:97) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1142) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:97) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator.internalCreate(AbstractNamespacedResourceOperator.java:271) ~[io.strimzi.operator-common-0.39.0.jar:0.39.0]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator.lambda$reconcile$0(AbstractNamespacedResourceOperator.java:105) ~[io.strimzi.operator-common-0.39.0.jar:0.39.0]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator$$Lambda$824/0x0000000100713948.apply(Unknown Source) ~[?:?]
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:60) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
	at io.vertx.core.impl.future.FutureBase$$Lambda$386/0x0000000100486338.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]

KafkaConnect YAML:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: strimzi-debezium-cluster
spec:
  version: 3.5.0
  image: 000.dkr.ecr.us-east-1.amazonaws.com/custom-kafka-connect:0.0.1
  replicas: 1
  bootstrapServers: b-1.devqacore.xxx.kafka.us-east-1.amazonaws.com:9092
  config:
    group.id: 2
    offset.storage.topic: connect-cluster-offsets
    config.storage.topic: connect-cluster-configs
    status.storage.topic: connect-cluster-status
    config.storage.replication.factor: 2
    offset.storage.replication.factor: 2
    status.storage.replication.factor: 2

Additional context

No response

scholzj · 2024-02-20T14:53:29Z

scholzj
Feb 20, 2024
Maintainer

That exception on its own just means that something is taking more time than it should. It does not really reveal any root cause which can be basically anything from unresponsive operands, insufficient resources for the operator or the operand, or in this case it could be also that the Kubernetes API server is not responding for some reason (or some admission controller in your Kubernetes cluster blocking the operation) etc. Having a DEBUG log might reveal more details about where exactly the issue is.

0 replies

udleinati · 2024-02-20T16:45:09Z

udleinati
Feb 20, 2024
Author

@scholzj

After changing the logLevel to TRACE (logLevel: TRACE for those using Helm), I encountered the following error

2024-02-20 16:38:12 TRACE HttpLoggingInterceptor:182 - -HTTP END-
2024-02-20 16:38:12 DEBUG StandardHttpClient:183 - HTTP operation on url: https://172.20.0.1:443/api/v1/namespaces/strimzi/services should be retried after 3200 millis because of IOException
java.net.http.HttpTimeoutException: request timed out
	at jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:63) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899) ~[java.net.http:?]
2024-02-20 16:38:13 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 63889 ms, time limit is 2000 ms

I accessed the operator POD and ran a curl command to access the same endpoint, resulting in the following output:

{
  "kind": "ServiceList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "4825213"
  },
  "items": []
}

Do you have any idea what might be causing this issue?

2 replies

udleinati Feb 20, 2024
Author

For more context, let me share more from the log:

2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:176 - -HTTP START-
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:141 - > GET https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/strimzi/leases/strimzi-cluster-operator
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:142 - > Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJjYjk3YTU3ZWM3ZTA5NDg2YmEwMjE2YjMyMmQ5OWQzNjhiZTM0NDUifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczOTk4MzU3NCwiaWF0IjoxNzA4NDQ3NTc0LCJpc3MiOiJodHRwczovL29pZGMuZWtzLnVzLWVhc3QtMS5hbWF6b25hd3MuY29tL2lkL0JDQ0MwQjM4Q0QwRkJCRkI1ODdFNDgwRDA2NUNFOTZGIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJzdHJpbXppIiwicG9kIjp7Im5hbWUiOiJzdHJpbXppLWNsdXN0ZXItb3BlcmF0b3ItN2JiNTQ2OGM1OS13ZGZ6cSIsInVpZCI6IjM1ZmYyM2ZkLWRmYWUtNDg4MC04ZWI1LWU3YzlhY2YwOTU4MiJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoic3RyaW16aS1jbHVzdGVyLW9wZXJhdG9yIiwidWlkIjoiYTA1NWJmMTMtNGNlYi00ZDNiLTg2MjgtYmRkOTVlZjY0OTNiIn0sIndhcm5hZnRlciI6MTcwODQ1MTE4MX0sIm5iZiI6MTcwODQ0NzU3NCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN0cmltemk6c3RyaW16aS1jbHVzdGVyLW9wZXJhdG9yIn0.Tpsbr5DFkXXApGJeogpDiQT1o282cKUadkz4-t7k9h7Z1unUMjuDSV45wo4Jj4adbg-rLqFvZwrzCzRTPnwU1q7fXF5hp9hHkiHc1BRjd4PTW2wMARaWa-dkulhYYTzncsQKvmbcWD5Z4Kw0xwco7xR4l_g-5adWRujGNEDTn0rqrj3RY7pbUP3zgm1rvdy5YXovkUrEVuf91LSq_0ED7QyM0SrQs4IJbEpbx5WFbkA_QE61SRLHy1La15kZWMNEG7kZJbOSztN54SC2pty-YTo2NxAj4Lu4hXDQOcO2vMhlJK9k7YwGodAfmfKz20oD19UF2-LXAS_CF_INafmgHA
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:142 - > User-Agent: strimzi-cluster-operator/0.39.0
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:151 - < 200 OK
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < :status: 200
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < audit-id: 927c8d48-b824-44dc-b89d-0db84ffab0f7
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < cache-control: no-cache, private
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < content-length: 747
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < content-type: application/json
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < date: Tue, 20 Feb 2024 16:46:43 GMT
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < x-kubernetes-pf-flowschema-uid: 15dcbc39-5319-4e5c-9b2b-907288c07240
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < x-kubernetes-pf-prioritylevel-uid: bbca2296-02ec-434d-a91d-6be0cffa5a7a
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:170 - {"kind":"Lease","apiVersion":"coordination.k8s.io/v1","metadata":{"name":"strimzi-cluster-operator","namespace":"strimzi","uid":"2f85abe0-85a8-4fff-b12a-eb0f947ce380","resourceVersion":"4827417","creationTimestamp":"2024-02-20T16:12:25Z","managedFields":[{"manager":"strimzi-cluster-operator","operation":"Update","apiVersion":"coordination.k8s.io/v1","time":"2024-02-20T16:46:41Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:acquireTime":{},"f:holderIdentity":{},"f:leaseDurationSeconds":{},"f:leaseTransitions":{},"f:renewTime":{}}}}]},"spec":{"holderIdentity":"strimzi-cluster-operator-7bb5468c59-wdfzq","leaseDurationSeconds":15,"acquireTime":"2024-02-20T16:46:27.409396Z","renewTime":"2024-02-20T16:46:41.499247Z","leaseTransitions":6}}

2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:182 - -HTTP END-
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:176 - -HTTP START-
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:141 - > PATCH https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/strimzi/leases/strimzi-cluster-operator
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:142 - > Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJjYjk3YTU3ZWM3ZTA5NDg2YmEwMjE2YjMyMmQ5OWQzNjhiZTM0NDUifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6MTczOTk4MzU3NCwiaWF0IjoxNzA4NDQ3NTc0LCJpc3MiOiJodHRwczovL29pZGMuZWtzLnVzLWVhc3QtMS5hbWF6b25hd3MuY29tL2lkL0JDQ0MwQjM4Q0QwRkJCRkI1ODdFNDgwRDA2NUNFOTZGIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJzdHJpbXppIiwicG9kIjp7Im5hbWUiOiJzdHJpbXppLWNsdXN0ZXItb3BlcmF0b3ItN2JiNTQ2OGM1OS13ZGZ6cSIsInVpZCI6IjM1ZmYyM2ZkLWRmYWUtNDg4MC04ZWI1LWU3YzlhY2YwOTU4MiJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoic3RyaW16aS1jbHVzdGVyLW9wZXJhdG9yIiwidWlkIjoiYTA1NWJmMTMtNGNlYi00ZDNiLTg2MjgtYmRkOTVlZjY0OTNiIn0sIndhcm5hZnRlciI6MTcwODQ1MTE4MX0sIm5iZiI6MTcwODQ0NzU3NCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN0cmltemk6c3RyaW16aS1jbHVzdGVyLW9wZXJhdG9yIn0.Tpsbr5DFkXXApGJeogpDiQT1o282cKUadkz4-t7k9h7Z1unUMjuDSV45wo4Jj4adbg-rLqFvZwrzCzRTPnwU1q7fXF5hp9hHkiHc1BRjd4PTW2wMARaWa-dkulhYYTzncsQKvmbcWD5Z4Kw0xwco7xR4l_g-5adWRujGNEDTn0rqrj3RY7pbUP3zgm1rvdy5YXovkUrEVuf91LSq_0ED7QyM0SrQs4IJbEpbx5WFbkA_QE61SRLHy1La15kZWMNEG7kZJbOSztN54SC2pty-YTo2NxAj4Lu4hXDQOcO2vMhlJK9k7YwGodAfmfKz20oD19UF2-LXAS_CF_INafmgHA
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:142 - > User-Agent: strimzi-cluster-operator/0.39.0
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:144 - {"apiVersion":"coordination.k8s.io/v1","kind":"Lease","metadata":{"name":"strimzi-cluster-operator","namespace":"strimzi","resourceVersion":"4827417"},"spec":{"acquireTime":"2024-02-20T16:46:27.409396Z","holderIdentity":"strimzi-cluster-operator-7bb5468c59-wdfzq","leaseDurationSeconds":15,"leaseTransitions":6,"renewTime":"2024-02-20T16:46:43.499375Z"}}
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:151 - < 200 OK
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < :status: 200
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < audit-id: 8e9a5955-0ded-4e55-a1db-ca7f676c0d94
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < cache-control: no-cache, private
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < content-length: 747
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < content-type: application/json
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < date: Tue, 20 Feb 2024 16:46:43 GMT
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < x-kubernetes-pf-flowschema-uid: 15dcbc39-5319-4e5c-9b2b-907288c07240
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:152 - < x-kubernetes-pf-prioritylevel-uid: bbca2296-02ec-434d-a91d-6be0cffa5a7a
2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:170 - {"kind":"Lease","apiVersion":"coordination.k8s.io/v1","metadata":{"name":"strimzi-cluster-operator","namespace":"strimzi","uid":"2f85abe0-85a8-4fff-b12a-eb0f947ce380","resourceVersion":"4827430","creationTimestamp":"2024-02-20T16:12:25Z","managedFields":[{"manager":"strimzi-cluster-operator","operation":"Update","apiVersion":"coordination.k8s.io/v1","time":"2024-02-20T16:46:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:acquireTime":{},"f:holderIdentity":{},"f:leaseDurationSeconds":{},"f:leaseTransitions":{},"f:renewTime":{}}}}]},"spec":{"holderIdentity":"strimzi-cluster-operator-7bb5468c59-wdfzq","leaseDurationSeconds":15,"acquireTime":"2024-02-20T16:46:27.409396Z","renewTime":"2024-02-20T16:46:43.499375Z","leaseTransitions":6}}

2024-02-20 16:46:43 TRACE HttpLoggingInterceptor:182 - -HTTP END-
2024-02-20 16:46:43 DEBUG StandardHttpClient:183 - HTTP operation on url: https://172.20.0.1:443/api/v1/namespaces/strimzi/services should be retried after 100 millis because of IOException
java.net.http.HttpTimeoutException: request timed out
	at jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:63) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270) ~[java.net.http:?]
	at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899) ~[java.net.http:?]
2024-02-20 16:46:44 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 10592 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
	at jdk.internal.misc.Unsafe.park(Native Method) ~[?:?]
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:211) ~[?:?]
	at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864) ~[?:?]
	at java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3465) ~[?:?]
	at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3436) ~[?:?]
	at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072) ~[?:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:491) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:340) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:753) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:97) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1142) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:97) ~[io.fabric8.kubernetes-client-6.9.2.jar:?]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator.internalCreate(AbstractNamespacedResourceOperator.java:271) ~[io.strimzi.operator-common-0.39.0.jar:0.39.0]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator.lambda$reconcile$0(AbstractNamespacedResourceOperator.java:105) ~[io.strimzi.operator-common-0.39.0.jar:0.39.0]
	at io.strimzi.operator.common.operator.resource.AbstractNamespacedResourceOperator$$Lambda$823/0x00000001006d3918.apply(Unknown Source) ~[?:?]
	at io.vertx.core.impl.future.Composition.onSuccess(Composition.java:38) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:60) ~[io.vertx.vertx-core-4.5.0.jar:4.5.0]
	at io.vertx.core.impl.future.FutureBase$$Lambda$386/0x0000000100487b40.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty.netty-transport-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.100.Final.jar:4.1.100.Final]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-02-20 16:46:45 DEBUG Reflector:269 - Event received MODIFIED Pod resourceVersion v4827441 for v1/namespaces/strimzi/pods
2024-02-20 16:46:45 DEBUG StrimziPodSetController:215 - Pod strimzi-cluster-operator-7bb5468c59-wdfzq in namespace strimzi was MODIFIED
2024-02-20 16:46:45 DEBUG StrimziPodSetController:243 - Did not found parent StrimziPodSet for Pod strimzi-cluster-operator-7bb5468c59-wdfzq in namespace strimzi based on owner reference. Will try selector labels next.
2024-02-20 16:46:45 DEBUG StrimziPodSetController:226 - Pod strimzi-cluster-operator-7bb5468c59-wdfzq in namespace strimzi which was MODIFIED does not seem to be controlled by any StrimziPodSet and will be ignored
2024-02-20 16:46:45 WARN  BlockedThreadChecker: - Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 11593 ms, time limit is 2000 ms

udleinati Feb 22, 2024
Author

I identified the issue. We are using Kuma Mesh, and it's blocking the request to the Kubernetes API after upgrading Kubernetes to version 1.29.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

[Bug]: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2457 ms, time limit is 2000 ms #9715

{{title}}

Replies: 2 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

[Bug]: Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2457 ms, time limit is 2000 ms #9715

udleinati Feb 20, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Replies: 2 comments · 2 replies

scholzj Feb 20, 2024 Maintainer

udleinati Feb 20, 2024 Author

udleinati Feb 20, 2024 Author

udleinati Feb 22, 2024 Author

udleinati
Feb 20, 2024

Replies: 2 comments 2 replies

scholzj
Feb 20, 2024
Maintainer

udleinati
Feb 20, 2024
Author

udleinati Feb 20, 2024
Author

udleinati Feb 22, 2024
Author