Strimzi
Operator stuck in loop trying to deserialise WatchEvent with empty map into string #4276
Answered
by
scholzj
raj-saxena
asked this question in
Q&A
-
|
Hi, I was trying to add few annotations to the Zookeeper pod template so that they are identified by Datadog's auto-discovery (details) that asks to add I added the following to the apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
# kafka cluster configuration
kafka:
# ...
zookeeper:
# Annotations for Datadog JMX auto-discovery
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka.check_names: '["zk"]'
ad.datadoghq.com/kafka.init_configs: {}
ad.datadoghq.com/kafka.instances: '[{"host": "%%host%%","port":"2181"}]'
ad.datadoghq.com/kafka.logs: '[{"source":"zookeeper","service":"zookeeper"}]'Unfortunately, the 2021-01-19 11:57:25 WARN WatchConnectionManager:205 - Exec Failure
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:249) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:168) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:153) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:153) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.WatchHTTPManager.readWatchEvent(WatchHTTPManager.java:292) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:228) [io.fabric8.kubernetes-client-4.12.0.jar:?]
at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105) ~[com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [com.squareup.okhttp3.okhttp-3.12.6.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.fabric8.kubernetes.api.model.WatchEvent["object"]->io.strimzi.api.kafka.model.Kafka["spec"]->io.strimzi.api.kafka.model.KafkaSpec["zookeeper"]->io.strimzi.api.kafka.
model.ZookeeperClusterSpec["template"]->io.strimzi.api.kafka.model.template.ZookeeperClusterTemplate["pod"]->io.strimzi.api.kafka.model.template.PodTemplate["metadata"]->io.strimzi.api.kafka.model.template.MetadataTemplate["annotations"]-
>java.util.LinkedHashMap["ad.datadoghq.com/kafka.init_configs"])
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1445) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1219) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2476) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2929) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at io.fabric8.kubernetes.internal.KubernetesDeserializer.fromObjectNode(KubernetesDeserializer.java:87) ~[io.fabric8.kubernetes-model-core-4.12.0.jar:4.12.0]
at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:57) ~[io.fabric8.kubernetes-model-core-4.12.0.jar:4.12.0]
at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:46) ~[io.fabric8.kubernetes-model-core-4.12.0.jar:4.12.0]
at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4218) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3259) ~[com.fasterxml.jackson.core.jackson-databind-2.10.4.jar:2.10.4]
at io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:247) ~[io.fabric8.kubernetes-client-4.12.0.jar:?]
... 14 moreAfter looking at the exception, I quickly updated the row to the following: ad.datadoghq.com/kafka.init_configs: ''However, the operator does not recovers. I believe, it is because it is trying to consume the faulty event and failing. This has been ongoing for hours now. Is there a way to make the operator drop the event & continue? How do I recover? |
Beta Was this translation helpful? Give feedback.
Answered by
scholzj
Jan 19, 2021
Replies: 1 comment 3 replies
-
|
I do not think the operator would be processing the same event again and again. If deleting the pod (and starting a new one) does not help, it means that the resource it does not understand is the one served by your Kubernetes API. Can you share the whole YAML of the resource as you have it? This might be ultimately a bug in the Fabric8 library from which we inherit the Metadata structure. |
Beta Was this translation helpful? Give feedback.
3 replies
-
Do you mean the Gist of broken spec. I removed the |
Beta Was this translation helpful? Give feedback.
-
|
I deleted the |
Beta Was this translation helpful? Give feedback.
-
|
Ok. glad the errors are gone now. I will try to look into it to see if we can avoid it in the future. |
Beta Was this translation helpful? Give feedback.
Answer selected by
raj-saxena
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I do not think the operator would be processing the same event again and again. If deleting the pod (and starting a new one) does not help, it means that the resource it does not understand is the one served by your Kubernetes API. Can you share the whole YAML of the resource as you have it? This might be ultimately a bug in the Fabric8 library from which we inherit the Metadata structure.