Kafka connect SSL handshake error with MSK #5797

sripathiacharya · 2021-10-28T10:29:11Z

sripathiacharya
Oct 28, 2021

Hi,

I am trying to deploy Kafka Connect pointing to AWS MSK broker with TLS authentication and encryption. Below is the configuration and error message on logs. Looks like Strimzi operator is defaulting to TLSv1.3 but MSK supports only TLSv1.2. Is there a way to control the TLS version for handshake?

Configurations:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: this-msk-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.8.0
  replicas: 1
  bootstrapServers: b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094,b-1.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094,b-2.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094
  tls:
    trustedCertificates:
      - secretName: msk-cluster-cluster-ca-cert
        certificate: ca.crt
  authentication:
    type: tls
    certificateAndKey:
      certificate: user.crt
      key: user.key
      secretName: super-user
  config:
    group.id: msk-this-msk-connect
    offset.storage.topic: this-msk-connect-offsets
    config.storage.topic: this-msk-connect-configs
    status.storage.topic: this-msk-connect-status
    config.storage.replication.factor: -1
    offset.storage.replication.factor: -1
    status.storage.replication.factor: -1

Logs:

Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Preparing keystore
Preparing keystore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094,b-1.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094,b-2.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094
# REST Listeners
rest.port=8083
rest.advertised.host.name=172.31.47.222
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
offset.storage.topic=this-msk-connect-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=this-msk-connect-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=ext-kafka-this-msk-connect
status.storage.topic=this-msk-connect-status
config.storage.replication.factor=-1
offset.storage.replication.factor=-1
status.storage.replication.factor=-1


security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
admin.security.protocol=SSL
# TLS / SSL
ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
ssl.truststore.password=[hidden]
ssl.truststore.type=PKCS12

producer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
producer.ssl.truststore.password=[hidden]

consumer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
consumer.ssl.truststore.password=[hidden]

admin.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
admin.ssl.truststore.password=[hidden]
ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
ssl.keystore.password=[hidden]
ssl.keystore.type=PKCS12

producer.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
producer.ssl.keystore.password=[hidden]
producer.ssl.keystore.type=PKCS12

consumer.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
consumer.ssl.keystore.password=[hidden]
consumer.ssl.keystore.type=PKCS12

admin.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
admin.ssl.keystore.password=[hidden]
admin.ssl.keystore.type=PKCS12


# Additional configuration
client.rack=

2021-10-28 10:02:45,632 INFO WorkerInfo values:
        jvm.args = -Xms128M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
        jvm.spec = Red Hat, Inc., OpenJDK 64-Bit Server VM, 11.0.12, 11.0.12+7-LTS
        jvm.classpath = /opt/kafka/bin/../libs/accessors-smart-2.4.7.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/automaton-1.11-8.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/commons-math3-3.6.1.jar:/opt/kafka/bin/../libs/connect-api-2.8.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-2.8.0.jar:/opt/kafka/bin/../libs/connect-file-2.8.0.jar:/opt/kafka/bin/../libs/connect-json-2.8.0.jar:/opt/kafka/bin/../libs/connect-mirror-2.8.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-2.8.0.jar:/opt/kafka/bin/../libs/connect-runtime-2.8.0.jar:/opt/kafka/bin/../libs/connect-transforms-2.8.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.5.57.jar:/opt/kafka/bin/../libs/error_prone_annotations-2.3.4.jar:/opt/kafka/bin/../libs/failureaccess-1.0.1.jar:/opt/kafka/bin/../libs/generex-1.0.2.jar:/opt/kafka/bin/../libs/gson-2.8.6.jar:/opt/kafka/bin/../libs/guava-30.1-jre.jar:/opt/kafka/bin/../libs/hamcrest-core-1.3.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/j2objc-annotations-1.3.jar:/opt/kafka/bin/../libs/jackson-annotations-2.10.5.jar:/opt/kafka/bin/../libs/jackson-core-2.10.5.jar:/opt/kafka/bin/../libs/jackson-databind-2.10.5.1.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.10.5.jar:/opt/kafka/bin/../libs/jackson-dataformat-yaml-2.11.2.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.10.5.jar:/opt/kafka/bin/../libs/jackson-datatype-jsr310-2.11.2.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.10.5.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-paranamer-2.10.5.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.10.5.jar:/opt/kafka/bin/../libs/jaeger-client-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-core-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.6.0.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.31.jar:/opt/kafka/bin/../libs/jersey-common-2.31.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.31.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.31.jar:/opt/kafka/bin/../libs/jersey-hk2-2.31.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.31.jar:/opt/kafka/bin/../libs/jersey-server-2.31.jar:/opt/kafka/bin/../libs/jetty-client-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-http-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-io-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-security-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-server-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-util-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.39.v20210325.jar:/opt/kafka/bin/../libs/jline-3.12.1.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.16.1.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/json-path-2.6.0.jar:/opt/kafka/bin/../libs/json-smart-2.4.7.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/junit-4.13.2.jar:/opt/kafka/bin/../libs/kafka-agent-0.25.0.jar:/opt/kafka/bin/../libs/kafka-clients-2.8.0.jar:/opt/kafka/bin/../libs/kafka-env-var-config-provider-0.1.0.jar:/opt/kafka/bin/../libs/kafka-kubernetes-config-provider-0.1.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-2.8.0.jar:/opt/kafka/bin/../libs/kafka-metadata-2.8.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.8.1.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.8.1.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.8.1.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.8.1.jar:/opt/kafka/bin/../libs/kafka-oauth-server-plain-0.8.1.jar:/opt/kafka/bin/../libs/kafka-quotas-plugin-0.1.0.jar:/opt/kafka/bin/../libs/kafka-raft-2.8.0.jar:/opt/kafka/bin/../libs/kafka-shell-2.8.0.jar:/opt/kafka/bin/../libs/kafka-streams-2.8.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-2.8.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-2.8.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-2.8.0.jar:/opt/kafka/bin/../libs/kafka-tools-2.8.0.jar:/opt/kafka/bin/../libs/kafka_2.13-2.8.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.13-2.8.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.4.10.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.4.0.jar:/opt/kafka/bin/../libs/kubernetes-client-5.4.0.jar:/opt/kafka/bin/../libs/kubernetes-model-core-5.4.0.jar:/opt/kafka/bin/../libs/libthrift-0.14.1.jar:/opt/kafka/bin/../libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/logging-interceptor-3.12.12.jar:/opt/kafka/bin/../libs/lz4-java-1.7.1.jar:/opt/kafka/bin/../libs/maven-artifact-3.6.3.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-2-extensions-1.0.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent-0.25.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.62.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.62.Final.jar:/opt/kafka/bin/../libs/nimbus-jose-jwt-9.10.jar:/opt/kafka/bin/../libs/okhttp-4.9.0.jar:/opt/kafka/bin/../libs/okio-2.8.0.jar:/opt/kafka/bin/../libs/opa-authorizer-1.0.0.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.15.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-5.18.4.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.3.0.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-0.9.1.jar:/opt/kafka/bin/../libs/scala-library-2.13.5.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.5.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snakeyaml-1.26.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.1.jar:/opt/kafka/bin/../libs/tracing-agent-0.25.0.jar:/opt/kafka/bin/../libs/zjsonpatch-0.3.0.jar:/opt/kafka/bin/../libs/zookeeper-3.5.9.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.5.9.jar:/opt/kafka/bin/../libs/zstd-jni-1.4.9-1.jar
        os.spec = Linux, amd64, 5.4.149-73.259.amzn2.x86_64
        os.vcpus = 1
 (org.apache.kafka.connect.runtime.WorkerInfo) [main]
2021-10-28 10:02:45,637 INFO Scanning for plugin classes. This might take a moment ... (org.apache.kafka.connect.cli.ConnectDistributed) [main]
2021-10-28 10:02:49,165 INFO Registered loader: jdk.internal.loader.ClassLoaders$AppClassLoader@75b84c92 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,165 INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,166 INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.runtime.PredicatedTransformation' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,167 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.common.config.provider.FileConfigProvider' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,168 INFO Added plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,169 INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MirrorCheckpointConnector' and 'MirrorCheckpoint' to plugin 'org.apache.kafka.connect.mirror.MirrorCheckpointConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MirrorHeartbeatConnector' and 'MirrorHeartbeat' to plugin 'org.apache.kafka.connect.mirror.MirrorHeartbeatConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MirrorSourceConnector' and 'MirrorSource' to plugin 'org.apache.kafka.connect.mirror.MirrorSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'SchemaSourceConnector' and 'SchemaSource' to plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,170 INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,171 INFO Added aliases 'DoubleConverter' and 'Double' to plugin 'org.apache.kafka.connect.converters.DoubleConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'FloatConverter' and 'Float' to plugin 'org.apache.kafka.connect.converters.FloatConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'IntegerConverter' and 'Integer' to plugin 'org.apache.kafka.connect.converters.IntegerConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'LongConverter' and 'Long' to plugin 'org.apache.kafka.connect.converters.LongConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'ShortConverter' and 'Short' to plugin 'org.apache.kafka.connect.converters.ShortConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,172 INFO Added aliases 'PredicatedTransformation' and 'Predicated' to plugin 'org.apache.kafka.connect.runtime.PredicatedTransformation' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'Filter' to plugin 'org.apache.kafka.connect.transforms.Filter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'HasHeaderKey' to plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'RecordIsTombstone' to plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'TopicNameMatches' to plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,173 INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,174 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2021-10-28 10:02:49,236 INFO DistributedConfig values:
        access.control.allow.methods =
        access.control.allow.origin =
        admin.listeners = null
        bootstrap.servers = [b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094, b-1.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094, b-2.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094]
        client.dns.lookup = use_all_dns_ips
        client.id =
        config.providers = []
        config.storage.replication.factor = -1
        config.storage.topic = this-msk-connect-configs
        connect.protocol = sessioned
        connections.max.idle.ms = 540000
        connector.client.config.override.policy = None
        group.id = ext-kafka-this-msk-connect
        header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
        heartbeat.interval.ms = 3000
        inter.worker.key.generation.algorithm = HmacSHA256
        inter.worker.key.size = null
        inter.worker.key.ttl.ms = 3600000
        inter.worker.signature.algorithm = HmacSHA256
        inter.worker.verification.algorithms = [HmacSHA256]
        internal.key.converter = class org.apache.kafka.connect.json.JsonConverter
        internal.value.converter = class org.apache.kafka.connect.json.JsonConverter
        key.converter = class org.apache.kafka.connect.json.JsonConverter
        listeners = null
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        offset.flush.interval.ms = 60000
        offset.flush.timeout.ms = 5000
        offset.storage.partitions = 25
        offset.storage.replication.factor = -1
        offset.storage.topic = this-msk-connect-offsets
        plugin.path = [/opt/kafka/plugins]
        rebalance.timeout.ms = 60000
        receive.buffer.bytes = 32768
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 40000
        response.http.headers.config =
        rest.advertised.host.name = 172.31.47.222
        rest.advertised.listener = null
        rest.advertised.port = 8083
        rest.extension.classes = []
        rest.host.name = null
        rest.port = 8083
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        scheduled.rebalance.max.delay.ms = 300000
        security.protocol = SSL
        send.buffer.bytes = 131072
        session.timeout.ms = 10000
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.client.auth = none
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = /tmp/kafka/cluster.keystore.p12
        ssl.keystore.password = [hidden]
        ssl.keystore.type = PKCS12
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = PKCS12
        status.storage.partitions = 5
        status.storage.replication.factor = -1
        status.storage.topic = this-msk-connect-status
        task.shutdown.graceful.timeout.ms = 5000
        topic.creation.enable = true
        topic.tracking.allow.reset = true
        topic.tracking.enable = true
        value.converter = class org.apache.kafka.connect.json.JsonConverter
        worker.sync.timeout.ms = 3000
        worker.unsync.backoff.ms = 300000
 (org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2021-10-28 10:02:49,239 INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [main]
2021-10-28 10:02:49,243 INFO AdminClientConfig values:
        bootstrap.servers = [b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094, b-1.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094, b-2.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com:9094]
        client.dns.lookup = use_all_dns_ips
        client.id =
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.mechanism = GSSAPI
        security.protocol = SSL
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = /tmp/kafka/cluster.keystore.p12
        ssl.keystore.password = [hidden]
        ssl.keystore.type = PKCS12
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
        ssl.truststore.password = [hidden]
        ssl.truststore.type = PKCS12
 (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'producer.ssl.keystore.type' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'producer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'admin.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'consumer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'producer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,683 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'consumer.ssl.keystore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'producer.ssl.keystore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'admin.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'admin.ssl.keystore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'consumer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'client.rack' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,684 WARN The configuration 'admin.ssl.keystore.type' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'producer.ssl.keystore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'consumer.ssl.keystore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'admin.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'consumer.ssl.keystore.type' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,685 WARN The configuration 'admin.ssl.keystore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2021-10-28 10:02:49,686 INFO Kafka version: 2.8.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-28 10:02:49,686 INFO Kafka commitId: ebb1d6e21cc92130 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-28 10:02:49,686 INFO Kafka startTimeMs: 1635415369685 (org.apache.kafka.common.utils.AppInfoParser) [main]
2021-10-28 10:02:50,199 INFO [AdminClient clientId=adminclient-1] Failed authentication with b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com/172.31.35.199 (SSL handshake failed) (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclie2021-10-28 10:02:50,204 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (b-3.dsc-msk-nonprod.ecxi4x.c8.kafka.us-east-1.amazonaws.com/172.31.35.199:9094) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [kafka-ad
min-client-thread | adminclient-1]
2021-10-28 10:02:50,206 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:437)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:303)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
        at java.base/java.lang.Thread.run(Thread.java:829)
2021-10-28 10:02:50,224 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2021-10-28 10:02:50,225 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1635415399689, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2021-10-28 10:02:50,225 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1635415400211, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2021-10-28 10:02:50,233 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-28 10:02:50,233 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-28 10:02:50,233 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2021-10-28 10:02:50,233 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
        at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
        at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
        at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
        at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
        at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
        at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
        ... 3 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Unexpected handshake message: server_hello
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:129)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:336)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:283)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:437)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:303)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
        at java.base/java.lang.Thread.run(Thread.java:829)

Answered by sripathiacharya

Oct 29, 2021

The issue was due to DSA keys. Replaced with RSA keys and worked well.

View full answer

scholzj · 2021-10-28T10:39:03Z

scholzj
Oct 28, 2021
Maintainer

Why do you think this has anything to do with TLSv1.2 / TLSv1.3? The logs suggest that both are enabled & the error IMHO does not suggest this is the issue. But if you want, you should be able to configure the TLS protocols in .spec.config section with the options from Kafka: http://kafka.apache.org/documentation/#connectconfigs

1 reply

sripathiacharya Oct 28, 2021
Author

Thanks @scholzj. In the log I noticed ssl.protocol = TLSv1.3 so assumed the issue could be due to this. I used the .spec.config to override ssl.protocol. Even with TLSv1.2 I am getting the same error. I will debug some more.

sripathiacharya · 2021-10-29T11:31:42Z

sripathiacharya
Oct 29, 2021
Author

The issue was due to DSA keys. Replaced with RSA keys and worked well.

1 reply

scholzj Oct 29, 2021
Maintainer

Hmm, interesting. Good to know, thanks.

dlydiard · 2024-04-12T00:25:52Z

dlydiard
Apr 12, 2024

We encountered a similar issue with the DSA signature scheme resulting in a server_hello error:

avax.net.ssl|WARNING|21|kafka-admin-client-thread | adminclient-1|2024-04-11 22:30:22.539 
UTC|SignatureScheme.java:446|Unsupported signature scheme: dsa_sha256

Switching to RSA resolved the problem. Specifically, we updated our key generation command to include the -keyalg RSA -keysize 2048 options. After making this change, mutual TLS authentication with MSK functioned correctly.

keytool -genkeypair -keyalg RSA -keysize 2048 ...

This modification might be beneficial for others facing similar SSL/TLS handshake issues with MSK.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Kafka connect SSL handshake error with MSK #5797

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Strimzi

Kafka connect SSL handshake error with MSK #5797

sripathiacharya Oct 28, 2021

Replies: 3 comments · 2 replies

scholzj Oct 28, 2021 Maintainer

sripathiacharya Oct 28, 2021 Author

sripathiacharya Oct 29, 2021 Author

scholzj Oct 29, 2021 Maintainer

dlydiard Apr 12, 2024

sripathiacharya
Oct 28, 2021

Replies: 3 comments 2 replies

scholzj
Oct 28, 2021
Maintainer

sripathiacharya Oct 28, 2021
Author

sripathiacharya
Oct 29, 2021
Author

scholzj Oct 29, 2021
Maintainer

dlydiard
Apr 12, 2024