SSL Kafka handshake failed over docker

[laurafbec]

Hi everyone,
Based on the example
https://github.com/confluentinc/cp-docker-images/tree/5.3.3-post/examples/kafka-cluster-ssl
I've tried to develop a docker-compose file with zookeeper, a broker and the connect API with SSL enabled.
The certificates have been generated by using the script included in the example and the content of the docker-compose file would be the next one:

`version: '3.6'

services:
zookeeper:
image: confluentinc/cp-zookeeper:7.0.1
hostname: zookeeper
container_name: zookeeper
ports:
- "2181:2181"
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
ZOOKEEPER_INIT_LIMIT: 5
ZOOKEEPER_SYNC_LIMIT: 2

broker:
image: confluentinc/cp-kafka:7.0.1
hostname: broker
container_name: broker
depends_on:
- zookeeper
ports:
- "39093:39093"
- "9093:9093"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_ADVERTISED_LISTENERS: SSL://broker:39093,SSL_HOST://localhost:9093
KAFKA_SSL_KEYSTORE_FILENAME: kafka.broker.keystore.jks
KAFKA_SSL_KEYSTORE_CREDENTIALS: broker_keystore_creds
KAFKA_SSL_KEY_CREDENTIALS: broker_sslkey_creds
KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.broker.truststore.jks
KAFKA_SSL_TRUSTSTORE_CREDENTIALS: broker_truststore_creds
KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: " "
KAFKA_SSL_CLIENT_AUTH: required
KAFKA_SECURITY_INTER_BROKER_PROTOCOL: SSL
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: SSL:SSL,SSL_HOST:SSL
volumes:
- $PWD/secrets2:/etc/kafka/secrets

connect:
image: confluentinc/cp-kafka-connect:7.0.1
hostname: connect
container_name: connect
depends_on:
- zookeeper
- broker
ports:
- "8083:8083"
environment:
CONNECT_BOOTSTRAP_SERVERS: 'broker:39093'
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: compose-connect-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
CONNECT_INTERNAL_KEY_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_INTERNAL_VALUE_CONVERTER: "org.apache.kafka.connect.json.JsonConverter"
CONNECT_LOG4J_ROOT_LOGLEVEL: "INFO"
CONNECT_LOG4J_LOGGERS: "org.apache.kafka.connect.runtime.rest=WARN,org.reflections=ERROR,com.mongodb.kafka=DEBUG"
CONNECT_PLUGIN_PATH: /usr/share/confluent-hub-components
CONNECT_ZOOKEEPER_CONNECT: 'zookeeper:2181'
CLASSPATH: /usr/share/java/monitoring-interceptors/monitoring-interceptors-6.2.2.jar
CONNECT_PRODUCER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor"
CONNECT_CONSUMER_INTERCEPTOR_CLASSES: "io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor"
CONNECT_SSL_KEYSTORE_FILENAME: kafka.connect.keystore.jks
CONNECT_SSL_KEYSTORE_CREDENTIALS: connect_keystore_creds
CONNECT_SSL_KEY_CREDENTIALS: connect_sslkey_creds
CONNECT_SSL_TRUSTSTORE_FILENAME: kafka.connect.truststore.jks
CONNECT_SSL_TRUSTSTORE_CREDENTIALS: connect_truststore_creds

 volumes:
   - $PWD/mongodb-kafka-connect:/usr/share/confluent-hub-components/kafka-connect-mongodb
   - $PWD/kafka/scripts:/scripts
 command:
   - bash
   - -c
   - |
     echo "Launching Kafka Connect worker"
     /etc/confluent/docker/run &
     #
     echo "Waiting for Kafka Connect to start listening on $$CONNECT_REST_ADVERTISED_HOST_NAME"
     while [ $$(curl -s -o /dev/null -w %{http_code} http://$$CONNECT_REST_ADVERTISED_HOST_NAME:$$CONNECT_REST_PORT/connectors) -ne 200 ]; do
       echo -e $$(date) "Kafka Connect listener HTTP state: "$$(curl -s -o /dev/null -w %{http_code} http://$$CONNECT_REST_ADVERTISED_HOST_NAME:$$CONNECT_REST_PORT/connectors)" (waiting for 200)"
       sleep 5 
     done
     nc -vz $$CONNECT_REST_ADVERTISED_HOST_NAME $$CONNECT_REST_PORT
     echo -e "\n--\n+> Creating Kafka Connect MongoDB sink"
     chmod 755 /scripts/sink-connect.sh
     echo -e "Permisos cambiados"
     /scripts/sink-connect.sh 
     sleep infinity

`

When running it I get the error from broker

[2022-01-10 11:08:03,163] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /172.19.0.4 (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Can anyone help me? thanks in advance

[OneCricketeer]

The error is suggesting you didn't setup SSL for Zookeeper. And your environment variables for Zookeeper seem to confirm that

The examples have moved, by the way https://github.com/confluentinc/kafka-images/blob/master/examples/kafka-cluster-ssl/docker-compose.yml

[laurafbec]

Thanks @OneCricketeer !! I've actually solved it after posting the error. I had a mismatch between kafka listeners and the certs, and now I've TLSv1.3 enabled between kafka client and broker. I haven't found examples about enabling TLS for Zokeeper of Kafka connect. Do you know where can I find some of them? Thanks again!

[OneCricketeer]

haven't found examples about enabling TLS for Zookeeper or Kafka connect.

Is this page what you're looking for?

https://docs.confluent.io/platform/current/security/security_tutorial.html

Otherwise, Zookeeper has its own official documentation, and Connect is configured like any other broker client. Both utilize KAFKA_JAVA_OPTS environment variables for setting JAAS or other JVM System properties

Connect doesn't depend on Zookeeper

[laurafbec]

Thanks again @OneCricketeer!!
I actually was looking for docker-compose examples with SSL enabled between zookeeper and Kafka. On that page is described SASL for Zookeeper, but, I think that SSL is not. But thanks anyway!! I'll check everything.