docker-kafka-connect

Dockerized Apache Kafka Connect (distributed mode)

Supported Tags

• 0.10.0.0 (2.11) (0.10.0.0/Dockerfile)
• 0.10.1.1 (2.11) (0.10.1.1/Dockerfile)
• latest 0.10.2.0 (2.12) (0.10.2.0/Dockerfile)

Quick Start

with Docker Compose

Write docker-compose.yml like and then execute docker-compose up

version: '2'
services:
  zk:
    image: 31z4/zookeeper:3.4.8

  kafka:
    image: ches/kafka
    links:
      - zk
    environment:
      KAFKA_BROKER_ID: 0
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ADVERTISED_PORT: 9092
      ZOOKEEPER_CONNECTION_STRING: zk:2181
      ZOOKEEPER_CHROOT: /broker-0

  connect:
    image: 1ambda/kafka-connect
    links:
      - kafka
    ports:
      - "8083:8083"
    environment:
      CONNECT_BOOTSTRAP_SERVERS: kafka:9092
      CONNECT_GROUP_ID: connect-cluster-A

with Docker CLI

$ docker run -d --name zk 31z4/zookeeper:3.4.8

$ docker run -d --name kafka \
    -e KAFKA_BROKER_ID=0 \
    -e KAFKA_ADVERTISED_PORT=9092 \
    -e ZOOKEEPER_CONNECTION_STRING=zk:2181 \ 
    --link zk:zk ches/kafka
    
$ docker run --rm --name connect \
    -p 8083:8083 \
    -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \
    -e CONNECT_GROUP_ID=connect-cluster-A \
    --link kafka:kafka 1ambda/kafka-connect

Environment Variables

Pass env variables starting with CONNECT_ to configure connect-distributed.properties.
For example, If you want to set offset.flush.interval.ms=15000, use CONNECT_OFFSET_FLUSH_INTERVAL_MS=15000

• (required) CONNECT_BOOTSTRAP_SERVERS
• (recommended): CONNECT_GROUP_ID (default value: connect-cluster)
• (recommended) CONNECT_REST_ADVERTISED_HOST_NAME
• (recommended) CONNECT_REST_ADVERTISED_PORT

Other connect configuration fields are optional. (see also Kafka Connect Configs)

How To Extend This Image

If you want to run additional connectors, add connector JARs to ${KAFKA_HOME}/connectors in container.

FROM 1ambda/kafka-connect:latest

# same as `cp -R connectors/ $KAFKA_HOME/`
# the entrypoint will extends `$CLASSPATH` 
# like `export CLASSPATH=${CLASSPATH}:${KAFKA_HOME}/connectors/*`

COPY connectors $KAFKA_HOME/connectors

Development

• SCALA_VERSION: 2.11
• KAFKA_VERSION: 0.10.0.0
• KAFKA_HOME: /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION}
• CONNECT_CFG: ${KAFKA_HOME}/config/connect-distributed.properties
• CONNECT_BIN: ${KAFKA_HOME}/bin/connect-distributed.sh
• CONNECT_PORT: 8083 (exposed)
• JMX_PORT: 9999 (exposed)

License

Apache 2.0