kafka-streams

This is the repository for the examples of using Kafka streams covered in the blog posts:

• Kafka Streams - The Processor API
• Kafka Streams - The KStreams API
• Machine Learning with Kafka Streams

Requirements to build this project

1. Java 8
2. Gradle

Requirements to run the examples

1. kafka version kafka_2.11-0.10.1.0 see the section marked "Running a task on a particular version of Scala"
2. The json-data-generator from ACES,Inc

Setup Instructions

Extact the kafka_2.11-0.10.1.0.tgz file

tar -xvzf kafka_2.11-0.10.1.0.tgz

Start zookeeper and kafka

      kafka-install-dir/bin/zookeeper-server-start.sh kafka-install-dir/conf/zookeeper.properties
      kafka-install-dir/bin/kafka-server-start.sh kafka-install-dir/conf/server.properties

Install the Json-Data-Generator

Download the latest json-data-generator release and follow the install instructions here

Setup the kafka-streams repo

Clone or fork the repo

     git clone [email protected]:bbejeck/kafka-streams    
     cd kafka-streams

Then copy the json config files to json generator conf directory

    cp streaming-workflows/* <dir>/json-data-generator-1.2.0/conf

Create all the topics required by the examples

     ./bin/create-topics.sh /usr/local/kafka_2.11-0.10.1.0 localhost 2181
     args are kafka home, zookeeper host and zookeeper port adjust accordingly

Running the Purchase Processor API KStreams API Examples

 cd <dir>/json-data-generator-1.2.0/
 java -jar json-data-generator-1.2.0 purchases-config.json
 cd kafka-streams
 ./gradlew runPurchaseProcessor | runPurchaseStreams

Running the Stock Trades Processor API or KStreams API Examples

 cd <dir>/json-data-generator-1.2.0/
 java -jar json-data-generator-1.2.0 stock-transactions-config.json
 cd kafka-streams
 ./gradlew runStockProcessor | runStockStreams

Running the Twitter KStreams Language Classification Example

rename src/main/resources/twitter-app.properties.template to twitter-app.properties 
fill out the properties file with all the required values

cd kafka-streams
./gradlew runTwitterKstreamNLP 

Viewing the results of the purchase streaming examples

cd kafka_install-dir/bin
./kafka-console-consumer --topic [patterns|rewards|purchases] --zookeeper localhost:2181

Viewing the results of the stock-trading streaming examples

cd kafka_install-dir/bin
./kafka-console-consumer --topic [stocks-out|transaction-summary] --zookeeper localhost:2181

Viewing the results of the Twitter KStreams Language Classification Example

cd kafka_install-dir/bin
./kafka-console-consumer --topic [english|french|spanish] --zookeeper localhost:2181