Kafka Monitor is a framework to implement and execute long-running kafka system tests in a real cluster. It complements Kafka’s existing system tests by capturing potential bugs or regressions that are only likely to occur after prolonged period of time or with low probability. Moreover, it allows you to monitor Kafka cluster using end-to-end pipelines to obtain a number of derived vital stats such as end-to-end latency, service availability and message loss rate. You can easily deploy Kafka Monitor to test and monitor your Kafka cluster without requiring any change to your application.
Kafka Monitor requires Gradle 2.0 or higher. Java 7 should be used for building in order to support both Java 7 and Java 8 at runtime.
Kafka Monitor supports Apache Kafka 0.8 and 0.9. Use branch 0.8.2.2 to monitor Apache Kafka cluster 0.8.
In order to run Kafka Monitor, you need to create a topic manually that is dedicated for use by Kafka Monitor. The topic should be specified in either the config file or the command line parameters in the examples below. For best performance the number of partitions in the topic should be proportational to the number of brokers in the cluster. We may avoid this manual step in the future by automatically creating topics in Kafka Monitor.
$ git clone https://github.com/linkedin/kafka-monitor.git
$ cd kafka-monitor
$ ./gradlew jar
$ ./bin/kafka-monitor-start.sh config/kafka-monitor.properties
$ ./bin/end-to-end-test.sh --topic test --broker-list localhost:9092 --zookeeper localhost:2181
Open localhost:8000/index.html in your web browser
You can edit webapp/index.html to easily add new metrics to be displayed.
curl localhost:8778/jolokia/read/kmf.services:type=produce-service,name=*/produce-availability-avg
You can query other JMX metric value as well by substituting object-name and attribute-name of the JMX metric in the query above.
./gradlew checkstyleMain checkstyleTest
./gradlew idea
./gradlew eclipse