Simple streaming test
git clone https://github.com/rssdev10/spark-kafka-streaming.git
cd spark-kafka-streaming
./gradlew setup
./gradlew test_data_prepare
To break data generation press CTRL-C. And continue it by same command
./gradlew test_data_prepare
And in other console just run:
./gradlew test_spark
or
./gradlew test_flink
Stop all:
./gradlew stop_all
Spark test must generate messages each 10 seconds like:
***************************************************************************
Processing time: 33477
Expected time: 10000
Processed messages: 2897866
Message example: {"message": 1, "uid":"dde09b16-248b-4a2b-8936-109c72eb64cc"}
Recovered json: {"message":1,"uid":"dde09b16-248b-4a2b-8936-109c72eb64cc"}
message is number of fist message in the window. Time values are in milliseconds.
src/main/java/KafkaDataProducer.java
MESSAGES_NUMBER = 100L * 1000 * 1000;
src/main/java/SparkStreamingConsumer.java
delayFactor = 10000; // 0..