🧐 About

This is a data engineering project for log analytics using kafka and spark

🏁 Getting Started

To get started and runing this program

Run kafka producer in terminal

python producer.py

Run spark streaming script to consume the log files

spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.4.1 consumer_2gb.py