ullala

ullala is a demo that showcase using kafka as a data pipeline and log aggregation through the ELK stack

Setting up

Download kafka command line tools: scripts/download-kafka.sh
Run Kafka, Zookeeper, Elasticsearch and Logstash: docker-compose up
Set kafka on your host file to point to the running kafka instance (docker-machine ip default on mac)
Setup topics scripts/setup-topics.sh
Install OpenCV for capturer, brew install opencv on mac.
Symlink opencv into capturer, for mac you can run scripts/symlink-opencv-bew.sh
Install capturer dependencies pip install -r capturer/requirements.txt
Run Capturer: KAFKA_HOSTS=KAFKA:9092 capturer/capturer.py
Install processor dependencies cd processor && npm insall
Run processor: cd processor && KAFKA_HOSTS=kafka:9092 node index.js
Install pusher dependencies cd pusher && npm install
Run pusher `cd pusher && KAFKA_HOSTS=kafka:9092 node index.js

Name	Name	Last commit message	Last commit date
Latest commit albertein Adds presentation Jul 26, 2016 a794ece · Jul 26, 2016 History 8 Commits
capturer	capturer	Adds image capturer	Jul 25, 2016
data	data	Adds Dockerfiles and docker-compose file	Jul 25, 2016
kafka	kafka	Adds Dockerfiles and docker-compose file	Jul 25, 2016
logstash	logstash	Adds Dockerfiles and docker-compose file	Jul 25, 2016
presentation	presentation	Adds presentation	Jul 26, 2016
processor	processor	Adds processor	Jul 25, 2016
pusher	pusher	Adds pusher	Jul 25, 2016
scripts	scripts	Adds image capturer	Jul 25, 2016
zookeeper	zookeeper	Adds Dockerfiles and docker-compose file	Jul 25, 2016
.gitignore	.gitignore	Adds image capturer	Jul 25, 2016
README.md	README.md	Adds pusher	Jul 25, 2016
docker-compose.yaml	docker-compose.yaml	Adds Dockerfiles and docker-compose file	Jul 25, 2016