ingest-avro

Ingesting Avro messages

In this recipe we'll learn how to ingest JSON files.

Pinot Version	1.0.0
Schema	config/schema.json
Table Config	config/table.json

---

Clone this repository and navigate to this recipe:

git clone [email protected]:startreedata/pinot-recipes.git
cd pinot-recipes/recipes/ingest-avro

Spin up a Pinot cluster using Docker Compose:

docker-compose up

Ingest data into Kafka

pip install avro confluent-kafka click faker requests

python datagen.py| python kafkaproducer.py

Check data is ingesting:

kcat -C -b localhost:9092 -t person-topic -r localhost:8081 -s value=avro

Add tables and schema:

docker run \
   --network ingestavro \
   -v $PWD/config:/config \
   apachepinot/pinot:1.0.0 AddTable \
     -schemaFile /config/schema.json \
     -tableConfigFile /config/table.json \
     -controllerHost "pinot-controller" \
    -exec

Query via the Pinot UI:

select * 
from people 
where person.friend_ids <> 'null'
AND person.interests <> 'null'
and ARRAYLENGTH(person.interests) > 1

limit 10