Add data ingest via kafka as an option

[mpankow]

All of the DDL and data is there so I am hoping this would be a relatively minor enhancement. Sometimes people want kafka ingest specifically. It would be great to have that option as part of this demo.

[carlsverre]

This is an interesting idea... the main requirement is being able to provide connection details to a kafka cluster with the correct data in the correct topics. Since the data is already available in s3 - in theory it's possible to ingest the data into kafka and then modify the pipelines accordingly.

Assuming we build something like this - which hosted kafka solution were you thinking of using? If this is primarily to unlock our sales team I can add an advanced option setting that lets you plug in kafka details - but leave it up to you guys to setup kafka correctly.

[mpankow]

The goal would be to unlock the sales team. I think leaving it to SEs to get a kafka environment setup is reasonable.