Reading JSON from external Google drive and Save it to Kafka topic

[vperi1730]

Hi All & Scholzj(reaching you after years :) )

I have a use case that follows 2 steps process like this

1. Read the JSON files from an external google drive system and store it inside Kafka topics.
2. Later i would need to consume this JSON and store is inside my DB.

Is it possible to use KafkaConnectors over here, primarly FileStream source & sink, If YES can someone help me with the references to it, i would like to try in my Kafka cluster.

Help is appreciated

[scholzj]

The File connectors read / write local files (plus they are also very simple when it comes to the format of the files). You would need to find connectors for what you are describing ->

• reading from Google Drive
• and writing to a database

I'm not sure if Google Drive has some Kafka Connect connector. For writing to a database, you would need to look for either some generic JDBC connector or for a connector for the database you need to use. The communities of those connectors should be able to provide you help with how exactly to configure them. You can also consider using some more generic integration technologies if you don't find any Kafka Connect connector such as Apache Camel.

[vperi1730]

In that case i am thinking of another approach

1. Use Kafka SDKs to produce data into topics since my application is a spring boot one.
2. Using any of the debezium connector for postgres(this is the DB we are using) i should be able to consume and feed into DB.

Let me know if this can be one good approach?

Also is there a way map JSON objects->DB columns using these connectors?, Any idea here @scholzj ?

[scholzj]

Re 1) Sure, that is an option as well.
Re 2) Probably yes. But I do not know Debezium in detail.

But overall this sounds reasonable.

Also is there a way map JSON objects->DB columns using these connectors?, Any idea here @scholzj ?

No idea. That might be more question on the Debezium folks.

[vperi1730]

Sure, Let me take it up with Debezium forum @scholzj , thanks for the inputs, will connect again!!!