[FEAT] : Implementation of event streaming through Kafka

[majorbruteforce]

Events are any real world match instances which contain a snapshot of the match. For instance, in cricket: a ball, a wicket, a substitution, an interruption by nature, a new batsman entering the crease etc are all events.

This particular service will be responsible for accepting the data from client (admin), processing it (if required) and pushing it to other layers of services managing SSE connections to the clients (viewers).

Need for Kafka:

• Ensuring no data is lost (consistent writes to DB )and the system is fault tolerant
• Managing dead letter queues to ensure chronological delivery of events
• Synchronising multiple microservices

Extras to be implemented:

• Dedicated logger for Kafka
• Unit and Integration testing

All implementations are to be done using the go library sarama

Refer past discussions regarding usage of Kafka and Sarama: #4

All queries and suggestions related to the event streaming service(s) can be done here.

[majorbruteforce]

CC: @soumil-kumar17 @zakhaev26 @punitkr03

[zakhaev26]

lets roll with the implementation for event streams once i m done with the core services 💪🏻

[majorbruteforce]

lets roll with the implementation for event streams once i m done with the core services 💪🏻

Aye aye, Sarge!

[majorbruteforce]

@zakhaev26 do share good resources that you have come across regarding Sarama

[zakhaev26]

From my experience, after going through various Kafka tutorials, I found implementing it in Go quite straightforward. I found the Sarama package to be more reliable than the packages provided by Confluent on Go. Even though there isn't a video specifically focused on implementing Sarama with Go, I managed to do it by referring to Sarama's documentation, GitHub examples, and some articles on Medium.

Here's a good video on the basics of Go + Kafka heer . (At around 11:10 in the video, the man also mentions that Sarama was a dominant package at the time)

BTW,i found a good stuff --> Transition sarama ownership

[zakhaev26]

BTW,are we looking into security measures of kafka?I think they provide a lot of things over security of kafka servers.

[majorbruteforce]

BTW,are we looking into security measures of kafka?I think they provide a lot of things over security of kafka servers.

Sure why not.

[majorbruteforce]

BTW,i found a good stuff --> Transition sarama ownership

Ahh. Interesting.

[majorbruteforce]

Here's a good video on the basics of Go + Kafka heer . (At around 11:10 in the video, the man also mentions that Sarama was a dominant package at the time)

Yup was following this one. OG. (Heeer!)

[zakhaev26]

As the api services are almost completed with some essential components being added simultaneously, we shall start implementing the live streaming feature - the last piece of the puzzle.
As the POC has been made and explored,we should discuss about schemas and structure.
Lets get this done boys!