Transaction Watcher

This is a repository to harden your SQL skill, learn Kafka (or Redpanda) and learn Docker.

I work in a stock broker company, yet I found one unique problem in which you might encounter while having to manage and account for every transaction that is done through the company.

The first one being a need for something that listens to the changes on the database. Someone or some application would insert each customer's transaction activity to the database. Let's say it's from a legacy application in which nobody in the room has the guts to do any modification out of it. If you have multiple application that listens to state changes from the database, you will meet a problem on too many connection being opened on the database itself. Hence, you will probably need a message bus (or a message broker, same stuff, different wording) to spread your message to multiple people.

Then, you could accumulate by using event sourcing to find out current customer's balance. Rather than keeping a single integer field for customer's balance, you must store every transaction that they made, and look up to it as a source of truth. In other terms, it's also called journaling on accounting field (on bookeeping).

How do people at the stock broker (or similar financial company) built them and solve the issue? You will have to find it out yourself by building one.

Will you eventually build the best one that's actually used on the production environment? Probably not, but it's worth a try!

Your Tasks

This project contains a few tasks for you to work with.

#1 The Watcher

Create a subdirectory called watcher that will do a SQL query periodically to the transactions table, and for every new row on that table, the watcher will produce a message to transactions topic on Redpanda. You will also need to create the transactions topic yourself.

The schema of the message is defined on the kafka-schemas directory. Search for transaction.json file.

You must create a Dockerfile for your application. You can choose any language.

You don't need to create a HTTP API for this one. Just create a single function that will run when the program is executed. The application must work without any interference from you. It must work without having anyone (including you) to trigger the run or consume function.

You can see that your watcher is emitting correct message to Redpanda through Redpanda Console that is running on your local machine on port 8080.

There will be environment variable available for you when you run it though Docker Compose:

DATABASE_URL: "postgresql://watcher:password@postgres:5432/watcher?sslmode=disable"
KAFKA_ADDRESSES: "kafka:9092"

#2 The Swimmer

Create a subdirectory called swimmer that will consume the balance topic from Redpanda, and do an event sourcing of a customer's current balance. You must expose a HTTP API that handles a single endpoint of:

GET /current-balance?customer_id=123
Accept: application/json

With a response schema of:

{
  "customer_id": 123,
  "current_balance": 123456
}

It's called swimmer because you will swim through the Redpanda balance topic records, and get an answer from that. I will not give a clue on how you should consume your topic and produce a result based on the customer ID.

For the balance topic schema, you can look for balance.json on kafka-schemas directory.

You must create a Dockerfile for your application. You can choose any language.

Please expose the HTTP API at port 3000.

There will be environment variable available for you when you run it though Docker Compose:

KAFKA_ADDRESSES: "kafka:9092"

#3 The Frontend

This is an optional task.

Create a frontend (on frontend directory) that shows list of customer ID from customer-list service. Then, show the amount or balance that each customer have. You can create a full SPA page, or an SSR website.

You can hit these two endpoints:

GET http://customer-list:7201/customers
Accept: application/json

# Returns list of customer ID
[1,2,3,4,5]

GET http://swimmer:3000/current-balance?customer_id=123
Accept: application/json

# Returns:
{
  "customer_id": 123,
  "current_balance": 123456
}

Hey, the swimmer one is the one you made!

If you are developing it from your local machine, you can replace both customer-list and swimmer hostname into localhost.

You can create a Dockerfile and add a new service schema on docker-compose.yml file to state your frontend container configuration.

How do I know if I did it?

Easy, run the Docker Compose. Build everything. See the frontend if it works. See the Redpanda Console if you published and consume your topics and messages flawlessly.

docker compose up -d

To stop it, use

docker compose down

If you'd like to run just a few services for your local development purposes, for example only postgres and kafka services, you can specify it on the up command, like so:

docker compose up -d postgres kafka

Words of affirmation

Good luck, you can do it!

License

MIT License

Copyright (c) 2023 Reinaldy Rafli <[email protected]>

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

See LICENSE