Run webhooks directly from your Postgres database using replication.
Explore the docs »
View Demo
Table of Contents
This project came about from MovingLake technological development. Webhooks directly from a database using a replication slot was one of our core desires since before we founded MovingLake. This is now a reality and we wanted to give back to the awesome Open-Source Community by creating this Repo. Hope you enjoy it!
Pg_webhook uses a logical replication slot to get real-time data from the main database. As such a few configuration steps must be followed to allow this to happen
Use the following commands to get started. We'll create a database, a user and setup postgres to create a logical replication slot.
create database pg_webhook;
create user pg_webhook with replication password 'pg_webhook';
Next you need to add some lines to Postgres' configuration files. If you do not know where this files are, you can run ps -ef | grep postgres. You shoud see a line such as:
/opt/homebrew/opt/postgresql/bin/postgres -D /opt/homebrew/var/postgres
The last piece of this line (/opt/homebrew/var/postgres) is where your postgres configuration files will be stored. Now open pg_hba.conf and add the following line:
host replication pg_webhook 127.0.0.1/32 md5
Next open postgres.conf and add these following lines:
wal_level=logical
max_wal_senders=5
max_replication_slots=5
Finally when specifying the postgres DNS string when running pg_webhook, make sure it has the replication query parameter ?replication=database. Eg postgres://pg_webhook:pg_webhook@localhost:5432/pg_webhook?replication=database
RDS does not let you run with a real superuser, and also doesn't let you change the configuration files. Most likely because of multitenant systems. To circumvent this, the easiest way to go is to use Postgres extension pglogical.
Please follow this link to set-up your RDS instance with the extension. Also checkout this other link if don't know how to turn on the preloaded libraries for RDS.
Once you follow these steps, try and follow the rest from #standalone-postgres
The easiest way to get started is to build the Dockerfile
docker build -t latest .
and then run the image with the appropriate environment variables
docker run --rm -it --net=host -e PG_DNS='postgres://postgres@localhost:5432/postgres?replication=database' -e WEBHOOK_SERVICE_URL='http://localhost:9000/' latest
You can also run directly the extractor, just use
go run main.go
Do ensure that the environment variables are set beforehand.
This extractor runs on Golang. It uses the pglogrepl library to create the replication slots and subscription. It then uses a producer consumer pattern to send received messages to the downstream service. If the downstream service cannot receive the requests, this code retries automatically for upto a day. After that it logs the failed requests into a Sqlite database on disk.
Consumer concurrency can be set higher if a high volume of transactions are being sent to the database.
Distributed under the GNU AGPL 3 License. See LICENSE for more information.
MovingLake - @MovingLake - [email protected]
Project Link: https://github.com/movinglake/pg_webhook
If you ever want to have a managed version of this system built for scale and completely maintenance free then we'd be happy to accommodate your requests. Please use this link to create a Postgres connector with a webhook destination.
Want to acknowledge the pglogrepl repo and its creators for such a great tool!