Skip to content

This repo plugs to a replication slot of a Postgres database, and sends webhooks for any change that happens in the database.

License

Notifications You must be signed in to change notification settings

MovingLake/pg_webhook

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation


Logo

Postgres Webhook

Run webhooks directly from your Postgres database using replication.
Explore the docs »

View Demo

Table of Contents
  1. About The Project
  2. Postgres Setup
  3. Getting Started
  4. System
  5. License
  6. Contact
  7. Managed Version
  8. Acknowledgments

About The Project

This project came about from MovingLake technological development. Webhooks directly from a database using a replication slot was one of our core desires since before we founded MovingLake. This is now a reality and we wanted to give back to the awesome Open-Source Community by creating this Repo. Hope you enjoy it!

(back to top)

Built With

  • Golang
  • Sqlite

(back to top)

Postgres Setup

Pg_webhook uses a logical replication slot to get real-time data from the main database. As such a few configuration steps must be followed to allow this to happen

Standalone Postgres

Use the following commands to get started. We'll create a database, a user and setup postgres to create a logical replication slot.

create database pg_webhook;
create user pg_webhook with replication password 'pg_webhook';

Next you need to add some lines to Postgres' configuration files. If you do not know where this files are, you can run ps -ef | grep postgres. You shoud see a line such as:

/opt/homebrew/opt/postgresql/bin/postgres -D /opt/homebrew/var/postgres

The last piece of this line (/opt/homebrew/var/postgres) is where your postgres configuration files will be stored. Now open pg_hba.conf and add the following line:

host replication pg_webhook 127.0.0.1/32 md5

Next open postgres.conf and add these following lines:

wal_level=logical
max_wal_senders=5
max_replication_slots=5

Finally when specifying the postgres DNS string when running pg_webhook, make sure it has the replication query parameter ?replication=database. Eg postgres://pg_webhook:pg_webhook@localhost:5432/pg_webhook?replication=database

RDS Postgres

RDS does not let you run with a real superuser, and also doesn't let you change the configuration files. Most likely because of multitenant systems. To circumvent this, the easiest way to go is to use Postgres extension pglogical.

Please follow this link to set-up your RDS instance with the extension. Also checkout this other link if don't know how to turn on the preloaded libraries for RDS.

Once you follow these steps, try and follow the rest from #standalone-postgres

Getting Started

Docker

The easiest way to get started is to build the Dockerfile

docker build -t latest .

and then run the image with the appropriate environment variables

docker run --rm -it --net=host -e PG_DNS='postgres://postgres@localhost:5432/postgres?replication=database' -e WEBHOOK_SERVICE_URL='http://localhost:9000/' latest

Compile and run

You can also run directly the extractor, just use

go run main.go

Do ensure that the environment variables are set beforehand.

System

This extractor runs on Golang. It uses the pglogrepl library to create the replication slots and subscription. It then uses a producer consumer pattern to send received messages to the downstream service. If the downstream service cannot receive the requests, this code retries automatically for upto a day. After that it logs the failed requests into a Sqlite database on disk.

Consumer concurrency can be set higher if a high volume of transactions are being sent to the database.

License

Distributed under the GNU AGPL 3 License. See LICENSE for more information.

(back to top)

Contact

MovingLake - @MovingLake - [email protected]

Project Link: https://github.com/movinglake/pg_webhook

(back to top)

Managed Version

If you ever want to have a managed version of this system built for scale and completely maintenance free then we'd be happy to accommodate your requests. Please use this link to create a Postgres connector with a webhook destination.

(back to top)

Acknowledgments

Want to acknowledge the pglogrepl repo and its creators for such a great tool!

(back to top)

About

This repo plugs to a replication slot of a Postgres database, and sends webhooks for any change that happens in the database.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published