debezium-postgres-hbase-sink

A kafka connect sink specifically from (source) debezium-postgres to (sink) HBase

Debezium-postgres

PostgreSQL’s logical decoding feature was first introduced in version 9.4 and is a mechanism which allows the extraction of the changes which were commited to the transaction log and the processing of these changes in a user-friendly manner via the help of an output plugin.

Debezium is a great project for Postgres CDC atop of the logical decoding feature(https://github.com/debezium/debezium).

debezium/decoderbufs

Any changes commit to PG tables are captured and the change events are sent to kafka through kafka connect source task. The events are Protobuf formated (https://github.com/debezium/postgres-decoderbufs).

We want to poll the change events out from kafka,'decode' them and put to HBase.

Goal

So, rather than a generic hbase-sink for kafka connect (such as https://github.com/mravi/kafka-connect-hbase), this project is very specific:

• it is a hbase sink for kafka connect,
• BUT it consumes only Debezium-Postgres CDC generated by decoderbufs (github.com/debezium/postgres-decoderbufs)