Skip to content

Support logical replication / change data capture (CDC) with Kafka #2122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fritzblue opened this issue Feb 13, 2018 · 5 comments
Closed

Support logical replication / change data capture (CDC) with Kafka #2122

fritzblue opened this issue Feb 13, 2018 · 5 comments
Labels
area/integrations Related to integrations with other projects. kind/feature Something completely new we should consider. priority/P2 Somehow important but would not block a release. status/accepted We accept to investigate/work on it.
Milestone
Unplanned (v1.x)

Comments

@fritzblue
Copy link

fritzblue commented Feb 13, 2018 •
edited by ghost
Loading

The following workflow is becoming more and more common in streaming architectures:

image

If Dgraph is to be used as a primary data store in a production system, it must support logical replication / change data capture.

Is this currently possible? If not, is it planned for the future?
@pawanrawal
Copy link
Contributor

It's not possible right now but we can do something like this in the future. Some users have asked for subscriptions. This would be a similar feature I suppose.

@pawanrawal pawanrawal added the kind/question Something requiring a response. label Feb 19, 2018
@manishrjain
Copy link
Contributor

manishrjain commented Jan 15, 2019 •
edited
Loading

We could potentially support this at Badger level first, and then expose it via Dgraph.

hypermodeinc/badger#684

@manishrjain manishrjain added kind/feature Something completely new we should consider. and removed kind/question Something requiring a response. labels Jan 15, 2019
@campoy campoy added priority/P2 Somehow important but would not block a release. status/accepted We accept to investigate/work on it. labels Jul 11, 2019
@campoy campoy added this to the Unplanned (v1.x) milestone Jul 11, 2019
@campoy
Copy link
Contributor

campoy commented Jul 11, 2019

After chatting with @gitlw, it seems like this could be implemented relatively easily on top on our current plans for datacenter to datacenter replication, which already uses Kafka.

Adding it to our 1.X plans!

@campoy campoy changed the title Does Dgraph support logical replication / change data capture (CDC)? Support logical replication / change data capture (CDC) with Kafka Jul 11, 2019
@campoy campoy added the area/integrations Related to integrations with other projects. label Jul 11, 2019
@campoy campoy mentioned this issue Sep 13, 2019
Closed
@fritzblue
Copy link
Author

For an example of a distributed DB that's doing this in what I would argue is the Right Way™, see CockroachDB https://www.cockroachlabs.com/docs/stable/change-data-capture.html

@MichelDiz MichelDiz mentioned this issue Mar 19, 2020
Closed
@minhaj-shakeel
Copy link
Contributor

Github issues have been deprecated.
This issue has been moved to discuss. You can follow the conversation there and also subscribe to updates by changing your notification preferences.

drawing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/integrations Related to integrations with other projects. kind/feature Something completely new we should consider. priority/P2 Somehow important but would not block a release. status/accepted We accept to investigate/work on it.
Milestone
Unplanned (v1.x)
Development

No branches or pull requests
5 participants
@manishrjain @campoy @fritzblue @pawanrawal @minhaj-shakeel