Support for initial state replication

[PhilLello]

Just a quick note to say I'm experimenting with capturing the initial state of a database; work will be over at https://github.com/dunlop-lello/mysql-binlog-connector-java until I've made enough progress to start filing pull-requests.

Please ping me if you're interested in collaboration.

[shyiko]

Hi Phil.

I think some of the projects under the https://github.com/shyiko/mysql-binlog-connector-java#real-world-applications section (like debezium) are already doing that. You might wanna check their sources.

Also, I don't think this would be a good fit for mysql-binlog-connector-java simply because it's out of scope (mysql-binlog-connector-java deals with binary log stream only, everything else is supposed to be built on top of it). BUT. If you build a standalone library/application that produces a snapshot and can be used together with mysql-binlog-connector-java to achieve what I think you have in mind I would gladly advertise it in readme.md as a go-to-reference in case people need something like this.

[PhilLello]

Hi Stanley,

I do understand hesitation about widening scope for the project, however I
think allowing a small bit of extra functionality to implementations of
BinaryLogClient.​LifecycleListener is useful. I've got a WIP patch to the
0.5.1-hotfix branch at
https://github.com/dunlop-lello/mysql-binlog-connector-java/tree/custom-0.5.2
that provides a minimal set of hooks to support this, and a simple demo app
at
https://github.com/dunlop-lello/mysql-binlog-connector-java/tree/custom-0.5.2
.

I'm a bit reluctant to expose the ability to execute random SQL statements
since that significantly widens room for errors due to client code
executing unexpected/unsupported/untested statements. My use-case is a
one-way MySQL => elasticsearch integration, so I'm most interested in
getting column definitions out of INFORMATION_SCHEMA.COLUMNS (which helps
with column names for the row updates) and reading full tables; being able
to slurp all the server variables too would help with sanity checking a
configuration/integration. However I'm aware adding dedicated functions for
this risks opening the floodgates for other wrapper functions - maybe the
compromise is to make execute protected and add wrappers in custom
subclasses?

Would appreciate your thoughts.

Phil

On Sat, Nov 12, 2016 at 8:28 PM, Stanley Shyiko [email protected]
wrote:

Hi Phil.

I think some of the projects under the https://github.com/shyiko/
mysql-binlog-connector-java#real-world-applications section (like
debezium) are already doing that. You might wanna check their sources.

Also, I don't this this would be a good fit for
mysql-binlog-connector-java simply because it's out of scope
(mysql-binlog-connector-java deals with binary log stream only, everything
else is supposed to be built on top of it). BUT. If you build a
standalone library/application that produces a snapshot and can be used
together with mysql-binlog-connector-java to achieve what I think you have
in mind I would gladly advertise it readme.md as a go-to-reference in
case people need something like this.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#128 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuHebmln5rK6Iaq82OX7y2OeXhvcTelks5q9iFYgaJpZM4KwY5e
.

[PhilLello]

The demo link should have read "https://github.com/dunlop-lello/my2es"

On Sun, Nov 13, 2016 at 4:56 PM, Phil Lello [email protected]
wrote:

Hi Stanley,

I do understand hesitation about widening scope for the project, however I
think allowing a small bit of extra functionality to implementations of
BinaryLogClient.​LifecycleListener is useful. I've got a WIP patch to the
0.5.1-hotfix branch at https://github.com/dunlop-
lello/mysql-binlog-connector-java/tree/custom-0.5.2 that provides a
minimal set of hooks to support this, and a simple demo app at
https://github.com/dunlop-lello/mysql-binlog-connector-
java/tree/custom-0.5.2.

I'm a bit reluctant to expose the ability to execute random SQL statements
since that significantly widens room for errors due to client code
executing unexpected/unsupported/untested statements. My use-case is a
one-way MySQL => elasticsearch integration, so I'm most interested in
getting column definitions out of INFORMATION_SCHEMA.COLUMNS (which helps
with column names for the row updates) and reading full tables; being able
to slurp all the server variables too would help with sanity checking a
configuration/integration. However I'm aware adding dedicated functions for
this risks opening the floodgates for other wrapper functions - maybe the
compromise is to make execute protected and add wrappers in custom
subclasses?

Would appreciate your thoughts.

Phil

On Sat, Nov 12, 2016 at 8:28 PM, Stanley Shyiko [email protected]
wrote:

Hi Phil.

I think some of the projects under the https://github.com/shyiko/mysq
l-binlog-connector-java#real-world-applications section (like debezium)
are already doing that. You might wanna check their sources.

Also, I don't this this would be a good fit for
mysql-binlog-connector-java simply because it's out of scope
(mysql-binlog-connector-java deals with binary log stream only, everything
else is supposed to be built on top of it). BUT. If you build a
standalone library/application that produces a snapshot and can be used
together with mysql-binlog-connector-java to achieve what I think you have
in mind I would gladly advertise it readme.md as a go-to-reference in
case people need something like this.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#128 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuHebmln5rK6Iaq82OX7y2OeXhvcTelks5q9iFYgaJpZM4KwY5e
.

[shyiko]

Why not to use mysql-connector-java (#24 (comment))? What about https://github.com/debezium/debezium/blob/master/debezium-connector-mysql/src/main/java/io/debezium/connector/mysql/SnapshotReader.java#L498? It's fairly to implement but why bother when official mysql-connector-java already supports that?

[shyiko]

@PhilLello

[az3]

@PhilLello Also check "mydumper": https://github.com/maxbube/mydumper/blob/master/docs/files.rst#metadata
MyDumper can dump the current state of a database, and give you the checkpoint (binlog position). So you can continue to track CDC via this project.

[shyiko]

Thank you.

I'm gonna close this ticket as there is nothing to be done (at this point). Feel free to reopen.