Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Immediately write checkpoint when copy begins #298

Open
samongyr-sq opened this issue Jun 13, 2024 · 2 comments
Open

Immediately write checkpoint when copy begins #298

samongyr-sq opened this issue Jun 13, 2024 · 2 comments

Comments

@samongyr-sq
Copy link
Collaborator

If a migration is interrupted before Spirit gets a chance to write to the _chkpnt table, progress is lost as Spirit is unable to pick up from where it left off and the following message is logged

INFO[0000] could not resume from checkpoint: reason=could not read from table '_sbtest1_chkpnt'

Ideally, Spirit would write to _chkpnt ASAP to avoid losing progress. Alternatively, an option to configure the interval at which checkpoints are taken may be sufficient.

@morgo morgo self-assigned this Jun 17, 2024
@morgo morgo removed their assignment Aug 16, 2024
@morgo
Copy link
Collaborator

morgo commented Aug 16, 2024

I have un-assigned because I'm not currently working on this - but I might pick it up again in future.

There is a tradeoff with saving progress immediately, in that if it fails immediately, and then there is a long period of delay before restarting, resuming will actually be more expensive than starting again because of all the binary logs that need to be processed (vs. starting again where they don't need to be).

This is a weak argument however, since we currently start the first checkpoint ~1 minute in, and checkpointing after 1m vs 1s doesn't make much of a difference.

@jayjanssen
Copy link
Collaborator

I think the bigger issue is that spirit is unhappy if the checkpoint table is totally empty and it just fails.

one quick fix might be to let spirit continue if the checkpoint table is empty as if the table just wasn't there to begin with

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants