Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for incremental extractions (only load records new since last run) #37

Open
hyfen opened this issue Mar 19, 2022 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@hyfen
Copy link
Member

hyfen commented Mar 19, 2022

If using Chronicle-ETL to do incremental backups of personal data or syncing to other services, it's annoying to extract a full set of records each time a job is run. An incremental extraction system could let users extract records created/modified since the last time the job was run.

The guts of it (some already half-implemented):

  • a system for persisting results from a job (a row saved in a sqlite db stored in $XDG_DATA_HOME)
  • a way to specify that a job should continue from the last run
  • setting the since option on a job automatically based on results of the last run

This system won't work with ad-hoc jobs specified with only CLI flags. It would require either a stable name for a job or a config file (perhaps this is one and the same?)

@hyfen hyfen added the enhancement New feature or request label Mar 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant