Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(restore): Introduce incremental restore #7942

Merged
merged 22 commits into from
Jul 7, 2021

Conversation

ahsanbarkati
Copy link
Contributor

@ahsanbarkati ahsanbarkati commented Jul 2, 2021

Incremental restore can be used to restore a set of incremental backups on a cluster with a part of the backup already restored.
No mutations should be made between a normal restore and an incremental restore.


This change is Reviewable

@ahsanbarkati ahsanbarkati requested a review from manishrjain as a code owner July 2, 2021 14:53
@github-actions github-actions bot added the area/graphql Issues related to GraphQL support on Dgraph. label Jul 2, 2021
Copy link
Contributor

@NamanJain8 NamanJain8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Nice change. 🎉

One issue that we discussed regarding the stale predicates not getting deleted bcoz of predicate move.

schema/schema.go Show resolved Hide resolved
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: Also compare the current manifest with the last manifest that was applied, and drop any predicates that have moved etc.

Secondly, use the incremental stream writer.

Reviewed 9 of 9 files at r1.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @ahsanbarkati)


worker/draft.go, line 677 at r1 (raw file):

		defer closer.Done()

		glog.Infof("Got restore proposal at Index:%d, ReadTs:%d",

Index: <space> %d


worker/online_restore.go, line 245 at r1 (raw file):

}

type IncrementalWriter struct {

Don't need this. We should ideally use incremental stream.


worker/online_restore.go, line 350 at r1 (raw file):

		if mapRes.shouldDropAll {
			if err := pstore.DropAll(); err != nil {
				return errors.Wrap(err, "failed to reduce incremental restore map")

failed to drop all ...

Copy link
Contributor

@NamanJain8 NamanJain8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nit picks. Rest looks good.

worker/online_restore.go Outdated Show resolved Hide resolved
worker/online_restore.go Outdated Show resolved Hide resolved
worker/online_restore.go Show resolved Hide resolved
worker/restore_map.go Outdated Show resolved Hide resolved
worker/online_restore.go Outdated Show resolved Hide resolved
Copy link
Contributor Author

@ahsanbarkati ahsanbarkati left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 48 files reviewed, 4 unresolved discussions (waiting on @manishrjain and @NamanJain8)


worker/draft.go, line 677 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Index: <space> %d

Done.


worker/online_restore.go, line 245 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Don't need this. We should ideally use incremental stream.

Done.


worker/online_restore.go, line 350 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

failed to drop all ...

Done.

@ahsanbarkati ahsanbarkati merged commit dfa5dae into release/v21.03-slash Jul 7, 2021
@ahsanbarkati ahsanbarkati deleted the ahsan/incremental-restore branch July 7, 2021 15:38
ahsanbarkati added a commit that referenced this pull request Aug 2, 2021
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5dae)
ahsanbarkati added a commit that referenced this pull request Aug 2, 2021
* feat(restore): Introduce incremental restore (#7942)

This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5dae)
all-seeing-code pushed a commit that referenced this pull request Dec 13, 2022
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.

(cherry picked from commit dfa5dae)
mangalaman93 pushed a commit that referenced this pull request Jan 20, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Feb 2, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Feb 14, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Feb 17, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Feb 18, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Feb 24, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Mar 1, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Mar 2, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Mar 6, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Mar 6, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 pushed a commit that referenced this pull request Mar 7, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 7, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 7, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 10, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 13, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 15, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups.
In between two incremental restores, the cluster is in draining mode.
mangalaman93 added a commit that referenced this pull request Mar 15, 2023
This commit introduces incremental restore. It allows incremental
backups to be restored on top of a set of already restored backups. In
between two incremental restores, the cluster is in draining mode.

fixes #8533
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/graphql Issues related to GraphQL support on Dgraph.
Development

Successfully merging this pull request may close these issues.

3 participants