Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup fails on bulk loader data. #3831

Closed
martinmr opened this issue Aug 16, 2019 · 2 comments
Closed

Backup fails on bulk loader data. #3831

martinmr opened this issue Aug 16, 2019 · 2 comments
Assignees
Labels
area/enterprise Related to proprietary features kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit)
Milestone

Comments

@martinmr
Copy link
Contributor

Starting with commit 30ad3a4, backup hangs forever when trying to backup an instance on which the bulk loader was used (even if mutations are added later). @danielmai reported that the rollup function seems to be taking a long time.
This commit introduces the backwards-compatible format, which must rollup the lists. Before that, rollup was not called during backup.

  • What version of Dgraph are you using?
    master

  • Have you tried reproducing the issue with latest release?
    yes

  • Steps to reproduce the issue (command/config used to run Dgraph).

  1. Load 1 million dataset into dgraph using the bulk loader.
  2. Send a backup request (either to minio/s3 or the local filesystem).
  • Expected behaviour and actual result.
    Backup completes
@martinmr martinmr added this to the Dgraph v1.1 milestone Aug 16, 2019
@martinmr martinmr added area/enterprise Related to proprietary features kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit) labels Aug 16, 2019
@martinmr
Copy link
Contributor Author

I debugged a little bit more and I've narrowed down the issue a bit. To summarize:

  • Rollup is not the issue, as it completes successfully and the output appears to be reasonable.
  • The function toBackupList in ee/backup/backup.go never exits. This function takes an iterator and goes through the posting lists, rolling them up and converting them to the new format. However, the iteration loops forever. I've tried to add more itr.Next in the places where the loop returns or breaks but that has not worked.
  • I still have no idea why this only happens when bulk loader is involved. Manual and automated tests using the client all work fine.

@gitlw
Copy link

gitlw commented Sep 3, 2019

This issue has been fixed in the PR referenced above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/enterprise Related to proprietary features kind/bug Something is broken. priority/P1 Serious issue that requires eventual attention (can wait a bit)
Development

Successfully merging a pull request may close this issue.

3 participants