fix(proposals): Incremental proposal key for zero proposals (#8005) #8567
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
all-seeing-code
requested review from
akon-dey,
skrdgraph,
darkn3rd,
meghalims,
matthewmcneely and
billprovince
as code owners
all-seeing-code
added
the
slash-to-main
|
What's the benefit of this change? Could it cause problems when old Zero's key and new Zero's keys after upgrade collide? |
all-seeing-code
force-pushed
the
anurag/cherry-pick-proposal-fix
branch
3 times, most recently
from
11ecf3a to
b3b4042
Compare
matthewmcneely
previously approved these changes
Jan 24, 2023
mangalaman93
previously approved these changes
Feb 5, 2023
mangalaman93
previously approved these changes
Feb 6, 2023
matthewmcneely
previously approved these changes
Feb 6, 2023
all-seeing-code
dismissed stale reviews from matthewmcneely and mangalaman93
via
dfdfb1e
mangalaman93
approved these changes
Feb 7, 2023
mangalaman93
requested changes
Feb 7, 2023
mangalaman93
previously approved these changes
Feb 7, 2023
matthewmcneely
previously approved these changes
Feb 7, 2023
This change ensures that the value in the mutation is not too big. The challenge here is that the keys in badger have a limitation on their size (< 2<<16). We need to ensure that no key, either primary or secondary index key is bigger than that. Fixes https://github.com/dgraph-io/projects/issues/73
Earlier, whenever the alpha starts(or restarts), we were upserting guardian and groot for all the namespaces. This is not actually needed. The change was made in the PR #7759 to fix a bulk loader edge case. This PR fixes that by generating the required RDFs in the bulk loader itself. Essentially, it inserts the ACL RDFs when force loading into non-Galaxy namespace. (cherry picked from commit 6730f10) Co-authored-by: Naman Jain <[email protected]>
Change: Added new automated tests for backup/restore (full as well as incremental ) on NFS. Test Scenarios covered:- - Full+Incremental backup on nfs mounted directory - Full+Incremental restore from nfs mounted directory.
Earlier the admin server mutex lock was used to protect the graphql schema map. But now we store that in schema store that internally handles the concurrency. Hence, we don't need to take the admin server's read lock to access schema. /probe/graphql is used as health check and is called very frequently. This rlock on adminserver mutex makes the /probe/graphql requests block while lazy loading when restore operation gets triggered at the startup. That leads to so many go routines being spun up. (cherry picked from commit 5ad40d8)
#8619) After snapshot stream is received from other running alpha, ensure that the new graphql schema is loaded into memory . Sequence of steps in the test: * Stop one alpha container * Add data into cluster + apply new graphql schema * Start alpha container * Check whether the newly started alpha got graphql schema in snapshot.
Whenever an error happens in CommitToDisk, we retry the function according to x.config.MaxRetries. However, the value was set to -1, which causes the function to get stuck in infinite loop. This leads to dgraph not being able to commit the transaction, nor move forward to newer queries. CommitToDisk function is required to be pushed to disk. In case of an error, different alphas in a group can end up having different data leading to data loss. To avoid such issues, we panic in case we are not able to CommitToDisk after 10 retries. Once the alpha is restarted, if the issue is fixed, the alpha would start to work again. This way, Alphas wont fail silently and we would know if there was any issue occuring. Fixes: https://github.com/dgraph-io/projects/issues/85
…t json marshal issues (#8601) We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: - Live loader using guardian of galaxy - Backup and List Backup - Http clients and Ratel - Schema and predicate Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <namespace>-<attribute> (- is the hyphen literal) <namespace> is a string "81" in hex We also update the manifest version after update. This diff takes care that older backups are still compatible and can be used to restore. Contains: #7838 #7828 #7825 #7815 #7810
…ing (#8622) Proposal key initialisation has a bug where we want to reserve first 2 bytes for node id but do not do because we read random bytes in all 8 bytes and do a logical OR. This PR adds a test case and fixes the logic of initialising the proposal key. --------- Co-authored-by: Ahsan Barkati <[email protected]>
…ize (#8652) This PR fixes issue in building docker-images during CI and releases. It uses cache busting to ensure `apt-get update` is always run. It also reduces the image size by removing `/var/lib/apt/lists`
## Problem Currently the minimum version of Go required to build Dgraph is 1.12. This can be found at the top of the `go.mod` file. In order to bring in features like generics, we would have to bump up the minimum required version to 1.19. This PR shows the difference after changing 1.12 to 1.18 and running `go mod tidy`. Note that we already build Dgraph using Go 1.18, so bumping up to 1.19 is not a huge leap and will allow us to bring in many new features in the future. ## Why are there now two require blocks? This PR actually is not making any substantive changes to Dgraph's dependencies. The huge apparent increase in indirect dependencies is actually just Go showing us transitive dependencies that already existed. For more info see [here](#8423 (comment)). This is simply a change in how dependencies are described in the go.mod, not in the actual dependencies themselves. The first require block lists all direct dependencies, and the second require block lists all secondary (or tertiary, etc.) dependencies. _These dependencies already existed_ - the go.mod just makes that more explicit.
all-seeing-code
dismissed stale reviews from matthewmcneely and mangalaman93
via
3246cbd
all-seeing-code
force-pushed
the
anurag/cherry-pick-proposal-fix
branch
from
ec3829e to
3246cbd
Compare
mangalaman93
approved these changes
Feb 8, 2023
harshil-goel
reviewed
Feb 8, 2023
harshil-goel
approved these changes
Feb 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Change the proposal's unique key to an atomic counter instead of using a randomly generated key.
(cherry picked from commit a515d0d) (cherry picked from commit 2aa3d3e)