Merge master into Release v20.11 #7116
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* remove <X> <uid> <X> type of triplets from the output * fix output for recurse queries * fix for recurse queries * fix test * add Test for ignoring uid * minor modification * fix solution for recurse
…onfig (#7013) Pass the encryption key to `raftwal.InitEncrypted()` instead of directly reading from the workerConfig. Fix the debug tool and raftwal migrate, to allow using them with encryption enabled.
Fix the incorrect log about "writing to db" despite no db write being done while Flush() in xidmap.
* added backup policy * rephrase
* Fix typos in upsert and deploy docs. * Update backup policy section to link to docs for backups (enterprise feature) and data exports (open-source feature).
* Use []*gql.GraphQuery instead for query and mutation rewriting * Remove unncessary code in addFilterToTopLevel * Address comments
* fix: added comment docstring for ExportInput format
* fixing plugin test to run with go script * making http login to respect refresh token in case they expire * fixing bulk test cases * skipping bulk test for now * changing scripts tests to go * fixing bulk, live test cases to be run under load test suite * basic refactoring * fixing test cases and removing ludicrous queries * fix custom plugins test * refactoring basic things + renaming functions * fixing review changes * merging conflict fix * removing flags because of recent merge changes * fixing test cases because of kebab case removal
Zero snapshots were being calculated in a round about way. This got worse due to a recent bug introduction, which caused Zeros to not get snapshots if we don't purge keyCommits. This PR make Zero leader calculate the snapshots based on the checkpoints from Alpha leaders (instead of their snapshots). It then proposes this snapshot to the Zero group, causing all members to create snapshot and purge keyCommit map at the same time.
In the last commit, I was picking up checkpoint index instead of the timestamp corresponding to the checkpoint from Alpha leaders. This is a fix for that.
- Add UncompressedBytes to pb.Tablet denoting the uncompressed size of tablet. - Rename pb.Tablet.Space -> pb.Tablet.OnDiskBytes - Simplify tablet size calculation as badger does not have internal keys now.
…oot auth query (GRAPHQL-854) (#7100) This PR moves the pagination parameters for root user query to root auth query. This greatly optimizes the number of nodes that need to be processed by child auth queries.
…888) (#7103) Fixes GRAPHQL-888. Previously, if you ran this request: ``` $ curl -H "Content-Type: application/json" http://localhost:8080/admin -d '{"query": "mutation {config(input: {logRequest: false}){response {code message}}}"}' ``` Alpha logs would also print this: ``` I1205 22:22:51.684693 2681396 middlewares.go:178] GraphQL admin mutation. Name = config I1205 22:22:51.684724 2681396 config.go:38] Got config update through GraphQL admin API I1205 22:22:51.684810 2681396 worker.go:138] Updating cacheMb to 0 ``` Indicating that cacheMb was also updated, even it wasn't specified in the request. This PR fixes this issue.
…ns for auth queries (GRAPHQL-854) (#7105) We don't need to query `uid` for auth queries, as they always have at least one field in their selection set. This optimizes the number of `touched_uids` greatly, while time is not much improved here.
… (GRAPHQL-854) (#7107) This PR moves the filter for user child queries to the first child auth variable. This will reduce the number of uids that need to be processed by the child auth rules.
Add missing Discard(ctx)
…7094) * Fix repeatitive fields in aggregate query * Address comments
…7059) In worker/draft.go, in Raft.Ready() when we get committed entries, we were unmarshalling them to pb.Proposals upfront. Instead, push them to applyCh as it is, and only when we need them, we unmarshal them. This should fix an OOM issue we see with pb.Proposals unmarshal taking up a lot of memory. We achieve this by switching proposal.Key which is a string to a uint64 key, which can be put directly in the raft entry.Data in the first 8 bytes. The rest are used to marshal proposal. Additional changes: * Bump up max pending proposal size to 256 MB to increase proposal throughput. Co-authored-by: Ahsan Barkati <[email protected]> Co-authored-by: NamanJain8 <[email protected]> Co-authored-by: Daniel Mai <[email protected]>
aman-bansal
requested review from
danielmai,
manishrjain and
MichaelJCompton
as code owners
aman-bansal
requested review from
pawanrawal and
vvbalaji-dgraph
as code owners
github-actions
bot
added
area/bulk-loader
|
✔️ Deploy preview for dgraph-docs ready! 🔨 Explore the source changes: 08ef3ad 🔍 Inspect the deploy logs: https://app.netlify.com/sites/dgraph-docs/deploys/5fd35dec2e6ada0007269c20 😎 Browse the preview: https://deploy-preview-7116--dgraph-docs.netlify.app |
aman-bansal
force-pushed
the
aman/merge_master_20.11
branch
from
7cb96d8 to
74ee434
Compare
…#7110) The bulk loader was not considering the last slice of cbuf in appendToList function. This change fixes the issue.
…red parents (GRAPHQL-854) (#7111) This PR optimizes auth query rewriting by deriving child auth queries from auth filtered parents instead of all possible parents. This reduces the number of `touched_uids` at deep levels. This performance benefit comes into effect when there are a lot of nested levels in auth rules.
aman-bansal
force-pushed
the
aman/merge_master_20.11
branch
from
49416fc to
f434397
Compare
pawanrawal
approved these changes
Dec 11, 2020
* Fix aggregate queries on empty data * Empty commit to make the CLA pass
aman-bansal
force-pushed
the
aman/merge_master_20.11
branch
from
f434397 to
08ef3ad
Compare
vvbalaji-dgraph
approved these changes
Dec 11, 2020
Use the latest ristretto to bring in the jemalloc-stats fix. (cherry picked from commit 188711a)
danielmai
approved these changes
Dec 11, 2020
There was a problem hiding this comment.
Reviewed 122 of 122 files at r1.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @manishrjain and @MichaelJCompton)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changelog
8b8b0ad Fix(GraphQL): Fix Aggregate queries on empty data
61ff62f fix(GraphQL): Correctly apply order for root auth query (GRAPHQL-854) (#7117)
e2d93de perf(GraphQL): Part-4: Child auth queries are derived from auth filtered parents (GRAPHQL-854) (#7111)
004ca03 Fix(bulk): Correctly append all the slices of cbuf while reduce phase (#7110)
2894f6b chore(GC): Increase discard ratio from 0.7 to 0.9 (#7114)
2e5499f [BREAKING] Fix(OOM): Don't unmarshal pb.Proposals until we need them (#7059)
7f12eee Fix(GraphQL): Add support for repeatitive fields in aggregate query (#7094)
67abf95 perf(GraphQL): Part-3: Filter child auth queries as early as possible (GRAPHQL-854) (#7107)
9baa3c7 perf(GraphQL): Part-2: Remove unnecessary
dgraph.uid : uidselections for auth queries (GRAPHQL-854) (#7105)c03c327 fix(GraphQL): don't update cacheMb if not specified by user (GRAPHQL-888) (#7103)
5042370 perf(GraphQL): Part-1: Pagination for root user query is applied at root auth query (GRAPHQL-854) (#7100)
843fe1f feat(tablet): add uncompressed size to pb.Tablet (#7095)
5efdfbf Fix(Zero): Fix how Zero snapshots and purge works (#7096)
73b36d4 [docs] added backup policy (#7084)
This change is