[Breaking flag] fix(Backups): new badger SuperFlag, NumGoroutines option solves OOM crashes

[karlmcguire]

(This PR is for DGRAPH-2953, paired with dgraph-io/badger#1656 which adds the Badger options.)

On master, Alpha nodes would crash during a backup on a machine with low memory (<6gb in my tests). This was because the default number of goroutines was hardcoded as 16 which is used to create local threads and used to overwrite Stream.numGo.

The solution is to add a --badger "goroutines=8; ..." SuperFlag to Alpha and anything else that might want to set Badger options. So far I've also added it to bulk/run.go, but that might be too much for this PR. The command line flags for Badger should be the same across each tool, for consistency.

The question now is how we should calculate defaults. The default goroutines value of 8 seems to be working fine on a 4gb container, but should we lessen it for machines with less memory?

---

This change is 

[jarifibrahim]

There's one more cache_mb flags in the x/flags.go file.

[jarifibrahim]

We also have the posting list cache. We can move both the badger caches inside the --badger flag and keep the posting list size flag outside.

Something like dgraph --badger="cache_mb=100; cache_percentage=50,50" --posting-cache-mb=50

[karlmcguire]

We need this goroutines flag in Alpha for creating local threads during backups, and Badger uses the goroutines value as the default Stream.numGo.

[karlmcguire]

We need this compression flag in Alpha for setting the compression type of Badger.

[karlmcguire]

We need this compression flag in Bulk for setting the compression type of Badger (similar to how Alpha uses it).

[karlmcguire]

We need this cache_mb flag in Bulk for setting the Badger cache size. The default value is set in worker/server_state.go.

[karlmcguire]

We need this cache_percentage flag in Bulk for setting the BlockCacheSize and IndexCacheSize.

[karlmcguire]

Some of these defaults aren't used by Alpha, which is kind of confusing...

We use goroutines and compression in Alpha.

We use compression, cache_mb, and cache_percentage in Bulk.

It doesn't make sense to split up the BadgerDefaults across packages, but it might be worth adding a comment explaining which packages use which options.

[manishrjain]

Reviewable status: 0 of 5 files reviewed, 14 unresolved discussions (waiting on @karlmcguire and @vvbalaji-dgraph)

---

dgraph/cmd/alpha/run.go, line 609 at r2 (raw file):

	walCache := (cachePercent[3] * (totalCache << 20)) / 100

	badger := x.NewSuperFlag(Alpha.Conf.GetString("badger")).MergeAndCheckDefault(worker.BadgerDefaults)

100 chars.

---

dgraph/cmd/bulk/run.go, line 120 at r2 (raw file):

	flag.String("badger", worker.BadgerDefaults,
		`Various badger options.
	compression=[none, zstd:level, snappy] specifies the compression algorithm and the compression

This string can be in one place. Also, we should be using Badger options directly. Whatever options Badger already has, should do directly to Badger.

---

dgraph/cmd/bulk/run.go, line 132 at r2 (raw file):

func run() {
	badger := x.NewSuperFlag(Bulk.Conf.GetString("badger")).MergeAndCheckDefault(worker.BadgerDefaults)

100 chars.

---

worker/server_state.go, line 34 at r2 (raw file):

Previously, karlmcguire (Karl McGuire) wrote…

Some of these defaults aren't used by Alpha, which is kind of confusing...

We use goroutines and compression in Alpha.

We use compression, cache_mb, and cache_percentage in Bulk.

It doesn't make sense to split up the BadgerDefaults across packages, but it might be worth adding a comment explaining which packages use which options.

Let's not change the default value of cache.

[karlmcguire]

This string can be in one place.

Where? We have Badger flags for Alpha and Bulk, both using different flags. Alpha used to expose compression and goroutine, Bulk used to expose compression, cache_mb, and cache_percentage. If we get rid of cache_mb and cache_percentage then we could probably just use the same compression and goroutine flag string for Alpha and Bulk.

Also, we should be using Badger options directly. Whatever options Badger already has, should do directly to Badger.

I'm not sure what this means. Are you saying to keep this rather than using badger.FromSuperFlag?

Let's not change the default value of cache.

We're currently using cache_mb and cache_percentage in Bulk here for the block and index caches. Should we not expose these flags?