Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add --cache_mb and --cache_percentage flag #6271

Merged
merged 11 commits into from
Aug 27, 2020
Merged

feat: add --cache_mb and --cache_percentage flag #6271

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1cf5d3b
add --cache_mv and --cache-percentage flag
NamanJain8 Aug 25, 2020
acb6bfa
address comments
NamanJain8 Aug 25, 2020
778efcf
address comments
NamanJain8 Aug 25, 2020
83a28a9
Use badger from release/v2.2007 branch
Aug 26, 2020
84673e4
make deepsource happy
NamanJain8 Aug 26, 2020
88ced28
Minor review comments addressed
Aug 26, 2020
42d8373
Fix go.mod and go.sum
Aug 26, 2020
6574a4c
Merge branch 'naman/add-cache-flag' of https://github.com/dgraph-io/d…
Aug 26, 2020
67cf784
fix line length in x.go
Aug 26, 2020
b81d301
Update badger to v2.2007.2
Aug 26, 2020
bd5f726
address review comments
NamanJain8 Aug 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion dgraph/cmd/alpha/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
_ "net/http/pprof" // http profiler
"os"
"os/signal"
"strconv"
"strings"
"sync/atomic"
"syscall"
Expand Down Expand Up @@ -203,6 +204,10 @@ they form a Raft group and provide synchronous replication.
flag.Bool("ludicrous_mode", false, "Run alpha in ludicrous mode")
flag.Bool("graphql_extensions", true, "Set to false if extensions not required in GraphQL response body")
flag.Duration("graphql_poll_interval", time.Second, "polling interval for graphql subscription.")

// Add cache flags
flag.Int64("cache_mb", 0, "Total size of cache (in MB) to be used in dgraph")
flag.String("cache-percentage", "30:30:20:20", "Cache percentages for various caches (pblocksize:pindexsize:windexsize:postingListsize)")
}

func setupCustomTokenizers() {
Expand Down Expand Up @@ -577,10 +582,45 @@ func run() {
}
bindall = Alpha.Conf.GetBool("bindall")

totalCache := int64(Alpha.Conf.GetInt("cache_mb"))
cachePercentage := Alpha.Conf.GetString("cache-percentage")

cp := strings.Split(cachePercentage, ":")
// Sanity checks
x.AssertTruef(totalCache >= 0, "ERROR: Cache size must be non-negative")
if len(cp) != 4 {
log.Fatalf("ERROR: cache percentage format is pblocksize:pindexsize:windexsize:postingListsize")
}

var cachePercent [4]int64
percentSum := 0
for idx, percent := range cp {
x, err := strconv.Atoi(percent)
if err != nil {
log.Fatalf("ERROR: unable to parse cache percentages")
}
if x < 0 {
log.Fatalf("ERROR: cache percentage cannot be negative")
}
cachePercent[idx] = int64(x)
percentSum += x
}

if percentSum != 100 {
log.Fatalf("ERROR: cache percentage does not sum up to 100")
}
pBlockSize := (cachePercent[0] * (totalCache << 20)) / 100
pIndexSize := (cachePercent[1] * (totalCache << 20)) / 100
wIndexSize := (cachePercent[2] * (totalCache << 20)) / 100
postingListSize := (cachePercent[3] * (totalCache << 20)) / 100

opts := worker.Options{
BadgerCompressionLevel: Alpha.Conf.GetInt("badger.compression_level"),
PostingDir: Alpha.Conf.GetString("postings"),
WALDir: Alpha.Conf.GetString("wal"),
PBlockSize: pBlockSize,
PIndexSize: pIndexSize,
WIndexSize: wIndexSize,

MutationsMode: worker.AllowMutations,
AuthToken: Alpha.Conf.GetString("auth_token"),
Expand Down Expand Up @@ -701,7 +741,7 @@ func run() {
// Posting will initialize index which requires schema. Hence, initialize
// schema before calling posting.Init().
schema.Init(worker.State.Pstore)
posting.Init(worker.State.Pstore)
posting.Init(worker.State.Pstore, postingListSize)
defer posting.Cleanup()
worker.Init(worker.State.Pstore)

Expand Down
8 changes: 4 additions & 4 deletions dgraph/cmd/bulk/reduce.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,10 +84,10 @@ func (r *reducer) run() error {
splitWriter := tmpDb.NewManagedWriteBatch()

ci := &countIndexer{
reducer: r,
writer: writer,
reducer: r,
writer: writer,
splitWriter: splitWriter,
tmpDb: tmpDb,
tmpDb: tmpDb,
}
sort.Slice(partitionKeys, func(i, j int) bool {
return bytes.Compare(partitionKeys[i], partitionKeys[j]) < 0
Expand Down Expand Up @@ -130,7 +130,7 @@ func (r *reducer) createBadgerInternal(dir string, compression bool) *badger.DB

opt := badger.DefaultOptions(dir).WithSyncWrites(false).
WithTableLoadingMode(bo.MemoryMap).WithValueThreshold(1 << 10 /* 1 KB */).
WithLogger(nil).WithMaxCacheSize(1 << 20).
WithLogger(nil).WithBlockCacheSize(1 << 20).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we use the flag configured instead of 1 << 20 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should. That would require adding a flag in dgraph bulk. @jarifibrahim What do you say?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bulk loader shouldn't even have a cache. I'll create a ticket for this. This Pr doesn't need to block on this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that those tmpDBs are being read. We need a cache here but it has to be configurable. I have created DGRAPH-2352

WithEncryptionKey(r.opt.EncryptionKey)

// Overwrite badger options based on the options provided by the user.
Expand Down
2 changes: 1 addition & 1 deletion dgraph/cmd/debug/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -784,7 +784,7 @@ func run() {
db, err = badger.OpenManaged(bopts)
}
x.Check(err)
posting.Init(db)
posting.Init(db, 0)
defer db.Close()

if isWal {
Expand Down
41 changes: 40 additions & 1 deletion dgraph/cmd/zero/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ package zero

import (
"context"
"strconv"
"strings"

// "errors"
"fmt"
"log"
Expand Down Expand Up @@ -55,6 +58,9 @@ type options struct {
w string
rebalanceInterval time.Duration
LudicrousMode bool

totalCache int64
cachePercentage string
}

var opts options
Expand Down Expand Up @@ -101,6 +107,10 @@ instances to achieve high-availability.
" exporter does not support annotation logs and would discard them.")
flag.Bool("ludicrous_mode", false, "Run zero in ludicrous mode")
flag.String("enterprise_license", "", "Path to the enterprise license file.")

// Add cache flags
flag.Int64("cache_mb", 0, "Total size of cache (in MB) to be used in dgraph")
flag.String("cache-percentage", "75:25", "Cache percentages for various caches (blockcache:indexCache)")
}

func setupListener(addr string, port int, kind string) (listener net.Listener, err error) {
Expand Down Expand Up @@ -183,6 +193,8 @@ func run() {
w: Zero.Conf.GetString("wal"),
rebalanceInterval: Zero.Conf.GetDuration("rebalance_interval"),
LudicrousMode: Zero.Conf.GetBool("ludicrous_mode"),
totalCache: int64(Zero.Conf.GetInt("cache_mb")),
cachePercentage: Zero.Conf.GetString("cache-percentage"),
}
glog.Infof("Setting Config to: %+v", opts)

Expand Down Expand Up @@ -215,6 +227,33 @@ func run() {
opts.rebalanceInterval)
}

cp := strings.Split(opts.cachePercentage, ":")
// Sanity checks
x.AssertTruef(opts.totalCache >= 0, "ERROR: Cache size must be non-negative")
if len(cp) != 2 {
log.Fatalf("ERROR: cache percentage format is blockcache:indexCache")
}

var cachePercent [2]int64
percentSum := 0
for idx, percent := range cp {
x, err := strconv.Atoi(percent)
if err != nil {
log.Fatalf("ERROR: ERROR: unable to parse cache percentages")
}
if x < 0 {
log.Fatalf("ERROR: cache percentage cannot be negative")
}
cachePercent[idx] = int64(x)
percentSum += x
}

if percentSum != 100 {
log.Fatalf("ERROR: cache percentage does not sum up to 100")
}
blockCache := (cachePercent[0] * (opts.totalCache << 20)) / 100
indexCache := (cachePercent[1] * (opts.totalCache << 20)) / 100

grpc.EnableTracing = false
otrace.ApplyConfig(otrace.Config{
DefaultSampler: otrace.ProbabilitySampler(Zero.Conf.GetFloat64("trace"))})
Expand All @@ -235,7 +274,7 @@ func run() {
// Open raft write-ahead log and initialize raft node.
x.Checkf(os.MkdirAll(opts.w, 0700), "Error while creating WAL dir.")
kvOpt := badger.LSMOnlyOptions(opts.w).WithSyncWrites(false).WithTruncate(true).
WithValueLogFileSize(64 << 20).WithMaxCacheSize(10 << 20).WithLoadBloomsOnOpen(false)
WithValueLogFileSize(64 << 20).WithBlockCacheSize(blockCache).WithIndexCacheSize(indexCache).WithLoadBloomsOnOpen(false)

kvOpt.ZSTDCompressionLevel = 3

Expand Down
21 changes: 11 additions & 10 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ require (
contrib.go.opencensus.io/exporter/jaeger v0.1.0
contrib.go.opencensus.io/exporter/prometheus v0.1.0
github.com/99designs/gqlgen v0.12.2
github.com/AndreasBriese/bbloom v0.0.0-20190825152654-46b345b51c96 // indirect
github.com/DataDog/datadog-go v0.0.0-20190425163447-40bafcb5f6c1 // indirect
github.com/DataDog/opencensus-go-exporter-datadog v0.0.0-20190503082300-0f32ad59ab08
github.com/DataDog/zstd v1.4.5 // indirect
Expand All @@ -17,10 +18,11 @@ require (
github.com/blevesearch/segment v0.0.0-20160915185041-762005e7a34f // indirect
github.com/blevesearch/snowballstem v0.0.0-20180110192139-26b06a2c243d // indirect
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd
github.com/dgraph-io/badger/v2 v2.2007.1
github.com/dgraph-io/badger v1.6.1 // indirect
github.com/dgraph-io/badger/v2 v2.0.1-rc1.0.20200825103817-c3b15824dc14
github.com/dgraph-io/dgo/v200 v200.0.0-20200805103119-a3544c464dd6
github.com/dgraph-io/graphql-transport-ws v0.0.0-20200715131837-c0460019ead2
github.com/dgraph-io/ristretto v0.0.3-0.20200630154024-f66de99634de
github.com/dgraph-io/ristretto v0.0.4-0.20200820164438-623d8ef1614b
github.com/dgrijalva/jwt-go v3.2.0+incompatible
github.com/dgrijalva/jwt-go/v4 v4.0.0-preview1
github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
Expand All @@ -31,10 +33,10 @@ require (
github.com/gogo/protobuf v1.3.1
github.com/golang/geo v0.0.0-20170810003146-31fb0106dc4a
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/protobuf v1.3.5
github.com/golang/protobuf v1.4.2
github.com/golang/snappy v0.0.1
github.com/google/codesearch v1.0.0
github.com/google/go-cmp v0.4.0
github.com/google/go-cmp v0.5.0
github.com/google/uuid v1.0.0
github.com/gorilla/websocket v1.4.2
github.com/graph-gophers/graphql-go v0.0.0-20200309224638-dae41bde9ef9
Expand All @@ -47,7 +49,6 @@ require (
github.com/pkg/errors v0.9.1
github.com/pkg/profile v1.2.1
github.com/prometheus/client_golang v0.9.3-0.20190127221311-3c4408c8b829
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 // indirect
github.com/prometheus/common v0.4.1 // indirect
github.com/prometheus/procfs v0.0.0-20190517135640-51af30a78b0e // indirect
github.com/spf13/cast v1.3.0
Expand All @@ -61,13 +62,13 @@ require (
github.com/willf/bitset v0.0.0-20181014161241-71fa2377963f // indirect
go.etcd.io/etcd v0.0.0-20190228193606-a943ad0ee4c9
go.opencensus.io v0.21.0
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
golang.org/x/net v0.0.0-20200822124328-c89045814202
golang.org/x/sync v0.0.0-20190423024810-112230192c58
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9
golang.org/x/sys v0.0.0-20200824131525-c12d262b63d8
golang.org/x/text v0.3.2
google.golang.org/genproto v0.0.0-20190516172635-bb713bdc0e52 // indirect
google.golang.org/grpc v1.23.0
google.golang.org/grpc v1.27.0
google.golang.org/protobuf v1.25.0 // indirect
gopkg.in/DataDog/dd-trace-go.v1 v1.13.1 // indirect
gopkg.in/ini.v1 v1.48.0 // indirect
gopkg.in/yaml.v2 v2.2.4
Expand Down
Loading