Skip to content

Commit

Permalink
Avoid crashing live loader in case the network is interrupted. (#5268)
Browse files Browse the repository at this point in the history
Live loader currently runs `x.Fatalf` the moment it has a connection interrupt. Instead, it should just retry indefinitely.

Also, remove a `%+v` error print for aborting transactions, which causes the entire error stack trace to be printed, which makes it look like a crash.
  • Loading branch information
manishrjain authored and danielmai committed Apr 22, 2020
1 parent 072f33e commit 31c16e2
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 2 deletions.
7 changes: 6 additions & 1 deletion dgraph/cmd/live/batch.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,12 @@ func handleError(err error, isRetry bool) {
s := status.Convert(err)
switch {
case s.Code() == codes.Internal, s.Code() == codes.Unavailable:
x.Fatalf(s.Message())
// Let us not crash live loader due to this. Instead, we should infinitely retry to
// reconnect and retry the request.
dur := time.Duration(1+rand.Intn(60)) * time.Second
fmt.Printf("Connection has been possibly interrupted. Got error: %v."+
" Will retry after %s.\n", err, dur.Round(time.Second))
time.Sleep(dur)
case strings.Contains(s.Message(), "x509"):
x.Fatalf(s.Message())
case s.Code() == codes.Aborted:
Expand Down
2 changes: 1 addition & 1 deletion worker/draft.go
Original file line number Diff line number Diff line change
Expand Up @@ -1289,7 +1289,7 @@ func (n *node) abortOldTransactions() {
glog.Infof("Found %d old transactions. Acting to abort them.\n", len(starts))
req := &pb.TxnTimestamps{Ts: starts}
err := n.blockingAbort(req)
glog.Infof("Done abortOldTransactions for %d txns. Error: %+v\n", len(req.Ts), err)
glog.Infof("Done abortOldTransactions for %d txns. Error: %v\n", len(req.Ts), err)
}

// calculateSnapshot would calculate a snapshot index, considering these factors:
Expand Down

0 comments on commit 31c16e2

Please sign in to comment.