Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OPTIMIZATION] Optimize snapshot creation #4901

Merged
merged 28 commits into from
Apr 9, 2020
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1bb3475
Improve CreateSnapshot()
ashish-goswami Mar 5, 2020
ef4f46d
Temp
ashish-goswami Mar 6, 2020
0b4e96f
Temp2
ashish-goswami Mar 8, 2020
ff70fa1
Refactor
ashish-goswami Mar 9, 2020
8f4160e
Remove defer
ashish-goswami Mar 10, 2020
b01a1ec
Fix FirstIndex
ashish-goswami Mar 10, 2020
27ad2ff
Fix NumEntries()
ashish-goswami Mar 10, 2020
42c1b0d
Minor changes
ashish-goswami Mar 10, 2020
3ff5f9e
Remove changes from draft.go
ashish-goswami Mar 10, 2020
d90b255
Minor fix
ashish-goswami Mar 10, 2020
4a65e7f
Delete entries before first at start
ashish-goswami Mar 11, 2020
a7aa99d
Minor fix
ashish-goswami Mar 12, 2020
ae8b9be
Address some review comments
ashish-goswami Mar 13, 2020
887ab67
Temp
ashish-goswami Mar 17, 2020
8acc9be
Fix broken tests
ashish-goswami Mar 17, 2020
e77b238
Refactor code
ashish-goswami Mar 18, 2020
71c0d96
Minor change
ashish-goswami Mar 18, 2020
81d1d3f
Address review comments
ashish-goswami Mar 18, 2020
e4e46f8
Address review comment
ashish-goswami Mar 19, 2020
0be3eca
Merge remote-tracking branch 'origin/master' into ashish/snapshot
ashish-goswami Mar 19, 2020
964d32d
Merge remote-tracking branch 'origin/master' into ashish/snapshot
ashish-goswami Mar 20, 2020
abd0ad1
docs: Add Releases page. (#4954)
danielmai Mar 20, 2020
c15189f
docs: Set v1.2.2 as latest docs. (#4993)
danielmai Mar 20, 2020
cf43770
Add comment regarding stream IDs for split lists in bulk loader. (#4982)
martinmr Mar 20, 2020
ca86bb6
Update CHANGELOG for v20.03.0-beta.20200320. (#4994)
lgalatin Mar 20, 2020
8269220
Merge remote-tracking branch 'origin/master' into ashish/snapshot
ashish-goswami Mar 21, 2020
233ba82
Minor change
ashish-goswami Mar 23, 2020
2ff7547
Merge remote-tracking branch 'origin/master' into ashish/snapshot
ashish-goswami Apr 3, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove changes from draft.go
ashish-goswami committed Mar 10, 2020

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 3ff5f9e4cdf905168ecfeeae974d0a8c38f62e36
8 changes: 3 additions & 5 deletions worker/draft.go
Original file line number Diff line number Diff line change
@@ -409,9 +409,8 @@ func (n *node) applyCommitted(proposal *pb.Proposal) error {
return nil
}
n.elog.Printf("Creating snapshot: %+v", snap)
// glog.Infof("Creating snapshot at index: %d. ReadTs: %d.\n", snap.Index, snap.ReadTs)
glog.Infof("Creating snapshot at index: %d. ReadTs: %d.\n", snap.Index, snap.ReadTs)

now := time.Now()
data, err := snap.Marshal()
x.Check(err)
for {
@@ -424,7 +423,6 @@ func (n *node) applyCommitted(proposal *pb.Proposal) error {
}
// Roll up all posting lists as a best-effort operation.
n.rollupCh <- snap.ReadTs
glog.Infof("Created snapshot at index: %d. ReadTs: %d. in time: %s\n", snap.Index, snap.ReadTs, time.Since(now))
return nil
}
x.Fatalf("Unknown proposal: %+v", proposal)
@@ -1361,8 +1359,8 @@ func (n *node) calculateSnapshot(startIdx uint64, discardN int) (*pb.Snapshot, e

if int(numDiscarding) < discardN {
span.Annotate(nil, "Skipping snapshot because insufficient discard entries")
glog.Infof("Skipping snapshot at index: %d. firstIndex: %d. Insufficient discard entries: %d."+
" MinPendingStartTs: %d\n", snapshotIdx, first, numDiscarding, minPendingStart)
glog.Infof("Skipping snapshot at index: %d. Insufficient discard entries: %d."+
" MinPendingStartTs: %d\n", snapshotIdx, numDiscarding, minPendingStart)
return nil, nil
}