Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check context more often while sorting. #1192

Closed
wants to merge 5 commits into from
Closed

Check context more often while sorting. #1192

wants to merge 5 commits into from

Conversation

ashwin95r
Copy link
Contributor

@ashwin95r ashwin95r commented Jul 14, 2017

This change is Reviewable

@manishrjain
Copy link
Contributor

:lgtm: Many comments. Address them and then submit.


Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, 3 unresolved discussions.


worker/sort.go, line 140 at r1 (raw file):

			// Copy, otherwise it'd affect the destUids and hence the srcUids of Next level.
			tempList := &protos.List{ts.UidMatrix[i].Uids}
			if err := sortByValue(ctx, ts.Attr, ts.Langs, tempList, sType, ts.Desc); err != nil {

This function is accessing so many ts values. Why not just give it the ts struct directly?


worker/sort.go, line 232 at r1 (raw file):

			// Intersect every UID list with the index bucket, and update their
			// results (in out).
			err := intersectBucket(ctx, ts, ts.Attr, token, out)

Again, what's the point of giving both ts and then ts.Attr?

If this is simple refactoring, just make it part of this PR.


worker/sort.go, line 282 at r1 (raw file):

	go func() {
		// Wait for 3ms before starting
		time.Sleep(3 * time.Millisecond)

Check context right after time.Sleep.


Comments from Reviewable

@ashwin95r
Copy link
Contributor Author

Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions.


worker/sort.go, line 140 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

This function is accessing so many ts values. Why not just give it the ts struct directly?

Done.


worker/sort.go, line 232 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Again, what's the point of giving both ts and then ts.Attr?

If this is simple refactoring, just make it part of this PR.

Done.


worker/sort.go, line 282 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Check context right after time.Sleep.

Done.


Comments from Reviewable

@manishrjain
Copy link
Contributor

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion.


worker/sort.go, line 284 at r2 (raw file):

		time.Sleep(3 * time.Millisecond)
		select {
		case <-ctx.Done():

select case between time.After(3*time.Millisecond) and ctx.Done. Once you're through it, only then do the sortWithoutIndex (it shouldn't be in default).


Comments from Reviewable

@manishrjain
Copy link
Contributor

Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions.


worker/sort.go, line 324 at r2 (raw file):

// intersectBucket intersects every UID list in the UID matrix with the
// indexed bucket.
func intersectBucket(ctx context.Context, ts *protos.SortMessage, token string, out []intersectedList) error {

100 chars.


worker/sort.go, line 335 at r2 (raw file):

	key := x.IndexKey(attr, token)
	// Don't put the Index keys in memory.
	pl, decr := posting.Get(key, 1)

How come the group is always 1?

Anyhow, looks like Get doesn't need group. Can you remove that as well?


Comments from Reviewable

@manishrjain
Copy link
Contributor

Reviewed 1 of 1 files at r2.
Review status: 0 of 1 files reviewed at latest revision, 3 unresolved discussions.


Comments from Reviewable

@ashwin95r
Copy link
Contributor Author

Review status: 0 of 2 files reviewed at latest revision, 3 unresolved discussions.


worker/sort.go, line 284 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

select case between time.After(3*time.Millisecond) and ctx.Done. Once you're through it, only then do the sortWithoutIndex (it shouldn't be in default).

Right! Done.


worker/sort.go, line 324 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

100 chars.

Done.


worker/sort.go, line 335 at r2 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

How come the group is always 1?

Anyhow, looks like Get doesn't need group. Can you remove that as well?

Done.


Comments from Reviewable

@manishrjain
Copy link
Contributor

:lgtm: Very important comment. Address before submitting.


Reviewed 2 of 2 files at r3.
Review status: all files reviewed at latest revision, 1 unresolved discussion.


worker/sort.go, line 285 at r3 (raw file):

			// Wait between ctx chan and time chan.
		case <-ctx.Done():
			resCh <- result{err: ctx.Err()}

No return??? Focus, man!


Comments from Reviewable

@ashwin95r
Copy link
Contributor Author

Review status: 1 of 2 files reviewed at latest revision, 1 unresolved discussion.


worker/sort.go, line 285 at r3 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

No return??? Focus, man!

Ah my bad. Done.


Comments from Reviewable

@ashwin95r
Copy link
Contributor Author

Merged.

@ashwin95r ashwin95r closed this Jul 14, 2017
@pawanrawal pawanrawal deleted the opt/sort branch December 19, 2017 08:39
jarifibrahim pushed a commit that referenced this pull request Mar 16, 2020
Important changes
```
 - Changes to overlap check in compaction.
 - Remove 'this entry should've been caught' log.
 - Changes to write stalling on levels 0 and 1.
 - Compression is disabled by default in Badger.
 - Bloom filter caching in a separate ristretto cache.
 - Compression/Encryption in background.
 - Disable cache by default in badger.
```

The following new changes are being added from badger
`git log ab4352b00a17...91c31ebe8c22`

```
91c31eb Disable cache by default (#1257)
eaf64c0 Add separate cache for bloom filters (#1260)
1bcbefc Add BypassDirLock option (#1243)
c6c1e5e Add support for watching nil prefix in subscribe API (#1246)
b13b927 Compress/Encrypt Blocks in the background (#1227)
bdb2b13 fix changelog for v2.0.2 (#1244)
8dbc982 Add Dkron to README (#1241)
3d95b94 Remove coveralls from Travis Build(#1219)
5b4c0a6 Fix ValueThreshold for in-memory mode (#1235)
617ed7c Initialize vlog before starting compactions in db.Open (#1226)
e908818 Update CHANGELOG for Badger 2.0.2 release. (#1230)
bce069c Fix int overflow for 32bit (#1216)
e029e93 Remove ExampleDB_Subscribe Test (#1214)
8734e3a Add missing package to README for badger.NewEntry (#1223)
78d405a Replace t.Fatal with require.NoError in tests (#1213)
c51748e Fix flaky TestPageBufferReader2 test (#1210)
eee1602 Change else-if statements to idiomatic switch statements. (#1207)
3e25d77 Rework concurrency semantics of valueLog.maxFid (#1184) (#1187)
4676ca9 Add support for caching bloomfilters (#1204)
c3333a5 Disable compression and set ZSTD Compression Level to 1 (#1191)
0acb3f6 Fix L0/L1 stall test (#1201)
7e5a956 Support disabling the cache completely. (#1183) (#1185)
82381ac Update ristretto to version  8f368f2 (#1195)
3747be5 Improve write stalling on level 0 and 1
5870b7b Run all tests on CI (#1189)
01a00cb Add Jaegar to list of projects (#1192)
9d6512b Use fastRand instead of locked-rand in skiplist (#1173)
2698bfc Avoid sync in inmemory mode (#1190)
2a90c66 Remove the 'this entry should've caught' log from value.go (#1170)
0a06173 Fix checkOverlap in compaction (#1166)
0f2e629 Fix windows build (#1177)
03af216 Fix commit sha for WithInMemory in CHANGELOG. (#1172)
23a73cd Update CHANGELOG for v2.0.1 release. (#1181)
465f28a Cast sz to uint32 to fix compilation on 32 bit (#1175)
ea01d38 Rename option builder from WithInmemory to WithInMemory. (#1169)
df99253 Remove ErrGCInMemoryMode in CHANGELOG. (#1171)
8dfdd6d Adding changes for 2.0.1 so far (#1168)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants