make index pruning configurable via index-rules.conf #924

Dieterbe · 2018-05-24T13:22:36Z

woodsaj · 2018-08-23T17:01:02Z

It seems like we should just add the max-stale setting to the existing storage-schemas.conf rather then having a whole new config file, like was done for the reorderBuffer.

Dieterbe · 2018-08-23T17:42:54Z

maybe. i've been going back and forth on that a bit myself. the benefit of separate file is that it easily allows to add more per-pattern index tunables in the future. though i don't know yet which those would be.
also, if the pruning rules are orthogonal to the storage schemas then having them in one file would mean we need a multiple of the rules.

eg schemas:

^a.* -> retain 1y
^b.* -> retain 2y
^c.* -> retain 3y

index rules:

\.containers$ -> max-stale 1d
\.biz$ -> max-stale 30d

becomes:

^a.*\.containers$ -> retain 1y, max-stale 1d
^a.*\.biz$ -> retain 1y, max-stale 30d
^b.*\.containers$ -> retain 2y, max-stale 1d
^b.*\.biz$ -> retain 2y, max-stale 30d
^c.*\.containers$ -> retain 3y, max-stale 1d
^c.*\.biz$ -> retain 3y, max-stale 30d

while for all the simple cases (and the majority of our deployments are simple) the extra file approach makes things a bit harder, having it all in 1 file makes the complication of more complicated cases a multiple. and that's a bit concerning in terms of operability.

the benefit of single file is we save some memory by not having to track the separate IrId.

I don't see a strong reason to go either way though.

woodsaj · 2018-08-23T17:49:16Z

ok, lets stick with the separate file for now.

woodsaj · 2018-08-23T17:56:39Z

conf/indexrules.go

+}
+
+type IndexCheck struct {
+	Keep   bool


why do we need the Keep field. Cant we just set Cutoff to 0 if maxStale==0?

woodsaj · 2018-08-23T17:59:07Z

docs/config.md

+# * Valid units are s/sec/secs/second/seconds, m/min/mins/minute/minutes, h/hour/hours, d/day/days, w/week/weeks, mon/month/months, y/year/years
+
+[default]
+pattern = 


should this have a pattern set? eg .*

not needed. empty string is a valid regex that matches everything. (the first test in conf/indexrules_test.go checks this)

woodsaj · 2018-08-23T18:05:48Z

idx/cassandra/cassandra.go

-		staleTs := time.Now().Add(maxStale * -1)
-		_, err := c.Prune(staleTs)
+	for now := range ticker.C {
+		log.Debug("cassandra-idx: pruning items")


I think we should have an info level log message for when a prune starts, and an info message when the prune completes. The message on completion should log how long the prune took and number of items pruned.

we already have such info logs for the memory index and the cassandra index doesn't really do any work beyond that, but I think it's still useful. in fact i'm making the logs from both more consistent so that it'll look obvious in the logs, both memory and cassandra index logging their own steps. in the future the cassandra-idx may do its own extra steps again.

woodsaj · 2018-08-23T18:10:53Z

idx/memory/memory.go

@@ -1251,9 +1269,13 @@ func (m *MemoryIdx) Prune(oldest time.Time) ([]idx.Archive, error) {
 	pre := time.Now()

 	m.RLock()
+
+	// getting all checks once saves having to recompute the cutoff everytime we have a match
+	indexChecks := IndexRules.Checks(now)


can we move this to before we acquire the RLock(). We want to keep lock times as small as possible

woodsaj · 2018-08-23T18:30:42Z

scripts/config/index-rules.conf

+# * Valid units are s/sec/secs/second/seconds, m/min/mins/minute/minutes, h/hour/hours, d/day/days, w/week/weeks, mon/month/months, y/year/years
+
+[default]
+pattern = 


should this have a match-all pattern?

empty string is a valid regex that matches everything. (the first test in conf/indexrules_test.go checks this)

Dieterbe · 2018-08-24T07:50:01Z

PTAL

woodsaj · 2018-08-24T08:12:34Z

conf/indexrules_test.go

+		},
+	}
+	for i, c := range cases {
+		err := ioutil.WriteFile("/tmp/indexrules-test-readindexrules", []byte(c.in), 0644)


Rather then a hard coded filename, you should use ioutil.TempFile()
https://golang.org/pkg/io/ioutil/#TempFile

You also need to make sure you delete the file when done.

fix #868

note that CassandraIdx.load() returns results out of order. No idea why our tests didn't fail before this but now we correctly work around this.

Dieterbe · 2018-10-24T07:34:11Z

rebased on master

Dieterbe force-pushed the index-rules branch from 860d9c5 to 9a0dd77 Compare May 24, 2018 13:25

Dieterbe added the breaks-config label Aug 17, 2018

Dieterbe added this to the 0.10.0 milestone Aug 17, 2018

Dieterbe force-pushed the index-rules branch 7 times, most recently from a39085a to 07a49fb Compare August 23, 2018 14:14

Dieterbe requested review from woodsaj and replay August 23, 2018 14:16

Dieterbe force-pushed the index-rules branch from 07a49fb to 06535d4 Compare August 23, 2018 14:17

woodsaj reviewed Aug 23, 2018

View reviewed changes

woodsaj reviewed Aug 24, 2018

View reviewed changes

woodsaj approved these changes Aug 24, 2018

View reviewed changes

Dieterbe added 6 commits October 23, 2018 21:25

order aggregations vs schemas

b20b74f

support per-pattern configurable index pruning

e37a16a

fix #868

fix cassandraIdx tests

cbe78aa

note that CassandraIdx.load() returns results out of order. No idea why our tests didn't fail before this but now we correctly work around this.

fix MemoryIdx tests

af1a7d2

PR feedback, simplify

942ffaf

format fix

7665a65

Dieterbe added 4 commits October 23, 2018 21:28

cleaner pruning logs

378b488

use tmp file

b2311ea

rebase -i fixups

c0adfad

small tweak

136a684

Dieterbe force-pushed the index-rules branch from 52acb3f to 136a684 Compare October 24, 2018 07:33

Dieterbe merged commit 82c333a into master Oct 24, 2018

Dieterbe changed the title ~~Index rules~~ make index pruning configurable via index-rules.conf Oct 24, 2018

Dieterbe mentioned this pull request Oct 24, 2018

index-rules.conf based pruning for bigtable #1108

Closed

Dieterbe deleted the index-rules branch October 29, 2018 09:06

Dieterbe modified the milestones: 1.0, 0.11.0 Dec 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make index pruning configurable via index-rules.conf #924

make index pruning configurable via index-rules.conf #924

Dieterbe commented May 24, 2018 •

edited

Loading

woodsaj commented Aug 23, 2018

Dieterbe commented Aug 23, 2018 •

edited

Loading

woodsaj commented Aug 23, 2018

woodsaj Aug 23, 2018

Dieterbe Aug 24, 2018

woodsaj Aug 23, 2018

Dieterbe Aug 24, 2018

woodsaj Aug 23, 2018

Dieterbe Aug 24, 2018 •

edited

Loading

woodsaj Aug 23, 2018

Dieterbe Aug 24, 2018

woodsaj Aug 23, 2018

Dieterbe Aug 24, 2018

Dieterbe commented Aug 24, 2018

woodsaj Aug 24, 2018 •

edited

Loading

Dieterbe commented Oct 24, 2018

make index pruning configurable via index-rules.conf #924

make index pruning configurable via index-rules.conf #924

Conversation

Dieterbe commented May 24, 2018 • edited Loading

woodsaj commented Aug 23, 2018

Dieterbe commented Aug 23, 2018 • edited Loading

woodsaj commented Aug 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dieterbe Aug 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dieterbe commented Aug 24, 2018

woodsaj Aug 24, 2018 • edited Loading

Choose a reason for hiding this comment

Dieterbe commented Oct 24, 2018

Dieterbe commented May 24, 2018 •

edited

Loading

Dieterbe commented Aug 23, 2018 •

edited

Loading

Dieterbe Aug 24, 2018 •

edited

Loading

woodsaj Aug 24, 2018 •

edited

Loading