Skip to content
This repository was archived by the owner on Aug 23, 2023. It is now read-only.

calculate TTL relative to now when inserting into cassandra #1448

Merged
merged 5 commits into from
Aug 30, 2019
Merged

calculate TTL relative to now when inserting into cassandra #1448

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5904a9a
when inserting we want to adjust the ttl to make it relative to now
replay Aug 29, 2019
a4a9675
determine chunk span to calculate its ttl on insert
replay Aug 30, 2019
3f3486d
saving one allocation
replay Aug 30, 2019
aca9ef7
removing unused error to save another allocation
replay Aug 30, 2019
b28f095
Rename method SpanOfChunk to ExtractChunkSpan
replay Aug 30, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion store/cassandra/cassandra.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -355,6 +355,21 @@ func (c *CassandraStore) insertChunk(key string, t0, ttl uint32, data []byte) er
return nil
}

// we calculate ttl like this:
// - the chunk's t0 plus maxChunkSpan is the ts of last possible datapoint in the chunk
// - the timestamp of the last datapoint + ttl is the timestamp until when we want to keep this chunk
// - then we subtract the current time stamp to get the difference relative to now
// - the result is the ttl in seconds relative to now
relativeTtl := int64(t0+mdata.MaxChunkSpan()+ttl) - time.Now().Unix()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concern with using MaxChunkSpan is that it makes the minimum ttl of data at MaxChunkSpan. eg, if you want raw data stored for 1h, but also have rollups being stored for longer and those rollups use a chunkSpan of 6h, then the raw data will be stored for a lot longer then intended.

This is not a concern for any of our use cases, but might cause problems for other users and perhaps end2end tests.

Copy link
Contributor Author

@replay replay Aug 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a valid concern, but I'm not sure how to improve that. There are basically two possibilities:

  1. we read the data byte slice to determine the chunk span. this should only require reading the first 2 bytes, it can be done like here: https://github.com/grafana/metrictank/blob/master/mdata/chunk/itergen.go#L76
  2. we add a span property to the chunk write request's payload. but this will require updating all locations that generate chunk write requests.

I'm leaning towards 1)

Copy link
Contributor Author

@replay replay Aug 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've given 1) a try

Copy link
Contributor Author

@replay replay Aug 30, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran another test with the latest version. I've started MT with an empty cassandra instance and this schema:

1s:1h:10min:1,1min:48h:6h:1

Then I fed it with 72h of data for one metric. It created two tables for the two different TTLs metric_1/metric_32. All the TTLs in these two tables look like what we want. In each table the TTLs are nicely adjusted to the span:

cqlsh:metrictank> select key, ts, TTL(data) from metric_1;

 key                                    | ts         | ttl(data)
----------------------------------------+------------+-----------
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567179600 |      3815
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567179000 |      3215
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567178400 |      2615
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567177800 |      2015
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567177200 |      1415
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567176600 |       815
 1.d0a8110e69b0d874610aa08ab6740dfa_647 | 1567176000 |       215

(7 rows)
cqlsh:metrictank> select key, ts, TTL(data) from metric_32;

 key                                           | ts         | ttl(data)
-----------------------------------------------+------------+-----------
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567144800 |    159128
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567123200 |    137528
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567101600 |    115928
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567080000 |     94328
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567058400 |     72728
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567036800 |     51128
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1567015200 |     29528
 1.d0a8110e69b0d874610aa08ab6740dfa_sum_60_647 | 1566993600 |      7928
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567144800 |    159128
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567123200 |    137528
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567101600 |    115928
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567080000 |     94328
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567058400 |     72728
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567036800 |     51128
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1567015200 |     29528
 1.d0a8110e69b0d874610aa08ab6740dfa_min_60_647 | 1566993600 |      7928
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567144800 |    159128
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567123200 |    137528
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567101600 |    115928
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567080000 |     94328
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567058400 |     72728
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567036800 |     51128
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1567015200 |     29528
 1.d0a8110e69b0d874610aa08ab6740dfa_cnt_60_647 | 1566993600 |      7928
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567144800 |    159128
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567123200 |    137528
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567101600 |    115928
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567080000 |     94328
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567058400 |     72728
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567036800 |     51128
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1567015200 |     29528
 1.d0a8110e69b0d874610aa08ab6740dfa_max_60_647 | 1566993600 |      7928

(32 rows)


// if the ttl relative to now is <=0 then we can omit the insert
if relativeTtl <= 0 {
if log.IsLevelEnabled(log.DebugLevel) {
log.Debugf("Omitting insert of chunk with ttl %d: %s, %d", relativeTtl, key, t0)
}
return nil
}

table, ok := c.TTLTables[ttl]
if !ok {
return errTableNotFound
Expand All @@ -363,7 +378,7 @@ func (c *CassandraStore) insertChunk(key string, t0, ttl uint32, data []byte) er
row_key := fmt.Sprintf("%s_%d", key, t0/Month_sec) // "month number" based on unix timestamp (rounded down)
pre := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
ret := c.Session.Query(table.QueryWrite, row_key, t0, data, ttl).WithContext(ctx).Exec()
ret := c.Session.Query(table.QueryWrite, row_key, t0, data, uint32(relativeTtl)).WithContext(ctx).Exec()
cancel()
cassPutExecDuration.Value(time.Now().Sub(pre))
return ret
Expand Down