Add a configuration for an index cache ttl

[SungJin1212]

Add a configuration for an index cache ttl. The index cache ttl can be configured by -blocks-storage.bucket-store.index-cache.redis.index-ttl and -blocks-storage.bucket-store.index-cache.memcached.index-ttl

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Thanks for the PR.

I think it probably needs more discussion on how we wanna do this. From my perspective, there are certain index cache items that worth a larger TTL than others.

For example, default TTL is 1 day but I might want to cache expanded postings for one week and other index cache types for only 1 day or shorter.

Cache entries from a 24h block might worth a longer TTL than 2h range uncompacted blocks because uncompacted blocks will be removed in a short period of time.

I am not against this PR. Just feel that the end state should be more flexible rather than a global TTL for everything.

[SungJin1212]

@yeya24
I agree with your thoughts. Different TTLs are necessary to consider block range. There is a necessary to set of different TTL based on block meta's MaxTime and MinTime. But, the cache interface is dependent on the thanos. I think if the TTL parameter is added to StorePostings, StoreExpandedPostings, and StoreSeries funcs, the implementation is possible.
c.f. https://github.com/thanos-io/thanos/blob/main/pkg/store/cache/cache.go#L43

[yeya24]

@SungJin1212 Agree. I think it would be useful to extend the interface to support TTL. The TTL could be a function that takes block metadata info as a parameter so that we know we use longer TTL for L4 blocks.

[SungJin1212]

@yeya24
Yes. If then, more optimization can be done.