Maybe this is not a valid issue now that I have discovered query-ingesters-within configuration option.

Do the chunks that are pulled from the store still get written to the chunk cache if the chunks are outside of the query-ingesters-within window?

Hi @mveitas, spoke with the Loki team today. @ashwanthgoli is going to investigate a little further but it still might be worth implementing this feature on initial glance.

@salvacorts has also mentioned it might be worth handing queries older than 30 days to L2 cache.

We use it ourselves: e.g. we store last 3d chunks on the L1 cache (SSD-backed) and anything older than 3d is stored on the L2 (memory-backed)

I haven't tested this myself yet but I have tracked down the cache config for you:
https://grafana.com/docs/loki/latest/configure/#chunk_store_config

# Chunks will be handed off to the L2 cache after this duration. 0 to disable L2
# cache.
# CLI flag: -store.chunks-cache-l2.handoff
[l2_chunk_cache_handoff: <duration> | default = 0s]

Thanks @Jayclifford345! l2 cache would be one way to reduce the churn for the desired duration. Any chunks older than l2_chunk_cache_handoff would be routed to l2 cache which can be configured under chunk_cache_config_l2

A configuration option is provided that would allow a duration to be specified that would allow the querier to write back to the chunk cache. If the chunk falls outside of this duration, it would not write the result back to the chunk cache.

afaik there is no support to do this, happy to help if you would like to contribute.

there is a hacky way to achieve what you want, i haven't tested it but it might work.

• configure chunk_cache_config_l2 to point to the same cache instance as l1 and set l2_chunk_cache_handoff to 30d
• update the writeback_buffer to a very small value for l2 cache so we drop most of the chunks by limiting the length of the queue that writes to the cache

edit: in addition setting writeback_goroutines to 0 might be more effective as it stops consuming from the queue

@ashwanthgoli This is something we are going to take on as we have a very large retention window that we need to have in place. While your workaround with the L2 cache would probably work, it's cleaner to have a separate configuration.

As an alternative would it make sense to create a no-op cache implementation and configure the L2 cache to use that?

prefer adding a new config to not store chunks older than the configured period
simpler than configuring chunk_cache_config_l2 with no-op cache, but not opposed to this either :)