Query ingesters within 12h too long?

[weeco]

I am not sure why it would make sense to set querier.query-ingesters-within to 12h as shown here: https://github.com/grafana/cortex-jsonnet/blob/master/cortex/config.libsonnet#L149 .

As the comment says the ingesters store chunks 6h and the configured retain period is 15m (see https://github.com/grafana/cortex-jsonnet/blob/master/cortex/ingester.libsonnet#L21-L22). Doesn't that mean chunks can only live up to 6h15min in ingesters or is there something else to consider as well?

One might want to change this value or add a better comment why 12h is reasonable I think.

[tomwilkie]

This is indeed a good question - your reasoning makes sense. Except it can take some time for the chunks to be flushed, and if you look at our alerts you'll see we sometimes see chunks take up to 10hrs to flush - so I suspect this is put in here to ensure users / customers never see gaps. Gaps are bad as they will be cached in the frontend.

[weeco]

Thanks for the explanation.

Because I am curious: Do you happen to know why it takes that long sometimes? Is it because BigTable is not able to process that many write requests sometimes and hence there might be a queue of up to 4h or what's the reason that it takes so much longer sometimes?

[gouthamve]

Also, one more reason is that people can write some old data. Like Prometheus writing to Cortex could be an hour or two behind for whatever reason. This is just an extra-safe buffer in those cases. Thanks, Goutham.

…

On Mon, May 4, 2020 at 7:14 PM Martin Schneppenheim < ***@***.***> wrote: Thanks for the explanation. Because I am curious: Do you happen to know why it takes that long sometimes? Is it because BigTable is not able to process that many write requests sometimes and hence there might be a queue of up to 4h or what's the reason that it takes so much longer sometimes? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABYDOH6KDWMWJIMCD3OBW63RP4AZXANCNFSM4MYSGPOA> .

[tomwilkie]

Is it because BigTable is not able to process that many write requests sometimes and hence there might be a queue of up to 4h or what's the reason that it takes so much longer sometimes?

I wouldn't characterise it like that - I'd say its more that users series churn is not always constant, so the number of chunks to flush is highly variable - and our bigtable is statically provisioned.