/flush handler doesn't wait for flush to finish

[pstibrany]

When /flush handler on ingester is called, it triggers flush of all chunks. It logs following lines:

level=info ts=2020-06-18T08:23:53.474269984Z caller=flush.go:42 msg="starting to flush all the chunks"
level=info ts=2020-06-18T08:24:03.286485064Z caller=flush.go:44 msg="flushing of chunks complete"

But this is misleading. Chunks flush is not complete at this point, all unflushed chunks are only enqueued for flushing, and are waiting in the queue when this log message is written and handler returns.

We should make /flush handler to wait for flush queues to be empty, if possible.

[pstibrany]

On the other hand, waiting for empty queues can take a long time. We already have cortex_ingester_flush_queue_length metric to monitor queue lengths, so maybe we should just fix the logged message, and write something like "123456 chunks enqueued for flushing" to the HTTP response (right now response is empty).

[pstibrany]

cortex_ingester_flush_queue_length alone is not enough -- there are still ongoing flushes even when cortex_ingester_flush_queue_length is 0.

PR #2778 introduces new cortex_ingester_flush_series_in_progress metric that can help to monitor that. Only when cortex_ingester_flush_queue_length + cortex_ingester_flush_series_in_progress reaches 0, all flushes are done.