Refactored TSDB opening concurrency at startup

[pracucci]

What this PR does:
This is an internal refactoring with no user impact. In the PR #3354 I received the feedback:

feel like error-handling code could be clearer by using errgroup. Also splitting the concerns would help -- one goroutine doing the scan, other N goroutines doing opening. Now both are mixed.

In this PR I'm addressing this feedback splitting the logic between the "scan" and the "opening".

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[pstibrany]

This is still bit more complicated than I would prefer. Suggestion to use errgroup would give us 1) handling of error (no need for errs and errsMx), 2) cancellation of context (right now if opening of one TSDB fails, other will still be tried), 3) simplified waiting (no need for wg).

When using errgroup, I'd suggest to spawn dir-walker as another goroutine, which can return its own error. Dir walker should react on group context cancellation, and by closing channel via defer, it will let tsdb-opening goroutines to stop as well.

Currently this code can return multiple errors – using errgroup will only return first one [I don't think this matters too much], but we can still log them all – as it is now.

[pstibrany]

By returning here, we lose a consumer of queue channel, which can lead to blocking of dir walker, when it tries to send another item to the channel.

[pracucci]

Uh good point!

[pracucci]

Thanks @pstibrany for your thoughtful code review. Could you take another look, please?

[pstibrany]

LGTM, Thanks!

[pstibrany]

Nit: I think this is unnecessary since dir-walker reacts on groupCtx.Err() and closes queue.