Clean up abandoned WAL directories

[56quarters]

Remove WAL directories that are no longer associated with a ManagedInstance
that this agent is responsible for. The storage directories used by active
instances are compared to all subdirectories under the wal_directory root.
Any that are not associated with an instance and that haven't been modified
in over a configured amount of time are removed.

How often the check is run and the cutoff for how recently a WAL must have
been modified are controlled by the wal_cleanup_period and
wal_cleanup_age settings respectively, under the top-level Prometheus
configuration. When omitted, they default to a period of 30 minutes and
recently modified threshold of 12 hours.

Fixes #132

• CHANGELOG updated
• Documentation added
• Tests updated

[CLAassistant]

All committers have signed the CLA.

[56quarters]

@rfratto I'm not really sure about the best way to integrate this into the rest of the agent. My initial thought was that a Cleaner instance would be a member of the Agent struct. The goroutine to run it periodically would be started on construction of the Agent struct. Your thoughts?

[rfratto]

Thanks for working on this!

I think creating it from the Agent struct is fine, but I think I'd prefer the goroutine to be launched by NewCleaner with a Stop method that signals the goroutine to exit. I'd like to keep the Agent acting as just the glue for other pieces of logic and not have any real logic itself. We do something similar in newManager, where it launches its own goroutine and has a Stop method to close it.

[rfratto]

Thanks for working on this, this will be great for people using the scraping service! 🎉 This is pretty close to a mergeable state, though I think there's some work that we can do to make the code a bit smaller.

Aside from my smaller pieces of feedback, I have one larger one. Some linters require function and type comments to be commented in the "godoc" style which are strings optionally starting with A/An/The and the name of the thing being commented:

// The Cleaner is responsible for ...
type Cleaner struct {}

// WithCleanerMinAge overrides the default age after which ....
func WithCleanerMinAge(d time.Duration) WALCleanerOpts 

We don't have that linting rule enabled (apparently, though I thought we did) and it only applies the restriction to exported functions/types, but I personally like doing it consistently for all types and functions. Can you make sure that at least all exported types and functions you added are written in this style?

[rfratto]

LGTM, thank you! Once you rebase and fix the merge conflict, feel free to merge.