Add Memcached Auto-discovery support

[zaldnoay]

Is your feature request related to a problem? Please describe.

Memcached Auto-discovery is using to auto discovery Memcached cluster node in some cloud provider. It automatically adapt to the addition and removal of node IP addresses that happens when you scale your Memcached instance. Current Mimir doesn't support this feature. And we need update Memcached node address manually which is inconvenience when we use cloud provider's Memcached service.

Describe the solution you'd like

Adapting Memcached Auto-discovery and add an option in memcached block configures.

Additional context

This feature is supported in recent Cortex.

More detail can be found here:

• https://cloud.google.com/memorystore/docs/memcached/auto-discovery-overview?hl=en
• https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.HowAutoDiscoveryWorks.html

[09jvilla]

At the moment we're actually looking for ways to eliminate Mimir's dependency on an external caching layer to simplify operations, so we won't be actively prioritizing this request.

That being said, PRs are welcome. :)

Assuming I'm understanding this correctly, it looks like the Cortex PR to support this was pretty simple - mostly updating the Thanos version and then adding some extra configuration options.
From a Mimir perspective, I think the Thanos version update may not even be needed - the version we're using in Mimir main already contains the feature (see link) - looks like Thanos introduced this in version 0.23.0 (see PR # 4487).
So that would mean all that's needed is a PR to Mimir that plumbs through the extra configuration options.

[TaylorMutch]

Would love to see this be implemented unless there are plans to do away with memcached. Running Mimir with managed memcached would greatly simplify operations for some of us!

[marcsantiago]

Look like grafana uses a fork of https://github.com/bradfitz/gomemcache

https://github.com/marcsantiago/gomemcache/blob/main/memcache/aws_discovery_selector.go

Note, I haven't tested this with Google. Not sure if they are using same protocol as AWS, but I personally don't use gcloud.

I've updated my fork of [bradfitz/gomemcache] to include auto discovery since no else has (so weird given the package is 13 years old with no real alternatives).
Feel free to just copy and paste, simply implement a new ServerSelector on your own grafana fork. You may have to bubble up the settings in the mimir api.

cache := memcache.NewFromSelector(
	memcache.NewAWSDiscoverySelector(
		os.Getenv("MEM_CACHE_CLUSTER_URL"),
	),
)
....

[aallawala]

@marcsantiago Is your change easily upstreamable? Wondering if we can get this merged in as I'm also looking for the same functionality