[mimir-distributed] Ingester rollout-group conflicts with other charts

[lindeskar]

Describe the bug

The mimir-distributed Helm chart generates Ingester StatefulSets with labels for rollout-operator. Ex.:

mimir/operations/helm/tests/large-values-generated/mimir-distributed/templates/ingester/ingester-statefulset.yaml

Lines 51 to 52 in 8571f6c

	name: "ingester-zone-a"
	rollout-group: ingester

The loki Helm chart with deploymentMode: Distributed and zoneAwareReplication enabled (default) generates StatefulSets with the same label values. Ex.:

name: ingester-zone-a
rollout-group: ingester

Deploying the two charts to the same Namespace means rollout-operator will select both Mimir and Loki StatefulSets and get confused about the rollout status. For me one of the Mimir Ingester Pods is constantly being recreated.

Issue grafana/loki#13168 is created for the same issue in the Loki repository.

To Reproduce

Steps to reproduce the behavior:

1. Deploy mimir-distributed chart with default values
2. Deploy loki chart with distributed-values.yaml

Expected behavior

rollout-operator handles Mimir and Loki Ingesters as separate rollout-groups.

Environment

• Infrastructure: Kubernetes
• Deployment tool: Helm
• Chart version: 5.3.0

Additional Context

From rollout-operator Pod:

level=info ts=2024-06-07T08:17:49.880277442Z msg="StatefulSet status is reporting all pods ready, but the rollout operator has found some not-Ready pods" statefulset=loki-ingester-zone-a not_ready_pods=mimir-ingester-zone-a-0

[dimitarvdimitrov]

you should deploy them in different namespaces

[lindeskar]

Sure, probably the easiest solution, though our setup runs LGTM in one Namespace today.

[lindeskar]

The related issue in Loki was resolved with an optional prefix added in grafana/loki#15063

Could this be ok to add for Mimir too?

[dimitarvdimitrov]

If the other charts are doing it, then it should be ok to add it to the mimir-distributed chart too

[dimitarvdimitrov]

on a second thought and after some discussion in GL slack I think we should close this and reject the patch. Here's our thought process:

The LGTM helm charts aren't designed to run in the same namespace. They surely can if configured properly and configuration added where necessary. Unfortunately, predicting what might break is a a bit futile. There might always be something no one anticipated. Even if we squash all bugs, the added configuration adds more complexity to the deployment model: more bells and whistles, more edge cases.

Namespaces in Kubernetes are purely a logical separation and creating an additional one isn't complicated. That may not be the reality of it, but this chart targets Kubernetes in general, not specific organiational structures that companies might have.