rabbitmq-cluster-operator pod is getting OOM killed in openshift

[RosarioVinoth]

Describe the bug

rabbitmq-cluster-operator pod is getting OOM killed in openshift, it appears that the 500MiB memory isn't enough.

To Reproduce

Steps to reproduce the behavior:
Deploy the rabbitmq-cluster-operator in a namespace and check the status of the operator pod.

Expected behavior
rabbitmq-cluster-operator to be running.

Screenshots

containerStatuses:
- restartCount: 3697
started: false
ready: false
name: operator
state:
waiting:
reason: CrashLoopBackOff
message: >-
back-off 5m0s restarting failed container=operator
pod=rabbitmq-cluster-operator-98545478d-w24nz_-test(8c7a7e6-86f-4b4-ac06-580012acc5)
imageID: >-
docker.io/rabbitmqoperator/cluster-operator@sha256:563f230500a3efb1a90ae52b3090cd9f20c5c08d57ff063ec7ce88fbb0b1ab4c
image: 'docker.io/rabbitmqoperator/cluster-operator:2.1.0'
lastState:
terminated:
exitCode: 137
reason: OOMKilled
startedAt: '2023-03-07T10:25:07Z'
finishedAt: '2023-03-07T10:25:52Z'
containerID: >-
cri-o://400a52f06fef804a644e2f22eb29f6a275d7d032588a00f3ea2c6c74d70
containerID: 'cri-o://400a5206fefd04a644d2f22eb29f6275cd7d032588008f3e6ac6c74d70'

Version and environment information

• rabbitmq-cluster-operator.v2.1.0
• Openshift-4.11

[DanielePalaia]

Hi @RosarioVinoth we do test our operator with an internal pipeline against Openshift (OKD 4.10) - we deploy the operator and run the operator system tests, but we are not detecting this issue currently. How are you deploying the operator? Using the main manifest? Is this happening consistently after every deployment? Is this error happening soon after the deployment or after a particular load/scenario? Any help on how to reproduce it? Another cause of a pod been OOMKilled can be an overcommitment of resources on the node of the cluster where the pod get scheduled. Has the cluster the required resources? Is there maybe some kind of Quota set up in the cluster? (For example at namespace level?)

[RosarioVinoth]

Hi,

It is bit weird, i don't know how the memory requirement works but when i install the operator in a small cluster, everything seems to be working fine. however when i install the operator on a comparatively bigger cluster, the operator pod is OOM killed. Once i update the memory limit to 1GiB, the OOM issue went away and it appears at least during start up it requires more than 500MB memory.

[RosarioVinoth]

Hi,

Any update ?
At the moment we have customized the memory limit on the deployment to make it work as workaround.

[DanielePalaia]

Hi @RosarioVinoth as you can imagine for us is not really easy to investigate on this issue which can have multiple root causes and that at the moment we can't reproduce. We didn't receive any other notifications on this kind of issues recently too.

Something that can help us on the investigation to see what component/library is consuming too much in this case would be to have a look to the memory stack trace of the operator.
Last year we added an integration with pprof for debug issues that you can see here:

#1064

You can add this environment variable ENABLE_DEBUG_PPROF to true in the operator container.
After this you can port forward the operator pod ex:

kubectl port-forward rabbitmq-cluster-operator-7bbd65996c-8q2d9 -n rabbitmq-system 9782

and then you can use all the entrypoints of pprof:
https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/

Then you can use pprof in interactive and non interactive mode.

For example:
go tool pprof http://localhost:9782/debug/pprof/heap

Should create a profile file in a zip format

But also info on the other entry points can help too.

Also can be useful to have more info on the environement (in case we can try to reproduce it). How many nodes, memory has your K8s cluster? What is the status of the node where the operator pod is deployed (any hint if you do a kubectl top nodes)? Are many other applications running on the cluster (are we under an high load)? Are you setting up any kind of Quota for example an namespace level? Is this happening consistently at every deployment?

[lukebakken]

@RosarioVinoth it's not polite to bump a discussion when you have not provided the information that @DanielePalaia requested here.

Please provide the information that @DanielePalaia has requested.

[RosarioVinoth]

Hi,

Please find the information you have requested for.

pprof.manager.alloc_objects.alloc_space.inuse_objects.inuse_space.134.pb.gz

pprof) top
Showing nodes accounting for 364.59MB, 96.98% of 375.95MB total
Dropped 104 nodes (cum <= 1.88MB)
flat flat% sum% cum cum%
138MB 36.71% 36.71% 138MB 36.71% k8s.io/apimachinery/pkg/runtime.(*Unknown).Unmarshal
118.39MB 31.49% 68.20% 125.40MB 33.35% k8s.io/api/core/v1.(*Secret).Unmarshal
72.08MB 19.17% 87.37% 74.58MB 19.84% k8s.io/api/core/v1.(*ConfigMap).Unmarshal
9.50MB 2.53% 89.90% 15.50MB 4.12% k8s.io/apimachinery/pkg/apis/meta/v1.(*ObjectMeta).Unmarshal
5.52MB 1.47% 91.37% 5.52MB 1.47% golang.org/x/net/http2.glob..func3
5MB 1.33% 92.70% 5MB 1.33% runtime.allocm
3.90MB 1.04% 93.74% 129.29MB 34.39% k8s.io/api/core/v1.(*SecretList).Unmarshal
3.50MB 0.93% 94.67% 3.50MB 0.93% k8s.io/apimachinery/pkg/apis/meta/v1.(*FieldsV1).Unmarshal
2.50MB 0.66% 95.33% 6MB 1.60% k8s.io/apimachinery/pkg/apis/meta/v1.(*ManagedFieldsEntry).Unmarshal
1.34MB 0.36% 95.69% 75.92MB 20.19% k8s.io/api/core/v1.(*ConfigMapList).Unmarshal
1.24MB 0.33% 96.02% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*threadSafeMap).Update
1.23MB 0.33% 96.34% 4.23MB 1.13% k8s.io/api/rbac/v1.(*RoleBindingList).Unmarshal
1.17MB 0.31% 96.66% 4.67MB 1.24% k8s.io/api/core/v1.(*ServiceList).Unmarshal
0.71MB 0.19% 96.84% 2.71MB 0.72% k8s.io/api/rbac/v1.(*RoleList).Unmarshal
0.50MB 0.13% 96.98% 3.01MB 0.8% runtime.main
0 0% 96.98% 219.14MB 58.29% github.com/gogo/protobuf/proto.Unmarshal
0 0% 96.98% 6.02MB 1.60% golang.org/x/net/http2.(*ClientConn).readLoop
0 0% 96.98% 5.52MB 1.47% golang.org/x/net/http2.(*clientConnReadLoop).processData
0 0% 96.98% 6.02MB 1.60% golang.org/x/net/http2.(*clientConnReadLoop).run
0 0% 96.98% 5.52MB 1.47% golang.org/x/net/http2.(*dataBuffer).Write
0 0% 96.98% 5.52MB 1.47% golang.org/x/net/http2.(*dataBuffer).lastChunkOrAlloc
0 0% 96.98% 5.52MB 1.47% golang.org/x/net/http2.(*pipe).Write
0 0% 96.98% 5.52MB 1.47% golang.org/x/net/http2.getDataBufferChunk
0 0% 96.98% 75.92MB 20.19% k8s.io/api/core/v1.(*ConfigMapList).XXX_Unmarshal
0 0% 96.98% 129.29MB 34.39% k8s.io/api/core/v1.(*SecretList).XXX_Unmarshal
0 0% 96.98% 3.50MB 0.93% k8s.io/api/core/v1.(*Service).Unmarshal
0 0% 96.98% 4.67MB 1.24% k8s.io/api/core/v1.(*ServiceList).XXX_Unmarshal
0 0% 96.98% 2MB 0.53% k8s.io/api/rbac/v1.(*Role).Unmarshal
0 0% 96.98% 3MB 0.8% k8s.io/api/rbac/v1.(*RoleBinding).Unmarshal
0 0% 96.98% 4.23MB 1.13% k8s.io/api/rbac/v1.(*RoleBindingList).XXX_Unmarshal
0 0% 96.98% 2.71MB 0.72% k8s.io/api/rbac/v1.(*RoleList).XXX_Unmarshal
0 0% 96.98% 357.14MB 95.00% k8s.io/apimachinery/pkg/runtime.WithoutVersionDecoder.Decode
0 0% 96.98% 357.14MB 95.00% k8s.io/apimachinery/pkg/runtime/serializer/protobuf.(*Serializer).Decode
0 0% 96.98% 219.14MB 58.29% k8s.io/apimachinery/pkg/runtime/serializer/protobuf.unmarshalToObject
0 0% 96.98% 3.77MB 1.00% k8s.io/apimachinery/pkg/util/wait.BackoffUntil
0 0% 96.98% 3.77MB 1.00% k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
0 0% 96.98% 3.24MB 0.86% k8s.io/apimachinery/pkg/util/wait.JitterUntil
0 0% 96.98% 2.74MB 0.73% k8s.io/apimachinery/pkg/util/wait.Until (inline)
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/rest.Result.Into
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/tools/cache.(*ListWatch).List
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/tools/cache.(*Reflector).list.func1
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/tools/cache.(*Reflector).list.func1.2
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*cache).Add
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*controller).Run
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*controller).processLoop
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*sharedIndexInformer).HandleDeltas
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.(*threadSafeMap).Add
0 0% 96.98% 2.24MB 0.6% k8s.io/client-go/tools/cache.processDeltas
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/tools/pager.(*ListPager).List
0 0% 96.98% 357.14MB 95.00% k8s.io/client-go/tools/pager.SimplePageFunc.func1
0 0% 96.98% 2.01MB 0.53% runtime.doInit
0 0% 96.98% 3.50MB 0.93% runtime.mstart
0 0% 96.98% 3.50MB 0.93% runtime.mstart0
0 0% 96.98% 3.50MB 0.93% runtime.mstart1
0 0% 96.98% 5MB 1.33% runtime.newm
0 0% 96.98% 5MB 1.33% runtime.resetspinning
0 0% 96.98% 5MB 1.33% runtime.schedule
0 0% 96.98% 5MB 1.33% runtime.startm
0 0% 96.98% 5MB 1.33% runtime.wakep
0 0% 96.98% 357.14MB 95.00% sigs.k8s.io/controller-runtime/pkg/cache/internal.createStructuredListWatch.func1
0 0% 96.98% 357.14MB 95.00% sigs.k8s.io/controller-runtime/pkg/client/apiutil.targetZeroingDecoder.Decode
0 0% 96.98% 6.03MB 1.60% sync.(*Pool).Get
(pprof)

I could only collect this after setting the memory limit to 1GiB, otherwise the pod was keep getting OOM killed.

About the cluster, we have 3 master and 21 worker nodes at the moment with cluster-autoscaler, we have quotas on namespaces but i have tested by removing the quotas a well and operator pod is still gettting OOM killed. And yes, this is happening consistently at every deployment with namespace scope, we have about 150 user namespaces in our cluster.

[RosarioVinoth]

[DanielePalaia]

Hi @RosarioVinoth thanks to have provided the profiling informations.
Yes from the profiling informations it seems like the operator is consuming too much memory unmarshalling K8s objects (especially Secrets and ConfigMaps).

I also attach the profiling image corresponding to the file for future reference:

Normally this shouldn't happen or we didn't notice this happening before so probably it has something to do with this specific environment but this is difficult now to determine what is causing this behaviour.

Just some of further questions: Can you confirm are you deploying the operator from the main manifests (https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml) and not from the openshift marketplace? Did you notice some similar behaviour deploying other operators (implemented with KUBEBUILDER) if any on this cluster? Did you try to deploy the operator in a newly created empty namespace?

I will leave this issue open in case we will able to reproduce this or someone else will hit the same issue in the future we may decide to do a more in deep investigation.

[RosarioVinoth]

Hi @DanielePalaia,

Thanks for the information.
I am installing the operator from "Operator HUB" within the OpenShift cluster.
And yes, i tested deployment of the operator in a new namespace, its same result, OOM killed.
Finally, I have seen similar behaviour in keda operator, though i have installed keda clusterwide, not sure if that of any significant difference.

[DanielePalaia]

@RosarioVinoth thanks! Can I ask you to try just to install from the main manifest?

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

Here The operator will be installed in the "rabbitmq-system" namespace

It shouldn't make any difference but just to double check and confirm if it may be something due to the installation through OLM packaging and "operator-hub".

[milosara78]

Hello @DanielePalaia For us recently we encountered the issue where Rabbitmq cluster was getting memory pressure and cluster was not getting up while pods were killed with OOM killed status . Post increasing the Memory limit of cluster we were able to see the pods running again . This is weird that it happened suddenly from only one environment. So is this something we are aware from Rabbitmq there is threshold which to be maintained for the limit of rabbitmq cluster ??

[mkuratczyk]

@saravanan-ai your issue is completely unrelated. This issue is about the Operator running out of memory in some rare cases, not about RabbitMQ running out of memory. Please create an issue/discussion in https://github.com/rabbitmq/rabbitmq-server with the necessary details of your problem if you want some help (and please make sure you are using a supported version first)

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.