Basic MongoDBCommunity resource fails to start

[mikelsid]

What did you do to encounter the bug?
Steps to reproduce the behavior:

1. Create basic mongodb manifests (basically the example + storageClass overrides). See attached.

What did you expect?
Working MongoDB cluster

What happened instead?
The mongod container stuck indefinitely waiting for mongodb-agent:

❯ k logs -f example-mongodb-0
Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
2025-08-04T18:04:37.140Z        INFO    versionhook/main.go:33  Running version change post-start hook
2025-08-04T18:04:37.141Z        INFO    versionhook/main.go:40  Waiting for agent health status...
2025-08-04T18:04:38.141Z        INFO    versionhook/main.go:46  Agent health status file not found, mongod will start
Waiting for config and keyfile files to be created by the agent...
Waiting...
Waiting...
Waiting...
Waiting...

Operator Information

• Operator Version v1.2.0
• MongoDB Image used 8.0.0

Kubernetes Cluster Information

• Distribution: kubeadm
• Version: 1.31.4
• Image Registry location (quay, or an internal registry): quay

Additional context
Add any other context about the problem here.

❯ k get pods
NAME                READY   STATUS    RESTARTS   AGE
example-mongodb-0   2/2     Running   0          19m
example-mongodb-1   2/2     Running   0          18m
example-mongodb-2   2/2     Running   0          18m

                                                                                     
❯ k get mdbc
NAME              PHASE     VERSION
example-mongodb   Running   8.0.0

❯ kubectl get mdbc -oyaml
apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
  kind: MongoDBCommunity
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"example-mongodb","namespace":"test"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"template":{"spec":{"containers":[{"name":"mongod","resources":{"limits":{"cpu":3,"memory":"2000Mi"},"requests":{"cpu":3,"memory":"2000Mi"}}}],"initContainers":[{"name":"mongod-posthook","resources":{"limits":{"cpu":1,"memory":"2000Mi"},"requests":{"cpu":0.5,"memory":"1000Mi"}}},{"name":"mongodb-agent-readinessprobe","resources":{"limits":{"cpu":0.5,"memory":"128Mi"},"requests":{"cpu":0.5,"memory":"128Mi"}}}]}},"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5G"}},"storageClassName":"directpv-min-io"}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"2G"}},"storageClassName":"directpv-min-io"}}]}},"type":"ReplicaSet","users":[{"db":"admin","name":"my-user","passwordSecretRef":{"name":"my-user-password"},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"my-scram"}],"version":"8.0.0"}}
      mongodb.com/v1.lastAppliedMongoDBVersion: 8.0.0
      mongodb.com/v1.lastSuccessfulConfiguration: '{"members":3,"type":"ReplicaSet","version":"8.0.0","arbiters":0,"security":{"authentication":{"modes":["SCRAM"],"ignoreUnknownUsers":true},"tls":{"enabled":false,"optional":false,"certificateKeySecretRef":{}}},"users":[{"name":"my-user","db":"admin","passwordSecretRef":{"name":"my-user-password","key":""},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"my-scram","additionalConnectionStringConfig":null}],"statefulSet":{"spec":{},"metadata":{}},"agent":{"logLevel":"","logFile":"","maxLogFileDurationHours":0},"additionalMongodConfig":{},"additionalConnectionStringConfig":{}}'
    creationTimestamp: "2025-08-04T18:33:49Z"
    generation: 1
    name: example-mongodb
    namespace: test
    resourceVersion: "5289912"
    uid: 54610384-4ca8-4643-b0d1-26db03cc87df
  spec:
    additionalMongodConfig:
      storage.wiredTiger.engineConfig.journalCompressor: zlib
    members: 3
    security:
      authentication:
        ignoreUnknownUsers: true
        modes:
        - SCRAM
    statefulSet:
      spec:
        template:
          spec:
            containers:
            - name: mongod
              resources:
                limits:
                  cpu: 3
                  memory: 2000Mi
                requests:
                  cpu: 3
                  memory: 2000Mi
            initContainers:
            - name: mongod-posthook
              resources:
                limits:
                  cpu: 1
                  memory: 2000Mi
                requests:
                  cpu: 0.5
                  memory: 1000Mi
            - name: mongodb-agent-readinessprobe
              resources:
                limits:
                  cpu: 0.5
                  memory: 128Mi
                requests:
                  cpu: 0.5
                  memory: 128Mi
        volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 5G
            storageClassName: directpv-min-io
        - metadata:
            name: logs-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 2G
            storageClassName: directpv-min-io
    type: ReplicaSet
    users:
    - db: admin
      name: my-user
      passwordSecretRef:
        name: my-user-password
      roles:
      - db: admin
        name: clusterAdmin
      - db: admin
        name: userAdminAnyDatabase
      scramCredentialsSecretName: my-scram
    version: 8.0.0
  status:
    currentMongoDBMembers: 3
    currentStatefulSetReplicas: 3
    mongoUri: mongodb://example-mongodb-0.example-mongodb-svc.test.svc.cluster.local:27017,example-mongodb-1.example-mongodb-svc.test.svc.cluster.local:27017,example-mongodb-2.example-mongodb-svc.test.svc.cluster.local:27017/?replicaSet=example-mongodb
    phase: Running
    version: 8.0.0
kind: List
metadata:
  resourceVersion: ""

❯ kubectl get sts -oyaml
apiVersion: v1
items:
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2025-08-04T18:33:49Z"
    generation: 1
    name: example-mongodb
    namespace: test
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: example-mongodb
      uid: 54610384-4ca8-4643-b0d1-26db03cc87df
    resourceVersion: "5289909"
    uid: 0b500896-58c8-450d-a3e8-86c9e898bab5
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: example-mongodb-svc
    serviceName: example-mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: example-mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - "\nif [ -e \"/hooks/version-upgrade\" ]; then\n\t#run post-start hook
            to handle version changes (if exists)\n    /hooks/version-upgrade\nfi\n\n#
            wait for config and keyfile to be created by the agent\necho \"Waiting
            for config and keyfile files to be created by the agent...\"\nwhile !
            [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile
            ]; do\n\tsleep 3;\n\techo \"Waiting...\"\ndone\n\n# sleep is important
            after agent issues shutdown command\n# k8s restarts the mongod container
            too quickly for the agent to realize mongod is down\necho \"Sleeping for
            15s...\"\nsleep 15\n\n# start mongod with this configuration\necho \"Starting
            mongod...\"\nexec mongod -f /data/automation-mongod.conf;\n"
          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: quay.io/mongodb/mongodb-community-server:8.0.0-ubi8
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "3"
              memory: 2000Mi
            requests:
              cpu: "3"
              memory: 2000Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
          env:
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: example-mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
          imagePullPolicy: Always
          name: mongod-posthook
          resources:
            limits:
              cpu: "1"
              memory: 2000Mi
            requests:
              cpu: 500m
              memory: 1000Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources:
            limits:
              cpu: 500m
              memory: 128Mi
            requests:
              cpu: 500m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-kubernetes-appdb
        serviceAccountName: mongodb-kubernetes-appdb
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: example-mongodb-config
        - emptyDir: {}
          name: example-mongodb-keyfile
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5G
        storageClassName: directpv-min-io
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        storageClassName: directpv-min-io
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 3
    collisionCount: 0
    currentReplicas: 3
    currentRevision: example-mongodb-554f488f9f
    observedGeneration: 1
    readyReplicas: 3
    replicas: 3
    updateRevision: example-mongodb-554f488f9f
    updatedReplicas: 3
- apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    creationTimestamp: "2025-08-04T18:33:49Z"
    generation: 1
    name: example-mongodb-arb
    namespace: test
    ownerReferences:
    - apiVersion: mongodbcommunity.mongodb.com/v1
      blockOwnerDeletion: true
      controller: true
      kind: MongoDBCommunity
      name: example-mongodb
      uid: 54610384-4ca8-4643-b0d1-26db03cc87df
    resourceVersion: "5289298"
    uid: 849bac65-3538-46ef-a61e-ced3f992b86b
  spec:
    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain
    podManagementPolicy: OrderedReady
    replicas: 0
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: example-mongodb-svc
    serviceName: example-mongodb-svc
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: example-mongodb-svc
      spec:
        containers:
        - args:
          - ""
          command:
          - /bin/sh
          - -c
          - "\nif [ -e \"/hooks/version-upgrade\" ]; then\n\t#run post-start hook
            to handle version changes (if exists)\n    /hooks/version-upgrade\nfi\n\n#
            wait for config and keyfile to be created by the agent\necho \"Waiting
            for config and keyfile files to be created by the agent...\"\nwhile !
            [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile
            ]; do\n\tsleep 3;\n\techo \"Waiting...\"\ndone\n\n# sleep is important
            after agent issues shutdown command\n# k8s restarts the mongod container
            too quickly for the agent to realize mongod is down\necho \"Sleeping for
            15s...\"\nsleep 15\n\n# start mongod with this configuration\necho \"Starting
            mongod...\"\nexec mongod -f /data/automation-mongod.conf;\n"
          env:
          - name: AGENT_STATUS_FILEPATH
            value: /healthstatus/agent-health-status.json
          image: quay.io/mongodb/mongodb-community-server:8.0.0-ubi8
          imagePullPolicy: IfNotPresent
          name: mongod
          resources:
            limits:
              cpu: "3"
              memory: 2000Mi
            requests:
              cpu: "3"
              memory: 2000Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /healthstatus
            name: healthstatus
          - mountPath: /hooks
            name: hooks
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        - command:
          - /bin/bash
          - -c
          - |-
            current_uid=$(id -u)
            declare -r current_uid
            if ! grep -q "${current_uid}" /etc/passwd ; then
            sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
            echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
            export NSS_WRAPPER_PASSWD=/tmp/passwd
            export LD_PRELOAD=libnss_wrapper.so
            export NSS_WRAPPER_GROUP=/etc/group
            fi
            agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -logLevel INFO -maxLogFileDurationHrs 24
          env:
          - name: AGENT_STATUS_FILEPATH
            value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          - name: AUTOMATION_CONFIG_MAP
            value: example-mongodb-config
          - name: HEADLESS_AGENT
            value: "true"
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          image: quay.io/mongodb/mongodb-agent-ubi:108.0.2.8729-1
          imagePullPolicy: Always
          name: mongodb-agent
          readinessProbe:
            exec:
              command:
              - /opt/scripts/readinessprobe
            failureThreshold: 40
            initialDelaySeconds: 5
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            limits:
              cpu: "1"
              memory: 500M
            requests:
              cpu: 500m
              memory: 400M
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
          - mountPath: /var/lib/automation/config
            name: automation-config
            readOnly: true
          - mountPath: /data
            name: data-volume
          - mountPath: /var/lib/mongodb-mms-automation/authentication
            name: example-mongodb-keyfile
          - mountPath: /var/log/mongodb-mms-automation/healthstatus
            name: healthstatus
          - mountPath: /var/log/mongodb-mms-automation
            name: logs-volume
          - mountPath: /tmp
            name: tmp
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - cp
          - version-upgrade-hook
          - /hooks/version-upgrade
          image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.9
          imagePullPolicy: Always
          name: mongod-posthook
          resources:
            limits:
              cpu: "1"
              memory: 2000Mi
            requests:
              cpu: 500m
              memory: 1000Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /hooks
            name: hooks
        - command:
          - cp
          - /probes/readinessprobe
          - /opt/scripts/readinessprobe
          image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.22
          imagePullPolicy: Always
          name: mongodb-agent-readinessprobe
          resources:
            limits:
              cpu: 500m
              memory: 128Mi
            requests:
              cpu: 500m
              memory: 128Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /opt/scripts
            name: agent-scripts
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext:
          fsGroup: 2000
          runAsNonRoot: true
          runAsUser: 2000
        serviceAccount: mongodb-kubernetes-appdb
        serviceAccountName: mongodb-kubernetes-appdb
        terminationGracePeriodSeconds: 30
        volumes:
        - emptyDir: {}
          name: agent-scripts
        - name: automation-config
          secret:
            defaultMode: 416
            secretName: example-mongodb-config
        - emptyDir: {}
          name: example-mongodb-keyfile
        - emptyDir: {}
          name: healthstatus
        - emptyDir: {}
          name: hooks
        - emptyDir: {}
          name: tmp
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: data-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5G
        storageClassName: directpv-min-io
        volumeMode: Filesystem
      status:
        phase: Pending
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: logs-volume
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 2G
        storageClassName: directpv-min-io
        volumeMode: Filesystem
      status:
        phase: Pending
  status:
    availableReplicas: 0
    collisionCount: 0
    currentRevision: example-mongodb-arb-554f488f9f
    observedGeneration: 1
    replicas: 0
    updateRevision: example-mongodb-arb-554f488f9f
kind: List
metadata:
  resourceVersion: ""

❯ k logs -f example-mongodb-0
Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
2025-08-04T18:33:53.361Z        INFO    versionhook/main.go:33  Running version change post-start hook
2025-08-04T18:33:53.361Z        INFO    versionhook/main.go:40  Waiting for agent health status...
2025-08-04T18:33:54.361Z        INFO    versionhook/main.go:46  Agent health status file not found, mongod will start
Waiting for config and keyfile files to be created by the agent...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...
Waiting...

❯ kubectl exec -it example-mongodb-0  -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json

{"version":1,"processes":[{"name":"example-mongodb-0","disabled":false,"hostname":"example-mongodb-0.example-mongodb-svc.test.svc.cluster.local","args2_6":{"net":{"port":27017},"replication":{"replSetName":"example-mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"8.0","processType":"mongod","version":"8.0.0","authSchemaVersion":5},{"name":"example-mongodb-1","disabled":false,"hostname":"example-mongodb-1.example-mongodb-svc.test.svc.cluster.local","args2_6":{"net":{"port":27017},"replication":{"replSetName":"example-mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"8.0","processType":"mongod","version":"8.0.0","authSchemaVersion":5},{"name":"example-mongodb-2","disabled":false,"hostname":"example-mongodb-2.example-mongodb-svc.test.svc.cluster.local","args2_6":{"net":{"port":27017},"replication":{"replSetName":"example-mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"8.0","processType":"mongod","version":"8.0.0","authSchemaVersion":5}],"replicaSets":[{"_id":"example-mongodb","members":[{"_id":0,"host":"example-mongodb-0","arbiterOnly":false,"votes":1,"priority":1},{"_id":1,"host":"example-mongodb-1","arbiterOnly":false,"votes":1,"priority":1},{"_id":2,"host":"example-mongodb-2","arbiterOnly":false,"votes":1,"priority":1}],"protocolVersion":"1","numberArbiters":0}],"auth":{"usersWanted":[{"mechanisms":[],"roles":[{"role":"clusterAdmin","db":"admin"},{"role":"userAdminAnyDatabase","db":"admin"}],"user":"my-user","db":"admin","authenticationRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"3GHlybjCxo4B+1YYh4LNMw80jCNma3fRsKQ3bA==","serverKey":"zDPAJOeUI+0nQDdbhImeUrQgDvp0nRQzhtnCS+auFKw=","storedKey":"hyTTo9TmNusWXM48Gh6mVtN6bVdqFAloFj7fVkJMzHg="},"scramSha1Creds":{"iterationCount":10000,"salt":"C18cmdKmNnfrGFFJXO2bwA==","serverKey":"u0mYhBWPTKmkjpBwZup+WhnsNLs=","storedKey":"LZLA7haWyO7l8yNy58298h7vb2U="}}],"disabled":false,"authoritativeSet":false,"autoAuthMechanisms":["SCRAM-SHA-256"],"autoAuthMechanism":"SCRAM-SHA-256","deploymentAuthMechanisms":["SCRAM-SHA-256"],"autoUser":"mms-automation","key":"DdDszqMjV1lcx69wKvlWTiq9FF5sY/ew6P/2ABO61FykQ/38EA0q0bcT39xDnAzZrmZBADBD6nK5jN+cRZdg6X/4Gk1a8g5LD4+IScQCemndTcrAIlrJ3eoiEq9n2SoAzT/nE6KdJnzyGXPmBhgKy5r0zkhy+80N2EYFFkQdPLjmnjRdh+ioZ5EVITt3z+olL/oJk2LrNvUeSw9ZUSBuKRau/PW0D5UanjbcHDFhr151UYPVPg0mOVJy84de0pWJV0lpaamrxuZpAor/S2SOTjN0Uaik4zcRWRHmG1hIDuXzIcso1rT/DC+J07XO9gquJkDi7TzAQtEHGsSIAgWdLg+h2Pg+a7RBOzPRTNVnKHrMiartInCU5ixj3b5Dx6Tfv7lsQPVBLKo761LGnh1D9cdxauRdIMq/znlt+5vot5RM6LVcI2o7hQa4rowk5IH2PXT5rzpseQRrZPddGrHhbsKa7lC+W1c8BOoa9svIweuWNScDEQzS+/hYhTzwSZmsWdFhCyn3sEyQvzxRM4cHe59gjVzLY1ZHA1X9QrJSIx+3ue/3JU9eZetT7NtiAll8ibCISH5FOsDXlWIiwSS23US7QZ62PydHMS2mTwesbaYULgEaxnpsAMU7ocKSk8QgfHR92amvvTPdxMaGE3RdcObTqPI=","keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile","keyfileWindows":"%SystemDrive%\\MMSAutomation\\versions\\keyfile","autoPwd":"7v7tril0XlIOapyj_oRz"},"tls":{"CAFilePath":"","clientCertificateMode":"OPTIONAL"},"mongoDbVersions":[{"name":"8.0.0","builds":[{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","gitVersion":"","architecture":"aarch64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","gitVersion":"","architecture":"aarch64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]}]}],"backupVersions":[],"monitoringVersions":[],"options":{"downloadBase":"/var/lib/mongodb-mms-automation"}}

❯ kubectl exec -it example-mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
{"statuses":{},"mmsStatus":{}}


❯ kubectl exec -it example-mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        GitCommitId = ec1573c1fd5d7da3acab288d628b9e9eaaec6b2b
[2025-08-04T18:33:53.516+0000] [header.info] [::0]  AutomationVersion = 108.0.2.8729
[2025-08-04T18:33:53.516+0000] [header.info] [::0]          localhost = example-mongodb-0.example-mongodb-svc.test.svc.cluster.prod.xxx
[2025-08-04T18:33:53.516+0000] [header.info] [::0] ErrorStateSleepTime = 10s
[2025-08-04T18:33:53.516+0000] [header.info] [::0] GoalStateSleepTime = 10s
[2025-08-04T18:33:53.516+0000] [header.info] [::0] NotGoalStateSleepTime = 1s
[2025-08-04T18:33:53.516+0000] [header.info] [::0]     PlanCutoffTime = 300000
[2025-08-04T18:33:53.516+0000] [header.info] [::0]       TracePlanner = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]               User = mongodb
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         Go version = go1.22.9
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         MmsBaseURL =
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         MmsGroupId =
[2025-08-04T18:33:53.516+0000] [header.info] [::0]          HttpProxy =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DisableHttpKeepAlive = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        HttpsCAFile =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] TlsRequireValidMMSServerCertificates = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0] TlsMMSServerClientCertificate =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] KMIPProxyCertificateDir = /tmp
[2025-08-04T18:33:53.516+0000] [header.info] [::0] EnableLocalConfigurationServer = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] LocalConfigurationServerPort = 0
[2025-08-04T18:33:53.516+0000] [header.info] [::0]   InternalPromPort = 0
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DialTimeoutSeconds = 40
[2025-08-04T18:33:53.516+0000] [header.info] [::0] KeepUnusedMongodbVersions = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DisallowDowngrades = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]   UseLocalBinaries = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]  BinariesFixedPath =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] UseLocalMongoDbTools = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        NoDaemonize = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]     SkipMongoStart = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]       operatorMode = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] healthCheckFilePath = /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
[2025-08-04T18:33:53.516+0000] [.info] [src/util/sysdep_unix.go:LockAutomationLockFile:322] [18:33:53.516] Locking automation lock file at /tmp/mongodb-mms-automation.lock
[2025-08-04T18:33:53.516+0000] [.info] [main/components/agent.go:NewAgent:132] [18:33:53.516] Constructing new agent object with desiredClusterConfigPath=/var/lib/automation/config/cluster-config.json
[2025-08-04T18:33:53.516+0000] [.info] [src/mongosqld/custodian.go:NewCustodian:141] <mongosqld custodian> [18:33:53.516] Started
[2025-08-04T18:33:53.517+0000] [.warn] [prometheus/collector/hardwarecollector.go:collectMetrics:77] <promHardwareCollector> [18:33:53.517] agent.MetricsCollector is not running. Check agent.go for initialization issues : collector is not running
[2025-08-04T18:33:53.517+0000] [.info] [realtime/rtcollector/rtcollector.go:controlLoop:85] <rtCollector> [18:33:53.517] Starting control loop
[2025-08-04T18:33:53.517+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:33:53.517] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:33:53.517+0000] [.info] [src/dataexplorer/dataexplorer.go:controlLoop:109] <dataExplorer> [18:33:53.517] Starting control loop
[2025-08-04T18:33:53.517+0000] [.info] [src/kmipproxy/custodian.go:mainLoop:204] <kmipProxyMaintainer> [18:33:53.517] Starting main loop
[2025-08-04T18:33:53.529+0000] [.info] [main/components/agent.go:LoadClusterConfig:273] [18:33:53.529] New cluster config received! 0 (<nil>) -> 1 (2025-08-04 18:33:51.450390612 +0000 UTC)
[2025-08-04T18:33:53.529+0000] [.info] [src/modules/agents_unix.go:KillAllAgents:43] [18:33:53.529] Killing all running mongodb-mms-monitoring-agent agents at /var/lib/mongodb-mms-automation/mongodb-mms-monitoring-agent-.+\..+_.+/mongodb-mms-monitoring-agent *$
[2025-08-04T18:33:53.533+0000] [.info] [src/modules/agents_unix.go:KillAllAgents:43] [18:33:53.533] Killing all running mongodb-mms-backup-agent agents at /var/lib/mongodb-mms-automation/mongodb-mms-backup-agent-.+\..+_.+/mongodb-mms-backup-agent *$
[2025-08-04T18:33:53.580+0000] [.info] [realtime/rtcollector/rtcollector.go:controlLoop:93] <rtCollector> [18:33:53.580] Received new cluster config version 1, edition 2025-08-04 18:33:51.450390612 +0000 UTC!
[2025-08-04T18:33:53.580+0000] [.info] [src/dataexplorer/dataexplorer.go:controlLoop:115] <dataExplorer> [18:33:53.580] Received new cluster config version 1, edition 2025-08-04 18:33:51.450390612 +0000 UTC!
[2025-08-04T18:33:53.580+0000] [.info] [metrics/collector/realcollector.go:collectLoop:308] <hardwareMetricsCollector> [18:33:53.580] Received new cluster config version 1, edition 2025-08-04 18:33:51.450390612 +0000 UTC!
[2025-08-04T18:33:53.580+0000] [.info] [metrics/prometheus/exporter.go:mainLoop:85] <prometheusExporter> [18:33:53.580] Received new cluster config version 1, edition 2025-08-04 18:33:51.450390612 +0000 UTC!
[2025-08-04T18:33:53.580+0000] [.info] [metrics/prometheus/server.go:ApplyNewPromConfig:70] <prometheusExporter> [18:33:53.580] Prometheus config is nil or has prometheus explicitly disabled, stopping the prometheus server if running...
[2025-08-04T18:33:53.580+0000] [.info] [metrics/prometheus/server.go:func1:68] <prometheusExporter> [18:33:53.580] ApplyNewPromConfig took 14.39µs
[2025-08-04T18:33:53.591+0000] [.info] [main/components/agent.go:Iterate:913] [18:33:53.591] All 0 Mongo processes are in goal state.
[2025-08-04T18:33:54.517+0000] [.info] [src/runtimestats/printer.go:mainLoop:190] <runtimestatsPrinter> [18:33:54.517] stats:rss=50MB   heapInUse=8MB   memoryUsage=0.16%       normalizedCPU=0.00%     absoluteCPU=0.00%       gcNum=2 totalGcPauseNs=232796
[2025-08-04T18:34:03.605+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:03.605] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:03.605+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:03.605] clusterConfig unchanged
[2025-08-04T18:34:03.605+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:03.605] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:13.610+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:13.610] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:13.611+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:13.611] clusterConfig unchanged
[2025-08-04T18:34:13.611+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:13.611] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:23.649+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:23.649] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:23.649+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:23.649] clusterConfig unchanged
[2025-08-04T18:34:23.649+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:23.649] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:33.691+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:33.691] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:33.691+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:33.691] clusterConfig unchanged
[2025-08-04T18:34:33.691+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:33.691] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:43.735+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:43.735] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:43.735+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:43.735] clusterConfig unchanged
[2025-08-04T18:34:43.735+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:43.735] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:53.748+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:34:53.748] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:34:53.748+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:34:53.748] clusterConfig unchanged
[2025-08-04T18:34:53.748+0000] [.info] [main/components/agent.go:Iterate:913] [18:34:53.748] All 0 Mongo processes are in goal state.
[2025-08-04T18:34:54.518+0000] [.info] [src/runtimestats/printer.go:mainLoop:190] <runtimestatsPrinter> [18:34:54.518] stats:rss=54MB   heapInUse=9MB   memoryUsage=0.17%       normalizedCPU=0.03%     absoluteCPU=0.50%       gcNum=4 totalGcPauseNs=493592
[2025-08-04T18:35:03.784+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:03.784] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:03.784+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:03.784] clusterConfig unchanged
[2025-08-04T18:35:03.784+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:03.784] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:13.812+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:13.812] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:13.812+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:13.812] clusterConfig unchanged
[2025-08-04T18:35:13.812+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:13.812] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:23.838+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:23.838] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:23.838+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:23.838] clusterConfig unchanged
[2025-08-04T18:35:23.838+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:23.838] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:33.878+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:33.878] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:33.878+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:33.878] clusterConfig unchanged
[2025-08-04T18:35:33.878+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:33.878] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:43.885+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:43.885] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:43.885+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:43.885] clusterConfig unchanged
[2025-08-04T18:35:43.885+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:43.885] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:53.919+0000] [.info] [src/config/config.go:ReadClusterConfig:447] [18:35:53.919] Retrieving cluster config from /var/lib/automation/config/cluster-config.json...
[2025-08-04T18:35:53.919+0000] [.info] [main/components/agent.go:LoadClusterConfig:276] [18:35:53.919] clusterConfig unchanged
[2025-08-04T18:35:53.919+0000] [.info] [main/components/agent.go:Iterate:913] [18:35:53.919] All 0 Mongo processes are in goal state.
[2025-08-04T18:35:54.520+0000] [.info] [src/runtimestats/printer.go:mainLoop:190] <runtimestatsPrinter> [18:35:54.520] stats:rss=55MB   heapInUse=11MB  memoryUsage=0.17%       normalizedCPU=0.03%     absoluteCPU=0.48%       gcNum=5 totalGcPauseNs=544383

❯ kubectl exec -it example-mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent.log
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        GitCommitId = ec1573c1fd5d7da3acab288d628b9e9eaaec6b2b
[2025-08-04T18:33:53.516+0000] [header.info] [::0]  AutomationVersion = 108.0.2.8729
[2025-08-04T18:33:53.516+0000] [header.info] [::0]          localhost = example-mongodb-0.example-mongodb-svc.test.svc.cluster.prod.xxx
[2025-08-04T18:33:53.516+0000] [header.info] [::0] ErrorStateSleepTime = 10s
[2025-08-04T18:33:53.516+0000] [header.info] [::0] GoalStateSleepTime = 10s
[2025-08-04T18:33:53.516+0000] [header.info] [::0] NotGoalStateSleepTime = 1s
[2025-08-04T18:33:53.516+0000] [header.info] [::0]     PlanCutoffTime = 300000
[2025-08-04T18:33:53.516+0000] [header.info] [::0]       TracePlanner = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]               User = mongodb
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         Go version = go1.22.9
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         MmsBaseURL =
[2025-08-04T18:33:53.516+0000] [header.info] [::0]         MmsGroupId =
[2025-08-04T18:33:53.516+0000] [header.info] [::0]          HttpProxy =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DisableHttpKeepAlive = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        HttpsCAFile =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] TlsRequireValidMMSServerCertificates = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0] TlsMMSServerClientCertificate =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] KMIPProxyCertificateDir = /tmp
[2025-08-04T18:33:53.516+0000] [header.info] [::0] EnableLocalConfigurationServer = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] LocalConfigurationServerPort = 0
[2025-08-04T18:33:53.516+0000] [header.info] [::0]   InternalPromPort = 0
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DialTimeoutSeconds = 40
[2025-08-04T18:33:53.516+0000] [header.info] [::0] KeepUnusedMongodbVersions = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] DisallowDowngrades = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]   UseLocalBinaries = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0]  BinariesFixedPath =
[2025-08-04T18:33:53.516+0000] [header.info] [::0] UseLocalMongoDbTools = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        NoDaemonize = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]     SkipMongoStart = true
[2025-08-04T18:33:53.516+0000] [header.info] [::0]       operatorMode = false
[2025-08-04T18:33:53.516+0000] [header.info] [::0] healthCheckFilePath = /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json

[mikelsid]

Well, the issue appears to be with our kubernetes cluster domain, which differs from the standard .cluster.local.
For this reason, mongodb-agent expected an fqdn:

❯ kubectl exec -it example-mongodb-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent.log
[2025-08-04T18:33:53.516+0000] [header.info] [::0]        GitCommitId = ec1573c1fd5d7da3acab288d628b9e9eaaec6b2b
[2025-08-04T18:33:53.516+0000] [header.info] [::0]  AutomationVersion = 108.0.2.8729
[2025-08-04T18:33:53.516+0000] [header.info] [::0]          localhost = example-mongodb-0.example-mongodb-svc.test.svc.cluster.prod.xxx

While in automation config we had:

{
    "version": 1,
    "processes": [
        {
            "name": "example-mongodb-0",
            "disabled": false,
            "hostname": "example-mongodb-0.example-mongodb-svc.test.svc.cluster.local",
        }],
}

From mongodb-community-operator/controllers/replica_set_controller.go it seems that it's possible to change the cluster domain by setting env variable in operator:

const (
	clusterDomain = "CLUSTER_DOMAIN"

...

func buildAutomationConfig(mdb mdbv1.MongoDBCommunity, isEnterprise bool, auth automationconfig.Auth, currentAc automationconfig.AutomationConfig, modifications ...automationconfig.Modification) (automationconfig.AutomationConfig, error) {
	domain := getDomain(mdb.ServiceName(), mdb.Namespace, os.Getenv(clusterDomain))        // nolint:forbidigo
	arbiterDomain := getDomain(mdb.ServiceName(), mdb.Namespace, os.Getenv(clusterDomain)) // nolint:forbidigo

And it really fixes the issue.
But I didn't find any documented options in the new Helm chart, that would allow to set extraEnv or set this option directly

[jbrinksmeier]

We hit this too, and it appears that you can set extraEnv to the operator using customEnvVars:

mongodb-kubernetes/helm_chart/templates/operator.yaml

Line 278 in 34f1910

{{- if .Values.customEnvVars }}

This worked for us:

helm upgrade --install mongodb-kubernetes-operator mongodb/mongodb-kubernetes --namespace mongodb-operator --create-namespace --set customEnvVars="CLUSTER_DOMAIN=my-cluster.local"

Maintainers: I think its worth having this in the docs specifically, as I believe having a dedicated cluster-domain is a common use-case