failed to verify certificate: x509: certificate signed by unknown authority

[ImDevinC]

Description

I have installed both cert-manager and opentelemetry-operator. Other services work fine with cert-manager. However opentelemetry-operator keeps showing this error in the log and I'm not getting metrics due to it.

Steps to Reproduce

1. Apply config
2. Look at logs

Expected Result

No errors about certificate, and metrics show up

Actual Result

Log is filled with errors about invalid certificate, no metrics showing up

Kubernetes Version

v1.31.4

Operator version

0.136.0

Collector version

0.136.0

Environment information

Environment

OS: Arch 6.11.8-arch1-2

Log output

2025-10-19T21:52:46.839Z	error	targetallocator/manager.go:126	Failed to retrieve job list	{"resource": {"service.instance.id": "c6954f62-10ff-49df-9515-1965610f81d4", "service.name": "otelcol-contrib", "service.version": "0.136.0"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "Get \"https://otelcol-targetallocator:443/scrape_configs\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"otelcol-ca-cert\")"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).sync
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.136.0/targetallocator/manager.go:126
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).Start.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.136.0/targetallocator/manager.go:81
github.com/cenkalti/backoff/v5.Retry[...]
	github.com/cenkalti/backoff/v5@v5.0.3/retry.go:87
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).Start
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.136.0/targetallocator/manager.go:91
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).Start
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.136.0/metrics_receiver.go:121
go.opentelemetry.io/collector/service/internal/graph.(*Graph).StartAll
	go.opentelemetry.io/collector/service@v0.136.0/internal/graph/graph.go:432
go.opentelemetry.io/collector/service.(*Service).Start
	go.opentelemetry.io/collector/service@v0.136.0/service.go:254
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
	go.opentelemetry.io/collector/otelcol@v0.136.0/collector.go:240
go.opentelemetry.io/collector/otelcol.(*Collector).Run
	go.opentelemetry.io/collector/otelcol@v0.136.0/collector.go:310
go.opentelemetry.io/collector/otelcol.NewCommand.func1
	go.opentelemetry.io/collector/otelcol@v0.136.0/command.go:39
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.10.1/command.go:1015
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.10.1/command.go:1148
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.10.1/command.go:1071
main.runInteractive
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:70
main.run
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:10
main.main
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:63
runtime.main
	runtime/proc.go:283

Additional context

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
  - name: opentelemetry-operator
    releaseName: opentelemetry-operator
    repo: https://open-telemetry.github.io/opentelemetry-helm-charts
    version: 0.97.0
    valuesFile: values.yaml
    namespace: monitoring
namespace: monitoring
resources:
  - opentelemetry-collector.yaml

values.yaml

manager:
  featureGatesMap:
    operator.targetallocator.mtls: true
  collectorImage:
    repository: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib

opentelemetry-collector.yaml

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol
spec:
  mode: statefulset
  upgradeStrategy: automatic
  serviceAccount: opentelemetry-allocator-sa
  targetAllocator:
    enabled: true
    serviceAccount: opentelemetry-allocator-sa
    prometheusCR:
      enabled: true
      podMonitorSelector: {}
      serviceMonitorSelector: {}
  config:
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: otel-collector
              scrape_interval: 10s
              static_configs:
                - targets: ["0.0.0.0:8888"]
            - job_name: node-exporter
              static_configs:
                - targets: ["archlinux:9100", "pi.hole:9100"]
            - job_name: nvidia-smi
              static_configs:
                - targets: ["archlinux:9835"]
            - job_name: rpi
              static_configs:
                - targets: ["pi.hole:9110"]
            - job_name: fail2ban
              static_configs:
                - targets: ["pi.hole:9191"]
            - job_name: crowdsec
              static_configs:
                - targets: ["pi.hole:6060", "pi.hole:60601"]

    processors:
      batch: {}
      attributes/cluster:
        actions:
          - key: cluster
            value: k8s
            action: insert

    exporters:
      prometheusremotewrite:
        endpoint: http://prometheus-operated:9090/api/v1/write
        tls:
          insecure: true

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [attributes/cluster, batch]
          exporters: [prometheusremotewrite]

kubectl describe certificate -n monitoring otelcol-ca-cert -o yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  creationTimestamp: "2025-10-19T21:47:43Z"
  generation: 1
  labels:
    app.kubernetes.io/component: opentelemetry-targetallocator
    app.kubernetes.io/instance: monitoring.otelcol
    app.kubernetes.io/managed-by: opentelemetry-operator
    app.kubernetes.io/name: otelcol-ca-cert
    app.kubernetes.io/part-of: opentelemetry
    app.kubernetes.io/version: latest
  name: otelcol-ca-cert
  namespace: monitoring
  ownerReferences:
  - apiVersion: opentelemetry.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TargetAllocator
    name: otelcol
    uid: c2d1aeef-5f1c-450d-8b26-ae55054a7507
  resourceVersion: "218930758"
  uid: f66329a9-aab1-437a-971d-dbecd904e2cc
spec:
  commonName: otelcol-ca-cert
  isCA: true
  issuerRef:
    kind: Issuer
    name: otelcol-self-signed-issuer
  secretName: otelcol-ca-cert
  subject:
    organizationalUnits:
    - opentelemetry-operator
status:
  conditions:
  - lastTransitionTime: "2025-10-19T21:47:43Z"
    message: Certificate is up to date and has not expired
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready
  notAfter: "2026-01-17T09:25:18Z"
  notBefore: "2025-10-19T09:25:18Z"
  renewalTime: "2025-12-18T09:25:18Z"

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[ImDevinC]

Of note, if I disable the mtls option in the operator flags, metrics work properly, but that prevents me from using API endpoints with authentication that requires secrets.

[pavolloffay]

I think this is related to the target allocator cc) @swiatekm

[alorenzo175]

I just saw the same kind of issue. In my case, the operator created the target allocator CA cert, TA client cert, and TA server cert all at the same time with the same (default) duration. So all three tried to renew at the same time, but the order of renewals was TA Server -> CA -> TA client causing a CA mismatch between the server and client

[swiatekm]

@ImDevinC are you able to reproduce this issue in an empty kind cluster? If it's the same issue as @alorenzo175 experienced, then the fix should be straightforward, but that can only really happen if you've been running this setup for at least the default cert renewal time.

Thank you for reporting these issues, by the way. The feature flag is still in Alpha precisely because we didn't want to enable it for all users without first working through these kinds of problems.

[ImDevinC]

I have been running this setup for awhile, so definitely possible. Using a fresh install, I could not replicate. @alorenzo175 can you clarify how you fixed it? Just delete all three certificates?

[swiatekm]

I'd be fine with a fix that increases the duration on the CA certificate or changes the renew period. It's not clear this is the root cause of the problem reported here, but it sounds like it can be problematic, so there's no reason to hold back on it.

[ImDevinC]

Is there a fix that can be implemented in the meantime manually?

[swiatekm]

Deleting the certs should cause the operator to recreate them.

[ImDevinC]

I deleted all three certs, but after they are recreated it is still failing

[swiatekm]

We might need to look at the secret values for the certificates then. In particular the target allocator cert and the CA cert, since that's where the error is coming from.

[ImDevinC]

Appreciate it, here's the three certificates

k get secret -n monitoring otelcol-ca-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout                
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            86:b1:5b:08:21:f5:ac:2f:10:bf:c4:8d:49:02:b9:a6
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU=opentelemetry-operator, CN=otelcol-ca-cert
        Validity
            Not Before: Oct 19 09:25:18 2025 GMT
            Not After : Jan 17 09:25:18 2026 GMT
        Subject: OU=opentelemetry-operator, CN=otelcol-ca-cert
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ba:80:39:91:e0:8e:bc:ca:85:ef:ae:1e:ee:bd:
                    81:d0:59:b7:49:18:15:9c:42:6a:41:61:d1:eb:1b:
                    dc:df:ac:e9:b8:71:a2:30:6e:e6:2c:3b:df:d7:f4:
                    5e:62:5d:0b:08:3c:79:ad:5b:8b:d9:d1:f1:e5:0b:
                    02:87:bc:d8:cb:52:37:22:db:a9:cd:56:9f:65:48:
                    67:27:dc:fa:82:b4:2c:3c:d0:03:ed:e9:b8:ca:d8:
                    c2:2b:47:80:12:5f:20:57:f0:9f:9b:10:36:af:d9:
                    53:22:b9:a4:9d:8c:95:e7:5a:8c:74:97:e3:2b:60:
                    9f:6b:50:50:30:ed:23:02:4e:7c:3d:1b:fa:8b:6e:
                    9f:1b:9c:d6:83:19:dc:b2:8c:ea:be:ec:53:35:14:
                    47:c3:db:70:69:c5:1c:b7:16:89:cd:ad:b6:a7:bf:
                    c0:9c:6c:66:b0:78:4d:d9:64:a8:82:5e:a6:4b:75:
                    db:9c:82:be:58:ad:24:6e:3d:90:09:99:4d:f9:78:
                    74:60:8b:99:26:0a:f8:b7:4f:88:7f:1a:f0:59:2c:
                    5b:f1:75:89:1c:ce:84:df:b3:b8:e1:d6:d6:6c:de:
                    aa:ba:93:18:b1:5c:25:cf:8d:b3:1d:1d:0e:2a:50:
                    30:3d:e9:47:ec:ba:fb:09:7c:19:ba:3f:8d:e0:5c:
                    92:2d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier: 
                DE:72:F3:6C:EF:F7:D2:CF:FE:AF:B1:7B:5E:86:2D:99:00:A7:AC:E5
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        57:c9:b3:1d:ea:2a:79:3e:c0:09:3a:b1:66:0a:26:74:19:f9:
        5f:74:d3:08:1f:b7:2e:5c:88:52:a9:12:1c:ba:ad:2e:cd:76:
        d0:a4:a0:8b:42:f4:a0:7c:c4:08:ec:1b:4e:ca:ed:f1:50:c5:
        25:d6:89:4d:6a:e3:fb:e6:48:4d:fc:49:93:1e:ec:ea:ee:46:
        ba:91:6e:a1:42:8e:1e:8c:c2:33:29:3b:d8:5e:29:a9:4e:e7:
        5c:99:6d:58:23:08:0a:0e:fe:f9:7d:f0:73:8f:68:86:cd:a5:
        70:2f:92:a6:03:90:0a:f9:a0:2c:4a:ea:1a:05:e3:67:d0:28:
        45:f9:7b:52:df:e3:4f:75:da:48:76:1b:22:dd:7f:6d:c9:08:
        ea:7f:be:8b:cd:93:74:36:86:4e:3b:81:a6:ac:ac:4c:8a:6b:
        fb:20:46:9d:c4:10:89:6f:52:d5:85:79:0e:ec:f5:ce:20:aa:
        2a:e7:7f:d9:56:69:30:81:8e:95:21:e7:5b:3f:0c:b5:f1:6d:
        dc:2c:f2:89:84:b6:a2:77:39:95:ed:90:41:96:96:a6:46:20:
        f4:c2:0e:2a:cd:f8:d7:09:23:66:a8:dd:99:07:cb:70:45:63:
        34:97:22:0f:47:77:a8:d0:3b:af:24:4a:e6:e8:cf:a8:c3:5c:
        f5:59:a9:9d

k get secret -n monitoring otelcol-ta-client-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            be:3d:4a:97:65:2b:b3:04:1c:d5:9f:77:cb:9b:16:68
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU=opentelemetry-operator, CN=otelcol-ca-cert
        Validity
            Not Before: Oct 19 07:58:53 2025 GMT
            Not After : Jan 17 07:58:53 2026 GMT
        Subject: OU=opentelemetry-operator
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cb:ee:0d:d0:ea:29:5a:a7:45:6c:d8:71:d9:cb:
                    28:67:5d:0a:83:d5:19:c6:a8:91:2e:5f:ab:ae:0a:
                    ca:ee:2e:7b:5f:8b:aa:26:58:cc:82:71:33:64:f5:
                    9f:ca:b5:8a:cb:3f:fd:a9:cd:14:73:b3:3b:ff:e7:
                    fd:19:6f:2a:07:cc:b9:54:54:77:a6:e5:aa:82:08:
                    48:85:b7:6c:e0:51:f1:be:8b:69:0b:0c:98:f5:f9:
                    ba:aa:0b:59:d6:87:9d:e7:ac:22:55:2c:2a:5f:22:
                    31:ee:dd:f7:4d:74:f0:de:b0:7c:c2:94:e4:50:6c:
                    d5:18:c4:0b:6d:c0:4e:0e:a9:26:3f:07:f0:84:12:
                    ec:07:c4:bc:d5:d9:22:d0:e6:87:e8:98:8c:c7:d2:
                    17:83:3e:fa:a6:ef:30:3f:d2:54:0c:dc:c6:fa:34:
                    a6:ad:5e:e9:18:32:17:4d:e7:e9:5a:4a:62:b5:6b:
                    0f:25:8e:e2:2c:72:d5:d4:3b:84:41:1b:41:bb:b5:
                    d3:ba:d6:df:12:b3:26:b2:25:c7:e0:51:20:ed:8c:
                    1b:62:23:64:1f:5b:da:70:3b:a0:0c:e4:a4:96:a0:
                    a2:a3:58:f2:cf:c3:ec:09:a0:17:d4:8c:64:10:13:
                    9d:b6:63:a5:2d:97:e1:48:16:99:33:44:03:f6:f4:
                    eb:db
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                78:7B:69:C9:8F:36:89:7A:D1:B3:08:B2:FF:81:61:60:AC:26:3A:15
            X509v3 Subject Alternative Name: 
                DNS:otelcol-targetallocator, DNS:otelcol-targetallocator.monitoring.svc, DNS:otelcol-targetallocator.monitoring.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        5d:12:0e:89:8d:47:84:4b:40:2f:cc:c6:41:f3:d3:b6:75:c9:
        c9:7a:8c:87:53:f5:ad:69:7e:0c:0f:cc:cd:0f:25:98:d5:29:
        73:29:f5:8e:4d:b5:ed:a1:c5:13:23:f8:2e:26:dc:3a:26:09:
        d7:81:f4:bd:13:a3:e6:60:34:8c:24:ea:31:53:76:d4:47:0c:
        c2:e2:73:fb:1e:fb:dc:22:c0:32:77:85:df:16:5a:0c:06:ae:
        6f:38:86:33:28:25:d8:d9:80:56:32:e8:e9:11:e3:3b:dd:45:
        4f:e3:79:31:aa:9e:e5:ae:28:92:41:63:2f:09:4e:3d:5d:f7:
        20:83:40:5f:91:40:85:30:3a:67:4f:e8:76:47:7d:62:fe:5c:
        c7:18:15:1a:e0:21:75:6b:23:e6:bf:36:13:e7:f7:28:ac:a6:
        b5:b3:7f:25:00:03:8b:b9:db:8f:3f:9b:d9:52:86:32:cb:02:
        92:b2:f4:5a:5c:d9:01:ec:7b:4b:5e:e2:dc:09:d4:66:75:cf:
        af:57:c8:17:f3:60:83:71:e2:24:05:50:67:7f:bb:9c:73:90:
        e6:d7:74:9b:16:d2:a6:78:1d:bd:78:ab:33:82:65:2f:4c:18:
        38:93:d6:9b:5d:7f:e8:5a:ac:24:19:b2:2a:9d:0d:a8:f8:44:
        ff:42:d6:0a

k get secret -n monitoring otelcol-ta-server-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            39:53:80:3f:ca:8d:5f:fa:4c:62:95:a9:98:5b:ff:eb
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: OU=opentelemetry-operator, CN=otelcol-ca-cert
        Validity
            Not Before: Oct 19 07:58:51 2025 GMT
            Not After : Jan 17 07:58:51 2026 GMT
        Subject: OU=opentelemetry-operator
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:ba:10:bf:df:6c:5b:9b:34:c2:5c:88:f9:d4:7c:
                    0c:2b:04:0c:b8:47:5a:2a:06:b9:5c:17:f4:bf:7f:
                    2a:ce:d6:3d:cb:0b:8b:2a:24:f6:59:65:30:54:ae:
                    7b:d6:fb:72:3f:6a:07:3d:7b:85:f6:c4:5d:b5:28:
                    74:a0:74:6e:c8:48:9f:9a:b6:e6:a5:2f:48:5c:24:
                    5c:87:c0:1c:18:72:68:0f:83:cf:43:a0:09:4c:4e:
                    84:22:a8:ab:ea:84:48:3d:c4:c9:d6:e9:c4:a0:e0:
                    2d:09:e9:c5:dd:45:9a:17:f6:bc:a2:f0:79:f7:45:
                    05:7c:6b:4f:2e:80:71:47:e5:ed:fe:7b:12:b7:7b:
                    46:45:07:e2:a5:58:19:a7:eb:19:0f:b4:2b:9e:9f:
                    d5:27:48:0f:62:bd:58:72:af:2d:57:b4:59:0b:8e:
                    75:ae:ac:fa:f4:93:17:27:5c:cc:8d:7a:ff:f4:5f:
                    15:26:56:7c:01:d5:fa:2c:80:92:5e:b9:33:98:cd:
                    49:c8:9a:ea:ed:86:aa:e7:98:c2:e3:ac:b5:80:f0:
                    96:65:90:6e:c4:e2:55:5b:06:d1:4f:5e:8c:46:97:
                    0b:31:bb:8a:98:6c:22:1d:7f:28:01:1d:0c:b7:f0:
                    91:ca:4f:9c:b2:75:fc:37:c0:f5:ef:ab:d6:ef:06:
                    35:6f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier: 
                36:79:DA:BC:0E:4D:1C:1E:65:44:CB:E3:E6:CB:A7:1D:19:95:36:43
            X509v3 Subject Alternative Name: 
                DNS:otelcol-targetallocator, DNS:otelcol-targetallocator.monitoring.svc, DNS:otelcol-targetallocator.monitoring.svc.cluster.local
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        28:ca:ed:ed:95:77:e2:c5:a4:0a:ce:aa:5a:6f:db:0c:7b:93:
        f1:83:3c:8e:1a:d6:43:70:29:b2:47:30:1a:9e:c2:05:4e:cf:
        91:46:8e:10:2f:ed:1e:98:93:5a:d6:0d:fa:9e:3a:1c:1d:08:
        a2:33:49:a4:40:5e:fd:81:0d:f3:6a:2e:4d:7c:01:05:03:2e:
        82:5f:1d:d9:3d:08:8c:44:d7:94:0f:cd:cb:51:79:a2:f1:a6:
        47:4d:14:a0:87:32:6f:92:23:fc:f2:96:2a:f7:b4:6f:5c:7f:
        b8:0e:13:9c:23:11:5c:2f:e9:ed:20:73:31:68:d9:62:9a:d9:
        be:db:be:ff:64:0d:1c:68:a3:d6:fc:dd:66:f1:d4:43:0e:0f:
        22:df:c1:03:f1:87:f0:47:a8:61:02:48:12:2d:55:a0:bb:bc:
        99:90:b8:74:0a:df:76:15:e2:b4:5f:2f:23:9b:8f:7b:36:09:
        31:4c:d8:4c:5f:97:08:b7:8a:0d:34:8e:db:be:79:9f:c4:d8:
        45:f3:84:5b:08:35:4d:65:c8:81:ab:0d:ee:5f:31:30:bb:00:
        24:1b:e0:08:2f:a2:c7:a2:28:63:9a:8a:4d:09:d0:f1:60:89:
        d2:cd:8f:08:15:71:22:b6:da:7a:ab:c5:ab:e1:c8:12:61:3d:
        ab:d0:f2:4a