Skip to content

Renewed certificates are not reloaded by target allocator #4368

@noentry12

Description

@noentry12

Component(s)

No response

What happened?

Description

Target Allocator workload does not reload certificates after expiry

Steps to Reproduce

  1. Enable mtls between Opentelemetry and target allocator using Operator
  2. Wait till the certs are getting expired or renew them
  3. Updated certs via k8s secrets are mounted in the filesystem of target allocator.
  4. Check connections between Opentelemetry collector pods and target allocator pods

Target allocator goes on crashing as the probe fails with the expired cert.
https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/internal/config/config.go#L389

The older certs are passed to the listener for the incoming requests

Expected Result

Target allocator should hot-reload the renewed certificates so that connections wont be interrupted.
Both server cert and CA file needs to be reloaded, otherwise there will be connection failures
when CA is renewed or server-cert is renewed.

Actual Result

No reload is happening and crashing both Opentelemetry collector and targetallocator instances

Kubernetes Version

1.29.14

Operator version

0.131.0

Collector version

0.134.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

Error: cannot start pipelines: failed to start \"prometheus/metrics\" receiver: Get \"https://abcd-targetallocator:443/scrape_configs\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-09-17T07:48:37Z is after 2025-09-17T07:46:52Z; failed to shutdown pipelines: no existing monitoring routine is running

Additional context

Both server and CA files need to be reloaded to ensure proper connectivity.
Generally CA validity is kept more, so we may not notice the issue even after fixing the hot reload for server cert.
But both server and CAs need to be watched and passed along periodically so that latest renewed certs are used after expiration

Tip

React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions