Component(s)
No response
What happened?
Description
Target Allocator workload does not reload certificates after expiry
Steps to Reproduce
- Enable mtls between Opentelemetry and target allocator using Operator
- Wait till the certs are getting expired or renew them
- Updated certs via k8s secrets are mounted in the filesystem of target allocator.
- Check connections between Opentelemetry collector pods and target allocator pods
Target allocator goes on crashing as the probe fails with the expired cert.
https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/internal/config/config.go#L389
The older certs are passed to the listener for the incoming requests
Expected Result
Target allocator should hot-reload the renewed certificates so that connections wont be interrupted.
Both server cert and CA file needs to be reloaded, otherwise there will be connection failures
when CA is renewed or server-cert is renewed.
Actual Result
No reload is happening and crashing both Opentelemetry collector and targetallocator instances
Kubernetes Version
1.29.14
Operator version
0.131.0
Collector version
0.134.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
Log output
Error: cannot start pipelines: failed to start \"prometheus/metrics\" receiver: Get \"https://abcd-targetallocator:443/scrape_configs\": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-09-17T07:48:37Z is after 2025-09-17T07:46:52Z; failed to shutdown pipelines: no existing monitoring routine is running
Additional context
Both server and CA files need to be reloaded to ensure proper connectivity.
Generally CA validity is kept more, so we may not notice the issue even after fixing the hot reload for server cert.
But both server and CAs need to be watched and passed along periodically so that latest renewed certs are used after expiration
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.
Component(s)
No response
What happened?
Description
Target Allocator workload does not reload certificates after expiry
Steps to Reproduce
Target allocator goes on crashing as the probe fails with the expired cert.
https://github.com/open-telemetry/opentelemetry-operator/blob/main/cmd/otel-allocator/internal/config/config.go#L389
The older certs are passed to the listener for the incoming requests
Expected Result
Target allocator should hot-reload the renewed certificates so that connections wont be interrupted.
Both server cert and CA file needs to be reloaded, otherwise there will be connection failures
when CA is renewed or server-cert is renewed.
Actual Result
No reload is happening and crashing both Opentelemetry collector and targetallocator instances
Kubernetes Version
1.29.14
Operator version
0.131.0
Collector version
0.134.0
Environment information
Environment
OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
Log output
Additional context
Both server and CA files need to be reloaded to ensure proper connectivity.
Generally CA validity is kept more, so we may not notice the issue even after fixing the hot reload for server cert.
But both server and CAs need to be watched and passed along periodically so that latest renewed certs are used after expiration
Tip
React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding
+1orme too, to help us triage it. Learn more here.