Collector unable to use TLS Config from Target Allocator with ScrapeConfig

[CharlieTLe]

Component(s)

collector, target allocator

What happened?

Description

The OpenTelemetry Collector is unable to use the TLS Configuration values that are specified in a ScrapeConfig CR from the Target Allocator.

Steps to Reproduce

1. Setup OTel Operator with PrometheusCR to have a collector and target allocator setup.
2. Specify a ScrapeConfig to scrape the nodes in the cluster

apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: scrape-config
spec:
  scheme: HTTPS
  kubernetesSDConfigs:
    - role: Node
  authorization:
    type: Bearer
    credentials:
      key: "token"
      name: "opentelemetry-collector"
  tlsConfig:
    ca:
      secret:
        key: "ca.crt"
        name: "opentelemetry-collector"
    insecureSkipVerify: false

Expected Result

Successful scraping of the Kubernetes node.

Actual Result

Error while reading CA certificate needed to verify communication with Kubernetes node.

Kubernetes Version

1.31.0

Operator version

0.118.0

Collector version

0.118.0

Environment information

The operator is configured with --feature-gates=operator.targetallocator.mtls to allow for the collector to get secret information from the target allocator.

Log output

The collector logs this error:

error	scrape/manager.go:258	error reloading scrape pool	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "error": "error creating HTTP client: unable to read CA cert: unable to read file /etc/prometheus/certs/opentelemetry-collector_ca.crt: open /etc/prometheus/certs/opentelemetry-collector_ca.crt: no such file or directory", "scrape_pool": "scrapeConfig/static-config"}
github.com/prometheus/prometheus/scrape.(*Manager).ApplyConfig
	github.com/prometheus/prometheus@v0.54.1/scrape/manager.go:258
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).applyCfg
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.118.0/targetallocator/manager.go:177
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).sync
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.118.0/targetallocator/manager.go:156
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).Start.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.118.0/targetallocator/manager.go:80

Additional context

The scrape config from the Target Allocator looks something like this.

curl -s localhost:8080/scrape_configs | jq
{
  "scrapeConfig/static-config": {
    "authorization": {
      "credentials": "<secret>",
      "type": "Bearer"
    },
    "job_name": "scrapeConfig/static-config",
    "kubernetes_sd_configs": [
      {
        "role": "node"
      }
    ],
    "metrics_path": "/metrics",
    "scheme": "https",
    "tls_config": {
      "ca_file": "/etc/prometheus/certs/opentelemetry-collector_ca.crt",
      "insecure_skip_verify": false
    },
    "track_timestamps_staleness": false
  }
}

It's possible to have the ca_file become just ca with the contents of the file being referenced. The change would happen in https://github.com/prometheus-operator/prometheus-operator/blob/8b110a62bb134bad46fc5382776943a9059c4fa0/pkg/prometheus/promcfg.go#L628-L630 similar to how the authorization credentials are given to the collector in https://github.com/prometheus-operator/prometheus-operator/blob/8b110a62bb134bad46fc5382776943a9059c4fa0/pkg/prometheus/promcfg.go#L503-L511

[CharlieTLe]

Probably related to #3572

[CharlieTLe]

Now that prometheus-operator/prometheus-operator#7359 is merged, we should be able to set inlineTLSConfig to true in the ConfigGenerator to get the cert details needed for the collector from the TA.

[fasher]

@CharlieTLe is there any workaround to this issue ?