Receiver : Prometheus : Authorization Password not replaced

[joaolcorreia]

Component(s)

No response

What happened?

Description

When you have a receiver prometheus that requires authentication the password is not being adding to the header authorization: Basic xxxxxxx

The xxxxx is replaces the username: and shows instead of replacing the password given in the config file

Steps to Reproduce

Configure a prometheus receiver with authentication pointing to Pipedream and inspect the headers.

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'warpstream'
          metrics_path: '/api/v1/monitoring/prometheus/virtual_clusters/xxxxxx'
          scheme: https
          basic_auth:
            username: prometheus
            password: "xxxxxxxxx"
          scrape_interval: 60s
          static_configs:
            - targets: ['xxxxx.m.pipedream.net']
              labels:
                service: 'warpstream'
                environment: 'production'

Expected Result

You should see username:password base 64 encoded

Actual Result

You see username:<secret> you see <secret> instead of the actual secret.

Collector version

latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Log output

Additional context

No response

[axw]

I can reproduce this issue.

Searching for <secret> in the Prometheus code base, I came across https://github.com/prometheus/common/blob/75c3814dc66c571cc82cee2d3a6bf5f37ee73f1a/config/config.go#L35-L44

It looks like it's related to how the config gets re-marshaled to YAML when reloading. See

opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go

Lines 147 to 158 in b467589

	func reloadPromConfig(dst *PromConfig, src any) error {
	yamlOut, err := yaml.Marshal(src)
	if err != nil {
	return fmt.Errorf("prometheus receiver: failed to marshal config to yaml: %w", err)
	}
	newCfg, err := promconfig.Load(string(yamlOut), slog.Default())
	if err != nil {
	return fmt.Errorf("prometheus receiver: failed to unmarshal yaml to prometheus config object: %w", err)
	}
	*dst = PromConfig(*newCfg)
	return nil
	}

There's a global (MarshalSecretValue) that can be set to true to marshal these secrets as-is. I don't know if that would have any other negative side-effects.

[github-actions]

Pinging code owners for receiver/prometheus: @Aneurysm9 @dashpole @ArthurSens @krajorama. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

[ArthurSens]

Seems like a duplicate of #40520, the fix is being worked on in #40780

[ridwanmsharif]

It looks like it's related to how the config gets re-marshaled to YAML when reloading. See

opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go

Lines 147 to 158 in b467589

func reloadPromConfig(dst *PromConfig, src any) error {
yamlOut, err := yaml.Marshal(src)
if err != nil {
return fmt.Errorf("prometheus receiver: failed to marshal config to yaml: %w", err)
}
newCfg, err := promconfig.Load(string(yamlOut), slog.Default())
if err != nil {
return fmt.Errorf("prometheus receiver: failed to unmarshal yaml to prometheus config object: %w", err)
}
*dst = PromConfig(*newCfg)
return nil
}
There's a global (MarshalSecretValue) that can be set to true to marshal these secrets as-is. I don't know if that would have any other negative side-effects.

We don't even have to use a global setting here, we can update the reloadPromConfig to use a custom marshaller to preserve the secret value on reload. See here for an example: https://github.com/GoogleCloudPlatform/ops-agent/blob/1c8f5801ca9e65e9e45dcdb90e570fc6d86fd908/confgenerator/prometheus.go#L137-L153

I updated the PR with the fix with this suggestion: #40780 (comment)

[erikburt]

@ridwanmsharif - I will try this out. Thanks 🙏