[receiver/prometheusreceiver] basic authentication for urls stopped working

[Ltty]

Component(s)

No response

What happened?

Description

The Prometheus receiver fails to scrape and endpoint with basic authentication enabled.

Steps to Reproduce

• Set up basic authentication for a Prometheus endpoint, in this case 192.168.0.8:5572/metrics
• Configure basic_auth in the scrape config
• Start collector

Expected Result

Prometheus receiver is able to scrape the endpoint.

Actual Result

Scrape fails with 401 Unauthorized

Collector version

v0.127.0

Environment information

Environment

OS: Ubuntu 24.04.2 LTS

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: integrations/rclone
          metrics_path: /metrics
          scrape_interval: 30s
          static_configs:
            - targets: ['192.168.0.8:5572']
          basic_auth:
            username: "USR"
            password: "PWD"
          relabel_configs:
            - source_labels: [__address__]
              action: replace
              target_label: instance
          metric_relabel_configs:
            - source_labels: [__name__]
              action: keep
              regex: "up|rclone_.*"

processors:
  cumulativetodelta:
  resourcedetection/dynatrace:
   override: false
   detectors: [dynatrace]

Log output

2025-06-06T09:48:43.196Z        debug   Scrape failed   {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_pool": "integrations/rclone", "target": "http://192.168.0.8:5572/metrics", "err": "server returned HTTP status 401 Unauthorized"}

2025-06-06T09:48:43.196Z        warn    internal/transaction.go:150     Failed to scrape Prometheus endpoint    {"resource": {}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "scrape_timestamp": 1749203323194, "target_labels": "{__name__=\"up\", instance=\"192.168.0.8:5572\", job=\"integrations/rclone\"}"}

Additional context

• Verified that the enpoint is reachable
• Verified that enpont returns 401 Unauthorized without providing basic authentication using curl 192.168.0.8:5572/metrics
• Verified that the endpoint returns the Prometheus metrics using curl -u USER:PWD 192.168.0.8:5572/metrics

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole @ArthurSens @krajorama

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@Ltty Does your prometheus serve show any errors why this might happen?

You mentioned that basic authentication for urls stopped working. Did it work in older versions and not working now?

[Ltty]

Unfortunately not. I just see that trying to scrape the endpoint returns a 401 Unauthorized with debug logs.

This has been working for almost a year since I initially set up the server. It stopped working with the latest distro update though.

And 401 is only returned if basic authentication is not working. Since I verified that that username and password is correct and that the endpoint is reachable, I concluded that the scrape is not authenticating correctly.

[dashpole]

What versions did you upgrade to/from?

[jage]

We're also experiencing this, broke when upgrading from 0.126.0 to 0.127.0. Still broken in 0.128.0.

When trying to understand what happened we captured this traffic:

$ sudo tcpdump -i lo -A -s 10240 'tcp port 15672 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
...
Host: 127.0.0.1:15672
User-Agent: otelcol/0.0.42
Accept: application/openmetrics-text;version=1.0.0;escaping=allow-utf-8;q=0.6,application/openmetrics-text;version=0.0.1;escaping=allow-utf-8;q=0.5,text/plain;version=1.0.0;escaping=allow-utf-8;q=0.4,text/plain;version=0.0.4;escaping=allow-utf-8;q=0.3,*/*;q=0.2
Accept-Encoding: gzip
Authorization: Basic YWRtaW46PHNlY3JldD4=
X-Prometheus-Scrape-Timeout-Seconds: 5

Decoding Base64 YWRtaW46PHNlY3JldD4= gives admin:<secret> which led me to https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L25 but haven't figured out if/how it's used in opentelemetry.

We are using a YAML config, but seeing same issue with JSON config, and putting the secret in a file, same issue.

This is our own build of otel-collector that fails:

dist:
  name: otelcol
  description: CloudAMQP OTel Collector
  output_path: ./otelcol
  version: 0.0.42

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.127.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/rabbitmqexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/googlecloudexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/azuremonitorexporter v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/coralogixexporter v0.127.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.127.0
  - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/cumulativetodeltaprocessor v0.127.0

receivers:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/journaldreceiver v0.127.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/encoding/otlpencodingextension v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/basicauthextension v0.127.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.127.0

providers:
  - gomod: go.opentelemetry.io/collector/confmap/provider/envprovider v1.33.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/fileprovider v1.33.0
  - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.33.0

[dashpole]

#40103 seems like the most likely culprit based on timing, and was the only relevant PR I can see.

cc @ArthurSens or @npordash who might have more context on why basic auth scrape config could have stopped working.

[erikburt]

I was able to replicate locally by pointing a scrape config at a local http server that logged the basic auth. Saw the same thing @jage did. It seems to use the configured username but uses <secret> for the password, instead of the configured one.

2025/06/16 15:29:11 Authentication attempt: Username='username', Password='<secret>'

Config

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: test
          basic_auth:
            username: username
            password: password
          metrics_path: /metrics
          scrape_interval: 1s
          static_configs:
            - targets:
              - localhost:8080

exporters:
  debug:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [debug]

Simple go http server

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"
)

// basicAuth is a middleware function to handle Basic Authentication.
func basicAuth(handler http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        user, pass, ok := r.BasicAuth()
        if !ok {
            log.Println("Authentication failed: No authentication provided.")
            w.Header().Set("WWW-Authenticate", `Basic realm="Restricted"`)
            http.Error(w, "Unauthorized.", http.StatusUnauthorized)
            return
        }

        // Log the username and password regardless of what they are.
        log.Printf("Authentication attempt: Username='%s', Password='%s'\n", user, pass)

        // In a real application, you would validate user and pass here.
        // For this example, we accept any authentication.

        handler(w, r)
    }
}

// metricsHandler handles requests to the /metrics endpoint.
func metricsHandler(w http.ResponseWriter, r *http.Request) {
    log.Println("Accessed /metrics endpoint successfully.")
    fmt.Fprintf(w, "Metrics data would be here.\n")
}

func main() {
    // Define a command-line flag for the port.
    port := flag.String("port", "8080", "Port to listen on")
    flag.Parse()

    // Register the /metrics endpoint with Basic Authentication.
    http.HandleFunc("/metrics", basicAuth(metricsHandler))

    // Start the HTTP server.
    listenAddr := ":" + *port
    log.Printf("Server starting on port %s\n", *port)
    log.Printf("Access the metrics endpoint at http://localhost:%s/metrics\n", *port)
    if err := http.ListenAndServe(listenAddr, nil); err != nil {
        log.Fatalf("Could not start server: %s\n", err)
    }
}
   

[erikburt]

I think I've root caused the issue. The problem lies with the new reloadPromConfig function.

opentelemetry-collector-contrib/receiver/prometheusreceiver/config.go

Lines 147 to 158 in dfc894e

	func reloadPromConfig(dst *PromConfig, src any) error {
	yamlOut, err := yaml.Marshal(src)
	if err != nil {
	return fmt.Errorf("prometheus receiver: failed to marshal config to yaml: %w", err)
	}
	newCfg, err := promconfig.Load(string(yamlOut), slog.Default())
	if err != nil {
	return fmt.Errorf("prometheus receiver: failed to unmarshal yaml to prometheus config object: %w", err)
	}
	*dst = PromConfig(*newCfg)
	return nil
	}

The password field in the basic_auth is of type Secret, and if you try and Marshal it, by default it will return the placeholder <secret>.

https://github.com/prometheus/common/blob/95acce133ca2c07a966a71d475fb936fc282db18/config/config.go#L36-L44

I believe the issue is because reloadPromConfig is now marshalling the config to then call promconfig.Load. Based on the PR comment this was needed to support target_allocator configs.

Here's a patch for a minimal reproduction in config_test.

Patch

diff --git i/receiver/prometheusreceiver/config_test.go w/receiver/prometheusreceiver/config_test.go
index c020b53744..5a5e6ac862 100644
--- i/receiver/prometheusreceiver/config_test.go
+++ w/receiver/prometheusreceiver/config_test.go
@@ -5,6 +5,7 @@ package prometheusreceiver
 
 import (
        "context"
+       "log/slog"
        "path/filepath"
        "strings"
        "testing"
@@ -12,6 +13,7 @@ import (
 
        promConfig "github.com/prometheus/common/config"
        promModel "github.com/prometheus/common/model"
+       promconfig "github.com/prometheus/prometheus/config"
        "github.com/stretchr/testify/assert"
        "github.com/stretchr/testify/require"
        "go.opentelemetry.io/collector/component"
@@ -399,3 +401,79 @@ func TestLoadPrometheusAPIServerExtensionConfig(t *testing.T) {
        require.NoError(t, sub.Unmarshal(cfg))
        require.Error(t, xconfmap.Validate(cfg))
 }
+
+func TestReloadPromConfigSecretHandling(t *testing.T) {
+       // This test specifically focuses on the secret marshaling issue mentioned
+       // where config entries might get marshaled to "<secret>" instead of proper values
+
+       tests := []struct {
+               name       string
+               configYAML string
+               checkFn    func(t *testing.T, dst *PromConfig)
+       }{
+               {
+                       name: "basic auth password preservation",
+                       configYAML: `
+scrape_configs:
+  - job_name: "test-basic-auth"
+    basic_auth:
+      username: "testuser"
+      password: "mysecretpassword"
+    static_configs:
+      - targets: ["localhost:8080"]
+`,
+                       checkFn: func(t *testing.T, dst *PromConfig) {
+                               require.Len(t, dst.ScrapeConfigs, 1)
+                               scrapeConfig := dst.ScrapeConfigs[0]
+                               assert.Equal(t, "test-basic-auth", scrapeConfig.JobName)
+
+                               // The critical check: ensure the password is not "<secret>"
+                               require.NotNil(t, scrapeConfig.HTTPClientConfig.BasicAuth, "basic auth should be configured")
+                               password := string(scrapeConfig.HTTPClientConfig.BasicAuth.Password)
+                               assert.NotEqual(t, "<secret>", password, "password should not be marshaled as '<secret>'")
+                               assert.Equal(t, "mysecretpassword", password, "password should preserve original value")
+                               assert.Equal(t, "testuser", scrapeConfig.HTTPClientConfig.BasicAuth.Username)
+                       },
+               },
+               {
+                       name: "bearer token preservation",
+                       configYAML: `
+scrape_configs:
+  - job_name: "test-bearer-token"
+    authorization:
+      type: "Bearer"
+      credentials: "mySecretBearerToken123"
+    static_configs:
+      - targets: ["localhost:9090"]
+`,
+                       checkFn: func(t *testing.T, dst *PromConfig) {
+                               require.Len(t, dst.ScrapeConfigs, 1)
+                               scrapeConfig := dst.ScrapeConfigs[0]
+                               assert.Equal(t, "test-bearer-token", scrapeConfig.JobName)
+
+                               // Check that bearer token is preserved
+                               require.NotNil(t, scrapeConfig.HTTPClientConfig.Authorization, "authorization should be configured")
+                               credentials := string(scrapeConfig.HTTPClientConfig.Authorization.Credentials)
+                               assert.NotEqual(t, "<secret>", credentials, "credentials should not be marshaled as '<secret>'")
+                               assert.Equal(t, "mySecretBearerToken123", credentials, "credentials should preserve original value")
+                               assert.Equal(t, "Bearer", scrapeConfig.HTTPClientConfig.Authorization.Type)
+                       },
+               },
+       }
+
+       for _, tt := range tests {
+               t.Run(tt.name, func(t *testing.T) {
+                       // First, load the config using promconfig.Load to get a proper prometheus config
+                       initialCfg, err := promconfig.Load(tt.configYAML, slog.Default())
+                       require.NoError(t, err)
+
+                       // Now test reloadPromConfig by passing the loaded config through it
+                       // This tests the marshaling/unmarshaling round-trip that could corrupt secrets
+                       dst := &PromConfig{}
+                       err = reloadPromConfig(dst, initialCfg)
+                       require.NoError(t, err)
+
+                       tt.checkFn(t, dst)
+               })
+       }
+}