[receiver/prometheus] retry connections to targetallocator by CharlieTLe · Pull Request #40982 · open-telemetry/opentelemetry-collector-contrib

CharlieTLe · 2025-06-29T23:42:47Z

Description

The collector will crash if the target allocator that it is receiving its scrape configuration from is still starting up. This change adds retry logic to give the target allocator more time to become ready since the collector and target allocator are usually deployed at the same time by the otel operator.

Signed-off-by: Charlie Le <charlie_le@apple.com>

dashpole · 2025-06-30T13:12:27Z

Should the target allocator just add a readiness probe to ensure this doesn't happen?

It looks like it can only cause collectors to fail to start, not to crash if the collector is already started:

opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator/manager.go

Line 80 in 26733ec

hash, newErr := m.sync(savedHash, httpClient)

CharlieTLe · 2025-06-30T15:04:43Z

Should the target allocator just add a readiness probe to ensure this doesn't happen?

The target allocator could have a readiness probe setup, but the collector will still try to make the connection to the service's ClusterIP even the target allocator is not ready yet and see the same error.

Here's how to reproduce:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  mode: statefulset
  config:
    receivers: 
      prometheus:
        config:
          scrape_configs: []
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [debug]
    exporters:
      debug: {}
  replicas: 1
  targetAllocator:
    enabled: true
    replicas: 0
    prometheusCR:
      enabled: true

Then when the collector starts up, it will crash loop since no target allocator is ready (it was configured to have 0 replicas).

It looks like it can only cause collectors to fail to start, not to crash if the collector is already started:

opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator/manager.go

Line 80 in 26733ec

hash, newErr := m.sync(savedHash, httpClient)

It seems like the collector could crash here at start up if the initial sync fails:

opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator/manager.go

Lines 71 to 74 in 26733ec

    
           savedHash, err := m.sync(uint64(0), httpClient) 
        
           if err != nil { 
        
           	return err 
        
           }

When the collector crashes, its logs looks like this:

2025-06-29T23:25:23.911Z    info    service@v0.128.1-0.20250628171447-c6cd1aeb58b7/service.go:197    Setting up own telemetry...    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}}
2025-06-29T23:25:23.912Z    info    service@v0.128.1-0.20250628171447-c6cd1aeb58b7/service.go:257    Starting otelcontribcol...    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}, "Version": "0.128.0-dev", "NumCPU": 8}
2025-06-29T23:25:23.912Z    info    extensions/extensions.go:41    Starting extensions...    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}}
2025-06-29T23:25:23.912Z    info    prometheusreceiver@v0.128.0/metrics_receiver.go:157    Starting discovery manager    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics"}
2025-06-29T23:25:23.912Z    info    targetallocator/manager.go:70    Starting target allocator discovery    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics"}
2025-06-29T23:26:25.275Z    error    targetallocator/manager.go:107    Failed to retrieve job list    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}, "otelcol.component.id": "prometheus", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "failed to connect to target allocator after 1m0s, last error: Get \"http://simplest-targetallocator:80/scrape_configs\": dial tcp 10.96.206.86:80: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).sync
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.128.0/targetallocator/manager.go:107
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver/targetallocator.(*Manager).Start
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.128.0/targetallocator/manager.go:72
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver.(*pReceiver).Start
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver@v0.128.0/metrics_receiver.go:121
go.opentelemetry.io/collector/service/internal/graph.(*Graph).StartAll
    go.opentelemetry.io/collector/service@v0.128.1-0.20250628171447-c6cd1aeb58b7/internal/graph/graph.go:431
go.opentelemetry.io/collector/service.(*Service).Start
    go.opentelemetry.io/collector/service@v0.128.1-0.20250628171447-c6cd1aeb58b7/service.go:272
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
    go.opentelemetry.io/collector/otelcol@v0.128.1-0.20250628171447-c6cd1aeb58b7/collector.go:242
go.opentelemetry.io/collector/otelcol.(*Collector).Run
    go.opentelemetry.io/collector/otelcol@v0.128.1-0.20250628171447-c6cd1aeb58b7/collector.go:312
go.opentelemetry.io/collector/otelcol.NewCommand.func1
    go.opentelemetry.io/collector/otelcol@v0.128.1-0.20250628171447-c6cd1aeb58b7/command.go:39
github.com/spf13/cobra.(*Command).execute
    github.com/spf13/cobra@v1.9.1/command.go:1015
github.com/spf13/cobra.(*Command).ExecuteC
    github.com/spf13/cobra@v1.9.1/command.go:1148
github.com/spf13/cobra.(*Command).Execute
    github.com/spf13/cobra@v1.9.1/command.go:1071
main.runInteractive
    github.com/open-telemetry/opentelemetry-collector-contrib/cmd/otelcontribcol/main.go:70
main.run
    github.com/open-telemetry/opentelemetry-collector-contrib/cmd/otelcontribcol/main_others.go:10
main.main
    github.com/open-telemetry/opentelemetry-collector-contrib/cmd/otelcontribcol/main.go:63
runtime.main
    runtime/proc.go:272
2025-06-29T23:26:25.276Z    error    graph/graph.go:438    Failed to start component    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}, "error": "failed to connect to target allocator after 1m0s, last error: Get \"http://simplest-targetallocator:80/scrape_configs\": dial tcp 10.96.206.86:80: connect: connection refused", "type": "Receiver", "id": "prometheus"}
2025-06-29T23:26:25.276Z    info    service@v0.128.1-0.20250628171447-c6cd1aeb58b7/service.go:322    Starting shutdown...    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}}
2025-06-29T23:26:25.277Z    info    extensions/extensions.go:69    Stopping extensions...    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}}
2025-06-29T23:26:25.277Z    info    service@v0.128.1-0.20250628171447-c6cd1aeb58b7/service.go:336    Shutdown complete.    {"resource": {"service.instance.id": "89dca754-b50b-4207-98b9-4052ef337b34", "service.name": "otelcontribcol", "service.version": "0.128.0-dev"}}
Error: cannot start pipelines: failed to start "prometheus" receiver: failed to connect to target allocator after 1m0s, last error: Get "http://simplest-targetallocator:80/scrape_configs": dial tcp 10.96.206.86:80: connect: connection refused
2025/06/29 23:26:25 collector server run finished with error: cannot start pipelines: failed to start "prometheus" receiver: failed to connect to target allocator after 1m0s, last error: Get "http://simplest-targetallocator:80/scrape_configs": dial tcp 10.96.206.86:80: connect: connection refused

Signed-off-by: Charlie Le <charlie_le@apple.com>

atoulme · 2025-07-14T16:41:27Z

Needs a make tidy

Signed-off-by: Charlie Le <charlie_le@apple.com>

…metry#40982)  #### Description The collector will crash if the target allocator that it is receiving its scrape configuration from is still starting up. This change adds retry logic to give the target allocator more time to become ready since the collector and target allocator are usually deployed at the same time by the otel operator.  --------- Signed-off-by: Charlie Le <charlie_le@apple.com>

This PR upgrades the otelcollector to the latest version available for the opentelemetry-collector and opentelemetry-operator. It was automatically generated by the GitHub Actions workflow. The summary of the OSS changelog is below: # Prometheusreceiver Changes ## v0.127.0 to v0.131.0 Generated on: 2025-08-04 17:36:24 --- ### v0.131.0 - [**FEATURE**] `prometheusreceiver`: Add retry logic for connection refused errors so the collector doesn't crash at startup. ([#40982](open-telemetry/opentelemetry-collector-contrib#40982)) This change adds retry logic for connection refused errors. The target allocator could be busy starting up the receiver and the first connection attempt may fail. - [**FEATURE**] `receiver/prometheus`: Add support for otel_scope_schema_url label mapping to OpenTelemetry ScopeMetrics schema URL field ([#41488](open-telemetry/opentelemetry-collector-contrib#41488)) - [**FEATURE**] `receiver/prometheusremotewrite`: Add support for Native Histogram Custom Buckets (NHCB). ([#41043](open-telemetry/opentelemetry-collector-contrib#41043)) - [**BUG FIX**] `receiver/prometheus`: Fix otel_scope_name and otel_scope_version labels not being dropped from metric attributes ([#41456](open-telemetry/opentelemetry-collector-contrib#41456)) ### v0.130.0 - [**BUG FIX**] `receiver/prometheusreceiver`: Fixes masking of authentication credentials in Prometheus receiver, when reloading the Prometheus config. ([#40520](open-telemetry/opentelemetry-collector-contrib#40520), [#40916](open-telemetry/opentelemetry-collector-contrib#40916)) - [**BUG FIX**] `receiver/prometheusremotewrite`: Handle metrics with unspecified types without panicking. ([#41005](open-telemetry/opentelemetry-collector-contrib#41005)) ### v0.129.0 - [**FEATURE**] `prometheusreceiver`: Promote the receiver.prometheusreceiver.RemoveLegacyResourceAttributes featuregate to stable ([#40572](open-telemetry/opentelemetry-collector-contrib#40572)) It has been beta since v0.126.0 - [**BUG FIX**] `prometheusreceiver`: Fix invalid metric name validation error in scrape start from target allocator. ([#35459](open-telemetry/opentelemetry-collector-contrib#35459), [#40788](open-telemetry/opentelemetry-collector-contrib#40788)) Prometheus made setting metric_name_validation_scheme, metric_name_escaping_scheme mandatory mandatory, use sane defaults. ## Summary | Category | Count | |----------|-------| | Breaking Changes | 0 | | Features | 4 | | Bug Fixes | 4 | | Other Changes | 0 | | **Total** | **8** | # Target-allocator Changes ## v0.127.0 to v0.131.0 Generated on: 2025-08-04 17:36:38 --- ### 0.131.0 - [**FEATURE**] `manager, target-allocator, opamp-bridge, must-gather`: add -trimpath when building binaries ([#4078](open-telemetry/opentelemetry-operator#4078)) - [**FEATURE**] `collector, targer allocator, opamp`: Require Go 1.24+ to build the collector, target allocator, and opamp. ([#4173](open-telemetry/opentelemetry-operator#4173)) - [**BUG FIX**] `target allocator`: check CRD availability before registering informers ([#3987](open-telemetry/opentelemetry-operator#3987)) - [**BUG FIX**] `target allocator`: Allow collector to use TLS Config from Target Allocator with ScrapeConfig ([#3724](open-telemetry/opentelemetry-operator#3724)) This change allows the target allocator to configure TLS Config for a collector using the ScrapeConfig. ### 0.129.1 - [**BREAKING**] `targetallocator, collector`: Remove stable feature gate PrometheusOperatorIsAvailable ([#4141](open-telemetry/opentelemetry-operator#4141)) - [**FEATURE**] `target allocator`: Adds support for HTML output in the target allocator. ([#3622](open-telemetry/opentelemetry-operator#3622)) - [**BUG FIX**] `target allocator`: ensure stable iteration order of target labels when generating hash ([#4082](open-telemetry/opentelemetry-operator#4082)) - [**BUG FIX**] `target allocator`: Fix OpenShift must-gather for Target Allocator ([#4084](open-telemetry/opentelemetry-operator#4084)) ## Summary | Category | Count | |----------|-------| | Breaking Changes | 1 | | Features | 3 | | Bug Fixes | 4 | | Other Changes | 0 | | **Total** | **8** | --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Grace Wehner <grace.wehner@microsoft.com> Co-authored-by: rashmichandrashekar <rashmy@microsoft.com>

[receiver/prometheus] retry connections to targetallocator

41a710c

Signed-off-by: Charlie Le <charlie_le@apple.com>

CharlieTLe requested review from a team, ArthurSens and dashpole as code owners June 29, 2025 23:42

github-actions Bot assigned atoulme Jun 29, 2025

github-actions Bot added the receiver/prometheus Prometheus receiver label Jun 29, 2025

github-actions Bot requested review from Aneurysm9 and krajorama June 29, 2025 23:43

CharlieTLe added 4 commits June 30, 2025 08:16

lint fixes

e60567f

Signed-off-by: Charlie Le <charlie_le@apple.com>

add changelog

bd37c41

Signed-off-by: Charlie Le <charlie_le@apple.com>

Merge branch 'main' into prom-receiver-retry-connection-refused

07a69b3

Merge branch 'main' into prom-receiver-retry-connection-refused

d4c917d

dashpole requested changes Jun 30, 2025

View reviewed changes

Comment thread receiver/prometheusreceiver/targetallocator/manager.go Outdated

Comment thread receiver/prometheusreceiver/targetallocator/manager.go Outdated

CharlieTLe added 4 commits June 30, 2025 22:49

use github.com/cenkalti/backoff/v5 and syscall.ECONNREFUSED

0dcdd3d

Signed-off-by: Charlie Le <charlie_le@apple.com>

Merge branch 'main' into prom-receiver-retry-connection-refused

a9c7c6e

Update prom-receiver-retry-connection-refused.yaml

e3cdba8

run make gotidy

6064efe

Signed-off-by: Charlie Le <charlie_le@apple.com>

CharlieTLe requested a review from fatsheep9146 as a code owner July 1, 2025 16:56

github-actions Bot added receiver/simpleprometheus receiver/purefb receiver/purefa labels Jul 1, 2025

github-actions Bot requested review from chrroberts-pure and dgoscn July 1, 2025 16:57

CharlieTLe added 3 commits July 1, 2025 14:59

lint

7828162

Signed-off-by: Charlie Le <charlie_le@apple.com>

Merge branch 'main' into prom-receiver-retry-connection-refused

9c00997

Merge branch 'main' into prom-receiver-retry-connection-refused

0601976

CharlieTLe requested a review from dashpole July 2, 2025 16:54

Merge branch 'main' into prom-receiver-retry-connection-refused

a1211c2

dashpole approved these changes Jul 7, 2025

View reviewed changes

CharlieTLe added 2 commits July 7, 2025 12:41

Merge branch 'main' into prom-receiver-retry-connection-refused

a0365c6

Merge branch 'main' into prom-receiver-retry-connection-refused

f36de6f

CharlieTLe added 3 commits July 15, 2025 10:14

make gotidy

545559a

Signed-off-by: Charlie Le <charlie_le@apple.com>

Merge branch 'main' into prom-receiver-retry-connection-refused

dafc326

Merge branch 'main' into prom-receiver-retry-connection-refused

abf27da

ArthurSens added the ready to merge Code review completed; ready to merge by maintainers label Jul 16, 2025

atoulme merged commit b8dda6e into open-telemetry:main Jul 16, 2025
187 checks passed

github-actions Bot added this to the next release milestone Jul 16, 2025

azure-monitor-assistant Bot mentioned this pull request Aug 4, 2025

build(deps): Upgrade otelcollector to v0.131.0 Azure/prometheus-collector#1257

Merged

github-actions Bot mentioned this pull request Apr 27, 2026

Lychee found some broken links odubajDT/dynatrace-otel-collector#54

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[receiver/prometheus] retry connections to targetallocator#40982

[receiver/prometheus] retry connections to targetallocator#40982
atoulme merged 18 commits into
open-telemetry:mainfrom
CharlieTLe:prom-receiver-retry-connection-refused

CharlieTLe commented Jun 29, 2025

Uh oh!

dashpole commented Jun 30, 2025

Uh oh!

CharlieTLe commented Jun 30, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

atoulme commented Jul 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CharlieTLe commented Jun 29, 2025

Description

Uh oh!

dashpole commented Jun 30, 2025

Uh oh!

CharlieTLe commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

atoulme commented Jul 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CharlieTLe commented Jun 30, 2025 •

edited

Loading