fix(prometheus.scrape): Update arguments and targets even if scrape_native_histograms and extra_metrics are updated#5787
Merged
Merged
Conversation
x1unix
approved these changes
Mar 13, 2026
kalleep
reviewed
Mar 16, 2026
ptodev
force-pushed
the
ptodev/scrape-args
branch
from
1df5e34 to
a017750
Compare
3 tasks
kgeckhart
pushed a commit
that referenced
this pull request
Mar 16, 2026
…native_histograms` and `extra_metrics` are updated [backport] (#5792) ## Backport of #5787 This PR backports #5787 to release/v1.14. ### Original PR Author @ptodev ### Description ### Pull Request Details Changes to `scrape_native_histograms` and `extra_metrics` can only take effect after a restart of the component. In a future PR we could get the scrape pool to restart so that changes to those components can take effect even if the config is reloaded. The main issue with the current code is that targets won't be updated. So even though the component attempts to run the old ("good") config, that old config contains stale targets. Therefore the scraper isn't picking up any new targets. It may also try to scrape old targets which no longer exist. This PR also fixes a problem where an invalid config in general can cause stale targets. It's because Update() returns before targets can be updated. <details> <summary>The tests fail when I comment out the change in scrape.go</summary> ``` └─▪ go test . ts=2026-03-13T20:25:39.826105Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:39.826579Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:39.829549Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:39.829657Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:39.83196Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:39.832063Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:39.833304Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:39.833419Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:39.838636Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:39.838759Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.92322Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:44.92322Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:44.92337Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.923383Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.930673Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:44.934526Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:44.934687Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.936428Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:44.936503Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.942781Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:44.94289Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:25:44.944278Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:49.848985Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:49.947027Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:50.011818Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:50.016656Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:50.016691Z level=debug msg="passed new targets to scrape manager" level=debug test=TestCustomDialer ts=20:25:55.071 msg="accepted new in-memory connection" level=debug test=TestCustomDialer ts=20:25:55.071 msg="dialed to in-memory listener" ts=2026-03-13T20:25:55.073626Z level=info msg="scrape manager stopped" ts=2026-03-13T20:25:55.204507Z level=debug msg="scrape config was updated" ts=2026-03-13T20:25:55.205275Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:26:00.306499Z level=info msg="scrape manager stopped" ts=2026-03-13T20:26:00.422209Z level=debug msg="scrape config was updated" ts=2026-03-13T20:26:00.422455Z level=debug msg="passed new targets to scrape manager" ts=2026-03-13T20:26:05.523946Z level=info msg="scrape manager stopped" ts=2026-03-13T20:26:05.528546Z level=debug msg="scrape config was updated" component_path=prometheus.scrape component_id=targets_are_updated ts=2026-03-13T20:26:05.528653Z level=debug msg="passed new targets to scrape manager" component_path=prometheus.scrape component_id=targets_are_updated ts=2026-03-13T20:26:10.601378Z level=debug msg="scrape config was updated" component_path=prometheus.scrape component_id=targets_are_updated ts=2026-03-13T20:26:10.601752Z level=debug msg="passed new targets to scrape manager" component_path=prometheus.scrape component_id=targets_are_updated ts=2026-03-13T20:26:15.552585Z level=info msg="scrape manager stopped" component_path=prometheus.scrape component_id=targets_are_updated ts=2026-03-13T20:26:15.57111Z level=debug msg="scrape config was updated" component_path=prometheus.scrape component_id=scrape_native_histograms_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:15.571333Z level=debug msg="passed new targets to scrape manager" component_path=prometheus.scrape component_id=scrape_native_histograms_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:20.623134Z level=info msg="scrape manager stopped" component_path=prometheus.scrape component_id=scrape_native_histograms_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:20.629026Z level=debug msg="scrape config was updated" component_path=prometheus.scrape component_id=extra_metrics_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:20.629441Z level=debug msg="passed new targets to scrape manager" component_path=prometheus.scrape component_id=extra_metrics_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:25.680582Z level=info msg="scrape manager stopped" component_path=prometheus.scrape component_id=extra_metrics_change_is_ignored_but_component_continues_with_updated_targets ts=2026-03-13T20:26:25.705794Z level=debug msg="scrape config was updated" component_path=prometheus.scrape component_id=targets_are_updated_even_when_ApplyConfig_fails ts=2026-03-13T20:26:25.706469Z level=debug msg="passed new targets to scrape manager" component_path=prometheus.scrape component_id=targets_are_updated_even_when_ApplyConfig_fails ts=2026-03-13T20:26:30.827503Z level=error msg="error reloading scrape pool" component_path=prometheus.scrape component_id=targets_are_updated_even_when_ApplyConfig_fails err="error creating HTTP client: unable to read CA cert: unable to read file /nonexistent/ca.pem: open /nonexistent/ca.pem: no such file or directory" scrape_pool=test_job ts=2026-03-13T20:26:40.829039Z level=info msg="scrape manager stopped" component_path=prometheus.scrape component_id=targets_are_updated_even_when_ApplyConfig_fails --- FAIL: TestRuntimeUpdate (35.31s) --- FAIL: TestRuntimeUpdate/scrape_native_histograms_change_is_ignored_but_component_continues_with_updated_targets (5.07s) scrape_test.go:1175: Error Trace: /Users/paulintodev/Documents/GitHub/alloy-1/internal/component/prometheus/scrape/scrape_test.go:1175 Error: Received unexpected error: scrape_native_histograms cannot be updated at runtime Test: TestRuntimeUpdate/scrape_native_histograms_change_is_ignored_but_component_continues_with_updated_targets --- FAIL: TestRuntimeUpdate/extra_metrics_change_is_ignored_but_component_continues_with_updated_targets (5.06s) scrape_test.go:1175: Error Trace: /Users/paulintodev/Documents/GitHub/alloy-1/internal/component/prometheus/scrape/scrape_test.go:1175 Error: Received unexpected error: extra_metrics cannot be updated at runtime Test: TestRuntimeUpdate/extra_metrics_change_is_ignored_but_component_continues_with_updated_targets --- FAIL: TestRuntimeUpdate/targets_are_updated_even_when_ApplyConfig_fails (15.15s) scrape_test.go:1178: Error Trace: /Users/paulintodev/Documents/GitHub/alloy-1/internal/component/prometheus/scrape/scrape_test.go:1146 /Users/paulintodev/Documents/GitHub/alloy-1/internal/component/prometheus/scrape/scrape_test.go:1179 /Users/paulintodev/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.7.darwin-arm64/src/runtime/asm_arm64.s:1268 Error: Should be true Messages: server_b_up should appear even when ApplyConfig fails scrape_test.go:1178: Error Trace: /Users/paulintodev/Documents/GitHub/alloy-1/internal/component/prometheus/scrape/scrape_test.go:1178 Error: Condition never satisfied Test: TestRuntimeUpdate/targets_are_updated_even_when_ApplyConfig_fails Messages: post-update check timed out FAIL FAIL github.com/grafana/alloy/internal/component/prometheus/scrape 61.987s FAIL ``` </details> ### PR Checklist <!-- Remove items that do not apply. For completed items, change [ ] to [x]. --> - [ ] Documentation added - [x] Tests updated - [ ] Config converters updated --- *This backport was created automatically.* Co-authored-by: Paulin Todev <paulin.todev@gmail.com>
blewis12
pushed a commit
that referenced
this pull request
Mar 30, 2026
🤖 I have created a release *beep* *boop* --- ## [1.15.0](v1.14.0...v1.15.0) (2026-03-26) ### ⚠ BREAKING CHANGES * **otelcol:** Upgrade to OTel Collector v0.147.0 ([#5784](#5784)) * Renamed undocumented metrics that was previously prefixed with <component_id>_<metric_name> to loki_source_awsfirehose_<metric_name> ### Features 🌟 * **alloy-mixin:** Add filters, groupBy, and multi-select dashboard variables ([#5611](#5611)) ([3ef714e](3ef714e)) * **beyla.ebpf:** Add support for Prometheus native histograms ([#5812](#5812)) ([7d806fb](7d806fb)) * **beyla.ebpf:** Bump Beyla to v3.6 ([#5833](#5833)) ([cd878d5](cd878d5)) * **converters:** Support converting Promtail limits_config ([#5777](#5777)) ([9491385](9491385)) * **database_observability.mysql:** Add filtering of query samples and wait events by minimum duration ([#5678](#5678)) ([5a4d03b](5a4d03b)) * **database_observability.mysql:** Embed prometheus exporter within db-o11y component ([#5711](#5711)) ([88bffb0](88bffb0)) * **database_observability.postgres:** Add configurable limit to `pg_stat_statements` query ([#5639](#5639)) ([0de0a3f](0de0a3f)) * **database_observability.postgres:** Embed prometheus exporter within db-o11y component ([#5714](#5714)) ([9dc2e83](9dc2e83)) * **database_observability:** Add scaffolding for db-o11y integration tests ([#5575](#5575)) ([ca637d8](ca637d8)) * **database_observability:** Promote components to stable ([#5736](#5736)) ([21a9af6](21a9af6)) * Expose Functionality to Handle syslogs with Empty MSG Field ([#5687](#5687)) ([178b1e6](178b1e6)) * **helm:** Allow setting `revisionHistoryLimit` in the helm chart ([#5847](#5847)) ([9713ad4](9713ad4)) * **loki.process:** Support structured metadata as source type of stage.labels for loki.process ([#5055](#5055)) ([eda3152](eda3152)) * **loki.secretfilter:** Add sampling for secretfilter entries ([#5663](#5663)) ([9997802](9997802)) * **loki.source.gcplog:** Add alloy config for MaxOutstandingBytes and MaxOutstandingMessages ([#5760](#5760)) ([c2b9f0b](c2b9f0b)) * **loki.write:** Add loki pipeline latency metric ([#5702](#5702)) ([cc744a1](cc744a1)) * **mixin:** Update loki dashboard ([#5848](#5848)) ([b616d58](b616d58)) * **otelcol.receiver.datadog:** Expose intake proxy and trace_id_cache_size settings ([#5776](#5776)) ([0384ad4](0384ad4)) * **otelcol:** Upgrade to OTel Collector v0.147.0 ([#5784](#5784)) ([a9b5396](a9b5396)) * **prometheus.exporter.cloudwatch:** Use aws-sdk-go-v2 by default ([#5768](#5768)) ([a2f3489](a2f3489)) * **pyroscope.ebpf:** Add comm, pid labels and kernel frame options ([#5769](#5769)) ([4fa7068](4fa7068)) * **pyroscope.ebpf:** Expose OTel eBPF profiler internal metrics to Prometheus ([#5774](#5774)) ([e713392](e713392)) * **pyroscope:** Copy prometheus common/config HTTP client into promhttp2 package ([#5810](#5810)) ([0b31aaa](0b31aaa)) ### Bug Fixes 🐛 * **beyla:** Inject Beyla version into binary via ldflags ([#5735](#5735)) ([71c03ec](71c03ec)) * Correctly handle the deprecated topic field in otelcol.receiver.kafka configuration ([#5726](#5726)) ([538ac75](538ac75)) * **database_observability.mysql:** Ensure result sets are properly closed ([#5893](#5893)) ([f28f91c](f28f91c)) * **database_observability:** Ensure all collectors are properly stopped ([#5796](#5796)) ([6bfa2a7](6bfa2a7)) * **database_observability:** Ensure that `connection_info` metric is only emitted for a given DB instance when it is available ([#5707](#5707)) ([bf0c3dc](bf0c3dc)) * **database_observability:** Solve test flakiness in MySQL and Postgres sample collectors ([#5130](#5130)) ([a7590d1](a7590d1)) * **deps:** Update module github.com/buger/jsonparser to v1.1.2 [SECURITY] ([#5834](#5834)) ([b2fee8a](b2fee8a)) * **deps:** Update module github.com/buger/jsonparser to v1.1.2 [SECURITY] ([#5870](#5870)) ([698b4e7](698b4e7)) * **deps:** Update module google.golang.org/grpc to v1.79.3 [SECURITY] ([#5825](#5825)) ([5cfbcc4](5cfbcc4)) * **deps:** Update module google.golang.org/grpc to v1.79.3 [SECURITY] ([#5871](#5871)) ([259152d](259152d)) * **deps:** Update npm dependencies ([#5876](#5876)) ([f0f6a11](f0f6a11)) * **deps:** Update npm deps across repo to address CVE-2026-26996 and CVE-2026-22029 ([#5872](#5872)) ([df518dd](df518dd)) * **go:** Update build image to go v1.25.8 ([#5832](#5832)) ([f9b3043](f9b3043)) * **go:** Update go to 1.25.8 ([#5844](#5844)) ([534e7db](534e7db)) * Helm: alloy.extraPorts not working with service.type=NodePort [COPY] ([#5892](#5892)) ([162c6f7](162c6f7)) * **loki.enrich:** Use shared loki functions and fix locking ([#5821](#5821)) ([f916c72](f916c72)) * **loki.process:** Multiline no longer pass empty entry if start was flushed ([#5746](#5746)) ([7bdedf1](7bdedf1)) * **loki.process:** Protect against json that does not look like docker json format ([#5761](#5761)) ([0af6eaa](0af6eaa)) * **loki.secretfilter:** Fix bug where entries were being shadow dropped ([#5786](#5786)) ([90243f9](90243f9)) * **loki.source.file:** Fix position tracking when component stops ([#5800](#5800)) ([9762946](9762946)) * **loki.source.file:** Keep positions for compressed files when reading is finished ([#5723](#5723)) ([fb41d0a](fb41d0a)) * **loki.source.gcplog:** Update to pubsub v2 and fix shutdown semantics ([#5713](#5713)) ([e9d9b69](e9d9b69)) * **loki.source.heroku:** Fix shutdown semantics and consume logs in batches ([#5804](#5804)) ([deda452](deda452)) * **loki.write:** Remove noisy log ([#5837](#5837)) ([8e28f35](8e28f35)) * **loki:** Make drain forward entries with fallback timeout ([#5830](#5830)) ([cfbca90](cfbca90)) * **prometheus.scrape:** Update arguments and targets even if `scrape_native_histograms` and `extra_metrics` are updated ([#5787](#5787)) ([dc4cb0a](dc4cb0a)) * **pyroscope.ebpf:** Update opentelemetry-ebpf-profiler ([#5904](#5904)) ([dfaec47](dfaec47)) * Stop components in a deterministic order ([#5613](#5613)) ([00cd371](00cd371)) ### Chores * Use shared source structures for aws firehose ([#5739](#5739)) ([aef19dc](aef19dc)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: grafana-alloybot[bot] <167359181+grafana-alloybot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Details
Changes to
scrape_native_histogramsandextra_metricscan only take effect after a restart of the component. In a future PR we could get the scrape pool to restart so that changes to those components can take effect even if the config is reloaded.The main issue with the current code is that targets won't be updated. So even though the component attempts to run the old ("good") config, that old config contains stale targets. Therefore the scraper isn't picking up any new targets. It may also try to scrape old targets which no longer exist.
This PR also fixes a problem where an invalid config in general can cause stale targets. It's because Update() returns before targets can be updated.
The tests fail when I comment out the change in scrape.go
PR Checklist