Skip to content

[chore] RFC for scraper controller extension#14469

Merged
mx-psi merged 5 commits into
open-telemetry:mainfrom
axw:rfc-scraper-controller-extensions
Apr 10, 2026
Merged

[chore] RFC for scraper controller extension#14469
mx-psi merged 5 commits into
open-telemetry:mainfrom
axw:rfc-scraper-controller-extensions

Conversation

@axw
Copy link
Copy Markdown
Contributor

@axw axw commented Jan 23, 2026

Description

This RFC proposes introducing an extension interface for scraper controllers to enable event-driven scraping patterns.
The current time-based scraper controller will be maintained with an option to disable the timer, and new configuration will allow specifying one or more scraper controller extensions that can trigger scrapes based on external events.

This is a proposed solution to #12449

RFC announced

@axw axw requested review from a team, bogdandrutu, codeboten, dmitryax and mx-psi as code owners January 23, 2026 05:12
@axw axw changed the title RFC for scraper controller extension [chore] RFC for scraper controller extension Jan 23, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jan 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.23%. Comparing base (91ba721) to head (ffca8c0).
⚠️ Report is 120 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #14469      +/-   ##
==========================================
- Coverage   91.83%   91.23%   -0.60%     
==========================================
  Files         680      698      +18     
  Lines       43043    44708    +1665     
==========================================
+ Hits        39528    40790    +1262     
- Misses       2445     2774     +329     
- Partials     1070     1144      +74     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread docs/rfcs/scraper-controller-extension.md

## Open Questions

1. Should there be a way to pass dynamic configuration from the extension to the scraper, e.g. `targets` for httpcheck?
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably better handled with something like receivercreator (e.g. scrapercreator), which dynamically creates one-shot scrapers in response to some event(s). The event payload could then be referenced with variables in the scraper's configuration.

@github-actions
Copy link
Copy Markdown
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions Bot added the Stale label Feb 19, 2026
@axw axw removed the Stale label Feb 19, 2026
@axw
Copy link
Copy Markdown
Contributor Author

axw commented Feb 19, 2026

@open-telemetry/collector-approvers can I please get a review on this? 🙏

@atoulme
Copy link
Copy Markdown
Contributor

atoulme commented Mar 4, 2026

Could we mesh this proposal and its use cases with telemetry policies? open-telemetry/opentelemetry-specification#4738

@axw
Copy link
Copy Markdown
Contributor Author

axw commented Mar 4, 2026

@atoulme did you have something in particular in mind?

I read through that proposal, and I think I get it (composeable, declarative policies to apply on telemetry data) -- but I don't see very clearly how my proposal would overlap with it.

From the description of that PR:

Policies are independent rules to enforce on generated telemetry

From my read, the generation of telemetry is outside the scope of policies; they're policies to apply on telemetry. I guess where the line becomes blurry is with things like metrics interval, which effectively controls generation (like scraper controllers).

I suppose you could extend that to generating telemetry in exactly one instance of a fleet of collectors, on an interval, and how that happens is an implementation detail.

This RFC is really about making those implementation detail possible. I think we can extend telemetry policies to take advantage of them later.

@mx-psi
Copy link
Copy Markdown
Member

mx-psi commented Mar 4, 2026

Can you add to the PR description how you have fulfilled the requirements regarding announcing this RFC/RFC Amendment?

@axw
Copy link
Copy Markdown
Contributor Author

axw commented Mar 4, 2026

@mx-psi done

@atoulme
Copy link
Copy Markdown
Contributor

atoulme commented Mar 16, 2026

I suppose you could extend that to generating telemetry in exactly one instance of a fleet of collectors, on an interval, and how that happens is an implementation detail.

This RFC is really about making those implementation detail possible. I think we can extend telemetry policies to take advantage of them later.

Yes, I had this link in mind. Agree the relationship is tenuous, just want to make sure we try to converge when at all sensible.

Copy link
Copy Markdown
Contributor

@codeboten codeboten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This proposal looks good to me. I think the details of the one-shot execution can sorted out later, but i like the flexibility of the extension proposed here.

@mx-psi mx-psi added the rfc:final-comment-period This RFC is in the final comment period phase label Apr 6, 2026
@mx-psi
Copy link
Copy Markdown
Member

mx-psi commented Apr 6, 2026

I have marked this as "final comment period". I will merge this on Friday EU morning unless there are blocking comments by then.

Comment thread docs/rfcs/scraper-controller-extension.md
Comment thread docs/rfcs/scraper-controller-extension.md
@axw axw requested a review from dmitryax April 7, 2026 03:20
Copy link
Copy Markdown
Member

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mx-psi mx-psi added this pull request to the merge queue Apr 10, 2026
Merged via the queue into open-telemetry:main with commit 41c3a76 Apr 10, 2026
66 checks passed
@axw axw deleted the rfc-scraper-controller-extensions branch April 14, 2026 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rfc:final-comment-period This RFC is in the final comment period phase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants