[Prototype] Pipeline Components - Extension Support

[gouslu]

Pipeline Components - Extension Support Proposal

Extensions are pipeline components alongside receivers, processors, and exporters -- but they do not participate in data-path connections (no PData channels, control messages only).

Extensions can be implemented as local (!Send futures, single-threaded LocalSet) or shared (Send futures, multi-threaded), giving extension authors flexibility with some caveats based on the Send only extension traits in this design.

Extensions can optionally implement extension traits (e.g., BearerTokenProvider) defined in the engine crate to expose capabilities to other components, or opt out and serve purely as background tasks.

Extension traits require Send + Clone + 'static. If an extension publishes any trait, the concrete type implementing that trait must satisfy these bounds -- meaning shared state must be managed via Arc (or similar) by the extension author.

The extension's lifecycle method (start) takes Box<Self> by move -- the extension instance is consumed, not cloned. What is cloned is the extension struct itself during extension_traits(): the macro clones self into each TraitRegistration, and those clones are inserted into the ExtensionRegistry. After trait collection, the original extension is consumed by start(). Each consumer (receiver/exporter) receives a clone of the registry, and calling registry.get::<dyn Trait>(name) returns a fresh clone of the stored trait object.

Extensions are configured as a sibling to nodes in the pipeline YAML (dedicated extensions: section), not inside nodes.

Extension trait types are sealed -- new trait types can only be added inside the engine crate; external crates can implement existing traits but cannot define new ones.

Extensions are started first, before exporters, processors, and receivers, so their capabilities are available at component initialization.

Extension registry is passed via start methods of exporters and receivers. Can be added to "process" as well if needed. Considered putting it into the EffectHandler, but it didn't feel like the right place for that.

Created a separate control message channel for Extensions based on the changes that were done by @utpilla 's PR #2141.

Alternatives Considered

• Arc-based registry -- true single instance with Arc clones, but enforces a Sync boundary on all extensions.
• Rc-based registry -- only works for local components. It might be acceptable to say that shared components don't have extension support, but this limits flexibility.
• All other alternatives seem to require at least one deep clone or multiple instances and never achieve a true single instance. For this reason I opted for a design where deep clone is accepted and shared extension state is the responsibility of the extension author via an explicit Clone requirement.

Why Local and Shared Variants

Local and shared extension variants exist for two reasons:

1. Consistency -- receivers and exporters already have local/shared variants; extensions follow the same pattern.
2. Event-loop optimizations -- despite extension traits requiring Send (because the registry is Send), the start() async body of a local extension can use !Send types (Rc, RefCell, LocalSet spawning, etc.). This means a local extension that publishes Send traits can still use non-thread-safe optimizations inside its own event loop, even though its struct fields must be Send + Clone for trait registration.

[codecov]

Codecov Report

❌ Patch coverage is 59.79381% with 585 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.22%. Comparing base (fc73f05) to head (4e8aa37).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2141      +/-   ##
==========================================
- Coverage   87.44%   87.22%   -0.23%     
==========================================
  Files         558      566       +8     
  Lines      185764   186872    +1108     
==========================================
+ Hits       162447   163003     +556     
- Misses      22791    23343     +552     
  Partials      526      526              

Components	Coverage Δ
otap-dataflow	89.32% <59.79%> (-0.33%)	⬇️
query_abstraction	80.61% <ø> (ø)
query_engine	90.30% <ø> (ø)
syslog_cef_receivers	∅ <ø> (∅)
otel-arrow-go	52.44% <ø> (ø)
quiver	91.83% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jmacd]

I reviewed extension-system.md. Looks good!

I think we can add more detail and flexibility in the future. We should focus on the configuration model, especially threads-vs-cores sharing questions, and the minimum required for our Azure auth core to serve as an extension for both azmon and parquet+object_store exporters.

As a nice-to-have, I think we should consider adding a core component that does some extremely-basic form of the extension we're adding, like a basicauth extension (receivers), like a headersetter extension (exporters).

[jmacd]

nit: Eventually, we will find ways to use processor extensions. In the Collector, we find cross cutting concerns like memory limiters and persistent key/value stores.

[jmacd]

Note: the admin component is in a similar position, needing to be decoupled from the PData type of the engine, yet interoperating with the engine.