[FEATURE REQ] Event Hubs clients exposing pipelines and policies (ClientOptions from Azure.Core)

[bartoszropski]

Library or service name.
Azure.Messaging.EventHub

Is your feature request related to a problem? Please describe.
EventHub is depending now on the Azure.Core package. One of the key concepts developers seems to be thrilled about, in the new Azure SDK guidelines, are pipelines and custom policies, that give you client side blocks similar to MessageHandler or OwinMiddleware concepts.

The pipelines provide clear and efficient way of enhancing the capabilities of the client, such as:

• logging
• telemetry
• throttling
• encryption
• retries
  and many others.

EventHub client option classes, however, are not following this concept.
I think it would be beneficial, if we had an option to inject custom logic this way, without a need to create wrappers/adapters on top of SDK interfaces and classes.

[jsquire]

For context, Bartosz and I have spoken offline discussed that Event Hubs is based on the AMQP protocol rather than HTTP, so many of the concepts of the Azure.Core pipeline do not have a direct equivalent or that they work in a different manner thus making full compatibility and feature parity a non-goal.

A portion of that conversation was spent discussing the desire for injecting a series of composable steps into the process of translating the EventData objects into the corresponding AMQP message and the inverse, allowing injection of concerns such as setting/reading message properties and manipulating the body of an event rather than building logic on top of the library.

@bartoszropski: If you would be so kind as to detail some of the specific scenarios that you're interested in, it would be helpful context for internal discussion as well as gauging wider community interest in proposed feature.

[jsquire]

//cc: @ramya-rao-a, @srinathnarayanan, @chradek, @richardpark-msft, @conniey, @KieranBrantnerMagee, @YijunXieMS

[bartoszropski]

Hi Jesse,
Let me give you an example - let's say current producer exposes API
Task SendAsync(EventDataBatch eventDataBatch)
let's say you allow to inject policies with interface having the same method.
I could then inject policies that would for example take the batch and encrypt the body and call next policy in the chain, where last one is provided by you, doing actual sending.
I could inject a policy for my integration tests that would mock the results (exceptions etc), while the test would still use the client in the same way as production code (no wrappers).

Now let's go step farther - let's imagine you would define common format of event, or maybe event and message that all the Azure.Messaging packages would use (maybe AMQP based?) where I could define this policy ones and reuse for my EventHub client, ServiceBus client, EventGrid and I'd have only one place to manage my telemetry for all my messaging based resources, same way as Azure.Core HttpPipeline allows me now to do it for all HTTP based resources.

Does it make sense?

[samuelkoppes]

We are investigating this ask and will report back with an update by 11/4/2020.

[jsquire]

@samuelkoppes: This is an ask on the Azure SDK team for the track two library. This isn't something that the Event Hubs service team needs to investigate.

[jsquire]

This aligns with the Plugin concept used by the legacy Service Bus library; there has been ongoing discussion around whether or not plugins should be adopted in the Azure.Messaging.* line of client libraries for message enrichment for the last few months. After prototyping, UX testing, and customer discussions around Service Bus, the discussion is still open but it looks unlikely that plugins will land.

As of now, there are no plans to bring plugins to Event Hubs so I'm going to close this out. We'll continue to monitor the outcome in Service Bus and, if plugins are incorporated and, based on customer feedback and experience, will revisit whether to consider plugins for Event Hubs.