HTTP webhook integration

[tomtom215]

Is there an existing proposal for this?

• I have searched the existing proposals

Is your feature request related to a problem?

The ability to send JSON via HTTP posts to a wehbook endpoint which then takes that JSON and adds it to the message queue.

Describe the solution you'd like

I know MQ's + HTTP is kind of an antipatern and usually the advice is to write your own middleware. So far I have not seen a message queue that offers this feature, but it seems like a common enough problem that people should not have to write their own customer middleware every time.

An example of this is can be found trying to search for ways to integrate HTTP messages with RabbitMQ and finding this project https://github.com/dougwatson/http2amqp

The ideal scenario would be if you could create a webhook endpoint on BlazingMQ which would then be able to receive JSON messages from HTTP (push/pub) sources which would convert the JSON into a message and place it into a queue.

Alternatives you considered

Custom middleware

[quarter-note]

Hi @tomtom215, thanks for your interest in BlazingMQ! Internally, we have an adapter which provides HTTP interface which a producer can use to post messages to BlazingMQ. This adapter is closely tied to enterprise layer and cannot be open sourced.

One solution here would be to implement a similar adapter for OSS. A simple application which listens over HTTP to the application on one side, and talks to BlazingMQ backend over BlazingMQ protocol on the other side (by using BlazingMQ SDK). This is not on our roadmap, but will keep in mind.

Out of curiosity, what interested you in BlazingMQ, and why are you looking for HTTP hook?

Thanks

[tomtom215]

Hi @quarter-note thanks for your response!

This seems to validate my initial assumption that if you want to have an HTTP interface for a MQ that you need to roll your own middleware and it is important enough that you would make it a licenses feature of your product.

So far one of the only instances I've found of this being out of the box by a OSS tool has been Apache SeaTunnel

I came across BlazingMQ because I am always keeping an eye of event/streaming platforms and components in the industry and when I saw BlazingMQ had been battle tested at Bloomberg and offered some architectural advantages over RabbitMQ, I decided to dig deeper.

Our use case for an HTTP webhook is that we interact with some systems and applications that have interfaces that push DB updates via JSON messages. It's close to CDC, but since we don't have influence over the source systems to setup something like Debezium, we considered a webhook -> MQ -> DB so that we didn't have to manage message lifecycle or delivery guarantees with our own custom middleware if it was available from BlazingMQ.

[quarter-note]

@tomtom215 If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

[tomtom215]

@tomtom215 If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

Honestly, we would. As mentioned it seems to be a big feature gap in the available message queue space and the main reason I've been looking around is to avoid either rolling our own middleware or by adding more complexity to our stack by introducing something as heavy as SeaTunnel or Kafka connect just to handle this use case.

If marketed correctly, I think this feature in an OSS offering will be a big advantage for BlazingMQ over other message queues and could help to simplify some architectural decisions.

Ps. Thanks for hearing me out and the good discussion!

[Montana]

If we provide a HTTP/HTTP2 proxy which will enable you to talk to BlazingMQ, would you consider using BlazingMQ in your setup?

It definitely would make it significantly more convenient to integrate BlazingMQ into setups requiring HTTP-based message ingestion.