Kafka Exactly Once implications

[ralph-tice]

Hi!

Awesome project! Can some information be added about how Kafka 0.11's exactly once functionality will or won't be incorporated into this work? Are they complementary or orthogonal?

[adamw]

I've been meaning to do that but didn't yet manage to :)

There are two ways in which you could leverage Kafka's new features in the kmq context:

• idempotent producers: that's a great way to reduce duplicate processing by eliminating duplicate message sends (so you won't get re-processing because of producer failures). No changes to kmq are necessary here, you just send the messages a bit differently
• transactions: maybe it would be possible to combine writing the consumed offsets and writing the markers into an atomic unit, decreasing the number of message re-processed on the consumer side

However, keep in mind that you'll still get at-least-once processing when interfacing with external clients - there's no way to help with that (*), and re-processing failed messages is the whole point of kmq

(*) unless your client is transactional and can store current message offset, but then you would just use plain Kafka consumers