Capture Kafka key without using it as ID

[kmcclellan]

I want to use the topic/partition/offset as the ES document ID while still capturing the Kafka message key for audit/search purposes. Unless I'm mistaken, there doesn't appear to be a way to do this with this connector.

[kmcclellan]

Ideally, all Kafka message metadata should be captured in a way that is searchable: topic, partition, offset, timestamp, headers

[kmcclellan]

A year later and there is still no way to capture/search the most basic message metadata within ES. The only reason I wanted Kafka messages copied to ES was to search it. IMO this library is pretty much worthless.

Kafka message keys are used for partitioning and are not intended to be unique. The fact that the implementors of this connector thought they should be used as an Elasticsearch document ID shows a pretty poor understanding of both domains.