Search messages with specific key/value in topic

[weeco]

Usecase:

As someone who works with Kafka I'd like to be able to find messages which have a specific key (which might have a semantic meaning - like a unique identifier for a customer). Equally I might want to filter/search for messages which have a specific value (either complete content or a subset of a JSON message like just some properties).

Kowl can do so by streaming all messages in a topic and evaluate each message against a set of user defined filters.

Examples:

• Search all messages in topic orders which have key xzy-3711
• Search all messages in topic orders where value customer.UUID = "abc123" && order.value > 6000
• Search all messages in topic orders where key and value filter from above bullets need to apply

Challenges:

• Value & Key can be plaintext, JSON, binary, avro, XML, ...
• In Kowl Business you want to apply additional limits what a user can do (e. g. max 1GB streaming / day is allowed) otherwise excessive usage can cause further troubles (traffic costs, resource usage of Kowl/Kafka, ...)
• Filters which do not just match a string but requiring the use of the correct type such as greater than equal, date comparisons, ..

Implementation hints:

Finding kafka messages by key: There are multiple partitioning strategies, but the default partitioner uses a 32-bit murmur2 to compute choose what PartitionID the message should go to. This could potentially be used to improve performance when searching a whole topic for a message by key.

[weeco]

UI could look like this:

Add an Add filter button where users can add filters (for key and value). If at least one filter is added Start offset should always be locked at Oldest offset with the hint that this can not be changed if a search filter is applied.

Clicking on New Filter should open a popover with a simple query builder where the kafka message key or value property can be chosen (supported by intellisense if possible), an operator selector (==, >=, ...) and the desired value to match with.

Additionally a execution plan should be added there, so that a user knows how many messages Kowl is going to process during the search. It should be clear that this can be a resource intensive operation and should be used with care.

Example:

Propertyname: value.battleTime
Operator: >
Value: Date("2020-06-01")

[rikimaru0345]

js parameters

key, value: string|object|null
offset, partitionID: number
timestamp: Date
keyLength, valueLength: number // size of each in bytes

js return

returns: false (skip message), true (return message)

2 filter editors

1. simple editor, just like in kibana
   • a list of simple expressions
   • you can select some property + operator + value
   • later: should probably either support * in the "property path", or just take the first property with a matching name
2. advanced editor: where you literally type the javascript code
   • also shown as a tag in the ui; user can give a name to the filter; if empty the first line is shown