configurable kafka partitioner #2284
Conversation
urso
commented
Aug 16, 2016
urso
commented
- Store kafka message wrapper in outputs.Data.Values for re-use on retry.
- Add support for configuring the partitioner strategy
- configurable support ignoring unavailable partitions
- partitioning is mostly stable in error case (try to schedule events to same partition if returned to libbeat on error)
urso
added
enhancement
in progress
|
Requires #2295 to fix test build |
urso
force-pushed
the
enh/kafka-partitioner
branch
2 times, most recently
from
bb9c463
to
8fa7888
Compare
urso
changed the title
urso
added
review
and removed
in progress
|
Will update docs in another PR, as a number of other settings have been added/changed for kafka output. |
urso
force-pushed
the
enh/kafka-partitioner
branch
from
e8b4ad2
to
8523bf6
Compare
| var testOptions = outputs.Options{} | ||
| func TestKafkaPublish(t *testing.T) { | ||
| single := modetest.SingleEvent | ||
| // multi := modetest.MultiEvent |
There was a problem hiding this comment.
What is the idea with these 2 commented out lines?
There was a problem hiding this comment.
they can be removed as not being used. events of type []modetest.EventInfo hold a number of events + setting to bulk or not bulk publish those events. It's used to drive sending events through output plugin.
In test randMulti is used.
|
LGTM. What about the updates to the config file? Will this also follow in a separate PR? |
|
config files can be done in this PR. |
urso
force-pushed
the
enh/kafka-partitioner
branch
from
8523bf6
to
86068dc
Compare
|
@ruflin added configs |
| @@ -35,6 +35,10 @@ packetbeat.protocols.amqp: | |||
| # the AMQP protocol by commenting out the list of ports. | |||
| ports: [5672] | |||
|
|
|||
| packetbeat.protocols.cassandra: | |||
| #Cassandra port for traffic monitoring. | |||
There was a problem hiding this comment.
It's not part of this PR, but needs a space in front of Cassandra.
There was a problem hiding this comment.
oh, this should be fixed by another PR. will have to rebase.
|
LGTM, left a minor comment/question. |
|
LGTM |
urso
force-pushed
the
enh/kafka-partitioner
branch
from
86068dc
to
0a3a90c
Compare
- add config settings to configure partition strategy - add config settings to configure partitioner selecting only reachable partitions - Store kafka message in data.values so original message including partition selection is still available on retry - make kafka message key configurable - update changelog
- replace kafka integration tests with table driven integration tests - add integrations tests using batch publishing - add integration tests using different partitioner configurations
urso
force-pushed
the
enh/kafka-partitioner
branch
from
0a3a90c
to
3597027
Compare
|
renamed |
