-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support message key for Kafka external stream #434
Conversation
5fa351a
to
d69c088
Compare
d69c088
to
f0501ae
Compare
@@ -304,8 +287,8 @@ void KafkaSink::addMessageToBatch(char * pos, size_t len) | |||
.partition = next_partition, | |||
.payload = const_cast<void *>(static_cast<const void *>(payload)), | |||
.len = len, | |||
.key = const_cast<void *>(static_cast<const void *>(key.data())), | |||
.key_len = key.size(), | |||
.key = const_cast<void *>(static_cast<const void *>(key.data)), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need all of this cast ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same for payload
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The difference here is that key.data
has type of const char *
, if we remove the const_cast
, it will report Cannot initialize a member subobject of type 'void *' with an lvalue of type 'const char *'. But the
static_cast` can be removed.
PR checklist:
proton: starts/ends
for new code in existing community code base ?Please write user-readable short description of the changes:
closes #433
Added a new external stream setting
message_key
which is an expression that returns a string value, the values return by the expression will be used as the message key for each row. Examples:message_key
can be used withsharding_expr
together, andsharding_expr
will take priority (exactly what a Kafka client does).Together with this change, a couple more changes are worth mentioning:
rd_kafka_produce_batch
to send messages to Kafka, which is expected to be more efficient ( at least, it removes the overhead of callingrd_kafka_produce
for each message ). This should benefitone_message_per_row
even more.