Add support for `max.request.size` and `batch.size` configuration for Kafka Event Listener #26132

vmtuan12 · 2025-07-03T21:57:06Z

Description

Add support for max.request.size and batch.size configuration for Kafka producer of Kafka Event Listener
Fixes #26129

Release notes

(x) Release notes are required, with the following suggested text:

## Kafka Event Listener
* Add support for configuring max request size with the `kafka-event-listener.max-request-size` config property. ({issue}`26129`)
* Add support for configuring batch size with the `kafka-event-listener.batch-size` config property. ({issue}`26129`)

cla-bot · 2025-07-03T21:57:09Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

ebyhr

I left some reivew comments in your older PR. Please address those comments.

cla-bot · 2025-07-03T22:37:12Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

vmtuan12 · 2025-07-03T23:00:04Z

I left some reivew comments in your older PR. Please address those comments.

I have already resolved these comments. Thanks!

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java

cla-bot · 2025-07-03T23:35:12Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java

cla-bot · 2025-07-04T03:22:02Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-07-04T03:38:57Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2025-07-04T03:59:59Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

docs/src/main/sphinx/admin/event-listeners-kafka.md

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java

ebyhr · 2025-07-06T22:20:32Z

@vmtuan12 Have you already submitted CLA?

vmtuan12 · 2025-07-06T22:26:11Z

@vmtuan12 Have you already submitted CLA?

Yes, I sent the email 5 days ago, but I think it has not been processed. My email is [email protected], can you check that for me?

cla-bot · 2025-07-06T22:54:37Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

chenjian2664 · 2025-07-07T12:11:16Z

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java

+        return batchSize;
+    }
+
+    @ConfigDescription("Attempt to batch records/messages together up to batch.size bytes before sending")


Could you update the config description? to me it's unclear what is batch.size bytes means?

it means that messages will be buffered before being sent, and sent when the sum of message size >= batch.size, instead of sending every individual message

If the description is modified to "Query events will be buffered, and sent when their total size reaches batch.size", will it be better?

How about follow the document part that you described the config seems better

Ok thanks! I have editted it

cla-bot · 2025-07-08T01:28:20Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

vmtuan12 · 2025-07-11T03:53:31Z

@ebyhr please review again

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java

chenjian2664

Looks good

chenjian2664 · 2025-07-17T01:08:59Z

The commit message : Add support for max.request.size and batch.size configuration Kafka Event Listener should be Add support for max.request.size and batch.size configuration for Kafka Event Listener?

ebyhr · 2025-07-17T01:14:22Z

That looks too long. I changed to "Support max.request.size and batch.size config in Kafka Event Listener"

vmtuan12 · 2025-07-17T01:58:08Z

Can this be merged?

ebyhr · 2025-07-17T02:02:46Z

@vmtuan12 Please use rebase when you apply changes from master. I re-forced pushed.

vmtuan12 · 2025-07-17T02:26:28Z

I do not have permission to merge into Master. Can you merge it @ebyhr

ebyhr reviewed Jul 3, 2025

View reviewed changes

vmtuan12 mentioned this pull request Jul 3, 2025

Vmhtuan/kafka producer config #26130

Closed

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from de6fd23 to a6cda56 Compare July 3, 2025 22:37

github-actions bot added the docs label Jul 3, 2025

ebyhr reviewed Jul 3, 2025

View reviewed changes

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java Outdated Show resolved Hide resolved

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from a6cda56 to d655ca2 Compare July 3, 2025 23:35

chenjian2664 reviewed Jul 4, 2025

View reviewed changes

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java Outdated Show resolved Hide resolved

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from d655ca2 to f2df89f Compare July 4, 2025 03:21

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from f2df89f to 6805c6c Compare July 4, 2025 03:38

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from 6805c6c to 20fded6 Compare July 4, 2025 03:59

vmtuan12 requested review from chenjian2664 and ebyhr July 4, 2025 06:04

ebyhr reviewed Jul 6, 2025

View reviewed changes

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from 20fded6 to 45ab732 Compare July 6, 2025 22:54

vmtuan12 requested a review from ebyhr July 7, 2025 01:27

chenjian2664 approved these changes Jul 7, 2025

View reviewed changes

vmtuan12 force-pushed the vmhtuan/add-kafka-producer-config branch from 45ab732 to 9fce061 Compare July 8, 2025 01:28

cla-bot bot added the cla-signed label Jul 11, 2025

ebyhr force-pushed the vmhtuan/add-kafka-producer-config branch from f7053e9 to 98270f8 Compare July 11, 2025 03:55

ebyhr reviewed Jul 15, 2025

View reviewed changes

...ent-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java Outdated Show resolved Hide resolved

chenjian2664 reviewed Jul 17, 2025

View reviewed changes

ebyhr force-pushed the vmhtuan/add-kafka-producer-config branch 2 times, most recently from a3963b9 to 07729fd Compare July 17, 2025 01:04

ebyhr approved these changes Jul 17, 2025

View reviewed changes

ebyhr force-pushed the vmhtuan/add-kafka-producer-config branch from 07729fd to f45f965 Compare July 17, 2025 01:12

Support max.request.size and batch.size config in Kafka Event Listener

01beda7

ebyhr force-pushed the vmhtuan/add-kafka-producer-config branch from 72870e0 to 01beda7 Compare July 17, 2025 01:59

ebyhr merged commit e6c903a into trinodb:master Jul 17, 2025
18 checks passed

github-actions bot added this to the 477 milestone Jul 17, 2025

ebyhr mentioned this pull request Aug 6, 2025

Add Trino 477 release notes #26350

Merged

Add support for max.request.size and batch.size configuration for Kafka Event Listener #26132

Add support for max.request.size and batch.size configuration for Kafka Event Listener #26132

Uh oh!

Conversation

vmtuan12 commented Jul 3, 2025 • edited by ebyhr Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Uh oh!

cla-bot bot commented Jul 3, 2025

Uh oh!

ebyhr left a comment

Choose a reason for hiding this comment

Uh oh!

cla-bot bot commented Jul 3, 2025

Uh oh!

vmtuan12 commented Jul 3, 2025

Uh oh!

Uh oh!

cla-bot bot commented Jul 3, 2025

Uh oh!

Uh oh!

cla-bot bot commented Jul 4, 2025

Uh oh!

cla-bot bot commented Jul 4, 2025

Uh oh!

cla-bot bot commented Jul 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ebyhr commented Jul 6, 2025

Uh oh!

vmtuan12 commented Jul 6, 2025

Uh oh!

cla-bot bot commented Jul 6, 2025

Uh oh!

chenjian2664 Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

vmtuan12 Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

vmtuan12 Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

chenjian2664 Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

vmtuan12 Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

cla-bot bot commented Jul 8, 2025

Uh oh!

vmtuan12 commented Jul 11, 2025

Uh oh!

Uh oh!

chenjian2664 left a comment

Choose a reason for hiding this comment

Uh oh!

chenjian2664 commented Jul 17, 2025

Uh oh!

ebyhr commented Jul 17, 2025

Uh oh!

vmtuan12 commented Jul 17, 2025

Uh oh!

ebyhr commented Jul 17, 2025

Uh oh!

vmtuan12 commented Jul 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

Add support for `max.request.size` and `batch.size` configuration for Kafka Event Listener #26132

Add support for `max.request.size` and `batch.size` configuration for Kafka Event Listener #26132

vmtuan12 commented Jul 3, 2025 •

edited by ebyhr

Loading