Skip to content

Vmhtuan/kafka producer config #26130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Vmhtuan/kafka producer config #26130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c915156
Add maxRequestSize and batchSize configuration to KafkaEventListenerC…
Jul 3, 2025
9b03063
Use maxRequestSize and batchSize configuration in BaseKafkaProducerFa…
Jul 3, 2025
08b5896
Add maxRequestSize and batchSize configuration to TestKafkaEventListe…
Jul 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions ...-listener/src/main/java/io/trino/plugin/eventlistener/kafka/KafkaEventListenerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ public class KafkaEventListenerConfig
private boolean publishCreatedEvent = true;
private boolean publishCompletedEvent = true;
private boolean publishSplitCompletedEvent;
private Long maxRequestSize = 5242880L;
private Long batchSize = 16384L;
Comment on lines +46 to +47
Copy link
Member

@ebyhr ebyhr Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you leave a code comment explaining how these values are chosen?

Please use long instead.

Copy link
Contributor Author

@vmtuan12 vmtuan12 Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reviewing! This PR has some issues, and I have closed it. I have opened a new PR at #26132. And I'm gonna resolve your comments. Thanks!

private Optional<String> completedTopicName = Optional.empty();
private Optional<String> createdTopicName = Optional.empty();
private Optional<String> splitCompletedTopicName = Optional.empty();
Expand Down Expand Up @@ -115,6 +117,30 @@ public KafkaEventListenerConfig setCreatedTopicName(String createdTopicName)
return this;
}

public Long getMaxRequestSize()
Copy link
Member

@ebyhr ebyhr Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getters/setters place should be consistent with fields.

{
return maxRequestSize;
}

@Config("kafka-event-listener.max-request-size")
Copy link
Member

@ebyhr ebyhr Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ConfigDescription is missing. Same for kafka-event-listener.batch-size

public KafkaEventListenerConfig setMaxRequestSize(Long maxRequestSize)
{
Optional.ofNullable(maxRequestSize).ifPresent(value -> this.maxRequestSize = value);
Copy link
Member

@ebyhr ebyhr Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

this.maxRequestSize = maxRequestSize;

return this;
}

public Long getBatchSize()
{
return batchSize;
}

@Config("kafka-event-listener.batch-size")
public KafkaEventListenerConfig setBatchSize(Long batchSize)
{
Optional.ofNullable(batchSize).ifPresent(value -> this.batchSize = value);
Copy link
Member

@ebyhr ebyhr Jul 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

this.batchSize = batchSize;

return this;
}

public Optional<String> getSplitCompletedTopicName()
{
return splitCompletedTopicName;
Expand Down
3 changes: 2 additions & 1 deletion .../src/main/java/io/trino/plugin/eventlistener/kafka/producer/BaseKafkaProducerFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@ protected Map<String, Object> baseConfig(KafkaEventListenerConfig config)
kafkaClientConfig.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
kafkaClientConfig.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
kafkaClientConfig.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
kafkaClientConfig.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, "5242880");
kafkaClientConfig.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, Long.toString(config.getMaxRequestSize()));
kafkaClientConfig.put(ProducerConfig.BATCH_SIZE_CONFIG, Long.toString(config.getBatchSize()));
kafkaClientConfig.put(ProducerConfig.REQUEST_TIMEOUT_MS_CONFIG, Long.toString(config.getRequestTimeout().toMillis()));
return kafkaClientConfig;
}
Expand Down
6 changes: 6 additions & 0 deletions ...tener/src/test/java/io/trino/plugin/eventlistener/kafka/TestKafkaEventListenerConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ void testDefaults()
.setCreatedTopicName(null)
.setSplitCompletedTopicName(null)
.setBrokerEndpoints(null)
.setBatchSize(null)
.setMaxRequestSize(null)
.setClientId(null)
.setExcludedFields(Set.of())
.setRequestTimeout(new Duration(10, TimeUnit.SECONDS))
Expand All @@ -66,6 +68,8 @@ void testExplicitPropertyMappings()
.put("kafka-event-listener.publish-completed-event", "false")
.put("kafka-event-listener.publish-split-completed-event", "true")
.put("kafka-event-listener.broker-endpoints", "kafka-host-1:9093,kafka-host-2:9093")
.put("kafka-event-listener.max-request-size", "5242880")
.put("kafka-event-listener.batch-size", "16384")
.put("kafka-event-listener.created-event.topic", "query_created")
.put("kafka-event-listener.completed-event.topic", "query_completed")
.put("kafka-event-listener.split-completed-event.topic", "split_completed")
Expand All @@ -84,6 +88,8 @@ void testExplicitPropertyMappings()
.setPublishCompletedEvent(false)
.setPublishSplitCompletedEvent(true)
.setBrokerEndpoints("kafka-host-1:9093,kafka-host-2:9093")
.setMaxRequestSize(5242880L)
.setBatchSize(16384L)
.setCreatedTopicName("query_created")
.setCompletedTopicName("query_completed")
.setSplitCompletedTopicName("split_completed")
Expand Down
Loading