Skip to content

KAFKA-19625: Consistency of command-line arguments for verifiable producer/consumer #20390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AndrewJSchofield merged 8 commits into from
Aug 27, 2025
Merged

KAFKA-19625: Consistency of command-line arguments for verifiable producer/consumer #20390

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d2d51fe
Consistency of command-line arguments for verifiable producer/consumer
JimmyWang6 Aug 21, 2025
211e9c5
Consistency of command-line arguments for verifiable producer/consumer
JimmyWang6 Aug 25, 2025
3863c85
Consistency of command-line arguments for verifiable producer/consumer
JimmyWang6 Aug 25, 2025
dc5135e
address comments
JimmyWang6 Aug 25, 2025
2aeb62a
address comments
JimmyWang6 Aug 25, 2025
f1096ef
address comments
JimmyWang6 Aug 26, 2025
17f0878
address comment
JimmyWang6 Aug 27, 2025
ad7a696
fix typo
JimmyWang6 Aug 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions tests/kafkatest/services/verifiable_client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@
* `--enable-autocommit`
* `--max-messages <n>`
* `--assignment-strategy <s>`
* `--consumer.config <config-file>` - consumer config properties (typically empty)
* `--consumer.config <config-file>` - (DEPRECATED) consumer config properties (typically empty). This option will be removed in a future version. Use --command-config instead.
* `--command-config <config-file>` - command config properties

Environment variables:
* `LOG_DIR` - log output directory. Typically not needed if logs are written to stderr.
Expand All @@ -97,7 +98,8 @@
* `--broker-list <brokers>`
* `--max-messages <n>`
* `--throughput <msgs/s>`
* `--producer.config <config-file>` - producer config properties (typically empty)
* `--producer.config <config-file>` - producer config properties (typically empty). This option will be removed in a future version. Use --command-config instead.
* `--command-config <config-file>` - command config properties

Environment variables:
* `LOG_DIR` - log output directory. Typically not needed if logs are written to stderr.
Expand Down
2 changes: 1 addition & 1 deletion tests/kafkatest/services/verifiable_consumer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,7 @@ def start_cmd(self, node):
if self.max_messages > 0:
cmd += " --max-messages %s" % str(self.max_messages)

cmd += " --consumer.config %s" % VerifiableConsumer.CONFIG_FILE
cmd += " --command-config %s" % VerifiableConsumer.CONFIG_FILE

@chia7712 chia7712 Sep 3, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this breaks consumer_protocol_migration_test.py

cmd += " 2>> %s | tee -a %s &" % (VerifiableConsumer.STDOUT_CAPTURE, VerifiableConsumer.STDOUT_CAPTURE)
return cmd

Expand Down
2 changes: 1 addition & 1 deletion tests/kafkatest/services/verifiable_producer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ def start_cmd(self, node, idx):
if self.repeating_keys is not None:
cmd += " --repeating-keys %s " % str(self.repeating_keys)

cmd += " --producer.config %s" % VerifiableProducer.CONFIG_FILE
cmd += " --command-config %s" % VerifiableProducer.CONFIG_FILE

@chia7712 chia7712 Sep 3, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change could break the compatibility_test_new_broker_test.py, since the VerifiableProducer is running under old kafka.

@omkreddy omkreddy Sep 3, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upgrade_test.py will also fail with these changes.

cc @JimmyWang6

@AndrewJSchofield AndrewJSchofield Sep 3, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chia7712 @omkreddy Thanks for this. These tests are an area of the code which I am unfamiliar with. Next time I will be aware.


cmd += " 2>> %s | tee -a %s &" % (VerifiableProducer.STDOUT_CAPTURE, VerifiableProducer.STDOUT_CAPTURE)
return cmd
Expand Down
27 changes: 24 additions & 3 deletions tools/src/main/java/org/apache/kafka/tools/VerifiableConsumer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -597,7 +597,7 @@ private static ArgumentParser argParser() {
.setDefault("earliest")
.type(String.class)
.dest("resetPolicy")
.help("Set reset policy (must be either 'earliest', 'latest', or 'none'");
.help("Set reset policy (must be either 'earliest', 'latest', or 'none')");

parser.addArgument("--assignment-strategy")
.action(store())
Expand All @@ -611,8 +611,17 @@ private static ArgumentParser argParser() {
.action(store())
.required(false)
.type(String.class)
.metavar("CONFIG_FILE")
.help("Consumer config properties file (config options shared with command line parameters will be overridden).");
.metavar("CONFIG-FILE")
.help("(DEPRECATED) Consumer config properties file" +
"This option will be removed in a future version. Use --command-config instead");

parser.addArgument("--command-config")
.action(store())
.required(false)
.type(String.class)
.metavar("CONFIG-FILE")

@AndrewJSchofield AndrewJSchofield Aug 25, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The names of the metavars use a mixture of - and _ as separators. Please could you choose one or the other (I have a slight preference for -), and then apply to the delinquent arguments.

.dest("commandConfigFile")
.help("Config properties file (config options shared with command line parameters will be overridden).");

return parser;
}
Expand All @@ -622,16 +631,28 @@ public static VerifiableConsumer createFromArgs(ArgumentParser parser, String[]

boolean useAutoCommit = res.getBoolean("useAutoCommit");
String configFile = res.getString("consumer.config");
String commandConfigFile = res.getString("commandConfigFile");
String brokerHostAndPort = res.getString("bootstrapServer");

Properties consumerProps = new Properties();
if (configFile != null && commandConfigFile != null) {
throw new ArgumentParserException("Options --consumer.config and --command-config are mutually exclusive.", parser);
}
if (configFile != null) {
System.out.println("Option --consumer.config has been deprecated and will be removed in a future version. Use --command-config instead.");
try {

@AndrewJSchofield AndrewJSchofield Aug 25, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a deprecation message here like System.out.println("Option --consumer.config has been deprecated and will be removed in a future version. Use --command-config instead.");.

@JimmyWang6 JimmyWang6 Aug 25, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm really sorry—I just woke up, so I made a lot of silly typo errors. :( I’ve already addressed all the comments. Thanks for your review.

@AndrewJSchofield AndrewJSchofield Aug 25, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, not quite. The deprecation message should be printed if the user specified the old --consumer.config and we'd like to encourage them to use --command-config. So, I think the System.out.println you added ought to be added just after if (configFile != null) on line 642.

consumerProps.putAll(Utils.loadProps(configFile));
} catch (IOException e) {
throw new ArgumentParserException(e.getMessage(), parser);
}
}
if (commandConfigFile != null) {
try {
consumerProps.putAll(Utils.loadProps(res.getString(commandConfigFile)));

@omkreddy omkreddy Sep 4, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be
consumerProps.putAll(Utils.loadProps(commandConfigFile)); ?

} catch (IOException e) {
throw new ArgumentParserException(e.getMessage(), parser);
}
}

GroupProtocol groupProtocol = GroupProtocol.of(res.getString("groupProtocol"));
consumerProps.put(ConsumerConfig.GROUP_PROTOCOL_CONFIG, groupProtocol.name());
Expand Down
27 changes: 24 additions & 3 deletions tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,8 +161,9 @@ private static ArgumentParser argParser() {
.action(store())
.required(false)
.type(String.class)
.metavar("CONFIG_FILE")
.help("Producer config properties file.");
.metavar("CONFIG-FILE")
.help("(DEPRECATED) Producer config properties file. " +
"This option will be removed in a future version. Use --command-config instead.");

parser.addArgument("--message-create-time")
.action(store())
Expand All @@ -189,6 +190,14 @@ private static ArgumentParser argParser() {
.dest("repeatingKeys")
.help("If specified, each produced record will have a key starting at 0 increment by 1 up to the number specified (exclusive), then the key is set to 0 again");

parser.addArgument("--command-config")
.action(store())
.required(false)
.type(String.class)
.metavar("CONFIG-FILE")
Comment thread
JimmyWang6 marked this conversation as resolved.
.dest("commandConfigFile")
.help("Config properties file (config options shared with command line parameters will be overridden).");

return parser;
}

Expand Down Expand Up @@ -217,6 +226,7 @@ public static VerifiableProducer createFromArgs(ArgumentParser parser, String[]
String topic = res.getString("topic");
int throughput = res.getInt("throughput");
String configFile = res.getString("producer.config");
String commandConfigFile = res.getString("commandConfigFile");
Integer valuePrefix = res.getInt("valuePrefix");
Long createTime = res.getLong("createTime");
Integer repeatingKeys = res.getInt("repeatingKeys");
Expand All @@ -240,14 +250,25 @@ public static VerifiableProducer createFromArgs(ArgumentParser parser, String[]
producerProps.put(ProducerConfig.ACKS_CONFIG, Integer.toString(res.getInt("acks")));
// No producer retries
producerProps.put(ProducerConfig.RETRIES_CONFIG, "0");
if (configFile != null && commandConfigFile != null) {
throw new ArgumentParserException("Options --producer.config and --command-config are mutually exclusive.", parser);
}

if (configFile != null) {
System.out.println("Option --producer.config has been deprecated and will be removed in a future version. Use --command-config instead.");
try {
producerProps.putAll(loadProps(configFile));
} catch (IOException e) {
throw new ArgumentParserException(e.getMessage(), parser);
}
}

if (commandConfigFile != null) {
try {
producerProps.putAll(loadProps(commandConfigFile));
} catch (IOException e) {
throw new ArgumentParserException(e.getMessage(), parser);
}
}
StringSerializer serializer = new StringSerializer();
KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps, serializer, serializer);

Expand Down