KAFKA-19624: Improve consistency of command-line arguments for consumer performance tests by aheev · Pull Request #20385 · apache/kafka

aheev · 2025-08-20T14:40:54Z

resolves https://issues.apache.org/jira/browse/KAFKA-19624

Reviewers: @brandboat, @AndrewJSchofield, @m1a2st

…umer performance tests

aheev · 2025-08-20T14:41:28Z

@AndrewJSchofield can you please review this?

brandboat · 2025-08-25T02:11:38Z

ping @aheev, could you please fix the conflicts?

# Conflicts: # tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java # tools/src/test/java/org/apache/kafka/tools/ConsumerPerformanceTest.java

aheev · 2025-08-25T03:07:42Z

ping @aheev, could you please fix the conflicts?

done

brandboat

Thanks for the patch, left some comments below.

aheev · 2025-08-25T07:16:09Z

FAILED ❌ RestoreIntegrationTest > "shouldInvokeUserDefinedGlobalStateRestoreListener(boolean).useNewProtocol=false"

This test seems to be flaky. It succeeds on my machine and fails sometimes. Also the changes are not even related to the test

Just tested. Same issue on trunk too

m1a2st

Thanks for this patch, some comments left

AndrewJSchofield

Thanks for the PR. Please add some tests for the variation combinations of new/old options to make sure the rules in the KIP are correctly implemented.

…nd `messages` traces from the tools

brandboat

Thanks for the update, some minor comments.

brandboat

LGTM, thanks!

…losed

aheev · 2025-09-07T16:04:36Z

@Yunyung ConsumerPerformanceService and ShareConsumerPerformanceService are used in benchmark_test.py(which uses DEV_BRANCH and test_performance_services.py(which uses LATEST_2_1 and DEV_BRANCH)

ShareConsumerPerformanceService runs only in test_performance_services.py runs on branches >= 4.1. Hence, we can ignore this
As for ConsumerPerformanceService, I have added changes, but, while running the test, ProducerPerformanceService, which is used to produce data in turn consumed by ConsumerPerformanceService, fails, thereby blocking the ConsumerPerformanceService to consume on LATEST_2_1. It fails on trunk too

[INFO:2025-09-07 09:01:58,715]: RunnerClient: kafkatest.sanity_checks.test_performance_services.PerformanceServiceTest.test_version.version=2.1.1.metadata_quorum=ISOLATED_KRAFT: FAIL: Exception('ProducerPerformanceService-0-125869363841104-worker-1: Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 36, in _protected_worker\n    self._worker(idx, node)\n  File "/opt/kafka-dev/tests/kafkatest/services/performance/producer_performance.py", line 130, in _worker\n    wait_until(lambda: self.alive(node), timeout_sec=20, err_msg="ProducerPerformance failed to start")\n  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 58, in wait_until\n    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception\nducktape.errors.TimeoutError: ProducerPerformance failed to start\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 351, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 411, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.10/dist-packages/ducktape/mark/_mark.py", line 438, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/sanity_checks/test_performance_services.py", line 58, in test_version
    self.producer_perf.run()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/service.py", line 345, in run
    self.wait()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 72, in wait
    self._propagate_exceptions()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 103, in _propagate_exceptions
    raise Exception(self.errors)
Exception: ProducerPerformanceService-0-125869363841104-worker-1: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/services/background_thread.py", line 36, in _protected_worker
    self._worker(idx, node)
  File "/opt/kafka-dev/tests/kafkatest/services/performance/producer_performance.py", line 130, in _worker
    wait_until(lambda: self.alive(node), timeout_sec=20, err_msg="ProducerPerformance failed to start")
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: ProducerPerformance failed to start

I think we need to fix our test suite first
@AndrewJSchofield what is the way forward here?

aheev · 2025-09-13T11:49:45Z

@AndrewJSchofield @Yunyung gentle ping on this one

AndrewJSchofield · 2025-09-14T01:28:52Z

@aheev Sorry for the delay. I'll review it early next week.

AndrewJSchofield · 2025-09-16T14:51:02Z

@aheev This is what I get when I run the tests on your branch:

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2025-09-16--001
run time:         68 minutes 36.085 seconds
tests run:        74
passed:           74
flaky:            0
failed:           0
ignored:          0
================================================================================

I do get the following warnings on trunk and your branch:

[WARNING - 2025-09-16 07:49:08,187 - service_registry - stop_all - lineno:53]: Error stopping service <ProducerPerformanceService-0-281473709848272: num_nodes: 1, nodes: ['ducker07']>: 1

AndrewJSchofield

Thanks for the updates.

The share_consumer_performance.py test file also needs changes to account for --messages changing into --num-records.

I've been running the ducktape tests and tests/kafkatest/sanity_checks/test_performance_services.py does seem flaky. However, I have had passes on v2.1.1 and dev across all variations. So, I think we can rule out problems being introduced by this PR.

Please merge latest changes from trunk.

aheev · 2025-09-18T06:33:08Z

@aheev This is what I get when I run the tests on your branch:

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2025-09-16--001
run time:         68 minutes 36.085 seconds
tests run:        74
passed:           74
flaky:            0
failed:           0
ignored:          0
================================================================================

I do get the following warnings on trunk and your branch:

[WARNING - 2025-09-16 07:49:08,187 - service_registry - stop_all - lineno:53]: Error stopping service <ProducerPerformanceService-0-281473709848272: num_nodes: 1, nodes: ['ducker07']>: 1

I got similar results

aheev · 2025-09-18T06:55:40Z

The share_consumer_performance.py test file also needs changes to account for --messages changing into --num-records.

done

aheev · 2025-09-20T02:35:24Z

@AndrewJSchofield addressed your comments. Can you please take a re-look?

AndrewJSchofield

Thanks for the PR. Looks good to me.

@brandboat

…er performance tests (apache#20385) resolves https://issues.apache.org/jira/browse/KAFKA-19624 Reviewers: @brandboat, @AndrewJSchofield, @m1a2st

mkansara23 · 2025-09-29T04:40:31Z

Hi @AndrewJSchofield ,

I noticed 36 rename refactoring in your recent commit:

Variable Declaration Statement messagesReadByConsumer -> recordsReadByConsumer.
Single Variable Declaration totalMessagesRead -> totalRecordsRead.
Method Declaration updateMessagesConsumed -> updateRecordsConsumed.
Method Declaration numMessages -> numRecords.
Single Variable Declaration messagesRead -> recordsRead.
Field Declaration messagesConsumed -> recordsConsumed.
Single Variable Declaration lastMessagesRead -> lastRecordsRead.
Variable Declaration Statement messagesRead -> recordsRead.
Variable Declaration Statement totalMessagesRead -> totalRecordsRead.
Variable Declaration Statement numMessages -> numRecords.
Variable Declaration Statement lastMessagesRead -> lastRecordsRead.
Method Declaration messagesConsumed -> recordsConsumed.
Variable Declaration Statement intervalMessagesRead -> intervalRecordsRead.
Single Variable Declaration messagesConsumed -> recordsConsumed.
Method Declaration consumeMessagesForSingleShareConsumer -> - consumeRecordsForSingleShareConsumer.
Variable Declaration Statement intervalMessagesPerSec -> intervalRecordsPerSec.

How did you identify that these 36 elements were connected and should be renamed together?

Did you find this task challenging in any way?
Did you perform the renames manually or with tools?
Would it be helpful if tools could detect and rename connected program elements together instead of one by one?

AndrewJSchofield · 2025-09-29T07:44:18Z

@mkansara23 The code changes were performed by @aheev. Maybe they have an opinion.

aheev · 2025-09-29T08:52:06Z

Hi @AndrewJSchofield ,

I noticed 36 rename refactoring in your recent commit:

Variable Declaration Statement messagesReadByConsumer -> recordsReadByConsumer.

Single Variable Declaration totalMessagesRead -> totalRecordsRead.

Method Declaration updateMessagesConsumed -> updateRecordsConsumed.

Method Declaration numMessages -> numRecords.

Single Variable Declaration messagesRead -> recordsRead.

Field Declaration messagesConsumed -> recordsConsumed.

Single Variable Declaration lastMessagesRead -> lastRecordsRead.

Variable Declaration Statement messagesRead -> recordsRead.

Variable Declaration Statement totalMessagesRead -> totalRecordsRead.

Variable Declaration Statement numMessages -> numRecords.

Variable Declaration Statement lastMessagesRead -> lastRecordsRead.

Method Declaration messagesConsumed -> recordsConsumed.

Variable Declaration Statement intervalMessagesRead -> intervalRecordsRead.

Single Variable Declaration messagesConsumed -> recordsConsumed.

Method Declaration consumeMessagesForSingleShareConsumer -> - consumeRecordsForSingleShareConsumer.

Variable Declaration Statement intervalMessagesPerSec -> intervalRecordsPerSec.

How did you identify that these 36 elements were connected and should be renamed together?

Did you find this task challenging in any way?

Did you perform the renames manually or with tools?

Would it be helpful if tools could detect and rename connected program elements together instead of one by one?

We wanted to get rid of message naming convention in tests, so while I was going through the tests, I just a did a search and replace for message keyword. Delegating the task to an AI agent would have been simpler though

Can I know what's the motive behind your query?

mkansara23 · 2025-09-29T17:08:57Z

Hi @aheev ,
I noticed 36 rename refactoring in your recent commit:

Variable Declaration Statement messagesReadByConsumer -> recordsReadByConsumer.

Single Variable Declaration totalMessagesRead -> totalRecordsRead.

Method Declaration updateMessagesConsumed -> updateRecordsConsumed.

Method Declaration numMessages -> numRecords.

Single Variable Declaration messagesRead -> recordsRead.

Field Declaration messagesConsumed -> recordsConsumed.

Single Variable Declaration lastMessagesRead -> lastRecordsRead.

Variable Declaration Statement messagesRead -> recordsRead.

Variable Declaration Statement totalMessagesRead -> totalRecordsRead.

Variable Declaration Statement numMessages -> numRecords.

Variable Declaration Statement lastMessagesRead -> lastRecordsRead.

Method Declaration messagesConsumed -> recordsConsumed.

Variable Declaration Statement intervalMessagesRead -> intervalRecordsRead.

Single Variable Declaration messagesConsumed -> recordsConsumed.

Method Declaration consumeMessagesForSingleShareConsumer -> - consumeRecordsForSingleShareConsumer.

Variable Declaration Statement intervalMessagesPerSec -> intervalRecordsPerSec.

How did you identify that these 36 elements were connected and should be renamed together?

Did you find this task challenging in any way?

Did you perform the renames manually or with tools?

Would it be helpful if tools could detect and rename connected program elements together instead of one by one?

We wanted to get rid of message naming convention in tests, so while I was going through the tests, I just a did a search and replace for message keyword. Delegating the task to an AI agent would have been simpler though

Can I know what's the motive behind your query?

Thank you for your response! I am conducting a study to understand patterns and challenges in cluster-renames—situations where multiple related program elements are renamed together in the same commit.
The goal is to learn how developers approach these cluster-renames and whether parts of this process can be supported or automated through AI Agents.

If you were to estimate, how long did those renames take you, and what part of the process did you find the most challenging?

@brandboat

…er performance tests (apache#20385) resolves https://issues.apache.org/jira/browse/KAFKA-19624 Reviewers: @brandboat, @AndrewJSchofield, @m1a2st

@brandboat

…er performance tests (apache#20385) resolves https://issues.apache.org/jira/browse/KAFKA-19624 Reviewers: @brandboat, @AndrewJSchofield, @m1a2st

KAFKA-19487: Improving consistency of command-line arguments for cons…

931dfaf

…umer performance tests

github-actions Bot added triage PRs from the community tools labels Aug 20, 2025

aheev changed the title ~~KAFKA-19487: Improving consistency of command-line arguments for consumer performance tests~~ KAFKA-19624: Improving consistency of command-line arguments for consumer performance tests Aug 20, 2025

AndrewJSchofield added ci-approved and removed triage PRs from the community labels Aug 22, 2025

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

17460c7

# Conflicts: # tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java # tools/src/test/java/org/apache/kafka/tools/ConsumerPerformanceTest.java

brandboat reviewed Aug 25, 2025

View reviewed changes

Comment thread tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java

Comment thread tools/src/main/java/org/apache/kafka/tools/ShareConsumerPerformance.java

Add deprecation cycle for num-records

34651a9

aheev requested a review from brandboat August 25, 2025 06:27

brandboat reviewed Aug 25, 2025

View reviewed changes

Comment thread tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java Outdated

Comment thread tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java

m1a2st reviewed Aug 25, 2025

View reviewed changes

Comment thread tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java

Comment thread tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java Outdated

AndrewJSchofield self-requested a review August 25, 2025 14:42

AndrewJSchofield requested changes Aug 25, 2025

View reviewed changes

aheev added 4 commits August 26, 2025 16:32

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

529eb36

Add deprecation signatures to old options; Remove consumer config a…

d82ce4f

…nd `messages` traces from the tools

Add checks for deprecated options

c4cd51e

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

b8b2dc4

aheev requested review from AndrewJSchofield, brandboat and m1a2st August 26, 2025 20:01

brandboat reviewed Aug 27, 2025

View reviewed changes

aheev added 2 commits August 27, 2025 14:55

Rename tests to old config names

5c8fd2b

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

d252457

brandboat approved these changes Aug 27, 2025

View reviewed changes

Comment thread tools/src/test/java/org/apache/kafka/tools/ConsumerPerformanceTest.java Outdated

Rename consumer config files to unique prefix-suffixes

9d9a8c6

Yunyung reviewed Sep 5, 2025

View reviewed changes

Comment thread tools/src/test/java/org/apache/kafka/tools/ConsumerPerformanceTest.java

Yunyung reviewed Sep 5, 2025

View reviewed changes

Comment thread tests/kafkatest/services/performance/consumer_performance.py Outdated

Comment thread tests/kafkatest/services/performance/share_consumer_performance.py Outdated

aheev added 4 commits September 6, 2025 01:03

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

116f7cd

rename messages to num-records in testMetricsRetrievedBeforeConsumerC…

d9f8854

…losed

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

037a177

Add support for running consumer_performance.py on older kafka brokers

f624975

AndrewJSchofield requested changes Sep 16, 2025

View reviewed changes

Comment thread tests/kafkatest/services/performance/share_consumer_performance.py Outdated

Merge branch 'trunk' of https://github.com/apache/kafka into KAFKA-19487

4a3862f

change messages arg to num-records in share_consumer_performance.py

25da0ea

aheev requested a review from AndrewJSchofield September 18, 2025 07:20

aheev added 2 commits September 18, 2025 14:29

add support for old brokers in share_consumer_performance.py

0fca232

clean debug statements

d638653

aheev changed the title ~~KAFKA-19624: Improving consistency of command-line arguments for consumer performance tests~~ KAFKA-19624: Improve consistency of command-line arguments for consumer performance tests Sep 18, 2025

AndrewJSchofield approved these changes Sep 23, 2025

View reviewed changes

AndrewJSchofield merged commit dbe9d34 into apache:trunk Sep 23, 2025
24 checks passed

AndrewJSchofield mentioned this pull request Sep 29, 2025

KAFKA-16263 : Add Kafka Streams docs about available listeners/callback #20430

Merged

Uh oh!

Conversation

aheev commented Aug 20, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aheev commented Aug 20, 2025

Uh oh!

brandboat commented Aug 25, 2025

Uh oh!

aheev commented Aug 25, 2025

Uh oh!

brandboat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aheev commented Aug 25, 2025

Uh oh!

Uh oh!

Uh oh!

m1a2st left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandboat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brandboat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aheev commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aheev commented Sep 13, 2025

Uh oh!

AndrewJSchofield commented Sep 14, 2025

Uh oh!

AndrewJSchofield commented Sep 16, 2025

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aheev commented Sep 18, 2025

Uh oh!

aheev commented Sep 18, 2025

Uh oh!

aheev commented Sep 20, 2025

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mkansara23 commented Sep 29, 2025

Uh oh!

AndrewJSchofield commented Sep 29, 2025

aheev commented Aug 20, 2025 •

edited by github-actions Bot

Loading

aheev commented Sep 7, 2025 •

edited

Loading

mkansara23 commented Sep 29, 2025 •

edited

Loading