KAFKA-14596: Move TopicCommand to tools by OmniaGM · Pull Request #13201 · apache/kafka

OmniaGM · 2023-02-06T12:35:20Z

This pr include

The changes include switching Scala code to java
Move TopicCommand and all test cases to tools

Some implementation notes:

I added ToolsTestUtils.createBrokerProperties as a wrapper for TestUtils.createBrokerConfig to avoid converting between Map<Integer, String> and Map<Int, String>.
Replicated TestUtils.setReplicationThrottleForPartitions and TestUtils.removeReplicationThrottleForPartitions to ToolsTestUtils as the methods are used only TopicCommandIntegrationTest and ReassignPartitionsIntegrationTest. We need to remove it from TestUtils once we migrate ReassignPartitions
Replicated TestInfoUtils.TestWithParameterizedQuorumName to ToolsTestUtils as java convert this into a getter function which cannot be used with ParameterizedTest
Move CoreUtils.duplicate to ToolsUtils
~~Move TopicFilter out of core. (This is part of https://issues.apache.org/jira/browse/KAFKA-14647 which seems not moving for a while )~~ Used KAFKA-14647: Moving TopicFilter to server-common/utils #13158
~~Duplicate AdminCommandFailedException and AdminOperationException out of core.~~ Used the new exceptions from KAFKA-14591 DeleteRecordsCommand moved to tools #13278

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.

Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

github-actions · 2023-06-10T20:48:23Z

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has merge conflicts, please update it with the latest from trunk (or appropriate release branch)
If this PR is no longer valid or desired, please feel free to close it. If no activity occurrs in the next 30 days, it will be automatically closed.

divijvaidya · 2023-06-16T13:49:17Z

@OmniaGM can you please rebase this with trunk and I would be happy to begin a review for this.

OmniaGM · 2023-06-18T19:03:00Z

Hi @divijvaidya I just did rebase this with trunk

fvaleri · 2023-07-08T16:32:07Z

Hi @OmniaGM.

Move TopicFilter out of core. (This is part of https://issues.apache.org/jira/browse/KAFKA-14647 which seems not moving for a while )

I think #13158 is ready. I'll try to drive some attention to it.

Move CoreUtils.duplicate to ToolsUtils
Duplicate AdminCommandFailedException and AdminOperationException out of core.

I looks like AdminCommandFailedException and AdminOperationException are being moved in #13278. This also seems to be almost ready.

OmniaGM · 2023-07-09T14:31:06Z

Hi @OmniaGM.

Move TopicFilter out of core. (This is part of https://issues.apache.org/jira/browse/KAFKA-14647 which seems not moving for a while )

I think #13158 is ready. I'll try to drive some attention to it.

Move CoreUtils.duplicate to ToolsUtils
Duplicate AdminCommandFailedException and AdminOperationException out of core.

I looks like AdminCommandFailedException and AdminOperationException are being moved in #13278. This also seems to be almost ready.

Will update this PR once these prs are merged.

vamossagar12 · 2023-07-12T07:22:41Z

Thanks @OmniaGM , @fvaleri . #13158 has been approved by Federico. Just waiting for a committer to review/merge it.

OmniaGM · 2023-08-02T10:09:26Z

I have updated the PR to use #13158 and #13278. The test checks are failing because of an unrelated class.

However the :tools:compileTestJava is failing because it doesn't like JavaConverters.asScala and ~~I can not reproduce this problem locally. Logs are here any suggestion?~~

Edit: Actually running ./gradlew :tools:compileTestJava -PscalaVersion=2.12 clean check -x test --profile --continue -PxmlSpotBugsReport=true -PkeepAliveMode=session do reproduce it locally. Pushed a fix.

Out of curiosity are we planning to move KafkaServerTestHarness and RackAwareTest out of core?

fvaleri

Hi @OmniaGM, thanks. I left some comments.

Additionally, this error message is different from previous implementation:

### OLD
$ bin/kafka-topics.sh --bootstrap-server :9092 --describe --topic my-topic-foo --delete-config sdsad
Option "[delete-config]" can't be used with option "[describe]"

### NEW
$ bin/kafka-topics.sh --bootstrap-server :9092 --describe --topic my-topic-foo --delete-config sdsad
Option "[delete-config]" can't be used with option "[bootstrap-server]"

fvaleri

Hi @OmniaGM, there is still one formatting issue in the first output line:

$ bin/kafka-topics.sh --bootstrap-server :9092 --describe --topic my-topic
Topic: my-topic	TopicId: CITkda1vT5i6T7GdwrWCdA	PartitionCount: %s3	ReplicationFactor: %s2	Configs: %smin.insync.replicas=1
	Topic: my-topic	Partition: 0	Leader: 3	Replicas: 3,4	Isr: 3,4
	Topic: my-topic	Partition: 1	Leader: 4	Replicas: 4,2	Isr: 4,2
	Topic: my-topic	Partition: 2	Leader: 2	Replicas: 2,3	Isr: 2,3

After rebasing on your branch, I'm getting the following exception on the integration tests with quorum=kraft:

java.lang.NoClassDefFoundError: org/apache/kafka/server/log/remote/storage/RemoteStorageException

fvaleri

Hi @OmniaGM, I think we are almost there.

Thanks for the effort of finding a workaround for the dependency issue. Well done.

fvaleri

LGTM. Thanks.

mimaison · 2023-10-03T10:02:47Z

@OmniaGM Thanks for the PR. Can you rebase on trunk to resolve the conflicts?

OmniaGM · 2023-10-05T15:56:32Z

@OmniaGM Thanks for the PR. Can you rebase on trunk to resolve the conflicts?

Updated the pr waiting now for the pipeline to pass

mimaison · 2023-10-09T17:00:34Z

@OmniaGM Thanks for the update. I've not had time to take a look yet but noticed there's a compilation failure:

> Task :tools:compileTestJava

/home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-13201/tools/src/test/java/org/apache/kafka/tools/ToolsTestUtils.java:78: error: variable TEST_WITH_PARAMETERIZED_QUORUM_NAME is already defined in class ToolsTestUtils

    public static final String TEST_WITH_PARAMETERIZED_QUORUM_NAME = "{displayName}.quorum={0}";

OmniaGM · 2023-10-10T14:40:11Z

@mimaison fixed the compilation issue. I spotted another one with Java21 and fixed it as well.

mimaison

Thanks for the PR, it wasn't an easy one to convert! It looks good overall, I left a few comments and suggestions.

mimaison

Thanks for the updates! I left a few more small suggestions

mimaison · 2023-10-16T12:01:39Z

@OmniaGM thanks for the updates. I'm getting a test failure in TopicCommandIntegrationTest:

Gradle Test Run :tools:test > Gradle Test Executor 31 > TopicCommandIntegrationTest > testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(String) > testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(String).quorum=kraft FAILED
    java.lang.RuntimeException: java.lang.IllegalArgumentException: Topic 'testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized-HbRT1wjRyV' does not exist as expected
        at org.apache.kafka.tools.TopicCommandIntegrationTest.lambda$captureDescribeTopicStandardOut$46(TopicCommandIntegrationTest.java:1006)
        at org.apache.kafka.tools.ToolsTestUtils.captureStandardStream(ToolsTestUtils.java:66)
        at org.apache.kafka.tools.ToolsTestUtils.captureStandardOut(ToolsTestUtils.java:50)
        at org.apache.kafka.tools.TopicCommandIntegrationTest.captureDescribeTopicStandardOut(TopicCommandIntegrationTest.java:1009)
        at org.apache.kafka.tools.TopicCommandIntegrationTest.testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(TopicCommandIntegrationTest.java:943)

        Caused by:
        java.lang.IllegalArgumentException: Topic 'testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized-HbRT1wjRyV' does not exist as expected
            at org.apache.kafka.tools.TopicCommand.ensureTopicExists(TopicCommand.java:215)
            at org.apache.kafka.tools.TopicCommand.access$700(TopicCommand.java:78)
            at org.apache.kafka.tools.TopicCommand$TopicService.describeTopic(TopicCommand.java:559)
            at org.apache.kafka.tools.TopicCommandIntegrationTest.lambda$captureDescribeTopicStandardOut$46(TopicCommandIntegrationTest.java:1004)
            ... 4 more

OmniaGM · 2023-10-16T12:23:40Z

@OmniaGM thanks for the updates. I'm getting a test failure in TopicCommandIntegrationTest:

Gradle Test Run :tools:test > Gradle Test Executor 31 > TopicCommandIntegrationTest > testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(String) > testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(String).quorum=kraft FAILED
    java.lang.RuntimeException: java.lang.IllegalArgumentException: Topic 'testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized-HbRT1wjRyV' does not exist as expected
        at org.apache.kafka.tools.TopicCommandIntegrationTest.lambda$captureDescribeTopicStandardOut$46(TopicCommandIntegrationTest.java:1006)
        at org.apache.kafka.tools.ToolsTestUtils.captureStandardStream(ToolsTestUtils.java:66)
        at org.apache.kafka.tools.ToolsTestUtils.captureStandardOut(ToolsTestUtils.java:50)
        at org.apache.kafka.tools.TopicCommandIntegrationTest.captureDescribeTopicStandardOut(TopicCommandIntegrationTest.java:1009)
        at org.apache.kafka.tools.TopicCommandIntegrationTest.testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized(TopicCommandIntegrationTest.java:943)

        Caused by:
        java.lang.IllegalArgumentException: Topic 'testDescribeDoesNotFailWhenListingReassignmentIsUnauthorized-HbRT1wjRyV' does not exist as expected
            at org.apache.kafka.tools.TopicCommand.ensureTopicExists(TopicCommand.java:215)
            at org.apache.kafka.tools.TopicCommand.access$700(TopicCommand.java:78)
            at org.apache.kafka.tools.TopicCommand$TopicService.describeTopic(TopicCommand.java:559)
            at org.apache.kafka.tools.TopicCommandIntegrationTest.lambda$captureDescribeTopicStandardOut$46(TopicCommandIntegrationTest.java:1004)
            ... 4 more

Hi @mimaison I think this is flaky test I added waitForTopicCreated(testTopicName);. The test isn't failing locally with me so will wait for the pipeline to finish.

mimaison · 2023-10-16T13:08:55Z

With the latest updates it's not failing anymore on my laptop. I'll let the Apache CI run before merging.

mimaison

Thanks, LGTM

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

OmniaGM force-pushed the KAFKA-14596 branch 4 times, most recently from 1c25106 to d7c52df Compare February 7, 2023 11:16

vamossagar12 mentioned this pull request Jun 7, 2023

MINOR: Do not print an empty line when no topics exist #13762

Closed

3 tasks

github-actions Bot added the stale Stale PRs label Jun 10, 2023

OmniaGM force-pushed the KAFKA-14596 branch 3 times, most recently from b574ef2 to 68e39c0 Compare June 18, 2023 19:02

OmniaGM force-pushed the KAFKA-14596 branch from 68e39c0 to dc061f4 Compare June 18, 2023 20:31

github-actions Bot removed the stale Stale PRs label Jun 22, 2023

fvaleri added the tools label Jul 7, 2023

OmniaGM force-pushed the KAFKA-14596 branch from dc061f4 to c187f15 Compare July 9, 2023 15:19

OmniaGM force-pushed the KAFKA-14596 branch from c187f15 to 87a1171 Compare August 1, 2023 14:52

OmniaGM force-pushed the KAFKA-14596 branch 4 times, most recently from d126a39 to 993b6d6 Compare August 2, 2023 14:23

fvaleri suggested changes Aug 3, 2023

View reviewed changes

OmniaGM force-pushed the KAFKA-14596 branch 3 times, most recently from 07eaec5 to 81a0ef8 Compare August 23, 2023 15:19

fvaleri suggested changes Aug 28, 2023

View reviewed changes

fvaleri reviewed Sep 14, 2023

View reviewed changes

Comment thread build.gradle Outdated

fvaleri approved these changes Sep 15, 2023

View reviewed changes

fvaleri requested a review from mimaison September 15, 2023 14:06

OmniaGM added 6 commits October 5, 2023 16:10

KAFKA-14596: Move TopicCommand to tools

521cbed

Address fvaleri's feedback

06cb3d2

fix test

590fa47

Fix format

0574774

Fix invalidOptions to retains order of insertions

5b86ae0

reduce dependencies

f764089

OmniaGM force-pushed the KAFKA-14596 branch from 57c698e to f764089 Compare October 5, 2023 15:47

OmniaGM added 2 commits October 9, 2023 21:17

fix compilation

11f6a3e

fix [this-escape] warnings

1055c5b

mimaison reviewed Oct 12, 2023

View reviewed changes

address Mickael's feedback

d460fbd

OmniaGM force-pushed the KAFKA-14596 branch from 3004506 to d460fbd Compare October 13, 2023 16:10

mimaison reviewed Oct 16, 2023

View reviewed changes

Comment thread core/src/test/scala/unit/kafka/utils/TestUtils.scala Outdated

Comment thread tools/src/main/java/org/apache/kafka/tools/ToolsUtils.java Outdated

Comment thread tools/src/main/java/org/apache/kafka/tools/TopicCommand.java Outdated

address Mickael's feedback

6259143

OmniaGM requested a review from mimaison October 16, 2023 11:02

wait for topic creation

30864b5

mimaison approved these changes Oct 17, 2023

View reviewed changes

mimaison merged commit 9af1e74 into apache:trunk Oct 17, 2023

AnatolyPopov pushed a commit to aiven/kafka that referenced this pull request Feb 16, 2024

KAFKA-14596: Move TopicCommand to tools (apache#13201)

5891953

Reviewers: Mickael Maison <mickael.maison@gmail.com>, Federico Valeri <fedevaleri@gmail.com>

Uh oh!

Conversation

OmniaGM commented Feb 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Committer Checklist (excluded from commit message)

Uh oh!

github-actions Bot commented Jun 10, 2023

Uh oh!

divijvaidya commented Jun 16, 2023

Uh oh!

OmniaGM commented Jun 18, 2023

Uh oh!

fvaleri commented Jul 8, 2023

Uh oh!

OmniaGM commented Jul 9, 2023

Uh oh!

vamossagar12 commented Jul 12, 2023

Uh oh!

OmniaGM commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fvaleri left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fvaleri left a comment

Choose a reason for hiding this comment

Uh oh!

mimaison commented Oct 3, 2023

Uh oh!

OmniaGM commented Oct 5, 2023

Uh oh!

mimaison commented Oct 9, 2023

Uh oh!

OmniaGM commented Oct 10, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mimaison commented Oct 16, 2023

Uh oh!

OmniaGM commented Oct 16, 2023

Uh oh!

mimaison commented Oct 16, 2023

Uh oh!

mimaison left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

OmniaGM commented Feb 6, 2023 •

edited

Loading

OmniaGM commented Aug 2, 2023 •

edited

Loading

fvaleri left a comment •

edited

Loading