KAFKA-15793 Fix ZkMigrationIntegrationTest#testMigrateTopicDeletions

[mumrah]

Increases a few timeouts and fixes some retry logic. Also remove the technically unsupported 3.4 MV.

[mumrah]

After some timeout adjustments and removing the 3.4 case, this is looking a lot more stable

I did a 10x deflake run and all the tests passed

• Action: https://github.com/apache/kafka/actions/runs/10647646774
• Build scan https://ge.apache.org/s/ewu6ab2rf5lvk/tests/overview

[mumrah]

cc @soarez @divijvaidya who originally reported this flakiness

[soarez]

Thanks for fixing this @mumrah.

Many of the new timeout values are oddly specific. How did you determine them?

[mumrah]

@soarez, thanks for taking a look!

Many of the new timeout values are oddly specific

That's a trick I've used before when debugging timeout issues. Sometimes we don't get a good stacktrace or log message on timeouts, so having a unique value lets you correlate it back to a particular line.

[soarez]

@mumrah having a look at the test report, it seems that all attempts at running this test are being skipped.

If any of the replicas is offline during the deletion, the topic is kept under zkClient.getTopicDeletions. Would that be useful to ensure the test can always be performed?

[mumrah]

@soarez I never could figure out a reliable way to ensure there were pending deletions. You're suggestion could work, but I'd like to address that in another PR

[soarez]

@mumrah makes sense. Seems that was maybe an unlucky run anyway, as only one execution was skipped in the latest run:

Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [1] Type=ZK, MetadataVersion=3.5-IV2,Security=PLAINTEXT       11 sec  Passed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [1] Type=ZK, MetadataVersion=3.5-IV2,Security=PLAINTEXT       16 sec  Passed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [1] Type=ZK, MetadataVersion=3.5-IV2,Security=PLAINTEXT        9.1 sec Passed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [2] Type=ZK, MetadataVersion=3.6-IV2,Security=PLAINTEXT       5 min 20 sec    Failed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [2] Type=ZK, MetadataVersion=3.6-IV2,Security=PLAINTEXT       5 min 19 sec    Failed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [2] Type=ZK, MetadataVersion=3.6-IV2,Security=PLAINTEXT        9.5 sec Fixed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [3] Type=ZK, MetadataVersion=3.7-IV0,Security=PLAINTEXT        11 sec  Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [3] Type=ZK, MetadataVersion=3.7-IV0,Security=PLAINTEXT       5 min 22 sec    Failed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [3] Type=ZK, MetadataVersion=3.7-IV0,Security=PLAINTEXT       5 min 19 sec    Failed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [4] Type=ZK, MetadataVersion=3.7-IV1,Security=PLAINTEXT       5 min 20 sec    Failed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [4] Type=ZK, MetadataVersion=3.7-IV1,Security=PLAINTEXT        11 sec  Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [4] Type=ZK, MetadataVersion=3.7-IV1,Security=PLAINTEXT       12 sec  Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [5] Type=ZK, MetadataVersion=3.7-IV2,Security=PLAINTEXT       14 sec  Fixed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [5] Type=ZK, MetadataVersion=3.7-IV2,Security=PLAINTEXT        8.7 sec Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [5] Type=ZK, MetadataVersion=3.7-IV2,Security=PLAINTEXT       10 sec  Fixed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [6] Type=ZK, MetadataVersion=3.7-IV4,Security=PLAINTEXT        8.7 sec Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [6] Type=ZK, MetadataVersion=3.7-IV4,Security=PLAINTEXT       11 sec  Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [6] Type=ZK, MetadataVersion=3.7-IV4,Security=PLAINTEXT       9.6 sec Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [7] Type=ZK, MetadataVersion=3.8-IV0,Security=PLAINTEXT       5 min 19 sec    Failed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [7] Type=ZK, MetadataVersion=3.8-IV0,Security=PLAINTEXT        8.8 sec Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [7] Type=ZK, MetadataVersion=3.8-IV0,Security=PLAINTEXT       10 sec  Fixed
Build / JDK 8 and Scala 2.12 / testMigrateTopicDeletions [8] Type=ZK, MetadataVersion=3.9-IV0,Security=PLAINTEXT        9.6 sec Fixed
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [8] Type=ZK, MetadataVersion=3.9-IV0,Security=PLAINTEXT       16 sec  Skipped
Build / JDK 21 and Scala 2.13 / testMigrateTopicDeletions [8] Type=ZK, MetadataVersion=3.9-IV0,Security=PLAINTEXT       17 sec  Fixed

However, there are 6 failures, all hitting the same 5 minute timeout to delete topics:

org.opentest4j.AssertionFailedError: Timed out waiting for topics to be deleted
	at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
	at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
	at app//kafka.zk.ZkMigrationIntegrationTest.testMigrateTopicDeletions(ZkMigrationIntegrationTest.scala:386)

Does this mean there's some race condition wherein after the migration the topics cannot be deleted? I'm wondering if the test should timeout earlier and retry the deletion. If the topics cannot be deleted at all after the migration, that would be a serious bug.

[mumrah]

@soarez

Does this mean there's some race condition wherein after the migration the topics cannot be deleted?

At this point in the test, we are waiting for ZK or KRaft to finalize one of the pending deletions. We're not actually doing an explicit delete. I wonder if the 30s timeout on the listTopics call is too short (others are 60s).

[mumrah]

@soarez I may have found a fix. In https://github.com/apache/kafka/actions/runs/10729396763 I was able to see the "Timed out waiting for topics to be deleted" failure case. In the logs, it looks like the topic hadn't finished being created by the time I manually put the deletion in ZK. I've added a condition to wait on a full ISR before continuing, and the next run (both deflake x10 and this PR) were successful.

I've kicked off both again, if they both pass I'd like to merge this 🤞

[soarez]

Thanks for fixing this @mumrah.

LGTM