Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

flaky test report improvements #18212

Open
wants to merge 4 commits into
base: trunk
Choose a base branch
from
Open

flaky test report improvements #18212

wants to merge 4 commits into from
+278 −157

Conversation

santhoshct
Copy link
Contributor

@santhoshct santhoshct commented Dec 16, 2024 •
edited
Loading

  • Added test cases to cleared test reports
  • Added table view to all the remaining reports

Testing

Flaky Test Report for 2024-12-17

This report was run on 2024-12-17 08:45:18 UTC

Most Problematic Tests

Found 8 tests that have been quarantined for 7 days and are still failing frequently.

ClassTest CaseFailure RateBuild Scans
org.apache.kafka.tiered.storage.integration.OffloadAndTxnConsumeFromLeaderTest
executeTieredStorageTest(String, String)[1]100.00%2
executeTieredStorageTest(String, String)[2]100.00%2
kafka.api.CustomQuotaCallbackTest
testExecuteUpdateClusterMetadataMethodCallback(String, String)[1]100.00%2
org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest
testReadCommittedConsumerShouldNotSeeUndecidedData(String, String)[2]16.10%584
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]15.24%584
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]14.73%584
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]7.88%584
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]5.65%584
org.apache.kafka.common.network.Tls13SelectorTest
testCloseOldestConnection()25.00%4
org.apache.kafka.common.network.SslTransportLayerTest
testSelectorPollReadSize(Args)[1]25.00%4
kafka.api.TransactionsTest
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]10.14%582
testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]8.08%582
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]5.84%582
testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]5.50%582
kafka.api.PlaintextConsumerTest
testCloseLeavesGroupOnInterrupt(String, String)[2]9.59%584
testCoordinatorFailover(String, String)[2]3.08%584
testCloseLeavesGroupOnInterrupt(String, String)[1]2.40%584
testCoordinatorFailover(String, String)[1]2.23%584
org.apache.kafka.clients.consumer.internals.AbstractCoordinatorTest
testWakeupAfterSyncGroupReceived()25.00%4
Detailed Execution History

org.apache.kafka.tiered.storage.integration.OffloadAndTxnConsumeFromLeaderTest

  • Days Quarantined: 7
  • Recent Failure Rate: 100.00%
  • Total Runs: 2
  • Build Outcomes: Passed: 0 | Failed: 2 | Flaky: 0

executeTieredStorageTest(String, String)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-15 08:00  failed     lkhdsxpqn3mpa
2024-12-15 08:01  flaky      sqgdbmoepagkq

executeTieredStorageTest(String, String)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-15 08:00  flaky      lkhdsxpqn3mpa
2024-12-15 08:01  flaky      sqgdbmoepagkq

kafka.api.CustomQuotaCallbackTest

  • Days Quarantined: 7
  • Recent Failure Rate: 50.00%
  • Total Runs: 4
  • Build Outcomes: Passed: 2 | Failed: 2 | Flaky: 0

testExecuteUpdateClusterMetadataMethodCallback(String, String)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-16 17:09  failed     sbnxnoxopk3lw
2024-12-16 17:10  failed     jwr5anuivvwsw

org.apache.kafka.tiered.storage.integration.TransactionsWithTieredStoreTest

  • Days Quarantined: 7
  • Recent Failure Rate: 44.85%
  • Total Runs: 584
  • Build Outcomes: Passed: 320 | Failed: 8 | Flaky: 256

testReadCommittedConsumerShouldNotSeeUndecidedData(String, String)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  flaky      tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  flaky      dmojogsx5lrb4

org.apache.kafka.common.network.Tls13SelectorTest

  • Days Quarantined: 7
  • Recent Failure Rate: 25.00%
  • Total Runs: 4
  • Build Outcomes: Passed: 3 | Failed: 0 | Flaky: 1

testCloseOldestConnection()

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-12 20:45  flaky      hhdqo5lixoioi
2024-12-12 20:45  passed     4gllln4top7n6
2024-12-12 22:22  passed     ct6a5b7wnnosw
2024-12-12 22:22  passed     igseo3kh3m2z6

org.apache.kafka.common.network.SslTransportLayerTest

  • Days Quarantined: 7
  • Recent Failure Rate: 25.00%
  • Total Runs: 4
  • Build Outcomes: Passed: 3 | Failed: 0 | Flaky: 1

testSelectorPollReadSize(Args)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-12 20:45  passed     hhdqo5lixoioi
2024-12-12 20:45  passed     4gllln4top7n6
2024-12-12 22:22  flaky      ct6a5b7wnnosw
2024-12-12 22:22  passed     igseo3kh3m2z6

kafka.api.TransactionsTest

  • Days Quarantined: 7
  • Recent Failure Rate: 23.95%
  • Total Runs: 582
  • Build Outcomes: Passed: 449 | Failed: 8 | Flaky: 125

testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  flaky      ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Enabled(String, String, boolean)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testBumpTransactionalEpochWithTV2Disabled(String, String, boolean)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.PlaintextConsumerTest

  • Days Quarantined: 7
  • Recent Failure Rate: 15.84%
  • Total Runs: 584
  • Build Outcomes: Passed: 495 | Failed: 0 | Flaky: 89

testCloseLeavesGroupOnInterrupt(String, String)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[2]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCloseLeavesGroupOnInterrupt(String, String)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[1]

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.clients.consumer.internals.AbstractCoordinatorTest

  • Days Quarantined: 7
  • Recent Failure Rate: 25.00%
  • Total Runs: 8
  • Build Outcomes: Passed: 7 | Failed: 0 | Flaky: 1

testWakeupAfterSyncGroupReceived()

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-12 20:45  passed     hhdqo5lixoioi
2024-12-12 20:45  flaky      4gllln4top7n6
2024-12-12 22:22  passed     ct6a5b7wnnosw
2024-12-12 22:22  passed     igseo3kh3m2z6

Flaky Test Regressions

No flaky test regressions found.

Cleared Tests (Ready for Unquarantine)

Found 20 test classes with 42 test methods that have been consistently passing. These tests could be candidates for removing quarantine annotations at either class or method level.

Test ClassTest MethodSuccess RateTotal RunsRecent Status
kafka.integration.MetricsDuringTopicCreationDeletionTest
Class Overall100.00%7777 passed
testMetricsDuringTopicCreateDelete(String)[1]99.22%129passed
 
kafka.api.PlaintextConsumerSubscriptionTest
Class Overall99.80%509508 passed
testRe2JPatternExpandSubscription(String, String)[1]99.82%569passed
 
kafka.api.PlaintextAdminIntegrationTest
Class Overall99.61%517515 passed
testConsumerGroupWithMemberMigration(String)[1]99.65%575passed
 
kafka.api.SslAdminIntegrationTest
Class Overall99.60%251250 passed
testListNodesFromControllersIncludingFencedBrokers(String)[1]99.67%301passed
 
org.apache.kafka.streams.AutoOffsetResetTest
Class Overall99.48%382380 passed
shouldThrowExceptionOnDurationForEarliestReset()99.51%410passed
shouldThrowExceptionOnDurationForLastetReset()99.51%410passed
shouldThrowExceptionOnDurationForNoneReset()99.51%410passed
 
org.apache.kafka.streams.kstream.internals.KStreamImplTest
Class Overall99.45%362360 passed
shouldProcessWithProcessorAndState()99.51%410passed
 
org.apache.kafka.clients.producer.internals.TransactionManagerTest
Class Overall99.43%352350 passed
testTransactionV2AddPartitionAndOffsets()99.44%358passed
testMaybeAddPartitionToTransactionInTransactionV2()99.44%358passed
testFatalErrorWhenProduceResponseWithInvalidPidMapping()99.44%358passed
 
kafka.api.SslConsumerTest
Class Overall99.43%522519 passed
testCoordinatorFailover(String, String)[1]99.83%582passed
testCoordinatorFailover(String, String)[2]99.66%582passed
 
org.apache.kafka.streams.StreamsBuilderTest
Class Overall99.24%524520 passed
shouldWrapProcessorsForStreamStreamSelfJoinWithSharedStoreOptimization()99.39%327passed
 
kafka.api.SaslSslConsumerTest
Class Overall98.85%524518 passed
testCoordinatorFailover(String, String)[1]98.97%584passed
testCoordinatorFailover(String, String)[2]99.83%584passed
 
org.apache.kafka.clients.consumer.internals.ApplicationEventHandlerTest
Class Overall98.64%221218 passed
testRecordApplicationEventQueueSize()98.53%273passed
 
org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorderTest
Class Overall98.54%137135 passed
shouldThrowIfMetricRecorderIsInitialisedWithNullTaskId()98.94%189passed
shouldThrowIfMetricRecorderIsInitialisedWithNullMetrics()98.94%189passed
 
org.apache.kafka.controller.QuorumControllerTest
Class Overall98.53%273269 passed
testUncleanShutdownElrDisabled()99.64%277passed
testUncleanShutdownBrokerElrEnabled()98.92%277passed
 
kafka.test.api.ShareConsumerTest
Class Overall98.47%522514 passed
testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[1]99.83%582passed
testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[2]99.66%582passed
testAcknowledgementCommitCallbackInvalidRecordStateException(String)[1]99.14%582passed
testMultipleConsumersInGroupConcurrentConsumption(String)[1]99.83%582passed
testExplicitAcknowledgementCommitAsync(String)[1]99.83%582passed
 
org.apache.kafka.streams.integration.FineGrainedAutoResetIntegrationTest
Class Overall97.75%356348 passed
shouldResetByDuration()99.03%412passed
shouldFailForResetNone()99.03%412passed
 
kafka.api.SaslMultiMechanismConsumerTest
Class Overall96.95%524508 passed
testCoordinatorFailover(String, String)[1]97.77%584passed
testCoordinatorFailover(String, String)[2]98.97%584passed
 
org.apache.kafka.connect.integration.OffsetsApiIntegrationTest
Class Overall96.37%524505 passed
testResetSinkConnectorOffsetsOverriddenConsumerGroupId()98.80%584passed
testResetSinkConnectorOffsets()98.80%584passed
testAlterSinkConnectorOffsetsOverriddenConsumerGroupId()99.32%584passed
testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted()98.29%584passed
testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted()97.60%584passed
 
kafka.network.DynamicConnectionQuotaTest
Class Overall95.59%522499 passed
testDynamicConnectionQuota(String)[1]95.53%582passed
 
kafka.api.SaslPlaintextConsumerTest
Class Overall94.65%523495 passed
testCoordinatorFailover(String, String)[1]99.83%583passed
testCoordinatorFailover(String, String)[2]94.51%583passed
 
kafka.api.PlaintextConsumerTest
Class Overall84.16%524441 passed
testCoordinatorFailover(String, String)[2]96.92%584passed
testCoordinatorFailover(String, String)[1]97.77%584passed
testCloseLeavesGroupOnInterrupt(String, String)[1]97.60%584passed
testCloseLeavesGroupOnInterrupt(String, String)[2]90.41%584passed
 
Detailed Test Method History

kafka.integration.MetricsDuringTopicCreationDeletionTest

  • Overall Success Rate: 100.00%
  • Total Executions: 77
  • Consecutive Successful Runs: 77

testMetricsDuringTopicCreateDelete(String)[1]

  • Success Rate: 99.22%
  • Total Runs: 129

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 01:56  passed     jftbd3tqqv23e
2024-12-17 02:41  passed     kbajkeckvufya
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4

kafka.api.PlaintextConsumerSubscriptionTest

  • Overall Success Rate: 99.80%
  • Total Executions: 509
  • Consecutive Successful Runs: 508

testRe2JPatternExpandSubscription(String, String)[1]

  • Success Rate: 99.82%
  • Total Runs: 569

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.PlaintextAdminIntegrationTest

  • Overall Success Rate: 99.61%
  • Total Executions: 517
  • Consecutive Successful Runs: 515

testConsumerGroupWithMemberMigration(String)[1]

  • Success Rate: 99.65%
  • Total Runs: 575

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.SslAdminIntegrationTest

  • Overall Success Rate: 99.60%
  • Total Executions: 251
  • Consecutive Successful Runs: 250

testListNodesFromControllersIncludingFencedBrokers(String)[1]

  • Success Rate: 99.67%
  • Total Runs: 301

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.streams.AutoOffsetResetTest

  • Overall Success Rate: 99.48%
  • Total Executions: 382
  • Consecutive Successful Runs: 380

shouldThrowExceptionOnDurationForEarliestReset()

  • Success Rate: 99.51%
  • Total Runs: 410

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

shouldThrowExceptionOnDurationForLastetReset()

  • Success Rate: 99.51%
  • Total Runs: 410

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

shouldThrowExceptionOnDurationForNoneReset()

  • Success Rate: 99.51%
  • Total Runs: 410

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.streams.kstream.internals.KStreamImplTest

  • Overall Success Rate: 99.45%
  • Total Executions: 362
  • Consecutive Successful Runs: 360

shouldProcessWithProcessorAndState()

  • Success Rate: 99.51%
  • Total Runs: 410

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.clients.producer.internals.TransactionManagerTest

  • Overall Success Rate: 99.43%
  • Total Executions: 352
  • Consecutive Successful Runs: 350

testTransactionV2AddPartitionAndOffsets()

  • Success Rate: 99.44%
  • Total Runs: 358

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-14 03:24  passed     kxmtr5vf6jjos
2024-12-14 03:24  passed     2uthfvry5onu2
2024-12-14 14:27  passed     673h4uvqqp43i
2024-12-15 01:15  passed     657l6cmibrueo
2024-12-15 01:15  passed     sfttfmn2pt6am

testMaybeAddPartitionToTransactionInTransactionV2()

  • Success Rate: 99.44%
  • Total Runs: 358

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-14 03:24  passed     kxmtr5vf6jjos
2024-12-14 03:24  passed     2uthfvry5onu2
2024-12-14 14:27  passed     673h4uvqqp43i
2024-12-15 01:15  passed     657l6cmibrueo
2024-12-15 01:15  passed     sfttfmn2pt6am

testFatalErrorWhenProduceResponseWithInvalidPidMapping()

  • Success Rate: 99.44%
  • Total Runs: 358

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-14 03:24  passed     kxmtr5vf6jjos
2024-12-14 03:24  passed     2uthfvry5onu2
2024-12-14 14:27  passed     673h4uvqqp43i
2024-12-15 01:15  passed     657l6cmibrueo
2024-12-15 01:15  passed     sfttfmn2pt6am

kafka.api.SslConsumerTest

  • Overall Success Rate: 99.43%
  • Total Executions: 522
  • Consecutive Successful Runs: 519

testCoordinatorFailover(String, String)[1]

  • Success Rate: 99.83%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[2]

  • Success Rate: 99.66%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.streams.StreamsBuilderTest

  • Overall Success Rate: 99.24%
  • Total Executions: 524
  • Consecutive Successful Runs: 520

shouldWrapProcessorsForStreamStreamSelfJoinWithSharedStoreOptimization()

  • Success Rate: 99.39%
  • Total Runs: 327

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.SaslSslConsumerTest

  • Overall Success Rate: 98.85%
  • Total Executions: 524
  • Consecutive Successful Runs: 518

testCoordinatorFailover(String, String)[1]

  • Success Rate: 98.97%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[2]

  • Success Rate: 99.83%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.clients.consumer.internals.ApplicationEventHandlerTest

  • Overall Success Rate: 98.64%
  • Total Executions: 221
  • Consecutive Successful Runs: 218

testRecordApplicationEventQueueSize()

  • Success Rate: 98.53%
  • Total Runs: 273

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorderTest

  • Overall Success Rate: 98.54%
  • Total Executions: 137
  • Consecutive Successful Runs: 135

shouldThrowIfMetricRecorderIsInitialisedWithNullTaskId()

  • Success Rate: 98.94%
  • Total Runs: 189

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 01:56  passed     jftbd3tqqv23e
2024-12-17 02:41  passed     kbajkeckvufya
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4

shouldThrowIfMetricRecorderIsInitialisedWithNullMetrics()

  • Success Rate: 98.94%
  • Total Runs: 189

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 01:56  passed     jftbd3tqqv23e
2024-12-17 02:41  passed     kbajkeckvufya
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4

org.apache.kafka.controller.QuorumControllerTest

  • Overall Success Rate: 98.53%
  • Total Executions: 273
  • Consecutive Successful Runs: 269

testUncleanShutdownElrDisabled()

  • Success Rate: 99.64%
  • Total Runs: 277

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-14 03:24  passed     kxmtr5vf6jjos
2024-12-14 03:24  passed     2uthfvry5onu2
2024-12-14 14:27  passed     673h4uvqqp43i
2024-12-15 01:15  passed     657l6cmibrueo
2024-12-15 01:15  passed     sfttfmn2pt6am

testUncleanShutdownBrokerElrEnabled()

  • Success Rate: 98.92%
  • Total Runs: 277

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-14 03:24  passed     kxmtr5vf6jjos
2024-12-14 03:24  passed     2uthfvry5onu2
2024-12-14 14:27  passed     673h4uvqqp43i
2024-12-15 01:15  passed     657l6cmibrueo
2024-12-15 01:15  passed     sfttfmn2pt6am

kafka.test.api.ShareConsumerTest

  • Overall Success Rate: 98.47%
  • Total Executions: 522
  • Consecutive Successful Runs: 514

testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[1]

  • Success Rate: 99.83%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testAcknowledgementCommitCallbackCallsShareConsumerWakeup(String)[2]

  • Success Rate: 99.66%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testAcknowledgementCommitCallbackInvalidRecordStateException(String)[1]

  • Success Rate: 99.14%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testMultipleConsumersInGroupConcurrentConsumption(String)[1]

  • Success Rate: 99.83%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testExplicitAcknowledgementCommitAsync(String)[1]

  • Success Rate: 99.83%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.streams.integration.FineGrainedAutoResetIntegrationTest

  • Overall Success Rate: 97.75%
  • Total Executions: 356
  • Consecutive Successful Runs: 348

shouldResetByDuration()

  • Success Rate: 99.03%
  • Total Runs: 412

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

shouldFailForResetNone()

  • Success Rate: 99.03%
  • Total Runs: 412

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.SaslMultiMechanismConsumerTest

  • Overall Success Rate: 96.95%
  • Total Executions: 524
  • Consecutive Successful Runs: 508

testCoordinatorFailover(String, String)[1]

  • Success Rate: 97.77%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[2]

  • Success Rate: 98.97%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

org.apache.kafka.connect.integration.OffsetsApiIntegrationTest

  • Overall Success Rate: 96.37%
  • Total Executions: 524
  • Consecutive Successful Runs: 505

testResetSinkConnectorOffsetsOverriddenConsumerGroupId()

  • Success Rate: 98.80%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testResetSinkConnectorOffsets()

  • Success Rate: 98.80%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testAlterSinkConnectorOffsetsOverriddenConsumerGroupId()

  • Success Rate: 99.32%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted()

  • Success Rate: 98.29%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testAlterSinkConnectorOffsetsDifferentKafkaClusterTargeted()

  • Success Rate: 97.60%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.network.DynamicConnectionQuotaTest

  • Overall Success Rate: 95.59%
  • Total Executions: 522
  • Consecutive Successful Runs: 499

testDynamicConnectionQuota(String)[1]

  • Success Rate: 95.53%
  • Total Runs: 582

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.SaslPlaintextConsumerTest

  • Overall Success Rate: 94.65%
  • Total Executions: 523
  • Consecutive Successful Runs: 495

testCoordinatorFailover(String, String)[1]

  • Success Rate: 99.83%
  • Total Runs: 583

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[2]

  • Success Rate: 94.51%
  • Total Runs: 583

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

kafka.api.PlaintextConsumerTest

  • Overall Success Rate: 84.16%
  • Total Executions: 524
  • Consecutive Successful Runs: 441

testCoordinatorFailover(String, String)[2]

  • Success Rate: 96.92%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCoordinatorFailover(String, String)[1]

  • Success Rate: 97.77%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCloseLeavesGroupOnInterrupt(String, String)[1]

  • Success Rate: 97.60%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

testCloseLeavesGroupOnInterrupt(String, String)[2]

  • Success Rate: 90.41%
  • Total Runs: 584

Recent Executions:

Date/Time (UTC)      Outcome    Build ID
--------------------------------------------
2024-12-17 02:41  passed     ypf6siwbjjev2
2024-12-17 05:05  passed     n3v7qodjqjces
2024-12-17 05:05  passed     ie6eoylrybum4
2024-12-17 05:10  passed     tesh2oqgxr5qu
2024-12-17 05:10  passed     dmojogsx5lrb4

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@github-actions github-actions bot added triage PRs from the community build Gradle build or GitHub Actions labels Dec 16, 2024
@mumrah
Copy link
Member

mumrah commented Dec 16, 2024

@santhoshct some initial notes:

  1. In Cleared Tests, just include the test class and test case lines.
  2. See if we can move the detailed info (with recent builds) into a collapsible section (<summary><detail> or something?)
  3. Add a link to Develocity for test classes and test cases.

Thanks for continuing work on this!

@santhoshct
formatted the detailed info into collapsible section. added develocit…
bd1d83b 
…y links to the test classes and test methods
@santhoshct
Copy link
Contributor Author

All three successfully done.

@santhoshct santhoshct marked this pull request as ready for review December 16, 2024 17:06
@mumrah
Copy link
Member

mumrah commented Dec 16, 2024

This looks excellent! The links will be super helpful.

Looks like the "High-Priority Quarantined Tests" table is mostly redundant with the summary table at the top. Actually, with the collapsed details sections, I think we can restructure things a little bit.

How about the following layout:

Most Problematic Tests

<1 sentence text description>

<table>

<summary><details>

---

Flaky Regressions

<1 sentence text description>

<table>

<summary><details>

---

Cleared Tests

<1 sentence text description>

<table>

<summary><details>

I think this will give the report a fairly uniform feel and make it a bit more intuitive to navigate.

@mumrah mumrah removed the triage PRs from the community label Dec 16, 2024
@santhoshct
Copy link
Contributor Author

I was also thinking about removing the high priority quarantine test report. the layout looks uniform. I have added it and updated the results in PR description.

mumrah
mumrah reviewed Dec 17, 2024
View reviewed changes
Copy link
Member

@mumrah mumrah left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@santhoshct I ran this locally and copied it here https://gist.github.com/mumrah/b476eb6abe07eb37dd297078e39ebbe0

I'd like to see each flaky method for a given test class (like how the PR description looks). I probably should have been more specific with my suggestion yesterday, but in the tables I think we have to have the test cases in there. The only thing I consider too detailed for the tables is the build scan data.

The new layout looks excellent. Just need to add the test cases back inline to the tables.

edit: btw I also notice since we've hyperlinked the whole class/test name, it makes selecting the text tricky. Let's create a column with a clickable link, e.g. "develocity ↗️"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mumrah mumrah mumrah left review comments

Assignees
No one assigned
Labels
build Gradle build or GitHub Actions
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@santhoshct @mumrah