Skip to content

KAFKA-8164: Improve test passing rate by rerunning flaky tests #8019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ijuma merged 8 commits into from
Feb 6, 2020
Merged

KAFKA-8164: Improve test passing rate by rerunning flaky tests #8019

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
da14627
KAFKA-8164: Improve test passing rate by rerunning flaky tests
viktorsomogyi Jan 28, 2020
96baf78
Lower defaults, add configurable properties
viktorsomogyi Jan 31, 2020
9891b23
Update readme, correct namings
viktorsomogyi Feb 3, 2020
6d9841b
Add new options to common build options
viktorsomogyi Feb 5, 2020
0b3ac63
Update testRetry to 1.1.0
ijuma Feb 5, 2020
e2db38b
Adjust readme
ijuma Feb 5, 2020
0e47aa0
Mention that Readme should be updated if defaults are adjusted
ijuma Feb 5, 2020
078bc69
Change the defaults to 0.
ijuma Feb 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ buildscript {
classpath "org.owasp:dependency-check-gradle:$versions.owaspDepCheckPlugin"
classpath "com.diffplug.spotless:spotless-plugin-gradle:$versions.spotlessPlugin"
classpath "com.github.spotbugs:spotbugs-gradle-plugin:$versions.spotbugsPlugin"
classpath "org.gradle:test-retry-gradle-plugin:$versions.testRetryPlugin"
}
}

Expand Down Expand Up @@ -163,6 +164,7 @@ subprojects {
apply plugin: 'signing'
apply plugin: 'checkstyle'
apply plugin: "com.github.spotbugs"
apply plugin: 'org.gradle.test-retry'

sourceCompatibility = minJavaVersion
targetCompatibility = minJavaVersion
Expand Down Expand Up @@ -289,6 +291,11 @@ subprojects {
exceptionFormat = testExceptionFormat
}
logTestStdout.rehydrate(delegate, owner, this)()

retry {
maxRetries = 3
maxFailures = 50

@ijuma ijuma Jan 29, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would go with something conservative: 1 retry and max 5 failures.We want to fix flaky tests, but if something just fails once, it's not worth failing the build given Jenkins flakiness.

@ijuma ijuma Jan 29, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can adjust after we see how it works in practice, but this would be a good starting point IMO.

@viktorsomogyi viktorsomogyi Jan 31, 2020

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I figured we can also make these configurable. Updated the PR.

}
}

task integrationTest(type: Test, dependsOn: compileJava) {
Expand Down
1 change: 1 addition & 0 deletions gradle/dependencies.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ versions += [
spotbugs: "3.1.12",
spotbugsPlugin: "3.0.0",
spotlessPlugin: "3.27.1",
testRetryPlugin: "1.0.2",

@chia7712 chia7712 Feb 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using latest version (1.1.0)?

@ijuma ijuma Feb 5, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also noticed it was released shortly after this PR was submitted. Updated.

zookeeper: "3.5.6",
zstd: "1.4.4-7"
]
Expand Down