Skip to content

fix: mark flaky tests #21300

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Sep 30, 2020
Merged

fix: mark flaky tests #21300

merged 5 commits into from
Sep 30, 2020

Conversation

@kuisathaverat
Copy link
Contributor

@kuisathaverat kuisathaverat commented Sep 24, 2020 •
edited
Loading

What does this PR do?

It marks a bunch of tests as flaky based on the latest 21 builds.

Why is it important?

We try to move the master branch to successful builds, we can not trust on a build that contains flaky and broken tests

Checklist

  • [ ] My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

closes #21260

@kuisathaverat kuisathaverat added flaky-test Unstable or unreliable test cases. automation Team:Automation Label for the Observability productivity team labels Sep 24, 2020
@kuisathaverat kuisathaverat requested a review from a team as a code owner September 24, 2020 15:49
@kuisathaverat kuisathaverat self-assigned this Sep 24, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 24, 2020
@kuisathaverat kuisathaverat changed the title fix: mark flaky test fix: mark flaky tests Sep 24, 2020
kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
@mdelapenya
Copy link
Contributor

mdelapenya commented Sep 24, 2020 •
edited
Loading

I'd like to bring to the table the importance of stopping machines and fixing the "flaky" tests. A test could be flaky for many reasons: CI slowness, CI setup, or even an undetected race condition.

I encourage the team to stop machines and debug the tests before sending new code. Otherwise the build will be broken for ever

kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 24, 2020
View reviewed changes
@kuisathaverat kuisathaverat mentioned this pull request Sep 24, 2020
Closed
@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 24, 2020 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21300 updated]

  • Start Time: 2020-09-29T11:20:39.219+0000

  • Duration: 94 min 19 sec

Test stats 🧪

Test Results
Failed 0
Passed 20261
Skipped 1862
Total 22123

jsoriano
jsoriano requested changes Sep 24, 2020
View reviewed changes
jsoriano
jsoriano reviewed Sep 24, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 29, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 29, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 29, 2020
View reviewed changes
kuisathaverat
kuisathaverat commented Sep 29, 2020
View reviewed changes
jsoriano
jsoriano approved these changes Sep 29, 2020
View reviewed changes
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@kuisathaverat kuisathaverat merged commit 3deda87 into elastic:master Sep 30, 2020
v1v added a commit to v1v/beats that referenced this pull request Sep 30, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/2.0-skip-…
cb2bd71 
…ci-build-label-support

* upstream/master:
  [JJBB] Set shallow cloning to 10 (elastic#21409)
  docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419)
  docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403)
  fix: mark flaky tests (elastic#21300)
  fix: use a fixed version of setuptools (elastic#21393)
  Move Kubernetes events metricset to its own block in reference config (elastic#21407)
  [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391)
  docs: fix apt/yum formatting (elastic#21362)
  Fix shutdown tracking in s3 input (elastic#21380)
  [libbeat] Fix position writing in the disk queue
  Add UBI 8 image to the dependencies report (elastic#21374)
  Fix debug message to show actual SQS message ID (elastic#20614)
  [Elastic Agent] Rename *ConfigChange to PolicyChange (elastic#20779)
  [Elastic Agent] Add install/uninstall sub-command (elastic#21206)
  [Filebeat][httpjson] Make httpjson use cursor input when using date cursor (elastic#20751)
  feat: prepare release pipelines (elastic#21238)
  Add IP validation to Security module (elastic#21325)
v1v added a commit to v1v/beats that referenced this pull request Oct 2, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/windows-2016
ff9a47c 
* upstream/master: (27 commits)
  [Ingest Manager] Split index restrictions into type,dataset, namespace parts (elastic#21406)
  Update Filebeat module expected logs files (elastic#21454)
  Edit SQL module docs and fix broken doc structure (elastic#21233)
  [Ingest Manager] Send snapshot flag together with metadata (elastic#21285)
  Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)
  [JJBB] Use reference repo for fast checkouts (elastic#21410)
  Add initial skeleton of filestream input (elastic#21427)
  Initial spec file for apm-server (elastic#21225)
  [Ingest Manager] Upgrade Action: make source URI optional (elastic#21372)
  Add field limit check for AWS Cloudtrail flattened fields (elastic#21388)
  [Winlogbeat] Move winlogbeat javascript processor to libbeat (elastic#21402)
  ci: pipeline to generate the changelog (elastic#21426)
  [JJBB] Set shallow cloning to 10 (elastic#21409)
  docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419)
  docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403)
  fix: mark flaky tests (elastic#21300)
  fix: use a fixed version of setuptools (elastic#21393)
  Move Kubernetes events metricset to its own block in reference config (elastic#21407)
  [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391)
  docs: fix apt/yum formatting (elastic#21362)
  ...
@jsoriano
Copy link
Member

jsoriano commented Oct 19, 2020

@kuisathaverat could you please backport these changes to 7.x, 7.10 and 7.9 branches? They seem to be affected by tests skipped here https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats/detail/7.9/223/tests
Thanks!

@kuisathaverat kuisathaverat mentioned this pull request Oct 20, 2020
Merged
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Oct 20, 2020
@kuisathaverat
fix: mark flaky tests (elastic#21300)
b553ce0 
* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review
@kuisathaverat kuisathaverat mentioned this pull request Oct 20, 2020
Merged
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Oct 20, 2020
@kuisathaverat
fix: mark flaky tests (elastic#21300)
316a358 
* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Oct 20, 2020
@kuisathaverat
fix: mark flaky tests (elastic#21300)
8c7bc5c 
* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review
@kuisathaverat kuisathaverat mentioned this pull request Oct 20, 2020
Merged
kuisathaverat added a commit that referenced this pull request Oct 20, 2020
@kuisathaverat
Cherry-pick to 7.9: fix: mark flaky tests (#21300) (#22008)
972b507 
* fix: mark flaky tests (#21300)

* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* fix: skip python tests too
kuisathaverat added a commit that referenced this pull request Oct 21, 2020
@kuisathaverat
fix: mark flaky tests (#21300) (#22007)
679f599 
* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review
kuisathaverat added a commit that referenced this pull request Oct 21, 2020
@kuisathaverat
fix: mark flaky tests (#21300) (#22006)
74c7ef9 
* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review
@kuisathaverat kuisathaverat deleted the all_flaky_tests_in_a_nutshell branch December 24, 2020 11:06
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@kuisathaverat
Cherry-pick to 7.9: fix: mark flaky tests (elastic#21300) (elastic#22008
19dad39 
)

* fix: mark flaky tests (elastic#21300)

* fix: mark flaky test

* Apply suggestions from code review

* Update libbeat/outputs/elasticsearch/client_integration_test.go

* Apply suggestions from code review

* Apply suggestions from code review

* Apply suggestions from code review

* fix: skip python tests too
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsoriano jsoriano jsoriano approved these changes

@v1v v1v v1v approved these changes

@urso urso Awaiting requested review from urso

@andresrc andresrc Awaiting requested review from andresrc

@andrewkroh andrewkroh Awaiting requested review from andrewkroh

+1 more reviewer

@mdelapenya mdelapenya mdelapenya approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@kuisathaverat kuisathaverat

Labels

automation flaky-test Unstable or unreliable test cases. Team:Automation Label for the Observability productivity team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Mark flaky tests for retry 5 times

5 participants

@kuisathaverat @mdelapenya @elasticmachine @jsoriano @v1v