Skip to content

Deflake TestChaosUpload#33586

Merged
zmb3 merged 1 commit intomasterfrom
zmb3/chaos-upload-test
Oct 17, 2023
Merged

Deflake TestChaosUpload#33586
zmb3 merged 1 commit intomasterfrom
zmb3/chaos-upload-test

Conversation

@zmb3
Copy link
Copy Markdown
Collaborator

@zmb3 zmb3 commented Oct 17, 2023

This test attempted to verify that the uploader is able to operate correctly even when faults are injected into the system.

It ended up being flaky for a number of reasons, but the biggest issue is that it wasn't simulating a real world scenario.

  1. It used a fake clock, but never advanced it, so the uploader's scan loop didn't correctly "tick" and only ran once.
  2. It invoked many uploader scans in parallel while also running the uploader's own scan loop in the background. In any real deployment we only have one scan loop running, so this test is creating an environment that never exists in the wild.

Closes #33099

@github-actions github-actions Bot added audit-log Issues related to Teleports Audit Log size/sm labels Oct 17, 2023
This test attempted to verify that the uploader is able to operate
correctly even when faults are injected into the system.

It ended up being flaky for a number of reasons, but the biggest
issue is that it wasn't simulating a real world scenario.

1. It used a fake clock, but never advanced it, so the uploader's
   scan loop didn't correctly "tick" and only ran once.
2. It invoked many uploader scans in parallel while also running
   the uploader's own scan loop in the background. In any real
   deployment we only have one scan loop running, so this test
   is creating an environment that never exists in the wild.

Updates #33099
@zmb3 zmb3 force-pushed the zmb3/chaos-upload-test branch from 20dd1cc to 1bf747a Compare October 17, 2023 20:27
@zmb3 zmb3 enabled auto-merge October 17, 2023 20:27
@zmb3 zmb3 added this pull request to the merge queue Oct 17, 2023
Merged via the queue into master with commit faa9f2d Oct 17, 2023
@zmb3 zmb3 deleted the zmb3/chaos-upload-test branch October 17, 2023 21:10
@public-teleport-github-review-bot
Copy link
Copy Markdown

@zmb3 See the table below for backport results.

Branch Result
branch/v12 Failed
branch/v13 Create PR
branch/v14 Create PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

audit-log Issues related to Teleports Audit Log size/sm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestChaosUpload flakiness

3 participants