Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CANARY] Disable test retries #3189

Closed
wants to merge 1 commit into from

Conversation

apostasie
Copy link
Contributor

@apostasie apostasie commented Jul 5, 2024

IMHO we cannot keep on hiding problems by "retrying tests" until it works.
This is a doomed approach.

I perfectly understand how painful it is to go and debug these flakyness / concurrency issues, and that it is much easier to ignore the problem - but we have to solve these - they are definitely indicative of deep quality issues and the only way to do so is to make it as obvious as possible instead of hiding it under the retry-carpet.

This is a canary PR that will very likely keep failing for some time - until we fix ALL these issues.

I will rebase it against main daily, and eventually we should get it green and stable.

Finally note that this canary is usually AHEAD of main - using commits from pending PRs fixing flakyness

@apostasie apostasie mentioned this pull request Jul 5, 2024
3 tasks
@apostasie
Copy link
Contributor Author

Of course, please do not restart any of these manually.
It should be green on the first try immediately after push.

@apostasie apostasie force-pushed the dev-ci-kill-retry branch 5 times, most recently from 3eae9de to ef46b8f Compare July 6, 2024 16:24
@apostasie apostasie force-pushed the dev-ci-kill-retry branch 6 times, most recently from 2d0d19f to 320016c Compare July 9, 2024 17:42
@apostasie apostasie mentioned this pull request Jul 9, 2024
@apostasie apostasie force-pushed the dev-ci-kill-retry branch 4 times, most recently from 9ee9715 to d6747eb Compare July 16, 2024 18:57
@apostasie
Copy link
Contributor Author

Green again, but #3225 is still a clear issue before we can consider merging this.

@apostasie apostasie force-pushed the dev-ci-kill-retry branch 2 times, most recently from 10d0659 to 01713d3 Compare August 1, 2024 01:58
@apostasie apostasie force-pushed the dev-ci-kill-retry branch 3 times, most recently from 064f038 to 479e2f4 Compare August 9, 2024 17:54
@apostasie
Copy link
Contributor Author

08/09/2024:

  • TestLogsWithForegroundContainers
  • TestIPFSComposeUp

Run: https://github.com/containerd/nerdctl/actions/runs/10323534574/job/28581121980

The logs one is perplexing, but far from being the only log problem in CI. We should really have a look at logs overall.

The IPFS one is also happening often - use of closed network connection may suggest low quality code / failure to handle errors? IPFS overall has been very flaky so same here, a serious rewrite might be in order.

@apostasie apostasie marked this pull request as ready for review August 13, 2024 04:15
Signed-off-by: apostasie <[email protected]>
@apostasie
Copy link
Contributor Author

Closing in favor of #3305

@apostasie apostasie closed this Aug 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant