Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ci): retry flaky t0125-twonode.sh #9486

Merged
merged 1 commit into from
Jan 5, 2023
Merged

Conversation

lidel
Copy link
Member

@lidel lidel commented Dec 8, 2022

This PR is a sidequest to make is clear why flaky test fails:

expecting success:
    ipfsi 0 bitswap stat > stat0 &&
    test_should_contain "blocks sent: 126" stat0 &&
    test_should_contain "blocks received: 5" stat0 &&
    test_should_contain "data sent: 228113" stat0 &&
    test_should_contain "data received: 1000256" stat0

'stat0' does not contain 'blocks sent: 126', it contains:
bitswap status
	provides buffer: 0 / 256
	blocks received: 5
	blocks sent: 134
	data received: 1000256
	data sent: 231966
	dup blocks received: 0
	dup data received: 0
	wantlist [0 keys]
	partners [1]
not ok 31 - node0 data transferred looks correct

Seems this was always flaky, everyone just learned to re-run it.
I had no bandwidth to debug, but to save us all time, I made it retry until success or hits 60s timeout.

TODO

  • Fixes flaky test: It will re-run flaky advanced test until bitswap stats match expected value (something team has been doing anyway for the past year) or 60s timeout is reached.
  • adds /quic-v1 and /webtransport tests

@lidel lidel self-assigned this Dec 8, 2022
@Jorropo
Copy link
Contributor

Jorropo commented Dec 9, 2022

@lidel I had issues with this test when taking care of #9423, so I merged this pr into #9423.

Are you good with that or is there a reason this PR is draft ?

@Jorropo
Copy link
Contributor

Jorropo commented Dec 9, 2022

NVM I'll just run #9423 until it works.

@lidel
Copy link
Member Author

lidel commented Dec 11, 2022

This is draft because it is still flaky – only added verbosity, so we know why it fails.
Had no bandwidth to investigate fix.

@lidel lidel force-pushed the fix/flaky-twonode-sharness branch 5 times, most recently from a6d1d2f to 7da6d99 Compare January 5, 2023 20:04
@lidel lidel changed the title test: fix flaky t0125-twonode.sh fix(ci): retry flaky t0125-twonode.sh Jan 5, 2023
@lidel
Copy link
Member Author

lidel commented Jan 5, 2023

Seems this was always flaky, everyone on the team just learned to re-run CircleCI 🙃
I had no bandwidth to debug, and did not want to remove it, so to save us all time, I'll make it retry until success or hits timeout.

@lidel lidel marked this pull request as ready for review January 5, 2023 20:45
@lidel lidel marked this pull request as draft January 5, 2023 20:59
This makes is clear why test failed, and what were values.

Fixes flaky test: It will re-run flaky advanced test until bitswap stats
match expected value (something team has been doing anyway for the past
year).

It also adds /quic-v1 and /webtransport tests
@lidel lidel force-pushed the fix/flaky-twonode-sharness branch from 7da6d99 to f6ce21f Compare January 5, 2023 21:40
@lidel lidel marked this pull request as ready for review January 5, 2023 21:56
@lidel lidel merged commit 73e6ade into master Jan 5, 2023
@lidel lidel deleted the fix/flaky-twonode-sharness branch January 5, 2023 21:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants