-
Notifications
You must be signed in to change notification settings - Fork 489
chore: enable shuffle testing for most core packages #4061
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: enable shuffle testing for most core packages #4061
Conversation
BenchmarksBenchmark execution time: 2025-10-22 16:11:33 Comparing candidate commit 685c422 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 metrics, 0 unstable metrics. |
4e8aace to
74f17d8
Compare
This comment has been minimized.
This comment has been minimized.
Can't reproduce it locally, but flaked in CI as well.
55b0546 to
3265355
Compare
nsrip-dd
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I confirmed that the tests are split up appropriately in the CI run: https://github.com/DataDog/dd-trace-go/actions/runs/18722222366/job/53397550605?pr=4061#step:6:237
I also confirmed locally that gotestsum will log the shuffle seed if the tests fail:
% gotestsum -- -v -shuffle=on ./profiler
✖ profiler (21.383s) (-test.shuffle 1761161421624258000)
=== Skipped
=== SKIP: profiler TestDebugCompressionEnv (0.00s)
compression_test.go:77: Flaky. See #3681
=== Failed
=== FAIL: profiler TestFail (0.00s)
profiler_test.go:948: fail!
DONE 152 tests, 1 skipped, 1 failure in 21.383s
Thanks for doing this!
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What does this PR do?
Enable test shuffling for most of our core packages.
Packages that were unable to successfully pass their tests with
-shuffle onfor 10 times in a row (on my local machine) have been excluded. See #4060 for details.Additionally the following packages were excluded because they flaked in CI (this may or may not be due to shuffle):
The plan is to enable shuffling for the excluded packages in the future once their test suite has been fixed. Enabling shuffle testing for contrib packages will also be done separately.
If you see test flakes, and suspect that they are caused by this PR, please try to fix them instead of adding the packages to the shuffle exclusion list.
PROF-12813
Motivation
Some of our tests have come to rely on global state pollution over time. I.e. they need another test to run first in order to pass, or rely on another test not running first in order to pass. This is undesirable as it often causes problems for people debugging test failures (e.g. test passes when
go test -run TestXXXbut not when running all tests in the pkg). It may also hide implementation bugs.Reviewer's Checklist
./scripts/lint.shlocally.Unsure? Have a question? Request a review!