Aggressively reload A/B tests during spec runs#11693
Merged
Conversation
Tests can get stuck with mocked configs, leading to flakiness. [skip changelog]
Sgtpluck
approved these changes
Dec 31, 2024
Contributor
Sgtpluck
left a comment
There was a problem hiding this comment.
does this add time onto the spec run? (am curious if there's a way to benchmark that.)
Contributor
Author
|
Based on a quick test on my laptop, 1000 A/B test reloads execute in less than 1 second: it 'tests how fast reloading is' do
start = Time.zone.now
1000.times do
reload_ab_tests
end
expect(Time.zone.now - start).to be < 1.second
endSo I don't think this will add any appreciable time to CI runs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary of changes
TLDR: This PR updates our RSpec config to reload A/B tests after each spec file in a test run. This is because by default A/B tests are initialized once and only once, and this makes it possible for them to "hold on" to mocked configuration data, potentially leading to flakiness.
Previously in #11583 I added a cleanup hook to the
ab_tests_spec.rbfile, but there are two problems with this:Thus, this PR adds the cleanup hook to the main RSpec config. I'm adding it as an
after :contextblock, which here means "after each test file". We could theoretically run the same thing after each example, but that is probably overkill.📜 Testing Plan
To see an example of this causing a problem on
main, run:bundle exec rspec --seed 4734 --fail-fast \ spec/config/initializers/ab_tests_spec.rb \ spec/features/saml/ial2_sso_spec.rbial2_sso_spec.rbwill fail.Switch to this branch, and the same invocation will pass.