[Uptime] [Test] Repurpose unit test assertions to avoid flakiness#40650
Merged
andrewvc merged 4 commits intoelastic:masterfrom Aug 8, 2019
Merged
Conversation
justinkambic
added
test
v8.0.0
Team:Uptime - DEPRECATED
Contributor
|
Pinging @elastic/uptime |
justinkambic
changed the title
Contributor
💚 Build Succeeded |
justinkambic
force-pushed
the
uptime_es-monitors-flaky-unit-test
branch
from
666b7be to
7126990
Compare
Contributor
💚 Build Succeeded |
andrewvc
reviewed
Jul 12, 2019
...plugins/uptime/server/lib/adapters/monitors/__tests__/elasticsearch_monitors_adapter.test.ts
Outdated
Show resolved
Hide resolved
andrewvc
reviewed
Jul 12, 2019
...plugins/uptime/server/lib/adapters/monitors/__tests__/elasticsearch_monitors_adapter.test.ts
Outdated
Show resolved
Hide resolved
spalger
suggested changes
Jul 12, 2019
...plugins/uptime/server/lib/adapters/monitors/__tests__/elasticsearch_monitors_adapter.test.ts
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks for the feedback, I'll ping y'all when I come back around to this. |
Contributor
💚 Build Succeeded |
Contributor
|
jenkins, retest this please (to ensure that this is still not flakey) |
Contributor
💚 Build Succeeded |
Contributor
|
@spalger looks like @justinkambic made all the changes you requested, we still need your approval to merge this one. |
Contributor
💚 Build Succeeded |
andrewvc
pushed a commit
to andrewvc/kibana
that referenced
this pull request
Aug 8, 2019
…astic#40650) I had previously revised these tests to utilize Jest's toBeCloseTo assertion, because it provides a precision parameter. My understanding when I implemented this test was that it would check n significant digits, so if I ran code like expect(myValue).toBeCloseTo(36000, 3), a value like 36015 would be accepted by the assertion. After seeing this test fail in a flaky way while testing other changes (it failed when myValue === 36001), I researched the function more closely and found that it does not behave this way. This led me to opt for the current solution, which should help avoid any flakiness issues while continuing to rely on the end-user simplicity of a snapshot. I do not expect a variance greater than 1 when running these tests, so the threshold of 100 should be sufficient to consider these tests stable.
jloleysens
added a commit
to jloleysens/kibana
that referenced
this pull request
Aug 9, 2019
…p-metrics-selectall * 'master' of github.com:elastic/kibana: (306 commits) [ML] Adding job overrides to the module setup endpoint (elastic#42946) [APM] Fix missing RUM url (elastic#42940) close socket timeouts without message (elastic#42456) Upgrade elastic/charts to 8.1.6 (elastic#42518) [ML] Delete old AngularJS data visualizer and refactor folders (elastic#42962) Add custom formatting for Date Nanos Format (elastic#42445) [Vega] Shim new platform - vega_fn.js -> vega_fn.js , use ExpressionFunction (elastic#42582) add socket.getPeerCertificate to KibanaRequest (elastic#42929) [Automation] ISTANBUL PRESET PATH is not working fine with constructor(private foo) (elastic#42683) [ML] Data frames: Updated stats structure. (elastic#42923) [Code] fixed the issue that the repository can not be deleted in some cases. (elastic#42841) [kbn-es] Support for passing regex value to ES (elastic#42651) Connect to Elasticsearch via SSL when starting kibana with `--ssl` (elastic#42840) Add Elasticsearch SSL support for integration tests (elastic#41765) Fix duplicate fetch in Visualize (elastic#41204) [DOCS] TSVB and Timelion clean up (elastic#42953) [Maps] [File upload] Fix maps geojson upload hanging on index step (elastic#42623) [APM] Use rounded bucket sizes for transaction distribution (elastic#42830) [yarn.lock] consistent resolve domain (elastic#42969) [Uptime] [Test] Repurpose unit test assertions to avoid flakiness (elastic#40650) ...
andrewvc
added a commit
that referenced
this pull request
Sep 4, 2019
…0650) (#42977) I had previously revised these tests to utilize Jest's toBeCloseTo assertion, because it provides a precision parameter. My understanding when I implemented this test was that it would check n significant digits, so if I ran code like expect(myValue).toBeCloseTo(36000, 3), a value like 36015 would be accepted by the assertion. After seeing this test fail in a flaky way while testing other changes (it failed when myValue === 36001), I researched the function more closely and found that it does not behave this way. This led me to opt for the current solution, which should help avoid any flakiness issues while continuing to rely on the end-user simplicity of a snapshot. I do not expect a variance greater than 1 when running these tests, so the threshold of 100 should be sufficient to consider these tests stable.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I had previously revised these tests to utilize Jest's
toBeCloseToassertion, because it provides aprecisionparameter. My understanding when I implemented this test was that it would checknsignificant digits, so if I ran code likeexpect(myValue).toBeCloseTo(36000, 3), a value like36015would be accepted by the assertion. After seeing this test fail in a flaky way while testing other changes (it failed whenmyValue === 36001), I researched the function more closely and found that it does not behave this way.This led me to opt for the current solution, which should help avoid any flakiness issues while continuing to rely on the end-user simplicity of a snapshot. I do not expect a variance greater than
1when running these tests, so the threshold of100should be sufficient to consider these tests stable.Testing this PR
Run the unit test suites. Should be all green.
I'm ok with running these changes many times if we want to ensure there's no flakiness introduced.