-
Notifications
You must be signed in to change notification settings - Fork 166
Reduce Capybara wait time to zero #8013
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,11 +1,12 @@ | ||
| module InteractionHelper | ||
| def click_spinner_button_and_wait(...) | ||
| click_on(...) | ||
| expect(page).to have_no_css('lg-spinner-button.spinner-button--spinner-active', wait: 10) | ||
| end | ||
|
|
||
| def click_select_button_and_wait(...) | ||
| click_button(...) | ||
| expect(page).to have_no_css('button.usa-button', wait: 10) | ||
| begin | ||
| expect(page).to have_no_css('lg-spinner-button.spinner-button--spinner-active', wait: 10) | ||
| rescue Selenium::WebDriver::Error::StaleElementReferenceError | ||
| # A stale element error can occur when attempting to wait for the spinner to disappear if the | ||
| # context in which the button was clicked (e.g. a `within` block) itself disappears. This is | ||
| # fine, since if the ancestor disappears, it can be assumed that the button is gone too. | ||
| end | ||
| end | ||
| end |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why remove the capacity to set a longer wait time locally with an environment variable? I'm doing a full feature test run locally on this branch, and I saw some failures go by on document capture, which already gives the user a lot of feedback about the process taking a while.
Just finished! (45 minutes altogether) Failed examples:
rspec ./spec/features/accessibility/idv_pages_spec.rb:65 # Accessibility on IDV pages IDV pages doc auth steps accessibility on mobile
rspec './spec/features/idv/steps/phone_step_spec.rb[1:10:2:2:1]' # idv phone step after the max number of attempts behaves like verification step max attempts after completing one less than the max attempts allows the user to continue if their last attempt is successful
rspec ./spec/features/idv/steps/gpo_step_spec.rb:92 # idv gpo step Verified resets password, requests GPO, then signs in using SP shows the user the GPO code entry screen
rspec ./spec/features/users/sign_in_spec.rb:260 # Sign in session approaches timeout user sees warning before session times out
rspec ./spec/features/users/sign_in_spec.rb:279 # Sign in session approaches timeout user can continue browsing
rspec ./spec/features/idv/in_person_spec.rb:119 # In Person Proofing works for a happy path
rspec ./spec/features/idv/in_person_spec.rb:245 # In Person Proofing allows the user to cancel and start over from the beginning
rspec ./spec/features/idv/in_person_spec.rb:258 # In Person Proofing allows the user to go back to document capture from prepare step
rspec ./spec/features/idv/in_person_spec.rb:406 # In Person Proofing verify address by mail (GPO letter) lets the user clear and start over from gpo confirmation
rspec ./spec/features/idv/in_person_spec.rb:376 # In Person Proofing verify address by mail (GPO letter) requires address verification before showing instructions
rspec ./spec/features/idv/in_person_spec.rb:426 # In Person Proofing transliteration shows validation errors
rspec ./spec/features/idv/in_person_spec.rb:501 # In Person Proofing validate_id_and_residential_addresses feature flag enabled captures the address, address line 2, city, state and zip code
rspec './spec/features/idv/in_person_spec.rb[1:5:1:1]' # In Person Proofing after in-person proofing is completed and passed for a partner using oidc sends a survey when they share information with that partner
rspec './spec/features/idv/in_person_spec.rb[1:5:2:1]' # In Person Proofing after in-person proofing is completed and passed for a partner using saml sends a survey when they share information with that partner
rspec ./spec/features/idv/in_person_spec.rb:23 # In Person Proofing ThreatMetrix review pending allows the user to continue down the happy path
rspec ./spec/features/idv/in_person_spec.rb:325 # In Person Proofing with hybrid document capture resumes desktop session with in-person proofing
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:75 # doc capture document capture step does not advance original session with errors
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:61 # doc capture document capture step advances original session once complete
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:108 # doc capture document capture step when using async uploads advances original session once complete
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:125 # doc capture document capture step when using async uploads does not advance original session with errors
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:149 # doc capture document capture step when using async uploads with attention with barcode result advances original session only after confirmed
rspec ./spec/features/idv/doc_capture/document_capture_step_spec.rb:91 # doc capture document capture step with attention with barcode result advances original session only after confirmed
rspec ./spec/features/two_factor_authentication/sign_in_spec.rb:63 # Two Factor Authentication When the user has not set up 2FA with international phone that does not support voice delivery updates international code as user types
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you provide more detail about those errors? I had initially encountered some failures in my environment because of configuration in my local
config/application.ymlwas taking precedent, but they went away after resetting my configuration, and weren't related to wait timeouts.The description of the pull request explains why I think the default wait is an anti-pattern. Allowing this to be configurable would imply that there are instances where we want to tolerate the application to be unresponsive in areas that we aren't explicitly expecting it to be unresponsive. Setting this to a single value also avoids inconsistencies like where previously CI was more lax than local development, which contributed to flakiness of tests, difficulty in reproducing them, and lack of clarity that a distinction existed at all between CI and local development.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Emailed error output as far back as my terminal history goes.
I understand the reasoning behind changing it to 0 on CI/CD. And I think we need to be able to run feature tests locally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest changing the default from
0.5to0without hard-coding a magic number replacing the variable. Those of us with inconsistently failing tests still need this knob to twiddle as we attempt to improve the problem.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jskinne3 Could you share some detail about the specs you're seeing failing on this branch? My hope is that these changes address all of the intermittent failings related to this setting and normalize the behavior across systems to prevent the possibility of any "wait"-related intermittent failures from being introduced in the future.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure. I started a
bundle exec make_teston this branch. I'll email you the output when finished.UPDATE: I sent you test output with a bottom line of:
7791 examples, 1192 failures, 7 pendingsThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for sending the test output. Reviewing the errors, I don't see any which strike me as being related to this configuration. I still think a non-configurable zero timeout is the best option and I'd worry if the configuration could distract developers from other root causes of a test failure, but I'll reintroduce the environment variable if you expect that it could give some relief to issues you're encountering.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@soniaconnolly @jskinne3 I added the environment variable back in ebd4c5e. Can you please take a look?