Skip to content

Comments

Fix flaky redo document capture feature test 100% for sure, no chance of failure#8633

Closed
matthinz wants to merge 10 commits intomainfrom
matthinz/flaky-redo-doc-capture-test
Closed

Fix flaky redo document capture feature test 100% for sure, no chance of failure#8633
matthinz wants to merge 10 commits intomainfrom
matthinz/flaky-redo-doc-capture-test

Conversation

@matthinz
Copy link
Contributor

#8576 added a new feature test to check that, when using the "redo document capture" action, users are not able to access earlier steps of the IdV process (doc capture, ssn, etc.) after submitting the "Verify info" screen. We added this new test in among some existing redo document capture feature tests.

After adding the new test, we noticed it was flaky in CI. We temporarily disabled the new test in #8607, but the flakiness remained with other tests in the same file. Our current hypothesis is that some of the refactoring we did to allow adding the new test in amongst the existing tests has led to the flakiness.

This PR:

matthinz added 4 commits June 21, 2023 14:18
The changes to this spec introduced in #8576 made it really flaky in CI. This commit resets the spec to how it was before that change. Note that the file was renamed in the meantime.
This is the test added in #8576, reworked to be independent of other tests in this file. Hopefully this reduces the flakiness.

[skip changelog]
@matthinz matthinz requested a review from soniaconnolly June 21, 2023 22:13
@matthinz
Copy link
Contributor Author

(I'm going to run CI several times on this branch before marking ready to merge.)

@matthinz matthinz force-pushed the matthinz/flaky-redo-doc-capture-test branch from a88c30b to f474a4c Compare June 21, 2023 22:16
@aduth
Copy link
Contributor

aduth commented Jun 22, 2023

Looking at an example failure, I'd suspect a reason it may fail is because the submission of document capture involves an asynchronous submission we're not waiting to complete.

It seems like if we substitute the attach and submit behaviors with the attach_and_submit_images helper, it may help, since the helper explicitly waits for the load screen to disappear?

@matthinz
Copy link
Contributor Author

It looks like we're using attach_and_submit_images here. The click_idv_continue is to move past the "We couldn’t read the barcode on your ID" screen to get to the SSN entry screen. So we are waiting, but possibly just not waiting long enough in CI sometimes? Though 10 seconds seems like plenty of time to wait. I'll try lowering that locally and see if I can get a failure.

@aduth
Copy link
Contributor

aduth commented Jun 22, 2023

Oh, d'oh, I totally misread the test setup. That's what I get for commenting so early in the morning 🤦 You're right about it already using attach_and_submit_images.

That is strange, since it doesn't seem like there's anything asynchronous in clicking "Continue" from the barcode warning page that would delay the page transition (source).

@matthinz
Copy link
Contributor Author

Gonna close this PR--I'm working on this in the background, and we have a ticket now (LG-10192), but this PR is just noise ATM.

@matthinz matthinz closed this Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants