Test flakes on application tests

[conorsch]

Description

We're seeing occasional failures on the new TBB test logic.

Steps to Reproduce

1. Inspect CI run output: https://circleci.com/gh/freedomofpress/securedrop/27610
2. Observe failure

Expected Behavior

Application tests pass reliably.

Actual Behavior

Application tests fail occasionally.

Comments

The failing tests (observed so far) are:

• tests/test_2fa.py::test_totp_reuse_protections
• tests/test_2fa.py::test_totp_reuse_protections2
• tests/test_2fa.py::test_bad_token_fails_to_verify_on_admin_new_user_two_factor_page
• tests/test_2fa.py::test_bad_token_fails_to_verify_on_new_user_two_factor_page
• tests/test_journalist_api.py::test_user_cannot_get_an_api_token_with_wrong_2fa_token

Note that none of those are within the functional/ subdir, but several do emit webdriver-related errors:

        elif exception_class == UnexpectedAlertPresentException:
            alert_text = None
            if 'data' in value:
                alert_text = value['data'].get('text')
            elif 'alert' in value:
                alert_text = value['alert'].get('text')
            raise exception_class(message, screen, stacktrace, alert_text)
>       raise exception_class(message, screen, stacktrace)
E       selenium.common.exceptions.WebDriverException: Message: Socket timeout reading Marionette handshake data: Resource temporarily unavailable (os error 11)

Also worth noting that the tests are emitting a warning:

test_integration.py::test_filenames
  /usr/local/lib/python3.5/dist-packages/flask/sessions.py:208: UserWarning: "localhost" is not a valid cookie domain, it must contain a ".". Add an entry to your hosts file, for example "localhost.localdomain", and use that instead.
    ' "{rv}.localdomain", and use that instead.'.format(rv=rv)
  /usr/local/lib/python3.5/dist-packages/flask/sessions.py:208: UserWarning: "localhost" is not a valid cookie domain, it must contain a ".". Add an entry to your hosts file, for example "localhost.localdomain", and use that instead.
    ' "{rv}.localdomain", and use that instead.'.format(rv=rv)

[heartsucker]

Also worth noting that the tests are emitting a warning:

This happens all the time in flask testing when setting the server name to localhost. I have never seen it actually produce an error.

[eloquence]

Our goal for the 5/15-5/30 sprint is to at least understand the root cause of these test failures to determine whether they point to real dev or prod issues (@rmol's initial investigation suggests they do not, but we still don't know the root cause). Likely beyond the scope of this sprint, we'll want to revisit #3844, possibly in combination with a plugin like flaky, to deal with known-flaky functional tests more systematically, e.g., by automatically re-running them.

[redshiftzero]

another case: https://circleci.com/gh/freedomofpress/securedrop/27940