Fix #1942: Fix CI app module test flakiness/failures due to shared test processes #1943
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #1942
This started as #1939, but since the CI failure was also affecting that PR I decided to create a new PR that adds additional debugging information and fixes the issue.
The PR fixes CI tests by requiring each one run in its own process. Note that this was an existing issue, but it only recently started affecting CI tests (https://github.com/oppia/oppia-android/actions). This will result in tests running significantly slower than before because subsequent test suites cannot benefit from from previous tests' resources setup in Robolectric. This is exactly how Bazel works, so this is the correct long-term strategy for the tests. #1942 contains much more information, but the summary is that static state being shared and not properly synchronized across test suites was causing later suites to fail. Running tests in their own process fully mitigates the problem.
Note that parallelization is needed to run the tests locally, but it does not work in Android Studio's test runner UI which means that running whole modules of tests via the UI will take 3-4x longer than before. Running from the CLI is still quite fast since it does properly use Gradle's parallelization configuration introduced in this PR.
The debugging changes enable full stack traces for all Robolectric tests to ensure failures can actually be understood when they occur in CI. Note that full-stacktrace will specifically show Gradle task failures which is especially useful when running Espresso tests from the CLI (per #1831). The Gradle changes are needed for Robolectric tests to show full stack traces when failing. Note also that the Gradle configuration is resulting in all passed & skipped tests to print out which is resulting in much more output. That seems like a nice addition since it makes the results clearer, but please let me know if you disagree.
Sample of failures showing up with stack traces after this change: https://github.com/oppia/oppia-android/pull/1939/checks?check_run_id=1222719700.
Compared test times: