//src/test/shell/bazel/android:android_instrumentation_test_integration_test failing on CI

[katre]

Test test appears to be failing due to errors in the manifest:

ERROR: /home/ci/.cache/bazel/_bazel_ci/65ed5a6a0442d902ec235fee508231f8/bazel-sandbox/2532511943818503847/execroot/io_bazel/_tmp/19eb9a40c2097cd604c7dbc888ea3a61/workspace.VPkCoFRe/javatests/com/bin/BUILD:1:1: Validating the merged manifests of the target and instrumentation APKs failed 

See details at: https://ci.bazel.build/blue/organizations/jenkins/Global%2Fbazel-tests/detail/bazel-tests/482/tests

[jin]

Looking

[jin]

I can't reproduce this on macOS and Linux. Also, the test seems to be passing in the main pipeline: https://ci.bazel.build/blue/rest/organizations/jenkins/pipelines/Global/pipelines/bazel-tests/runs/482/nodes/30/steps/439/log/?start=0

//src/test/shell/bazel/android:android_instrumentation_test_integration_test PASSED in 173.7s
  Stats over 4 runs: max = 173.7s, min = 82.9s, avg = 131.9s, dev = 40.8s

Edit: Nvm, it's failing in this pipeline: https://ci.bazel.build/blue/organizations/jenkins/Global%2Fbazel-tests/detail/bazel-tests/482/pipeline/26

Seems like a Python 2 problem.

[katre]

This is failing fairly consistently on CI. I can work with you to try and reproduce it there.

[jin]

This seems to be the actual error:

Server crashed during startup. Now printing '/home/ci/.cache/bazel/_bazel_ci/65ed5a6a0442d902ec235fee508231f8/bazel-sandbox/2532511943818503847/execroot/io_bazel/_tmp/19eb9a40c2097cd604c7dbc888ea3a61/root/0a598e57f6c3319531a94fab05707c47/server/jvm.out':
------------------------------------------------------------------------
test_target_package_no_package_specified_build_failure FAILED: terminated because this command returned a non-zero status:
/home/ci/.cache/bazel/_bazel_ci/65ed5a6a0442d902ec235fee508231f8/bazel-sandbox/2532511943818503847/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/test/shell/bazel/android/android_instrumentation_test_integration_test.runfiles/io_bazel/src/test/shell/testenv.sh:438: in call to cleanup_workspace
/home/ci/.cache/bazel/_bazel_ci/65ed5a6a0442d902ec235fee508231f8/bazel-sandbox/2532511943818503847/execroot/io_bazel/bazel-out/k8-fastbuild/bin/src/test/shell/bazel/android/android_instrumentation_test_integration_test.runfiles/io_bazel/src/test/shell/testenv.sh:475: in call to tear_down
INFO[android_instrumentation_test_integration_test 2018-02-01 18:46:22 (+0000)] Cleaning up workspace```

The manifest error is what the `assert_build_fails` test is expecting, so it is expected in the log.

[aj-michael]

I logged into ubuntu_14.04-x86_64-4, cloned the git repo at head, added an Android SDK, ran this command line and it passed:

/home/ci/.bazel/custom/bazel_release/1282/variation_/binary/bazel --bazelrc=/home/ci/workspace/Global/bazel-tests-node=linux-x86_64,python_version=2/bazel.bazelrc test //src/test/shell/bazel/android:android_instrumentation_test_integration_test

[aj-michael]

@katre are you sure that it is failing consistently? I only see this single failure, which seems to have been caused by the Bazel server dying. Since the sandbox was enabled, I cannot recover the JVM log file. And since we cannot reproduce, this seems like a flake.

[aj-michael]

Ok, I was able to reproduce on the CI. It is 2% flaky.

/home/ci/.bazel/custom/bazel_release/1282/variation_/binary/bazel --bazelrc=/home/ci/workspace/Global/bazel-tests-node=linux-x86_64,python_version=2/bazel.bazelrc test //src/test/shell/bazel/android:android_instrumentation_test_integration_test --nocache_test_results --runs_per_test=100 --sandbox_debug
...
FAILED: //src/test/shell/bazel/android:android_instrumentation_test_integration_test (Summary)
      /home/ajmichael/.cache/bazel/_bazel_ajmichael/f57319d333225770f1fe38d4365c73a4/execroot/io_bazel/bazel-out/k8-fastbuild/testlogs/src/test/shell/bazel/android/android_instrumentation_test_integration_test/shard_1_of_4_run_79_of_100/test.log
      /home/ajmichael/.cache/bazel/_bazel_ajmichael/f57319d333225770f1fe38d4365c73a4/execroot/io_bazel/bazel-out/k8-fastbuild/testlogs/src/test/shell/bazel/android/android_instrumentation_test_integration_test/shard_2_of_4_run_46_of_100/test.log
Target //src/test/shell/bazel/android:android_instrumentation_test_integration_test up-to-date:
  bazel-bin/src/test/shell/bazel/android/android_instrumentation_test_integration_test
INFO: Elapsed time: 2226.389s, Critical Path: 230.22s
//src/test/shell/bazel/android:android_instrumentation_test_integration_test FAILED in 2 out of 400 in 230.2s
  Stats over 400 runs: max = 230.2s, min = 32.7s, avg = 170.9s, dev = 32.1s
  /home/ajmichael/.cache/bazel/_bazel_ajmichael/f57319d333225770f1fe38d4365c73a4/execroot/io_bazel/bazel-out/k8-fastbuild/testlogs/src/test/shell/bazel/android/android_instrumentation_test_integration_test/shard_1_of_4_run_79_of_100/test.log
  /home/ajmichael/.cache/bazel/_bazel_ajmichael/f57319d333225770f1fe38d4365c73a4/execroot/io_bazel/bazel-out/k8-fastbuild/testlogs/src/test/shell/bazel/android/android_instrumentation_test_integration_test/shard_2_of_4_run_46_of_100/test.log

Executed 1 out of 1 test: 1 fails locally.
INFO: Build completed, 1 test FAILED, 401 total actions

Unfortunately the jvm.out files were still deleted...

[jin]

@aj-michael thanks! looks like it's 0.5% and not 2%?

[katre]

Thank you for taking a look.

Looking at the RC bazel-tests invocation, I see related failures in the following tests:

• node=linux-x86_64,python_version=2 / [node=linux-x86_64,python_version=2] Results / test_target_package_no_package_specified_build_failure
• node=linux-x86_64,python_version=2 / [node=linux-x86_64,python_version=2] Results / test_incorrect_target_package_build_failure

Is it just chance that these flaked at the same time, or is there an underlying source of flakiness?

[jin]

There are two other tests in the file that test for similar assert_build_failure cases too, but they passed. It seems like there is an underlying source, but it does not seem to me that it's due to the test and the code under test.

[katre]

Wrong issue number on the patch.

[katre]

This seems to be no longer happening.