bazel shell test succeeds even when one test case reported failed

[meteorcloudy]

Description of the problem:

Bazel reports a shell test as passed when it actually fails.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. Adding the following test case to //src/test/shell/bazel/remote:remote_execution_test

diff --git a/src/test/shell/bazel/remote/remote_execution_test.sh b/src/test/shell/bazel/remote/remote_execution_test.sh
index 4964df8228..cf337e3f8e 100755
--- a/src/test/shell/bazel/remote/remote_execution_test.sh
+++ b/src/test/shell/bazel/remote/remote_execution_test.sh
@@ -2110,4 +2110,8 @@ EOF
   expect_log "INFO: Build completed successfully"
 }

+function test_blabla() {
+    fail "something"
+}
+

2. Run the test

$ bazel test //src/test/shell/bazel/remote:remote_execution_test --test_arg=test_blabla --test_output=streamed

WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
DEBUG: /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1804_java11 not using checked in configs as detect_java_home was set to True
DEBUG: /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/external/bazel_toolchains/rules/rbe_repo/version_check.bzl:68:14:
Current running Bazel is ahead of bazel-toolchains repo. Please update your pin to bazel-toolchains repo in your WORKSPACE file.
DEBUG: /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/external/bazel_toolchains/rules/rbe_repo/checked_in.bzl:103:14: rbe_ubuntu1604_java8 not using checked in configs as detect_java_home was set to True
INFO: Build option --test_sharding_strategy has changed, discarding analysis cache.
INFO: Analyzed target //src/test/shell/bazel/remote:remote_execution_test (3 packages loaded, 9290 targets configured).
INFO: Found 1 test target...
INFO[remote_execution_test 2020-10-28 15:47:56 (+0000)] bazel binary is at /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/bazel-out/darwin-fastbuild/bin/src/test/shell/bazel/remote/remote_execution_test.runfiles/io_bazel/src/test/shell/bin
INFO[remote_execution_test 2020-10-28 15:47:56 (+0000)] setting up client in /private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/_tmp/da8b39f188f68695621d53283fe46934/workspace

Remote execution and remote cache tests

Wed Oct 28 15:47:57 UTC 2020
** test_blabla *****************************************************************
-- Test log: -----------------------------------------------------------
Oct 28, 2020 3:47:57 PM com.google.devtools.build.lib.actions.LocalHostCapacity getNewLocalHostCapacity
INFO: Determined local resources: RAM=34359MB, CPU=16.0
201028 15:47:57.674:I 1 [com.google.devtools.build.remote.worker.RemoteWorker.startServer] Starting gRPC server on port 51483
201028 15:47:57.780:I 1 [com.google.devtools.build.remote.worker.RemoteWorker.main] Not starting HTTP cache server
------------------------------------------------------------------------
test_blabla FAILED: something .
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/bazel-out/darwin-fastbuild/bin/src/test/shell/bazel/remote/remote_execution_test.runfiles/io_bazel/src/test/shell/bazel/remote/remote_execution_test:2114: in call to test_blabla
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/bazel-out/darwin-fastbuild/bin/src/test/shell/bazel/remote/remote_execution_test.runfiles/io_bazel/src/test/shell/bazel/remote/remote_utils.sh: line 45: 92826 Killed: 9               "${REMOTE_WORKER}" --singlejar --jvm_flag=-Djava.library.path="${native_lib}" --work_path="${work_path}" --cas_path="${cas_path}" --listen_port=${worker_port} --pid_file="${pid_file}" "$@" >&$TEST_log
INFO[remote_execution_test 2020-10-28 15:48:00 (+0000)] Cleaning up workspace
-- Test log: -----------------------------------------------------------
$TEST_TMPDIR defined: output root default is '/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/_tmp/da8b39f188f68695621d53283fe46934' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Starting clean.
$TEST_TMPDIR defined: output root default is '/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/_tmp/da8b39f188f68695621d53283fe46934' and max_idle_secs default is '15'.
INFO: Starting clean.
------------------------------------------------------------------------
test_blabla FAILED: terminated because this command returned a non-zero status:
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/bazel-out/darwin-fastbuild/bin/src/test/shell/bazel/remote/remote_execution_test.runfiles/io_bazel/src/test/shell/bazel/remote/remote_execution_test:2117: in call to main
/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/bazel-out/darwin-fastbuild/bin/src/test/shell/bazel/remote/remote_execution_test.runfiles/io_bazel/src/test/shell/bazel/remote/remote_utils.sh: line 46: pid_file: unbound variable
$TEST_TMPDIR defined: output root default is '/private/var/tmp/_bazel_pcloudy/829441223e9fec5a5a2e3d1dd743fdf0/sandbox/darwin-sandbox/2388/execroot/io_bazel/_tmp/da8b39f188f68695621d53283fe46934' and max_idle_secs default is '15'.
Target //src/test/shell/bazel/remote:remote_execution_test up-to-date:
  bazel-bin/src/test/shell/bazel/remote/remote_execution_test
INFO: Elapsed time: 7.445s, Critical Path: 5.77s
INFO: 2 processes: 1 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 2 total actions
//src/test/shell/bazel/remote:remote_execution_test                      PASSED in 4.9s

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

3.7.0

[oquenchil]

Just to add one more detail, the fail call can be replaced with exit 1 and it will still pass when it should fail.

[meteorcloudy]

Maybe caused by #12377

[meteorcloudy]

After removing

function set_up() {
  start_worker \
        --incompatible_remote_symlinks
}

function tear_down() {
  bazel clean >& $TEST_log
  stop_worker
}

the test failed as expected... 😂

[meteorcloudy]

I only traced down to ${awefawef}, adding it to tear_down function will somehow shadow the exit code, but adding other explicit failure command like exit 1 or just call unknown_function won't...