Skip to content

Commit

Permalink
CI: Do not report test failures as errors
Browse files Browse the repository at this point in the history
Prior to this patch, the CircleCI continuous integration environment was
configured to report test failures in a negative light, displaying red
cross-marks and reporting that "some checks were not successful" in
commits and GitHub Pull Requests which included them.

The passing/failing status of tests does not influence their
desirability for Test262. (In practice, engines very commonly fail
newly-contributed tests.)

Although these conflicting interpretations does not technically
interfere with the maintainers' ability to merge new contributions, it
does create confusion for many contributors who interpreted the UI as a
rejection of their work.

In addition, this behavior made it impossible to distinguish between the
benign test failures and disruptive infrastructural problems (e.g. the
crashing of engines).

Reconfigure the continuous integration environment to accept passing and
failing tests equally, and to only report a problem when the Test262
project's testing infrastructure behaves unexpectedly.
  • Loading branch information
jugglinmike authored and rwaldron committed May 28, 2021
1 parent d454b83 commit 1ebd34b
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 9 deletions.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"homepage": "https://github.com/tc39/test262#readme",
"devDependencies": {
"esvu": "^1.0.0",
"test262-harness": "^7.3.0"
"test262-harness": "^8.0.0"
},
"scripts": {
"ci": "./tools/scripts/ci_test.sh",
Expand Down
9 changes: 1 addition & 8 deletions tools/scripts/ci_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,5 @@ if [ "$CIRCLE_PULL_REQUEST" != "" ]; then
echo ""

echo "Running the tests with test262-harness"
test262-harness -t 1 --hostType=$hostType --hostPath=$hostPath --hostArgs="$hostArgs" -- $paths | tee exec.out

if grep -q '^[1-9][0-9]* failed$' exec.out; then
rm exec.out
exit 1
fi

rm exec.out
test262-harness -t 1 --hostType=$hostType --hostPath=$hostPath --hostArgs="$hostArgs" -- $paths
fi

0 comments on commit 1ebd34b

Please sign in to comment.