allow `bail` setting to control when to bail out of a failing test run #7335

bookman25 · 2018-11-06T20:52:58Z

Summary

Sometimes when running a full suite of tests there might be a config change (or some other failure) that leads to a catastrophic failure. Instead of continuing to run the entire suite, have an early exit after say 15 failures, something must be seriously wrong so abort the run.

Typically in CI mode we'll want to try to let the tests run so we can see all the failures at one time (which is why bail is not quite enough). But in the event of everything failing there's no need to continue running. We had a case where a branch of code was generating 100GB worth of logs because of test failures, with this change it would have been able to early abort.

Test plan

Unit tests

SimenB · 2018-11-06T22:37:39Z

Interesting idea! I wonder if we could overload bail (making true mean 1) instead of adding a new option?

@thymikee @rickhanlonii

CHANGELOG.md

thymikee · 2018-11-07T07:59:40Z

I like passing a number to bail so we don't have to introduce yet another config option.

docs/CLI.md

docs/Configuration.md

SimenB · 2018-11-08T22:18:51Z

docs/WatchPlugins.md

@@ -155,7 +155,7 @@ class MyWatchPlugin {

 For stability and safety reasons, only part of the global configuration keys can be updated with `updateConfigAndRun`. The current white list is as follows:

- [`bail`](configuration.html#bail-boolean)
+- [`bail`](configuration.html#bail)


does the link work? You can run yarn start in the website folder to test links etc

Maybe I'm just doing something wrong, but I've tried yarn start and yarn build then yarn start in /website/ and it doesn't seem to be reflecting any of the changes from these *.md files.

Ok, I was able to find the link in /build folder. Updated to match now.

You need to open the next docs, by default it renders the versioned docs. No need to build :)
You can select the version by clicking the version number in the upper left

This document gives a 404, though, can yiu double check locally?

https://deploy-preview-7335--jest-preview.netlify.com/docs/en/next/watch-plugins

So the link itself for configuration#bail-number-boolean seems to work correctly. But the link you posted, as well as when running locally, and going to /next/watch-plugins both seem to return page not found. Seems to be partially broken in master: https://jestjs.io/docs/en/next/watch-plugins Looks like it's because of original_id inside the header. I've removed it because I don't see any references to that property outside of versioned docs and removing it seems to fix that link so we can test the updated link.

packages/jest-cli/src/cli/args.js

bookman25 · 2018-11-26T15:49:08Z

@SimenB anything else? I believe I've addressed all of the outstanding review comments.

thymikee · 2018-11-26T18:25:39Z

packages/jest-cli/src/TestScheduler.js

-    if (this._globalConfig.bail && aggregatedResults.numFailedTests !== 0) {
+    if (
+      this._globalConfig.bail !== 0 &&
+      aggregatedResults.numFailedTests >= this._globalConfig.bail


this will still be true if bail is explicitly set to false. Shouldn't we normalize it just in case?

There shouldn't be any case where bail gets to this point that isn't a number: https://github.com/facebook/jest/pull/7335/files#diff-7121e12d406ff9a3060a13ee808632de

types/Config.js

thymikee · 2018-11-28T14:31:13Z

packages/jest-config/src/normalize.js

+          value = options[key] ? 1 : 0;
+        } else if (isNaN(options[key])) {
+          value = 1;
+          argv._.push(options[key]);


can't we just remove this line? looks fishy

bookman25 · 2018-12-05T12:08:42Z

@thymikee @SimenB any outstanding concerns with this approach?

thymikee

👍

packages/jest-config/src/normalize.js

thymikee · 2018-12-05T13:04:52Z

@rickhanlonii you have anything to add here? if not, feel free to merge :)

codecov-io · 2018-12-06T12:56:17Z

Codecov Report

Merging #7335 into master will increase coverage by 0.29%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master    #7335      +/-   ##
==========================================
+ Coverage   67.15%   67.44%   +0.29%     
==========================================
  Files         247      247              
  Lines        9502     9511       +9     
  Branches        5        5              
==========================================
+ Hits         6381     6415      +34     
+ Misses       3119     3094      -25     
  Partials        2        2

Impacted Files	Coverage Δ
packages/jest-config/src/ValidConfig.js	`100% <ø> (ø)`	⬆️
packages/jest-config/src/Defaults.js	`100% <ø> (ø)`	⬆️
packages/jest-config/src/normalize.js	`83.13% <0%> (-2.41%)`	⬇️
packages/jest-cli/src/TestScheduler.js	`73.38% <100%> (+8.06%)`	⬆️
packages/jest-cli/src/lib/update_global_config.js	`93.75% <75%> (-1.91%)`	⬇️
packages/jest-cli/src/ReporterDispatcher.js	`87.5% <0%> (+12.5%)`	⬆️
packages/jest-cli/src/testResultHelpers.js	`72.72% <0%> (+63.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6a2237d...c807bb4. Read the comment docs.

rickhanlonii

LGTM just a couple questions!

docs/CLI.md

packages/jest-cli/src/cli/args.js

rickhanlonii · 2018-12-06T16:47:26Z

Thanks @bookman25, I'll merge when we're green

rickhanlonii · 2018-12-06T17:48:18Z

Thanks @bookman25!!

* master: (24 commits) Add `jest.isolateModules` for scoped module initialization (jestjs#6701) Migrate to Babel 7 (jestjs#7016) docs: changed "Great Scott!" link (jestjs#7524) Use reduce instead of filter+map in dependency_resolver (jestjs#7522) Update Configuration.md (jestjs#7455) Support dashed args (jestjs#7497) Allow % based configuration of max workers (jestjs#7494) chore: Standardize filenames: jest-runner pkg (jestjs#7464) allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) Add issue template labels (jestjs#7470) chore: standardize filenames in e2e/babel-plugin-jest-hoist (jestjs#7467) Add node worker-thread support to jest-worker (jestjs#7408) Add `testPathIgnorePatterns` to CLI documentation (jestjs#7440) pretty-format: Omit non-enumerable symbol properties (jestjs#7448) Add Jest Architecture overview to docs. (jestjs#7449) chore: run appveyor tests on node 10 chore: fix failures e2e test for node 8 (jestjs#7446) chore: update docusaurus to v1.6.0 (jestjs#7445) Enhancement/expect-to-be-close-to-with-infinity (jestjs#7444) Update CHANGELOG formatting (jestjs#7429) ...

jestjs#7335)

* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <[email protected]> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files

jestjs#7335)

* standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * standardizing jest-runner filenames to fit facebook standards * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names to fit facebook standards * updating filenames to better fit facebook standards * Add issue template labels (jestjs#7470) * Add issue template labels * Lint * standardizing jest-runner filenames to fit facebook standards * allow `bail` setting to control when to bail out of a failing test run (jestjs#7335) * standardizing file names in /e2e/before-all-filtered * resolving naming discrepencies inside of files * standardizing filenames in e2e/before-each-queue and corresponding tests * standardizing several tests in ./e2e/__test__/ * standardizing filenames for babel-plugin-jest-hoist and related test files * standardizing file-names for folders 'browser-support' and 'clear-cache' and their corresponding test files * resolving issue with karma referencing non-standard browserTest * resolving issue with karma referencing non-standard browserTest filename * standardizing file names in e2e/__test__ * standardizing file names in compare-dom-nodes * standardizing file names for coverage-report and coverage-remap * updating case sensitive filenames * renaming files to fit facebook standards * adding case sensitive file changes * resolving dependency issues after renaming case sensitive files. * standardizing file names to fit facebook standards * adjusting file names and snapshots to fit facebook standards * adjusting file names to fit facebook standards * resolves bug with nodeAssertionError not being found by test suite * changing references to match file name * changing file name to check test-jest-circus * readding a previously deleted snapshot * Update e2e/bad-source-map/__tests__/badSourceMap.js comment to reflect new file name Co-Authored-By: GGonryun <[email protected]> * improving file naming conventions letters g & j * changing filename to fit facebook standards * renaming filenames to fit facebook standards for 'L' * renaming filenames to fit facebook standards for 'M' * renaming filenames to fit facebook standards for 'N' * resolving leftover merge conflicts * reverting error in naming by changing filename back to uppercase * renaming filenames 'O' to fit facebook standards. * renaming filenames 'P' to fit facebook standards. * renaming filenames 'R' to fit facebook standards. * renaming filenames 'S' to fit facebook standards. * changing regex to match temporary folder name * renaming filenames 'T' to fit facebook standards. * renaming filenames 'U' to fit facebook standards. * renaming filenames 'V' to fit facebook standards. * renaming filenames 'W' to fit facebook standards. * updating snapshots to reflect standardized file names * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern-temp/package.json * Rename e2e/test-name-pattern/package.json to e2e/test-name-pattern-skipped/package.json * Rename e2e/test-name-pattern-skipped/package.json to e2e/test-name-pattern.package.json * Rename e2e/test-name-pattern-temp/package.json to e2e/test-name-pattern/package.json * Rename e2e/test-name-pattern.package.json to e2e/test-name-pattern-skipped/package.json * removing left over files and adjusting file names previously missed * deleting more leftover files

github-actions · 2021-05-12T05:14:15Z

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

facebook-github-bot added the cla signed label Nov 6, 2018

SimenB reviewed Nov 7, 2018

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

bookman25 changed the title ~~add bailAfter setting to control when to bail out of a failing test run~~ allow bail setting to control when to bail out of a failing test run Nov 8, 2018

SimenB requested changes Nov 8, 2018

View reviewed changes

SimenB requested review from thymikee and rickhanlonii November 26, 2018 16:20

SimenB approved these changes Nov 26, 2018

View reviewed changes

thymikee reviewed Nov 26, 2018

View reviewed changes

thymikee reviewed Nov 28, 2018

View reviewed changes

thymikee approved these changes Dec 5, 2018

View reviewed changes

packages/jest-config/src/normalize.js Show resolved Hide resolved

rickhanlonii approved these changes Dec 6, 2018

View reviewed changes

docs/CLI.md Outdated Show resolved Hide resolved

packages/jest-cli/src/cli/args.js Show resolved Hide resolved

allow bail setting to control when to bail out of a failing test run

60cb6e0

rickhanlonii merged commit 3e805bc into jestjs:master Dec 6, 2018

willsmythe pushed a commit to willsmythe/jest that referenced this pull request Dec 20, 2018

allow bail setting to control when to bail out of a failing test run (

1ebc5c4

jestjs#7335)

captain-yossarian pushed a commit to captain-yossarian/jest that referenced this pull request Jul 18, 2019

allow bail setting to control when to bail out of a failing test run (

8e30fd0

jestjs#7335)

github-actions bot locked as resolved and limited conversation to collaborators May 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

allow `bail` setting to control when to bail out of a failing test run #7335

allow `bail` setting to control when to bail out of a failing test run #7335

bookman25 commented Nov 6, 2018

SimenB commented Nov 6, 2018

thymikee commented Nov 7, 2018

SimenB Nov 8, 2018

bookman25 Nov 9, 2018

bookman25 Nov 9, 2018

SimenB Nov 10, 2018

SimenB Nov 10, 2018

bookman25 Nov 10, 2018

bookman25 commented Nov 26, 2018

thymikee Nov 26, 2018

bookman25 Nov 26, 2018

thymikee Nov 28, 2018

bookman25 commented Dec 5, 2018

thymikee left a comment

thymikee commented Dec 5, 2018

codecov-io commented Dec 6, 2018

rickhanlonii left a comment

rickhanlonii commented Dec 6, 2018

rickhanlonii commented Dec 6, 2018

github-actions bot commented May 12, 2021

allow bail setting to control when to bail out of a failing test run #7335

allow bail setting to control when to bail out of a failing test run #7335

Conversation

bookman25 commented Nov 6, 2018

Summary

Test plan

SimenB commented Nov 6, 2018

thymikee commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bookman25 commented Nov 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bookman25 commented Dec 5, 2018

thymikee left a comment

Choose a reason for hiding this comment

thymikee commented Dec 5, 2018

codecov-io commented Dec 6, 2018

Codecov Report

rickhanlonii left a comment

Choose a reason for hiding this comment

rickhanlonii commented Dec 6, 2018

rickhanlonii commented Dec 6, 2018

github-actions bot commented May 12, 2021

allow `bail` setting to control when to bail out of a failing test run #7335

allow `bail` setting to control when to bail out of a failing test run #7335