chore: improve vitest output

[jeluard]

Motivation

Current tests output can be quite verbose, and display interleaved console output.

Description

Default behavior is unchanged. 2 env variables allow to tweak vitest output:

• USE_BASIC_REPORTER to produce a basic tests summary
• STRIP_CONSOLE_OUTPUT to strip console output from tests summary

e.g. STRIP_CONSOLE_OUTPUT=true USE_BASIC_REPORTER=true yarn test:spec:mainnet will produce the following output:

 ✓ test/spec/presets/merkle.test.ts  (0 test)
 ✓ test/spec/presets/genesis.test.ts  (0 test)
 ✓ test/spec/presets/shuffling.test.ts  (300 tests) 565ms
 ✓ test/spec/presets/light_client/index.test.ts  (14 tests | 2 skipped) 4700ms
(node:55726) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 uncaughtException listeners added to [process]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
 ✓ test/spec/presets/fork.test.ts  (62 tests) 16235ms
 ✓ test/spec/presets/finality.test.ts  (30 tests) 21080ms
 ✓ test/spec/presets/rewards.test.ts  (219 tests) 26226ms
 ✓ test/spec/presets/fork_choice.test.ts  (117 tests | 2 skipped) 51488ms

 ✓ test/spec/presets/epoch_processing.test.ts  (453 tests | 6 skipped) 73137ms
 ❯ test/spec/presets/transition.test.ts  (178 tests | 3 failed) 80438ms
   ❯ test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_randomized_state
     → Invalid state root at slot 288, expected=0x708fd518df60ae030fecbaed6ef1668db0589de82e99700d25163fd8ffc9f131, actual=0x922edb35a7dd26410c63eb6548141dda28d522658f21988485eb86d451152fe4
   ❯ test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_with_deposit_right_before_fork
     → Invalid state root at slot 96, expected=0x5241d607d5383dd1daf9d059ca01506f1b80b837f85329fd6890a5612d22253d, actual=0xc474453d632eabffb5d408eaddf857918129b97e5bf40a7f4994bf8b04dc55fb
   ❯ test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_with_non_empty_activation_queue
     → Invalid state root at slot 96, expected=0xce646a5d98c6661bdd1c70634e1d2295449e3347b7915b6bb5ab3ac4e12ad4e7, actual=0x70461c653c78d1597687e396b0a7fbabcc45486044b34dc8f20520c964cd48f9
 ✓ test/spec/presets/ssz_static.test.ts  (1285 tests) 100868ms
 ✓ test/spec/presets/operations.test.ts  (1244 tests) 125409ms
 ✓ test/spec/presets/sanity.test.ts  (478 tests) 217204ms

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 3 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_randomized_state
Error: Invalid state root at slot 288, expected=0x708fd518df60ae030fecbaed6ef1668db0589de82e99700d25163fd8ffc9f131, actual=0x922edb35a7dd26410c63eb6548141dda28d522658f21988485eb86d451152fe4
 ❯ Module.stateTransition ../state-transition/src/stateTransition.ts:126:13
 ❯ testFunction test/spec/presets/transition.test.ts:57:19
     55|         for (let i = 0; i < meta.blocks_count; i++) {
     56|           const signedBlock = testcase[`blocks_${i}`] as allForks.SignedBeaconBlock;
     57|           state = stateTransition(state, signedBlock, {
       |                   ^
     58|             // Assume valid and available for this test
     59|             executionPayloadStatus: ExecutionPayloadStatus.valid,
 ❯ ../spec-test-util/src/single.ts:138:32

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/3]⎯

 FAIL  test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_with_deposit_right_before_fork
Error: Invalid state root at slot 96, expected=0x5241d607d5383dd1daf9d059ca01506f1b80b837f85329fd6890a5612d22253d, actual=0xc474453d632eabffb5d408eaddf857918129b97e5bf40a7f4994bf8b04dc55fb
 ❯ Module.stateTransition ../state-transition/src/stateTransition.ts:126:13
 ❯ testFunction test/spec/presets/transition.test.ts:57:19
     55|         for (let i = 0; i < meta.blocks_count; i++) {
     56|           const signedBlock = testcase[`blocks_${i}`] as allForks.SignedBeaconBlock;
     57|           state = stateTransition(state, signedBlock, {
       |                   ^
     58|             // Assume valid and available for this test
     59|             executionPayloadStatus: ExecutionPayloadStatus.valid,
 ❯ ../spec-test-util/src/single.ts:138:32

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/3]⎯

 FAIL  test/spec/presets/transition.test.ts > electra/transition/core/pyspec_tests > electra/transition/core/pyspec_tests/transition_with_non_empty_activation_queue
Error: Invalid state root at slot 96, expected=0xce646a5d98c6661bdd1c70634e1d2295449e3347b7915b6bb5ab3ac4e12ad4e7, actual=0x70461c653c78d1597687e396b0a7fbabcc45486044b34dc8f20520c964cd48f9
 ❯ Module.stateTransition ../state-transition/src/stateTransition.ts:126:13
 ❯ testFunction test/spec/presets/transition.test.ts:57:19
     55|         for (let i = 0; i < meta.blocks_count; i++) {
     56|           const signedBlock = testcase[`blocks_${i}`] as allForks.SignedBeaconBlock;
     57|           state = stateTransition(state, signedBlock, {
       |                   ^
     58|             // Assume valid and available for this test
     59|             executionPayloadStatus: ExecutionPayloadStatus.valid,
 ❯ ../spec-test-util/src/single.ts:138:32

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[3/3]⎯

 Test Files  1 failed | 12 passed (13)
      Tests  3 failed | 4367 passed | 10 skipped (4380)
   Start at  20:50:50
   Duration  218.15s (transform 2.94s, setup 220ms, collect 9.30s, tests 717.35s, environment 1ms, prepare 557ms)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.88%. Comparing base (8e875c6) to head (33bbae6).
Report is 7 commits behind head on unstable.

❗ Current head 33bbae6 differs from pull request most recent head f32d709. Consider uploading reports for the commit f32d709 to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #6779      +/-   ##
============================================
- Coverage     61.88%   61.88%   -0.01%     
============================================
  Files           562      562              
  Lines         59309    59331      +22     
  Branches       1916     1916              
============================================
+ Hits          36703    36715      +12     
- Misses        22563    22573      +10     
  Partials         43       43              

[nazarhussain]

It's good and aligned to what we already have before changing to verbose because other team members want to see all the tests which are passing as well not just the one which are failing.

We may change this flag to TEST_COMPACT_OUTPUT or something similar for clarity and to remember.

[nazarhussain]

May be use use TEST_CONSOLE_QUITE here.

[nflaig]

This should probably be merged after #6783 to document the env variables on the table @nazarhussain added in the other PR