How tests could be safeguarded for parallel running

[PavelVozenilek]

This is short followup to recent discord discussion about support for tests.

If tests are required to run in parallel (not a good idea, but whatever), then the "replace this and that function" feature is prone to nasty race conditions.

It could be made safe, by using read-write lock, no matter how the test runner is implemented. It would look like:

// ordinary test, no replacements inside, compiler know this
TEST() 
{
  enter-read-rw-lock(); // compiler inserted

  .. the test itself

  leave-read-rw-lock(); // compiler inserted
}


// test which replaces some function, compiler know this
TEST()
{
  enter-write-rw-lock(); // compiler inserted

  replace foo = (){} // compiler changes function pointer value here

  ... the test itself

  ... // here compiler returns previous function pointer value back
  
  leave-write-rw-lock(); // compiler inserted
}

This way by-design-MT-unsafe tests would be serialized and would not interfere with normal tests. I assume RW lock is fair and properly implemented. Performance penalty should be negligible.

This still does't help with tests not designed for parallel run and containing subtle bugs, but that's user's fault, wanting concurrency and not handling it properly.

---

Btw, there may be another requirement for a test runner, ability to run tests in random order, to detect certain class of bugs. If this is implemented (it is easy), then the randomizer seed for the current run should be available somewhere. If tests fail, one should be able to rerun them again with exactly the same seed (meaning no more randomness at this moment).

One could write the seed into a text file somewhere, before the tests run.

It is also possible to spawn small helper process in background, which stores the seed. If all tests finish OK as they should, the background process would close itself. If something failed, then the background process would stay alive, and the next time the tests are run, test runner would retrieve and use the stored seed from that background process. This way test-order-related bugs will be reproducible, and user would not be bothered about text files, how to reseed the randomizer, etc.

[PavelVozenilek]

Since the language has @static feature: each test should be conceptually seen as an independent program, which always expects the clean sheet. If tests are compiled in, the compiler could also generate helper function to reinitialize all static data in the program.

It would then be used like:

TEST()
{
  initialize-all-statics(); // invisibly called the compiler
  ...
}

TEST(some-parameters)
{
  initialize-all-statics(); // invisibly called the compiler
  ...
}

main :: proc()
{
  initialize-all-statics(); // invisibly called the compiler when in test mode
  ...
}

I guess number of static values will be rather low, so it shouldn't result in performance hit. I cannot think of any desired situation, when someone would like test modified value to stay. (Data about tests should be collected by the test runner.)

MT safety of statics is up to the user (and yet another argument, why to run tests only serially).