Lincheck benchmarks

[eupp]

This PR adds benchmarking infrastructure to the Lincheck. It includes:

• API to track progress of lincheck test run: LincheckRunTracker interface.
• Statistics gathering infrastructure based on LincheckRunTracker API.
• New gradle tasks to run benchmarks (tests and benchmarks are put into separate source sets).
• Several benchmarks for standard concurrent data structures and synchronization primitives.
• Script to draw various plots based on collected benchmark statistics (using Lets-Plot Kotlin library).

[eupp]

Hi @ndkoval !
I've rebased this PR on develop and fixed all previous review comments.
Please have a look.

[ndkoval]

Please take a look how it is done in the Kotlin coroutines repo

[eupp]

Could you please point out to some specific part or example of gradle scripts you mean, or describe what is the problem with the current code.

The kotlin coroutines build setup is quite large and complicated, what exactly should I look for?

It looks like they use subprojects approach, and benchmarks, in particular, is one of the subprojects.

Is that what you mean? If yes, what are the benefits of using subprojects for benchmarks compared to having them as another source set?

[ndkoval]

Making it a subproject makes the build script simpler and modularized, with all the dependencies and hacks necessary for benchmarks being configured for benchmarks only and in a separate file.

[ndkoval]

Also, it is much shorter 🙃

[eupp]

Ok, will try it.

[eupp]

I have tried the sub-projects approach and it haven't worked out.

The problem is that if we extract the benchmarks into a separate sub-project, then we have to add a dependency from benchmarks sub-module to a the lincheck itself.
But, because the lincheck itself is not a sub-project, but a top-level project, we cannot do (i.e., adding dependency { implementation(rootProject) } does not work).

It seems that currently Lincheck project structure does not adhere to the standard recommended project structure, where the top-level build.gradle.kts simply imports sub-projects and defines common configuration, and individual sub-projects' build.gradle.kts files define build scripts for sub-components, like lib, app, benchmarks, etc: https://docs.gradle.org/current/userguide/multi_project_builds.html

The kotlinx-coroutines project also follows this convention, and thus we cannot simply re-use it in Lincheck.
It defines a kotlinx-coroutines-core sub-project with core coroutine library primitives.
The benchmarks sub-project then
declares a dependency on the core subproject.

And it looks like other kotlin libraries follow the same approach, where the top-level build.gradle.kts is only an "umbrella" script and everything else is split into sub-projects. Most libraries also define "core" sub-project, which is either named core or $library-name:

• https://github.com/Kotlin/kotlinx-atomicfu
• https://github.com/Kotlin/kotlinx-rpc
• https://github.com/Kotlin/kotlinx-io

For us it means that we would have to move the current top-level build.gradle.kts file into a new "core" sub-project, which we could name either lincheck, lincheck-core, or core.
But this is a major change, which I think we should implement in a separate PR, not as a part of benchmarks PR.
Also this change is likely to produce a lot of merge-conflicts with existing active feature branches, so we should plan it thoughtfully and coordinate with other folks who currently work on various features in Lincheck.

I have very limited knowledge of gradle, so perhaps there is some way to overcome the problems described above. But I do not know how to do without major refactoring of project structure.