Modern Cross Platform Job Scheduler and Runner with Job Stealing for Game Engines

A job stealing implementation for high performance engines. This is a MoBaGEn module and part of a C++ game development course.

CI:

Join us:

Quality:

Metrics:

Documentation

https://infinibrains.github.io/concurrency/

Requirements

• Uses automated tests.
• Export documentation.
• Cross platform. Should work on Android, iOS, compiles for VS, Clang, GCC and Emscripten.
  • Windows Visual Studio
  • MacOS XCode
  • Linux Ubuntu GCC
  • Emscripten for Web
  • Android
  • iOS
• Work stealing thread runner
• Scheduler with dependency management
• Uses the most modern cpp version on all platforms listed.
• Relies on std where is possible, or acceptable.
• Small and simple.
• Uses CMake and CPM.
• Generate bindings for other languages if possible.
• Offload jobs to gpu if possible.
• Scheduler should manage coroutines/fibers too
• Stop a job, not only the thread
• A job can return a calculation to the caller
• Wait for a selected group of jobs to finish. Fill an array with responses
• Uses https://github.com/Naios/function2
• Automatically generate bindings for dynamic loading including webidl - scapix?.
• ???

References

• https://graphitemaster.github.io/fibers/
• https://www.youtube.com/watch?v=A7sVFJLJM-A - An Introduction to Multithreading in C++20 - Anthony Williams - CppCon 2022

Features

• Modern CMake practices
• Suited for single header libraries and projects of any scale
• Clean separation of library and executable code
• Integrated test suite
• Continuous integration via GitHub Actions
• Code coverage via codecov
• Code formatting enforced by clang-format and cmake-format via Format.cmake
• Reproducible dependency management via CPM.cmake
• Installable target with automatic versioning information and header generation via PackageProject.cmake
• Automatic documentation and deployment with Doxygen and GitHub Pages
• Support for sanitizer tools, and more

Usage

Adjust the template to your needs

• Use this repo as a template.
• Replace all occurrences of "Concurrency" in the relevant CMakeLists.txt with the name of your project
  • Capitalization matters here: Concurrency means the name of the project, while concurrency is used in file names.
  • Remember to rename the include/concurrency directory to use your project's lowercase name and update all relevant #includes accordingly.
• Replace the source files with your own
• For header-only libraries: see the comments in CMakeLists.txt
• Add your project's codecov token to your project's github secrets under CODECOV_TOKEN
• Happy coding!

Eventually, you can remove any unused files, such as the standalone directory or irrelevant github workflows for your project. Feel free to replace the License with one suited for your project.

To cleanly separate the library and subproject code, the outer CMakeList.txt only defines the library itself while the tests and other subprojects are self-contained in their own directories. During development it is usually convenient to build all subprojects at once.

Build and run the standalone target

Use the following command to build and run the executable target.

cmake -S standalone -B build/standalone
cmake --build build/standalone
./build/standalone/Concurrency --help

Build and run test suite

Use the following commands from the project's root directory to run the test suite.

cmake -S test -B build/test
cmake --build build/test
CTEST_OUTPUT_ON_FAILURE=1 cmake --build build/test --target test

# or simply call the executable: 
./build/test/ConcurrencyTests

To collect code coverage information, run CMake with the -DENABLE_TEST_COVERAGE=1 option.

Run clang-format

Use the following commands from the project's root directory to check and fix C++ and CMake source style. This requires clang-format, cmake-format and pyyaml to be installed on the current system.

cmake -S test -B build/test

# view changes
cmake --build build/test --target format

# apply changes
cmake --build build/test --target fix-format

See Format.cmake for details. These dependencies can be easily installed using pip.

pip install clang-format==14.0.6 cmake_format==0.6.11 pyyaml

Build the documentation

The documentation is automatically built and published whenever a GitHub Release is created. To manually build documentation, call the following command.

cmake -S documentation -B build/doc
cmake --build build/doc --target GenerateDocs
# view the docs
open build/doc/doxygen/html/index.html

To build the documentation locally, you will need Doxygen, jinja2 and Pygments installed on your system.

Build everything at once

The project also includes an all directory that allows building all targets at the same time. This is useful during development, as it exposes all subprojects to your IDE and avoids redundant builds of the library.

cmake -S all -B build
cmake --build build

# run tests
./build/test/ConcurrencyTests
# format code
cmake --build build --target fix-format
# run standalone
./build/standalone/Concurrency --help
# build docs
cmake --build build --target GenerateDocs

Additional tools

The test and standalone subprojects include the tools.cmake file which is used to import additional tools on-demand through CMake configuration arguments. The following are currently supported.

Sanitizers

Sanitizers can be enabled by configuring CMake with -DUSE_SANITIZER=<Address | Memory | MemoryWithOrigins | Undefined | Thread | Leak | 'Address;Undefined'>.

Static Analyzers

Static Analyzers can be enabled by setting -DUSE_STATIC_ANALYZER=<clang-tidy | iwyu | cppcheck>, or a combination of those in quotation marks, separated by semicolons. By default, analyzers will automatically find configuration files such as .clang-format. Additional arguments can be passed to the analyzers by setting the CLANG_TIDY_ARGS, IWYU_ARGS or CPPCHECK_ARGS variables.

Ccache

Ccache can be enabled by configuring with -DUSE_CCACHE=<ON | OFF>.

FAQ

Can I use this for header-only libraries?

Yes, however you will need to change the library type to an INTERFACE library as documented in the CMakeLists.txt. See here for an example header-only library based on the template.

I don't need a standalone target / documentation. How can I get rid of it?

Simply remove the standalone / documentation directory and according github workflow file.

Can I build the standalone and tests at the same time? / How can I tell my IDE about all subprojects?

To keep the template modular, all subprojects derived from the library have been separated into their own CMake modules. This approach makes it trivial for third-party projects to re-use the projects library code. To allow IDEs to see the full scope of the project, the template includes the all directory that will create a single build for all subprojects. Use this as the main directory for best IDE support.

I see you are using GLOB to add source files in CMakeLists.txt. Isn't that evil?

Glob is considered bad because any changes to the source file structure might not be automatically caught by CMake's builders and you will need to manually invoke CMake on changes. I personally prefer the GLOB solution for its simplicity, but feel free to change it to explicitly listing sources.

I want create additional targets that depend on my library. Should I modify the main CMakeLists to include them?

Avoid including derived projects from the libraries CMakeLists (even though it is a common sight in the C++ world), as this effectively inverts the dependency tree and makes the build system hard to reason about. Instead, create a new directory or project with a CMakeLists that adds the library as a dependency (e.g. like the standalone directory). Depending type it might make sense move these components into a separate repositories and reference a specific commit or version of the library. This has the advantage that individual libraries and components can be improved and updated independently.

You recommend to add external dependencies using CPM.cmake. Will this force users of my library to use CPM.cmake as well?

CPM.cmake should be invisible to library users as it's a self-contained CMake Script. If problems do arise, users can always opt-out by defining the CMake or env variable CPM_USE_LOCAL_PACKAGES, which will override all calls to CPMAddPackage with the according find_package call. This should also enable users to use the project with their favorite external C++ dependency manager, such as vcpkg or Conan.

Can I configure and build my project offline?

No internet connection is required for building the project, however when using CPM missing dependencies are downloaded at configure time. To avoid redundant downloads, it's highly recommended to set a CPM.cmake cache directory, e.g.: export CPM_SOURCE_CACHE=$HOME/.cache/CPM. This will enable shallow clones and allow offline configurations dependencies are already available in the cache.

Can I use CPack to create a package installer for my project?

As there are a lot of possible options and configurations, this is not (yet) in the scope of this template. See the CPack documentation for more information on setting up CPack installers.

This is too much, I just want to play with C++ code and test some libraries.

Perhaps the MiniCppStarter is something for you!

Related projects and alternatives

• ModernCppStarter & PVS-Studio Static Code Analyzer: Official instructions on how to use the ModernCppStarter with the PVS-Studio Static Code Analyzer.
• cpp-best-practices/gui_starter_template: A popular C++ starter project, created in 2017.
• filipdutescu/modern-cpp-template: A recent starter using a more traditional approach for CMake structure and dependency management.
• vector-of-bool/pitchfork: Pitchfork is a Set of C++ Project Conventions.

Star History

Coverage