From 11e881f6af3c9a01cf81fabc147671b27a48d058 Mon Sep 17 00:00:00 2001 From: "Daniel S. Katz" Date: Wed, 23 Aug 2023 07:38:59 -0500 Subject: [PATCH 1/4] bunch of changes in paper --- paper.md | 51 +++++++++++++++++++++++++-------------------------- 1 file changed, 25 insertions(+), 26 deletions(-) diff --git a/paper.md b/paper.md index c31e4fd..9923bab 100644 --- a/paper.md +++ b/paper.md @@ -26,20 +26,20 @@ bibliography: paper.bib # Summary -With libraries like Eigen[@eigen], Blaze[@blazelib], or CTRE[@ctre] being +With libraries like Eigen [@eigen], Blaze [@blazelib], and CTRE [@ctre] being developed with a large tempalte metaprogrammed implementation, we're seeing increasing computing needs at compile time. These needs might grow even larger -as C++ embeds more features over time to support and extend this kind of -practices, like compile-time containers[@more-constexpr-containers] or static -reflection[@static-reflection]. However, there is still no clear cut methodology -to compare the performance impact of different metaprogramming strategies. But, +as C++ embeds more features over time to support and extend these kinds of +practices, like compile-time containers [@more-constexpr-containers] and static +reflection [@static-reflection]. However, there is still no clear cut methodology +to compare the performance impact of different metaprogramming strategies. And as new C++ features allows for new techniques with claimed better compile-time -performance, no proper methodologies is provided to back up those claims. +performance, no proper methodologies are provided to back up those claims. -This paper introduces **ctbench**, which is a set of tools for compile-time +This paper introduces **ctbench**, a set of tools for compile-time benchmarking and analysis in C++. It aims to provide developer-friendly tools to declare and run benchmarks, then aggregate, filter out, and plot the data to -analyze it. As such, **ctbench** is meant to become the first layer for a proper +analyze it. As such, **ctbench** is meant to become the first layer of a proper scientific methodology for analyzing compile-time program behavior. We'll first have a look at current tools for compile-time profiling and @@ -51,49 +51,49 @@ what **ctbench** brings to overcome these limits. C++ template metaprogramming raised interest for allowing computing libraries to offer great performance with a very high level of abstraction. As a tradeoff for interpreting representations of calculations at runtime, they are represented at -compile-time, and transformed directly into their own programs. +compile time, and transformed directly into their own programs. As metaprogramming became easier with C++11 and C++17, it became more mainstream and consequently, developers have to bear with longer compilation times without being able to explain them. Therefore, being able to measure compilation times -is increasingly important, and being able to explain them as well. A first +is increasingly important, as is being able to explain them as well. A first generation of tools aims to tackle this issue with their own specific methodologies: -- Buildbench[@buildbench] measures compiler execution times for basic +- Buildbench [@buildbench] measures compiler execution times for basic A-B compile-time comparisons in a web browser, -- Metabench[@metabench] instantiates variably sized benchmarks using embedded +- Metabench [@metabench] instantiates variably sized benchmarks using embedded Ruby (ERB) templating and plots compiler execution time, allowing scaling analyses of metaprograms, -- Templight[@templight] adds Clang template instantiation inspection +- Templight [@templight] adds Clang template instantiation inspection capabilities with debugging and profiling tools. -Additionally, Clang has a built-in profiler[@time-trace] that provides in-depth +Additionally, Clang has a built-in profiler [@time-trace] that provides in-depth time measurements of various compilation steps, which can be enabled by passing the `-ftime-trace` flag. Its output contains data that can be directly linked to symbols in the source code, making it easier to study the impact of specific symbols on various stages of compilation. The output format is a JSON file meant -to be compatible with Chrome's flame graph visualizer, that contains a series of +to be compatible with Chrome's flame graph visualizer, which contains a series of time events with optional metadata like the mangled C++ symbol or the file related to an event. The profiling data can then be visualized using tools such as Google's [Perfetto UI](https://ui.perfetto.dev/). ![Perfetto UI displaying a sample Clang time trace file](docs/images/perfetto-ui.png) -Clang's profiler data is very exhaustive and insightful, however there is no +Clang's profiler data is very exhaustive and insightful; however, there is no tooling to make sense of it in the context of variable size compile-time benchmarks. **ctbench** tries to bridge the gap by providing a tool to analyze this valuable data. It also improves upon existing tools by providing a solution that's easy to integrate into existing CMake projects, and generates graphs in various formats that are trivially embeddable in documents like research papers, -web pages, or documentations. Additionally, relying on persistent configuration, -benchmark declaration and description files provides strong guarantees for +web pages, and documentation. Additionally, relying on persistent configuration, +benchmark declaration and description files provide strong guarantees for benchmark reproductibility, as opposed to web tools or interactive profilers. # Functionality -Originally inspired by Metabench[@metabench], **ctbench** development was -driven by the need for a similar tool that allows the observation of Clang's +Originally inspired by Metabench [@metabench], **ctbench** development was +driven by the need for a similar tool observes Clang's time-trace files to help get a more comprehensive view on the impact of metaprogramming techniques on compile times. A strong emphasis was put on developer friendliness, project integration, and component reusability. @@ -109,7 +109,7 @@ developer friendliness, project integration, and component reusability. In addition to **ctbench**'s time-trace handling, it has a compatibility mode for compilers that do not support it like GCC. This mode works by measuring -compiler execution time just like Metabench[@metabench] and generating a +compiler execution time just like Metabench [@metabench] and generating a time-trace file that contains compiler execution time. Moreover, the tooling allows setting different compilers per target within a same CMake build, allowing black-box compiler performance comparisons between GCC and Clang for @@ -122,10 +122,10 @@ scaling analysis. # Statement of interest -**ctbench** was first presented at the CPPP 2021 conference[@ctbench-cppp21] +**ctbench** was first presented at the CPPP 2021 conference [@ctbench-cppp21], which is the main C++ technical conference in France. It is being used to -benchmark examples from the poacher[@poacher] project, which was briefly -presented at the Meeting C++ 2022[@meetingcpp22] technical conference. +benchmark examples from the poacher [@poacher] project, which was briefly +presented at the Meeting C++ 2022 [@meetingcpp22] technical conference. # Related projects @@ -139,8 +139,7 @@ presented at the Meeting C++ 2022[@meetingcpp22] technical conference. # Acknowledgements -We acknowledge contributions from Philippe Virouleau and Paul Keir for their -insightful suggestions. +We acknowledge contributions and insightful suggestions from Philippe Virouleau and Paul Keir. # References From a05b29ac9a697f3c06aeb06bb2bc6c4547c5b67c Mon Sep 17 00:00:00 2001 From: "Daniel S. Katz" Date: Wed, 23 Aug 2023 07:41:34 -0500 Subject: [PATCH 2/4] bib case changes --- paper.bib | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/paper.bib b/paper.bib index a92e928..be5f58e 100644 --- a/paper.bib +++ b/paper.bib @@ -33,14 +33,14 @@ @misc{metabench } @misc{poacher, - title = {poacher: C++ compile-time compiling experiments}, + title = {poacher: {C}++ compile-time compiling experiments}, author = {Jules {Penuchot}}, year = {2020}, url = {https://github.com/jpenuchot/poacher/}, } @article{templight, - title = {Measuring Compilation Time of C++ Template Metaprograms}, + title = {Measuring Compilation Time of {C}++ Template Metaprograms}, author = {Porkol{\'a}b, Zolt{\'a}n and Mihalicza, J{\'o}zsef and Pataki, Norbert}, year = {2009}, @@ -50,7 +50,7 @@ @article{templight @inproceedings{metadebug, author = {Porkol\'{a}b, Zolt\'{a}n and Mihalicza, J\'{o}zsef and Sipos, \'{A}d \'{a}m}, - title = {Debugging C++ Template Metaprograms}, + title = {Debugging {C}++ Template Metaprograms}, year = {2006}, isbn = {1595932372}, publisher = {Association for Computing Machinery}, @@ -84,7 +84,7 @@ @inproceedings{metadebug @misc{ctre, author = {Hana {Dusíková}}, - title = {Compile Time Regular Expression in C++}, + title = {Compile Time Regular Expression in {C}++}, url = {https://github.com/hanickadot/compile-time-regular-expressions}, year = {2018}, } @@ -106,14 +106,14 @@ @online{time-trace @online{meetingcpp22, author = {Paul {Keir}, Joel {Falcou}, Jules {Penuchot}, Andrew {Gozillon}}, - title = {Meeting C++ - A totally constexpr standard library}, + title = {Meeting {C}++ - A totally constexpr standard library}, year = {2022}, url = {https://www.youtube.com/watch?v=ekFPm7e__vI}, } @online{sciplot, author = {Allan {Leal}}, - title = {Sciplot: A modern C++ scientific plotting library powered by gnuplot}, + title = {Sciplot: A modern {C}++ scientific plotting library powered by gnuplot}, year = {2018}, url = {https://sciplot.github.io/}, } @@ -127,14 +127,14 @@ @online{buildbench @misc{cest, author = {Paul {Keir}}, - title = {Towards a constexpr version of the C++ standard library}, + title = {Towards a constexpr version of the {C}++ standard library}, url = {https://github.com/pkeir/cest}, year = {2020}, } @misc{ctpg, author = {Piotr {Winter}}, - title = {C++ Compile Time Parser Generator}, + title = {{C}++ Compile Time Parser Generator}, url = {https://github.com/peter-winter/ctpg}, year = {2021}, } @@ -148,7 +148,7 @@ @misc{fmt @misc{blazelib, author = {Klaus {Iglberger}}, - title = {Blaze C++ Linear Algebra Library}, + title = {Blaze {C}++ Linear Algebra Library}, url = {https://bitbucket.org/blaze-lib}, year = {2012}, } From 204b806e76c2086c1e17722fba1afcb422d0de04 Mon Sep 17 00:00:00 2001 From: Jules P?nuchot Date: Sat, 2 Sep 2023 15:45:32 +0200 Subject: [PATCH 3/4] added JOSS status + bibtex --- readme.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/readme.md b/readme.md index 72fcaf5..70dfd1e 100644 --- a/readme.md +++ b/readme.md @@ -4,6 +4,8 @@ Compiler-assisted benchmarking for the study of C++ metaprogram compile times. - Online documentation: https://jpenuchot.github.io/ctbench-docs/ - Discord server: https://discord.gg/NvJFFrdS7p +[![DOI](https://joss.theoj.org/papers/10.21105/joss.05165/status.svg)](https://doi.org/10.21105/joss.05165) + ctbench allows you to declare and generate compile-time benchmark batches for given ranges, run them, aggregate and wrangle Clang profiling data, and plot them. @@ -233,3 +235,20 @@ per benchmark case. In this case, you would then get 3 graphs - [Pyperf - Tune the system for benchmarks]( https://pyperf.readthedocs.io/en/latest/system.html) - [Metabench](https://github.com/ldionne/metabench) + +## Citing ctbench + +```bibtex +@article{Penuchot2023, + doi = {10.21105/joss.05165}, + url = {https://doi.org/10.21105/joss.05165}, + year = {2023}, + publisher = {The Open Journal}, + volume = {8}, + number = {88}, + pages = {5165}, + author = {Jules Penuchot and Joel Falcou}, + title = {ctbench - compile-time benchmarking and analysis}, + journal = {Journal of Open Source Software}, +} +``` From 19bd4ca8a530d2ec96e0ccbc80b98f40cfe7bacc Mon Sep 17 00:00:00 2001 From: Jules P?nuchot Date: Wed, 15 Nov 2023 14:49:33 +0100 Subject: [PATCH 4/4] switch to steady_clock instead of high_resolution_clock for compiler execution time measurement --- compiler-launcher/compiler-launcher.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/compiler-launcher/compiler-launcher.cpp b/compiler-launcher/compiler-launcher.cpp index 5f2956f..f106ff5 100644 --- a/compiler-launcher/compiler-launcher.cpp +++ b/compiler-launcher/compiler-launcher.cpp @@ -52,7 +52,7 @@ inline int get_timetrace_file(std::filesystem::path const time_trace_file_dest, // Run program and measure CPU time - using exec_clock_t = ch::high_resolution_clock; + using exec_clock_t = ch::steady_clock; exec_clock_t::time_point const exec_t0 = exec_clock_t::now(); int const exit_code = boost::process::system(command_args);