Skip to content

Commit

Permalink
bunch of changes in paper
Browse files Browse the repository at this point in the history
  • Loading branch information
danielskatz authored Aug 23, 2023
1 parent 59c7060 commit 11e881f
Showing 1 changed file with 25 additions and 26 deletions.
51 changes: 25 additions & 26 deletions paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,20 +26,20 @@ bibliography: paper.bib

# Summary

With libraries like Eigen[@eigen], Blaze[@blazelib], or CTRE[@ctre] being
With libraries like Eigen [@eigen], Blaze [@blazelib], and CTRE [@ctre] being
developed with a large tempalte metaprogrammed implementation, we're seeing
increasing computing needs at compile time. These needs might grow even larger
as C++ embeds more features over time to support and extend this kind of
practices, like compile-time containers[@more-constexpr-containers] or static
reflection[@static-reflection]. However, there is still no clear cut methodology
to compare the performance impact of different metaprogramming strategies. But,
as C++ embeds more features over time to support and extend these kinds of
practices, like compile-time containers [@more-constexpr-containers] and static
reflection [@static-reflection]. However, there is still no clear cut methodology
to compare the performance impact of different metaprogramming strategies. And
as new C++ features allows for new techniques with claimed better compile-time
performance, no proper methodologies is provided to back up those claims.
performance, no proper methodologies are provided to back up those claims.

This paper introduces **ctbench**, which is a set of tools for compile-time
This paper introduces **ctbench**, a set of tools for compile-time
benchmarking and analysis in C++. It aims to provide developer-friendly tools to
declare and run benchmarks, then aggregate, filter out, and plot the data to
analyze it. As such, **ctbench** is meant to become the first layer for a proper
analyze it. As such, **ctbench** is meant to become the first layer of a proper
scientific methodology for analyzing compile-time program behavior.

We'll first have a look at current tools for compile-time profiling and
Expand All @@ -51,49 +51,49 @@ what **ctbench** brings to overcome these limits.
C++ template metaprogramming raised interest for allowing computing libraries to
offer great performance with a very high level of abstraction. As a tradeoff for
interpreting representations of calculations at runtime, they are represented at
compile-time, and transformed directly into their own programs.
compile time, and transformed directly into their own programs.

As metaprogramming became easier with C++11 and C++17, it became more mainstream
and consequently, developers have to bear with longer compilation times without
being able to explain them. Therefore, being able to measure compilation times
is increasingly important, and being able to explain them as well. A first
is increasingly important, as is being able to explain them as well. A first
generation of tools aims to tackle this issue with their own specific
methodologies:

- Buildbench[@buildbench] measures compiler execution times for basic
- Buildbench [@buildbench] measures compiler execution times for basic
A-B compile-time comparisons in a web browser,
- Metabench[@metabench] instantiates variably sized benchmarks using embedded
- Metabench [@metabench] instantiates variably sized benchmarks using embedded
Ruby (ERB) templating and plots compiler execution time, allowing scaling
analyses of metaprograms,
- Templight[@templight] adds Clang template instantiation inspection
- Templight [@templight] adds Clang template instantiation inspection
capabilities with debugging and profiling tools.

Additionally, Clang has a built-in profiler[@time-trace] that provides in-depth
Additionally, Clang has a built-in profiler [@time-trace] that provides in-depth
time measurements of various compilation steps, which can be enabled by passing
the `-ftime-trace` flag. Its output contains data that can be directly linked to
symbols in the source code, making it easier to study the impact of specific
symbols on various stages of compilation. The output format is a JSON file meant
to be compatible with Chrome's flame graph visualizer, that contains a series of
to be compatible with Chrome's flame graph visualizer, which contains a series of
time events with optional metadata like the mangled C++ symbol or the file
related to an event. The profiling data can then be visualized using tools such
as Google's [Perfetto UI](https://ui.perfetto.dev/).

![Perfetto UI displaying a sample Clang time trace file](docs/images/perfetto-ui.png)

Clang's profiler data is very exhaustive and insightful, however there is no
Clang's profiler data is very exhaustive and insightful; however, there is no
tooling to make sense of it in the context of variable size compile-time
benchmarks. **ctbench** tries to bridge the gap by providing a tool to analyze
this valuable data. It also improves upon existing tools by providing a solution
that's easy to integrate into existing CMake projects, and generates graphs in
various formats that are trivially embeddable in documents like research papers,
web pages, or documentations. Additionally, relying on persistent configuration,
benchmark declaration and description files provides strong guarantees for
web pages, and documentation. Additionally, relying on persistent configuration,
benchmark declaration and description files provide strong guarantees for
benchmark reproductibility, as opposed to web tools or interactive profilers.

# Functionality

Originally inspired by Metabench[@metabench], **ctbench** development was
driven by the need for a similar tool that allows the observation of Clang's
Originally inspired by Metabench [@metabench], **ctbench** development was
driven by the need for a similar tool observes Clang's
time-trace files to help get a more comprehensive view on the impact of
metaprogramming techniques on compile times. A strong emphasis was put on
developer friendliness, project integration, and component reusability.
Expand All @@ -109,7 +109,7 @@ developer friendliness, project integration, and component reusability.

In addition to **ctbench**'s time-trace handling, it has a compatibility mode
for compilers that do not support it like GCC. This mode works by measuring
compiler execution time just like Metabench[@metabench] and generating a
compiler execution time just like Metabench [@metabench] and generating a
time-trace file that contains compiler execution time. Moreover, the tooling
allows setting different compilers per target within a same CMake build,
allowing black-box compiler performance comparisons between GCC and Clang for
Expand All @@ -122,10 +122,10 @@ scaling analysis.

# Statement of interest

**ctbench** was first presented at the CPPP 2021 conference[@ctbench-cppp21]
**ctbench** was first presented at the CPPP 2021 conference [@ctbench-cppp21],
which is the main C++ technical conference in France. It is being used to
benchmark examples from the poacher[@poacher] project, which was briefly
presented at the Meeting C++ 2022[@meetingcpp22] technical conference.
benchmark examples from the poacher [@poacher] project, which was briefly
presented at the Meeting C++ 2022 [@meetingcpp22] technical conference.

# Related projects

Expand All @@ -139,8 +139,7 @@ presented at the Meeting C++ 2022[@meetingcpp22] technical conference.

# Acknowledgements

We acknowledge contributions from Philippe Virouleau and Paul Keir for their
insightful suggestions.
We acknowledge contributions and insightful suggestions from Philippe Virouleau and Paul Keir.

# References

Expand Down

0 comments on commit 11e881f

Please sign in to comment.