Evaluate Profile-Guided Optimization (PGO) usage

[zamazan4ik]

Hi!

Recently I checked optimizations like Profile-Guided Optimization (PGO) and Post-Link Optimizations (PLO) improvements on multiple projects. The results are available here. According to the tests, all these optimizations can achieve better performance in many cases for many applications. I think trying to enable them for the project could be a good idea. I already did some benchmarks and want to share my results here. Hopefully, they will be helpful.

Test environment

• Fedora 39
• Linux kernel 6.8.4
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler: Rustc 1.76
• nucleo version: the latest for now from the master branch on commit a82a24999b899e588a73da830d3a6957f0fbea2b
• Disabled Turbo boost (for better results consistency across runs)

Benchmark

For benchmark purposes, I use this workload. Release build is done with cargo build --release, PGO instrumentation - with cargo pgo build, PGO optimization - with cargo pgo optimize build. cargo-pgo is used for performing all PGO-related routines. The training scenario is running the whole program once.

All tests are done on the same machine, done multiple times (results are the same), with the same background "noise" (as much as I can guarantee of course) - the results are reproducible at least on my machine. taskset -c 0 is used for better stability across runs (to reduce OS scheduler influence).

Results

Here are the results:

• Release: https://gist.github.com/zamazan4ik/7f01adf81fa557817578c32550ee44fb
• PGO-optimized compared to Release: https://gist.github.com/zamazan4ik/7ffb9ab808efe6390cbb62d25df5fca3
• (just for reference) PGO-instrumented compared to Release: https://gist.github.com/zamazan4ik/d31d1b06e5f74a1e546c75901bc802a6

At least to the simple benchmarks above, PGO measurably improves the library's performance.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks on the project. If it shows improvements - add a note to the documentation (README file?) about possible improvements in the library performance with PGO.
• Test more advanced approaches like Post-Link Optimization (PLO) with tools like LLVM BOLT

Here are some examples of how PGO optimization is integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I have some examples of how PGO information looks in the documentation:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md

Please, do not treat the discussion like a bug or smth like that. It's just a benchmark report with possible improvement ideas for the project.