Evaluate using Profile-Guided Optimization (PGO) and Post Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. PGO results for LLVM-related tooling are here. According to the tests, PGO usually helps with the compiler and compiler-like workloads (like static analysis or code formatters) - e.g. Clang gets +20% compilation speed with PGO. Since this, I think trying to optimize Wgpu tools like naga-cli, would be a good idea. I already did some PGO benchmarks and want to share my results here.

Test environment

I performed tests on two configurations: Linux and macOS.

Linux:

• Fedora 39, KDE, Wayland
• Linux kernel 6.5.11
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.73
• GPU: Nvidia 3080 Ti
• Wgpu version: the latest for now from the trunk branch on commit 611471c4bc8b1adab4d7d96fd2ba8bae82dccb28

macOS:

• OS: macOS 13.6
• CPU: Macbook M1 Pro 6+2
• Wgpu version: the latest for now from the trunk branch on commit 611471c4bc8b1adab4d7d96fd2ba8bae82dccb28

Benchmark

For benchmark purposes, I run cargo bench --workspace --no-fail-fast -p naga --bench criterion. PGO training phase was done on the same scenario with cargo pgo bench -- --workspace --no-fail-fast -p naga --bench criterion. PGO optimization is done with cargo-pgo.

Benchmarks are done on the same machine, with the same background "noise". The benchmark was performed multiple times - the results are reproducible.

Results

I got the following results (in cargo bench format):

Linux:

• PGO-optimized compared to Release bench: https://gist.github.com/zamazan4ik/ef03ce9cee39839202f716098841bc9f
• (just for reference) PGO-instrumented compared to Release bench: https://gist.github.com/zamazan4ik/df96b8ddc081a354b63cce69710ed476

macOS:

• PGO-optimized compared to Release bench: https://gist.github.com/zamazan4ik/fe21310f2638cc7ce9355d5ec0a719d2
• (just for reference) PGO-instrumented compared to Release bench: https://gist.github.com/zamazan4ik/3958de6255af93ec68b7526ea125ab8c

At least in the scenario above PGO helps with achieving better performance.

One benchmark does not work on my setup but right I have no idea how to fix it.

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks on Wgpu tools. If it shows improvements - add a note to the documentation about possible improvements in Wgpu tools' performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize Wgpu tools according to their workloads.
• Optimize pre-built binaries with PGO (if it's possible to prepare a generic enough profile).

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Here are some examples of how PGO optimization is already integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I am not sure how performance is critical for Wgpu now. If it isn't the biggest priority right now - could be a nice feature to get in the future.

[cwfitzgerald]

This is all very lovely work and a good reference for anyone interested in PGO themselves - but I don't think there's anything actionable that we can do here.

We're fundamentally a library that people use in their own programs, so they are in charge of the compile settings used. We do have a few binaries (wgpu-info, naga-cli, etc) but we don't actually publish binaries for them as they're just developer tools. Everyone who uses them uses them through cargo install which recompiles the binary on the system. We are again not in control of building.

It's good to know that PGO does help naga (it makes sense, it's very branch heavy).

[zamazan4ik]

This is all very lovely work and a good reference for anyone interested in PGO themselves - but I don't think there's anything actionable that we can do here.

Even if Wgpu is a library-first project, I think it's possible to do several things.

We can add a note somewhere in the Wgpu documentation about possible PGO effects (at least based on the benchmarks above). This information could be useful for:

• For the people who use wgpu as a library. So they can estimate somehow benefits by enabling PGO for Wgpu and decide to recompile their project with PGO.
• For the people who use and/or maintain Wgpu binaries (naga-cli is probably the most performance-critical thing but I am not sure). Based on PGO benefits numbers, they can decide to recompile their local binaries with PGO to get an additional performance boost (if it's critical for them).

Here I have some PGO-related documentation examples in several projects:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md (yet, it's actually about PGO too)

If we are talking about binary optimization, even if wgpu does not distribute prebuilt binaries, we can make building Wgpu tools with PGO easier via some predefined build tweaks. The idea is quite simple:

• Prepare some generic enough training workload (if it's possible to do so)
• Write some Cargo/CMake/make/whatever integration to build a tool like naga-cli with the profile from the workload above. Something like make pgooptimize. Not sure about how it could be integrated into the Cargo build workflow - perhaps something with build.rs.
• A user locally runs simply make pgooptimize and gets PGO optimized version

In this case, for the users would be easier to get PGO optimized program version.

I have several examples of how it looks like in other projects:

• Rustc: a CI tool for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script.
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• Swift: CMake script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag
• ISPC: CMake scipts
• NodeJS: Configure script
• Android Open Source Project (AOSP):
  • Official documentation
  • Committed PGO profiles: repository
• DMD: Custom build rule
• LDC: GitHub action
• tsv-utils: Makefile
• Erlang OTP: Makefile
• Clingo (PGO enabled only in Spack): Package recipe

Which suggestions are worth it from the "benefits/implementation cost" point of view - is all up to you. IMO putting a note into the documentation is a cheap enough change, providing a build infra PGO integration could be a more time-consuming task to complete.