Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are available here. According to the tests, PGO can help with achieving better performance in many cases including compilers, parsers, log solutions, databases, etc. Since this, I think trying to optimize Zen performance with PGO can be a good idea to try.

I already did some benchmarks and want to share my results here.

Test environment

• Fedora 39
• Linux kernel 6.6.9
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.75
• Zen version: the latest for now from the master branch on commit 6bd9c0138c6c514246f7dfe9ec389f97f42eeed6
• Disabled Turbo boost (for more stable results across benchmark runs)

Benchmark

For benchmark purposes, I use cargo bench --workspace. For PGO optimization I use cargo-pgo tool. The same benchmark suite was used for the PGO training phase built with cargo pgo bench -- --workspace. PGO optimized results I got with cargo pgo optimize bench -- --workspace.

All measurements are done multiple times to check reproducibility - the results are stable across runs.

The only change to the codebase that was done is commenting this line due to Kobzol/cargo-pgo#44 . Not a big deal, honestly - it should not affect the benchmark results anyway.

Results

I got the following results:

• Release: https://gist.github.com/zamazan4ik/6f2b3b9342612ef9e8ee481f2c376b20
• PGO optimized compared to Release: https://gist.github.com/zamazan4ik/2371325422797b11d68678f427a5ac0d
• (just for reference) PGO instrumentation compared to Release: https://gist.github.com/zamazan4ik/ad82dca561d4e0614d353777b075d24d

At least in the benchmarks above, there are measurable improvements in many cases. However, there is a regression in the lexer scenario - it needs to be investigated further. One of the ideas why it's happening is that some noise from criterion-based benchmarks (it was previously discussed with other PGO people from the Rust community).

Further steps

I can suggest the following action points:

• Perform more PGO benchmarks on Zen. If it shows improvements - add a note to the documentation about possible improvements in Zen performance with PGO. So Zen users will be aware of PGO effects on Zen performance and can decide to enable PGO for their Zen-based applications to achieve better performance.

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

I would be happy to answer all your questions about PGO and PLO. I hope these benchmarks will be helpful.

[stefan-gorules]

Thanks for sharing this! It's super detailed and informative.

This is something that we can definitely look into promoting. I have some questions:

1. Is it possible to use PGO for NAPI-RS or PYO3 builds?
2. Would Rust library benefit from this? Would instructions on usage of PGO be useful instead for people who want super-charge performance since we don't build Rust binaries directly?

[zamazan4ik]

Is it possible to use PGO for NAPI-RS or PYO3 builds?

Yes, it should be possible to do since under the hood it's just a simple Rust library that can be PGO-optimized. Not exactly a NAPI-RS or PYO3 but I have an example of optimizing with a PGO Rust-based Python package: pydantic/pydantic-core#741 . NAPI-RS or PYO3 shouldn't make a difference here. However, please take a look at this comment. For some yet unknown reasons, JS bindings right now fail to compile with PGO instrumentation mode (due to some strange linker errors). The issue should be investigated and somehow resolved (I think I'll report the issue to the Rustc upstream). This issue does not affect Python bindings, by the way.

Would Rust library benefit from this?

Yes! I have many positive examples of applying PGO to libraries:

• native_model: GitHub issue
• pathfinding: GitHub issue
• lol-html: GitHub issue
• JSON libraries (serde_json, rustc_serialize, simd-json): GitHub issue
• xml-rs: GitHub issue
• quick-xml: GitHub issue
• tonic: GitHub issue
• tantivy: GitHub issue

Would instructions on usage of PGO be useful instead for people who want super-charge performance since we don't build Rust binaries directly?

Even if you don't prebuild Rust libraries, knowing about PGO effects on the library performance is still can be valuable for the users. In this case, they will be aware of an additional way how to speed up the library with PGO and what performance boost they can expect from this.

For libraries, I think documenting the whole PGO compilation process is an overhead - a link to the official Rustc PGO documentation and/or cargo-pgo should be enough.

However, if you prebuild JS and Python packages, having something more for them could be a good option. E.g. you can PGO-optimize the Python package and push the PGO-optimized wheel to the pip repo.