Running BenchmarkDotNet on Azure Kubernetes Service (AKS)

[mhadhbi]

Hello,

We are currently running CPU-intensive benchmarks using BenchmarkDotNet v0.13.0 inside containers deployed on Azure AKS (Linux). These benchmarks make heavy use of hardware intrinsics (e.g., AVX2, AVX-512), and some of them also involve moderate I/O operations (e.g, file access).

We are trying to identify the best practices and VM SKU choices for running these benchmarks reliably and reproducibly in a cloud-native environment.

We would appreciate your thought on the following topics:

• Azure VM SKU recommendations: Which VM SKU in Azure (e.g., AMD Dadsv6, Intel Fsv2) would you recommend for workloads that rely heavily on CPU intrinsics and some I/O?

• Consistency across VM instances: Have you observed stable results across different instances of the same SKU in cloud environments? Any tips to reduce variability?

• Iteration settings for convergence: What configuration (IterationCount, WarmupCount) works best to achieve statistically meaningful results in a cloud setup?

• Comparison strategies: We currently store a fixed reference benchmark in our Git repository and update it only when differences are justified.
  We are also considering comparing performance before and after a commit when code changes occur.
  Do you have best practices or tooling suggestions for these approaches?

• Running inside Kubernetes: Are there known caveats when running BenchmarkDotNet inside Kubernetes pods with CPU requests and limits? Does BenchmarkDotNet behave differently under cgroups or containerized environments?

• Cloud VM-specific issues: Have issues related to running benchmarks in cloud VMs (e.g., noisy neighbors, virtualization overhead) been addressed in recent versions of BenchmarkDotNet, or should we apply specific configuration tweaks?

• .NET Runtime configuration: Are there recommended runtime settings for accurate benchmarking in cloud environments? For example, should we explicitly configure GC mode (Server vs Workstation) or tweak JIT settings to reduce variability?

• Accepted variance between runs: What is a reasonable threshold for performance deviation between two benchmark runs in a cloud environment? For example, is +/- 10% considered acceptable, or should we aim for tighter bounds?

Current Benchmark Setup

• Benchmarks are run in containers based on .NET 8.0.
• Pods are scheduled on dedicated AKS node pools (currently testing AMD Dadsv6).
• Some benchmarks include I/O operations (e.g., reading/writing files).
• We are using version v0.13.0 of BenchmarkDotNet.

Any insights, recommendations, or community experiences would be greatly appreciated!

Thanks

[AndreyAkinshin]

You listed many important topics for anyone trying to set up a reliable and scalable benchmarking infrastructure. Unfortunately, a detailed answer would likely require a dedicated book. I don't have time to write such a book at the moment, but I recommend my blog post where I conducted a small study on GitHub Actions stability: https://aakinshin.net/posts/github-actions-perf-stability/. My primary advice is to start with a small set of artificial model benchmarks and explore how they behave in your infrastructure. This quick exercise may provide valuable insights into potential challenges.

I also recommend a book I wrote: Pro .NET Benchmarking. It was written between 2017 and 2019, so some .NET-specific advice may be outdated, and it doesn't cover all your questions. However, I hope it offers a methodological basis for designing your benchmarks and avoiding common pitfalls.

If you discover any interesting insights into how Azure AKS behaves, I would appreciate it if you could share your findings with the community.