We should setup a dedicated worker for benchmarks or explore using a service like bencher to get reliable benchmarks built into our CI workflow.

Hi, I work on bencher! If you wanted to try it out (it's free!), it should just take a few clicks to install from the GitHub marketplace
We've got some more-detailed installation instructions (with pictures!) as well.

Let me know if I can help with anything :)

@kirbyquerby I tried enabling it, but we use docker for integration testing in our benchmarks -- this is because we benchmark against various databases.

I see in the bencher configuration you can depend on services based on docker, but can we actually just get access to a docker socket?

@kirbyquerby @odeke-em Are there any rate limits on bencher? I've been playing with things over at https://github.com/jzelinskie/benchpress and half of my pushes don't run. I was pulling my hair out thinking I had an invalid config, but it's not starting builds even without a config.

I'm experimenting with using docker-in-docker so that our test suite can run unmodified.

Hey @jzelinskie, Bencher runs only when there is a code change in .go files or when there is a configuration update. The reason for that is that in a PR, lots of diverse changes can happen but shouldn't be wasting precious machine time and CPU re-benchmarking. We shall document this on https://bencher.orijtech.com/

@odeke-em Thanks for the response -- that makes perfect sense.

Do you have any recommendations on how we could get access to the Docker daemon? Our tests use the Docker API to spin up/down images it needs for end-to-end tests. I've been playing with docker-in-docker, but that hasn't seemed to work.

I'm hesitant to provide direct access to the docker daemon because we run benchmarks on bare metal. This makes it tricky to ensure the state of the machine is reset after a benchmark is run. The current setup provides a intentionally trimmed-down set of docker features to try and minimize the attack surface -- I'm not well-versed enough to understand all the potential pitfalls of allowing arbitrary access to all of docker.

docker-in-docker seems promising, though -- I'll take a look at what it would take to make it work with bencher.