Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: rebasing @localbtime and friends from LocalScopeBenchmarks.jl #150

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

miguelraz
Copy link

@miguelraz miguelraz commented Oct 1, 2019

This is a WIP PR to merge a @localbtime and friends macro into BenchmarkTools.jl .

There's a thread with reported issues here and this PR took existing code from LocalScopeBenchmarks.jl, with all credit to @rdeits.

This PR:

  • adds a dependency on MacroTools and Ordered collections for the local sourcewalking helper functions
  • exports @localbenchmark, @localbtime, @localbelapsed
  • adds tests for said functions.
  • bumps the end position semver number by 1
  • changes the statistical metric for the local benchmarks to be the mean and not the median (seemed more robust locally, modulo CI woes.)

All feedback is welcome.

Another welcome feedback will be on how the new tests for local time regression measurement should be done: I found that when running them locally, the first run would pass all tests, then fail due only to the regression threshold set. I think it's hitting the microbenchmarks remembering cache history.

Lastly, this PR does not:

  • make a judgement on how to/if at all to deprecate @btime et al.

That is left as a discussion for another time, and for now I will focus on landing this functionality.

Ping also to @jlperla, @ChrisRackauckas.

@miguelraz
Copy link
Author

The tests are failing / and passing due to a brittle judging function. One idea would be to judge loosely min, mean and median and if 2/3 pass then the judgement is considered invariant.

@gdalle gdalle marked this pull request as draft June 13, 2023 06:34
@gdalle
Copy link
Collaborator

gdalle commented Jun 13, 2023

Hey @miguelraz, just got onboarded trapped as a maintainer, and I wanted to see if this was still relevant some 4 years later?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants