v1.3.1

Chairmarks v1.3.1

Diff since v1.3.0

Merged pull requests:

• Keep old 9-arg sample constructor and discard checksum (#157) (@LilithHafner)

Closed issues:

• Sample no longer accepts evals::Int (#156)

v1.3.0

Chairmarks v1.3.0

Diff since v1.2.2

Major new features

• Experimental support for comparative benchmarking with @b f,g, @b setup f,g, etc.
• Add configurable defaults Chairmarks.DEFAULTS.seconds = 1

Technically breaking changes

• The parsing and return values of @b f,g, @b setup f,g, etc. has changed to accommodate comparative benchmarking
• The checksum keyword argument and the checksum field of Sample which have been obsolete since Julia 1.8 have been removed. We still compute checksums versions of Julia prior to 1.8

Significant bugfixes/measurement improvements

• Eliminate fractional allocations due to nonconstant global access in some cases

Relevant updates internals

• Major CI overhaul resulting in much faster results and more system coverage
• PkgEval should now pass

Merged pull requests:

• Support Chairmarks.DEFAULTS.seconds (#121) (@LilithHafner)
• Update doc typo in Chairmarks.jl (#126) (@mestinso)
• Allow nonconstant globals to not allocate in 1.12 tests (#130) (@LilithHafner)
• Support comparative benchmarking (#131) (@LilithHafner)
• Fix some bugs for early return cases when comparing and add tests (#132) (@LilithHafner)
• Remove checksum plumbing and configurability (#133) (@LilithHafner)
• Make benchmark only accept a tuple of fs instead of either a tuple or a single f (#134) (@LilithHafner)
• Share setup between samples of compared functions and add test (#135) (@LilithHafner)
• Fix and test summarize being broken for tuples of length other than 2 due to typo (#136) (@LilithHafner)
• Optimize load time and TTFX for non-comparison cases. (#137) (@LilithHafner)
• fix and test getting reasonable results when comparing more than 2 functions (#138) (@LilithHafner)
• remove two expected warnings from test output that are due to testing not Chairmarks itself (#140) (@LilithHafner)
• Make tests (excluding RegressionTests.jl) run 37% faster (#141) (@LilithHafner)
• Overhaul CI strategy (#142) (@LilithHafner)
• Fixups for 32-bit (#143) (@LilithHafner)
• Eliminate reported fractional allocations in the case of nonconstant global access (#144) (@LilithHafner)
• Switch some windows CI runts from x86 to x64. (#145) (@LilithHafner)
• Get codecov working reliably (#146) (@LilithHafner)
• Move donotdelete definition to compat (#147) (@LilithHafner)
• Tell the compiler about the value of the input to f if it is a type (and test) (#148) (@LilithHafner)
• Update docs for chagnes since 1.2.2 (#150) (@LilithHafner)
• Stop pretending to run RegressionTests on main (#151) (@LilithHafner)
• Only process and upload code coverage results in two configurations. (#152) (@LilithHafner)
• Improve error message in the absence of arguments to @b and add tests (#153) (@LilithHafner)
• Remove method redefinition from tests to avoid warning (#154) (@LilithHafner)
• Release 1.3.0! (#155) (@LilithHafner)

Closed issues:

• Support comparative benchmarking (#53)
• PSA: It is possible to use BenchmarkTools.BenchmarkGroup with Chairmarks (#70)
• [Discussion] Global settings/defaults (#71)
• benchmark results oscillate (#79)
• Use Base._stable_typeof (#107)
• Edit links in docs are sometimes broken (#113)
• feature request to pass out the test function output (like @Elapsed, @Btime can do) (#127)
• Fractional allocations on things that should clearly be deterministicc (#128)
• Forcing always to do a warm start (#129)
• Better CI strategy to reduce CI time and increase coverage (#139)
• @be with no args gives bad error message on main (#149)

v1.2.2

Chairmarks v1.2.2

Diff since v1.2.1

• Bugfix for seconds=Inf (thanks @gdalle for reporting)
• Bugfix for parsing standalone literal symbols
• Minor display formatting improvements
• Several small docs fixups and improvements (thanks @adrhill for one of them)

Merged pull requests:

• Recommend BasicAutoloads instead of a home-rolled autoloader (#96) (@LilithHafner)
• Update naming guidelines URL (#97) (@LilithHafner)
• Indent trailing lines of benchmark display by one space (#101) (@LilithHafner)
• Handle seconds=Inf and very high seconds args gracefully (unbounded runtime if samples and evals are specified, otherwise throw) (#105) (@LilithHafner)
• Fix time conversion in BenchmarkTools comparison (#110) (@adrhill)
• Make CI tests track lts, min, and pre (#112) (@LilithHafner)
• make show(::Benchmark) indent samples 1 space instead of 2 (#114) (@LilithHafner)
• Add explanations for some deviations from BenchmarkTools (#115) (@LilithHafner)
• Make the "no warmup" test less flaky (#116) (@LilithHafner)
• Release 1.2.2 (#117) (@LilithHafner)
• Make flaky Issue 74 test a regression test not an absolute test (#118) (@LilithHafner)
• Macro Expansion: Special case QuoteNode by Returnsing the unpacked value rather than the QuoteNode itself. (#119) (@LilithHafner)
• Replace recursion with mutation in the large/infinite seconds case (#120) (@LilithHafner)
• Refer to @be in why.md not @b which directs to @be (#123) (@LilithHafner)
• Update truthful section based on the use of Base.donotdelete (#125) (@LilithHafner)

Closed issues:

• Parsing error on standalone literal symbols (#99)
• Suggestion to Follow BenchmarkTools More Closely (#100)
• seconds=Inf does not work (#104)
• @b Produces Infs in matrix (#106)
• Histograms of benchmark times (#108)
• How to reproducibly sample random inputs? (#111)
• Update section on checksums to account for Base.donotdelete (#124)

v1.2.1

Chairmarks v1.2.1

Diff since v1.2.0

• Fix bug where the last sample is sometimes discarded (thanks @gdalle for reporting)
• Fix bug where evals is ignored when samples=0

Merged pull requests:

• bench/runbenchmarks.jl bugfixes (#83) (@LilithHafner)
• reduce runbenchmarks.jl workload by 5x (should drop CI times similarly) (#89) (@LilithHafner)
• Add aqua badge to readme (#90) (@LilithHafner)
• Fix off by one error in sample count (#92) (@LilithHafner)
• skip RegressionTests on local testing (#93) (@LilithHafner)
• Respect evals even when samples=0 (#94) (@LilithHafner)

Closed issues:

• Samples parameters is off by one? (#91)

v1.2.0

Chairmarks v1.2.0

Diff since v1.1.2

Merged pull requests:

• Declare soft semantic versioning (#77) (@LilithHafner)
• Use RegressionTests.jl! (#81) (@LilithHafner)
• Add option to disable GC (#85) (@LilithHafner)
• Add another gc=false test (#86) (@LilithHafner)
• Release 1.2.0 (#87) (@LilithHafner)

Closed issues:

• Does not suppress allocation & GC time as much as BenchmarkTools (#78)

v1.1.2

Chairmarks v1.1.2

Diff since v1.1.1

Merged pull requests:

• Test and fix issue 74 for Julia version 1.8+ (#75) (@LilithHafner)
• Release 1.1.2! (#76) (@LilithHafner)

Closed issues:

• Less accurate answers than BenchmarkTools on some microbenchmarks (#74)

v1.1.1

Chairmarks v1.1.1

Diff since v1.1.0

Merged pull requests:

• docstrings typo (#69) (@phlaster)
• Replace checksums with Base.donotdelete (#72) (@samuel3008)
• Release 1.1.1 (#73) (@LilithHafner)

Closed issues:

• use Base.donotdelete? (#68)

v1.1.0

Chairmarks v1.1.0

Diff since v1.0.2

Merged pull requests:

• Follow instructions in DocumenterVitepress.jl readme (Take 2) (#35) (@LilithHafner)
• Support interpolations! (#63) (@LilithHafner)
• Add more tests! (#64) (@LilithHafner)
• Delete unused median method (#65) (@LilithHafner)
• remove ... from docs file names because they don't look nice in Vitepress and are no longer necessary (#66) (@LilithHafner)
• Release 1.1.0 (#67) (@LilithHafner)

Closed issues:

• support $x variable interpolation (#62)

v1.0.2

Chairmarks v1.0.2

Diff since v1.0.1

Merged pull requests:

• Give a bit of justification for not interpolating and offer @eval (#59) (@LilithHafner)
• Fix explanation for the gc reason why Chairmarks is faster than BenchmarkTools (#60) (@Zentrik)
• Release 1.0.2 (#61) (@LilithHafner)

v1.0.1

Chairmarks v1.0.1

Diff since v1.0.0

Merged pull requests:

• Fix typo in tutorial.md (#49) (@simonp0420)
• Fixing typos (#50) (@etiennedeg)
• fix typos (#51) (@LilithHafner)
• Add proportion of time spent benchmarking to docs (#52) (@LilithHafner)
• Answer frequently asked design questions in new "Explanations" doc section (#55) (@LilithHafner)
• Fix and test reporting undef results bug (#57) (@LilithHafner)
• Release 1.0.1 (#58) (@LilithHafner)

Closed issues:

• Bug when specifying all of samples, seconds, and evals, and limited by seconds. (#56)