IterationSetup for micro-benchmarks (InvocationParam) #1782
Description
I remember seeing an issue about this, but I couldn't find it again.
Sometimes, in order to test a single responsibility, setup is necessary for each invocation. In cases where the benchmark method runs very quickly (micro-benchmarks), BDN complains about times less than 100ms due to inaccurate timings. Sometimes it's not possible to increase the number of operations while operating on the setup data (or doing so would affect the timing due to iterating over multiple setup datas).
Example:
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
TaskCompletionSource<T> _source;
[IterationSetup]
public void IterationSetup()
{
_source = new TaskCompletionSource<T>();
}
[Benchmark]
public void AwaitPendingTask()
{
Run(_source.Task);
_source.SetResult(default);
static async void Run(Task<T> awaitable)
{
_ = await awaitable;
}
}
}That could be changed to something like this.
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
const int invokeCount = 100_000;
TaskCompletionSource<T>[] _sources = new TaskCompletionSource<T>[invokeCount];
[IterationSetup]
public void IterationSetup()
{
for (int i = 0; i < invokeCount; ++i)
{
_sources[i] = new TaskCompletionSource<T>();
}
}
[Benchmark(OperationsPerInvoke = invokeCount)]
public void AwaitPendingTask()
{
for (int i = 0; i < invokeCount; ++i)
{
var source = _sources[i];
Run(source.Task);
source.SetResult(default);
}
static async void Run(Task<T> awaitable)
{
_ = await awaitable;
}
}
}Increasing invokeCount until the execution time >= 100ms. But that's a lot more code to write, it adds the overhead of iterating the array, it doesn't allow BDN to choose an optimal invocation count through heuristics, and the same code might still run faster than 100ms on a different machine.
So my idea is to have another attribute similar to ParamsSource that will update the value on each invocation.
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark<T>
{
[InvocationParamSource(nameof(CreateSource))]
public TaskCompletionSource<T> Source { get; set; }
public TaskCompletionSource<T> CreateSource() => new TaskCompletionSource<T>();
[Benchmark]
public void AwaitPendingTask()
{
Run(Source.Task);
Source.SetResult(default);
static async void Run(Task<T> awaitable)
{
_ = await awaitable;
}
}
}The implementation of this could still use a storage array (or other collection), setting its length to the invocation count on each iteration, filling each index with the return value from CreateSource as part of the setup, then iterating over that collection and updating the value before invoking the benchmark method.
Iterating over the collection and updating the value (without calling CreateSource) could be measured as part the overhead and subtracted from the final result, giving us very accurate microbenchmarks. And we'll be able to write IterationSetup benchmarks that are portable to different machines. :)
[Edit] To support teardown:
[InvocationParam(nameof(CreateTarget), nameof(DisposeTarget))]
public MyDisposable Target { get; set; }
public MyDisposable CreateTarget() => new MyDisposable();
public void DisposeTarget(MyDisposable value) => value.Dispose();