Releases: petabridge/NBench
NBench v1.0 Stable Release and .NET Standard Support
v1.0.0 March 14 2017
NBench v1.0.0 represents support for .NET Standard 1.6 for the NBench core library and a .NET Core 1.0 version of the NBench runner.
This release introduces a breaking change to NBench:
- NBench core library .NET Framework target increased from 4.5 -> 4.5.2. Client projects must target 4.5.2 or greater.
How to use the .NET Core NBench runner:
-
In your .NET Core performance test project, add the following dependency element to the .csproj file:
<ItemGroup> <DotNetCliToolReference Include="NBench.Runner.DotNetCli" Version="1.0.0" /> </ItemGroup>
-
Save the .csproj file (if using Visual Studio 2017) or run
dotnet restore
in the project location. -
From a command prompt within the project's parent directory, run
dotnet nbench project_name.dll arguments...
NBench v0.3.4 Release
v0.3.4 December 15 2016
NBench v0.3.4 is a bugfix for RunMode.ThroughPut
benchmarks where we regularly had false negatives on asserting number of operations per second.. This patch fixes this issue.
NBench v0.3.3 Release
0.3.3 December 07 2016
NBench v0.3.3 includes a handful of bug fixes, but also enables TeamCity output formatting for NBench specifications.
To enable TeamCity output formatting explicitly, pass in the following argument to the NBench.Runner.exe
NBench.Runner.exe [assembly names] [teamcity=true]
NBench v0.3.2 Release
Implemented first cut of TeamCity support, but had a bug with printing out failures
NBench v0.3.1 Release
0.3.1 August 15 2016
NBench v0.3.1 introduces full Mono support for cross-platform benchmarks to both NBench.Runner
as well as the core NBench
library.
No major new changes have been added feature wise, but we've worked around issues specific to Linux permissions and have tailored the core NBench package to be able to run benchmarks on any platform.
Here's an example of some side-by-side NBench benchmarks on Mono vs. .NET 4.6.2.
NBench.PerformanceCounters is platform-specific to Windows and thus can't be supported on Linux.
NBench v0.3.0 Release
0.3.0 May 24 2016
This release introduces some breaking changes to NBench:
Tracing
The biggest feature included in this release is the addition of tracing support, which is exposed directly to end-users so they can capture trace events and include them in the output produced by NBench.
You can access the IBenchmarkTrace
object through the BenchmarkContext
passed into any of your PerfSetup
, PerfBenchmark
, or PerfCleanup
methods like such:
public class TracingBenchmark
{
[PerfSetup]
public void Setup(BenchmarkContext context)
{
context.Trace.Debug(SetupTrace);
}
[PerfBenchmark(TestMode = TestMode.Test, NumberOfIterations = IterationCount, RunTimeMilliseconds = 1000)]
[MemoryMeasurement(MemoryMetric.TotalBytesAllocated)]
[MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThan, ByteConstants.EightKb)]
public void Run1(BenchmarkContext context)
{
context.Trace.Debug(RunTrace);
}
[PerfCleanup]
public void Cleanup(BenchmarkContext context)
{
context.Trace.Info(CleanupTrace);
}
}
NBench.Runner.exe
now takes a trace=true|false
commandline argument, which will enable the new tracing feature introduced in this release.
Tracing is disabled by default.
Skippable Warmups
You can now elect to skip warmups altogether for your specs. This feature is particularly useful for long-running iteration benchmarks, which are often used for stress tests. Warmups don't add any value here.
Here's how you can skip warmups:
[PerfBenchmark(TestMode = TestMode.Test, NumberOfIterations = IterationCount, RunTimeMilliseconds = 1000, SkipWarmups = true)]
[MemoryMeasurement(MemoryMetric.TotalBytesAllocated)]
[MemoryAssertion(MemoryMetric.TotalBytesAllocated, MustBe.LessThan, ByteConstants.EightKb)]
public void Run1(BenchmarkContext context)
{
context.Trace.Debug(RunTrace);
}
Just set SkipWarmups = true
on your PerfBenchmark
attribute wherever you wish to skip a warmup.
Foreground thread is no longer given high priority when concurrent mode is on.
If you are running the NBench.Runner
with concurrent=true
, we no longer give the main foreground thread high priority as this resulted in some unfair scheduling during concurrent tests. All threads within the NBench.Runner
process all share the same priority now.
Markdown reports include additional data
All markdown reports now include:
- The concurrency setting for NBench
- The tracing setting for NBench
- A flag indicating if warmups were skipped or not
All of these were added in order to make it easy for end-users reading the reports to know what the NBench settings were at the time the report was produced.
NBench V0.2.2 Release
0.2.2 May 03 2016
Warmup count is now equal to iteration count on all benchmarks, useful for users with long-running macro benchmarks and stress tests.
NBench V0.2.1 Release
0.2.1 March 22 2016
Fixed issue with NuGet logos and concurrency settings - we now still keep the process priority set to high, as we did before.
NBench v0.2 Release
0.2.0 March 22 2016
Major changes in NBench v0.2, beginning with our new logo:
First, we've added an extensible plugin API for capturing third-party metrics not natively provided by NBench. We will be providing more detailed documentation for this in a later release.
NBench.PerformanceCounters
The first example of this can be found in NBench.PerformanceCounters, a brand new NuGet package that allows you to instrument any arbitrary Windows PerformanceCounter
on any of your tests.
PS> Install-Package NBench.PerformanceCounters
This package introduces the three following attributes you can use on your benchmarks and performance tests:
PerformanceCounterMeasurementAttribute
- measures any available performance counter.PerformanceCounterThroughputAssertion
- asserts a performance counter's per-second value.PerformanceCounterTotalAssertion
- asserts a performance counter's total value.
TimingMeasurement and ElapsedTimeAssertion
Somewhat related to traditional CounterMeasurement
s, we've added two new attributes which allow you to measure and assert against the total amount of elapsed time it took to run a particular block of code.
TimingMeasurementAttribute
- reports on the elapsed time A run of a benchmark took in milliseconds. Designed to work withRunMode.Iterations
benchmarks.ElapsedTimeAssertionAttribute
- performs a bounds-checking assertion on amount of time it took to run a particular benchmark. Designed to work withRunMode.Iterations
benchmarks.
These are now available as part of the core NBench package.
Additional NBench.Runner Options
NBench.Runner now supports a new flag argument, concurrent=true|false
NBench.Runner.exe [assembly names] [output-directory={dir-path}] [configuration={file-path}] [include=MyTest*.Perf*,Other*Spec] [exclude=*Long*] [concurrent={true|false}]
concurrent=true|false - disables thread priority and processor affinity operations for all benchmarks. Used only when running multi-threaded benchmarks. Set to false
(single-threaded) by default.