more love from BenchmarkDotNet to CoreFX #27164
Conversation
adamsitnik
added
the
documentation
|
Great work Adam. I believe these changes will be beneficial to a lot of our contributors. Is it somehow possible to benchmark against the local coreclr/corefx build without consuming the nuget packages? In my inner loop scenario I just recompile a single assembly over and over again (e.g. System.Text.RegularExpressions.dll) and don't want to create a whole nuget package with all the coreclr/corefx binaries in it and invoke the .\build script everytime I do a change. The ideal situation for me would be to do the following:
This is just a suggestion for how you make my life much easier. I'm still very happy to about your changes 👍 |
|
@ViktorHofer you should rather not compare in-process vs out-process (too many things can be different) but speaking of the rebuild: I understand the problem and agree that the solution is not perfect. However, I don't have an idea yet. In general, if this is possible with dotnet cli, then it's possible with BenchmarkDotNet. My first two ideas:
I did not use |
|
cc @danmosemsft @eerhardt as we just talked about this in our standup |
|
Nice. Thanks, @adamsitnik. |
|
Yes, this is very cool. How hard would it be for BDN to consume and do reasonable things with tests that contain xunit-performance attributes? That way we could still author perf tests like we do now but run them under either perf harness. |
|
@AndyAyersMS, it should not be hard. In fact, the implementation of public static Summary Run<T>(IConfig config = null) =>
BenchmarkRunnerCore.Run(BenchmarkConverter.TypeToBenchmarks(typeof(T), config), ToolchainExtensions.GetToolchain);
public static Summary Run(Type type, IConfig config = null) =>
BenchmarkRunnerCore.Run(BenchmarkConverter.TypeToBenchmarks(type, config), ToolchainExtensions.GetToolchain);
public static Summary Run(Type type, MethodInfo[] methods, IConfig config = null) =>
BenchmarkRunnerCore.Run(BenchmarkConverter.MethodsToBenchmarks(type, methods, config), ToolchainExtensions.GetToolchain);
|
|
@ViktorHofer what is the fastest way to rebuild a part of corefx that developer is working on? do you always run I was wondering if the simplest way to run benchmarks after small code change would not be just copy+overwrite of selected files after BDN does From the user perspective it would be sth like: Add(Job.ShortRun.With(CustomCoreClrToolchain.CreateForLocalCoreFxBuild(
pathToNuGetFolder: @"C:\Projects\forks\corefx\bin\packages\Release",
privateCoreFxNetCoreAppVersion: "4.5.0-preview2-26313-0",
filesToCopy: new [] {
@"C:\Projects\forks\corefx\bin\AnyOS.AnyCPU.Release\System.Text.RegularExpressions\netcoreapp\System.Text.RegularExpressions.dll"
})));or Add(Job.ShortRun.With(CustomCoreClrToolchain.CreateForLocalCoreFxBuild(
pathToNuGetFolder: @"C:\Projects\forks\corefx\bin\packages\Release",
privateCoreFxNetCoreAppVersion: "4.5.0-preview2-26313-0",
filesToCopy: new [] {
typeof(Regex).Assembly.Location
}))); |
|
@adamsitnik, a developer working on a specific library generally just builds that library and runs the tests for that library. Once you've built the whole repo once, you can then build and test individual projects, e.g. |
|
@stephentoub thanks! I have written following benchmark: public class RegexBenchmarks
{
string pattern = @"^[0-9A-Z][-.\w]*(?<=[0-9A-Z])\$$";
string[] partNos = { "A1C$", "A4", "A4$", "A1603D$", "A1603D#" };
[Benchmark]
public int Sample()
{
int result = 0;
foreach (var input in partNos)
if (Regex.IsMatch(input, pattern))
result++;
return result;
}
}Modified the public static bool IsMatch(string input, string pattern)
{
int slow = 0;
for (int i = 0; i < 10000; i++)
slow += i;
return IsMatch(input, pattern, RegexOptions.None, DefaultMatchTimeout) && slow != 1345;
}And rebuilded it as explained by Stephen: After this I defined custom config: public class LocalCoreFxConfig : ManualConfig
{
public LocalCoreClrConfig()
{
Add(Job
.ShortRun
.With(CustomCoreClrToolchain.CreateForLocalCoreFxBuild(
pathToNuGetFolder: @"C:\Projects\forks\corefx\bin\packages\Release",
privateCoreFxNetCoreAppVersion: "4.5.0-preview2-26313-0",
displayName: "before"))
.AsBaseline()
.WithId("before"));
Add(Job
.ShortRun
.With(CustomCoreClrToolchain.CreateForLocalCoreFxBuild(
pathToNuGetFolder: @"C:\Projects\forks\corefx\bin\packages\Release",
privateCoreFxNetCoreAppVersion: "4.5.0-preview2-26313-0",
displayName: "after",
filesToCopy: new [] {
@"c:\Projects\forks\corefx\bin\AnyOS.AnyCPU.Release\System.Text.RegularExpressions\netcoreapp\System.Text.RegularExpressions.dll"
}))
.WithId("after"));
KeepBenchmarkFiles = true;
// rest of the config..
}
}And run the benchmark: BenchmarkDotNet=v0.10.12.20180216-develop, OS=Windows 10 Redstone 3 [1709, Fall Creators Update] (10.0.16299.248)
Intel Core i7-3687U CPU 2.10GHz (Ivy Bridge), 1 CPU, 4 logical cores and 2 physical cores
Frequency=2533321 Hz, Resolution=394.7388 ns, Timer=TSC
.NET Core SDK=2.1.300-preview2-008162
[Host] : .NET Core 2.0.5 (CoreCLR 4.6.26020.03, CoreFX 4.6.26018.01), 64bit RyuJIT
after : .NET Core ? (CoreCLR 4.6.26130.05, CoreFX 4.6.26314.0), 64bit RyuJIT
before : .NET Core ? (CoreCLR 4.6.26130.05, CoreFX 4.6.26313.0), 64bit RyuJIT
LaunchCount=1 TargetCount=3 WarmupCount=3
The result shows the difference, but to verify that the new feature work I verified the files in
moreover, we can check the produced asm: System.Text.RegularExpressions.Regex.IsMatch(System.String, System.String)
mov rcx,7FF7E1837B88h
mov edx,6
call coreclr!coreclr_execute_assembly+0x18580
mov rcx,1FE36946C88h
mov rcx,qword ptr [rcx]
mov rbx,qword ptr [rcx+8]
mov rcx,7FF7E1838420h
call coreclr!coreclr_execute_assembly+0x18180
mov rbp,rax
mov dword ptr [rsp+20h],1
mov rcx,rbp
mov rdx,rdi
mov r9,rbx
xor r8d,r8d
call System.Text.RegularExpressions.Regex..ctor(System.String, System.Text.RegularExpressions.RegexOptions, System.TimeSpan, Boolean)
mov rcx,rbp
mov rdx,rsi
mov rax,7FF7E1749238hSystem.Text.RegularExpressions.Regex.IsMatch(System.String, System.String)
xor ebx,ebx
xor ecx,ecx
M01_L00
add ebx,ecx
inc ecx
cmp ecx,2710h
jl M01_L00
mov rcx,7FF7E1837740h
mov edx,6
call coreclr!coreclr_execute_assembly+0x18580
mov rcx,2ACD13B6C88h
mov rcx,qword ptr [rcx]
mov rbp,qword ptr [rcx+8]
mov rcx,7FF7E1837FD8h
call coreclr!coreclr_execute_assembly+0x18180
mov r14,rax
mov dword ptr [rsp+20h],1
mov rcx,r14
mov rdx,rdi
mov r9,rbp
xor r8d,r8d
call System.Text.RegularExpressions.Regex..ctor(System.String, System.Text.RegularExpressions.RegexOptions, System.TimeSpan, Boolean)
mov rcx,r14
mov rdx,rsi
call System.Text.RegularExpressions.Regex.IsMatch(System.String)
test al,al
je M01_L01
cmp ebx,541h
setne al
movzx eax,al
add rsp,30h
pop rbx
pop rbp
pop rsi
pop rdi
pop r14
ret
M01_L01
xor eax,eax@stephentoub @ViktorHofer Is adding the ugly (but working) Btw I have improved the perf of BechmarkDotNet for |
|
Amazing, I will check it out over the weekend 👍 As Stephen stated we build individual projects like (after doing a full build): And the path you posted can even be shortened, as we copy all the assemblies into the runtime folder. But that's just a config setting.
I think your changes make sense and could help us a lot. As you pointed out, the filesToCopy feature isn't ideal but maybe there are also other use cases for it outside of corefx/coreclr. |
|
Instead of /t:rebuild you can do /t:build so it doesn't clean and /p:forceruntests if you want to rerun them without product changes. |
|
@danmosemsft are you talking about msbuild in a project folder or about the build script in root? |
|
@ViktorHofer MSBuild in a project folder. |
|
@eerhardt I updated the docs, there is no need to create a dedicated config type |
|
@ViktorHofer did you have some time to try it? |
|
Sorry Adam, haven't had time yet. I'll reserve some time for it today. |
|
As discussed with Adam offline this currently doesn't work for me as I get some weird dotnet restore errors that shouldn't occur because of the |
karelz
added
area-Infrastructure-libraries
and removed
documentation
|
@adamsitnik what is status of the PR? Is it blocked on further validation? (all of it vs. just a part?) |
|
@karelz it works on my dev box and both VMs I possess, but I still need somebody to test the feature that I have implemented in BenchmarkDotNet before I merge the docs change. |
|
@adamsitnik maybe it would be best to just check it in (it is "just" documentation after all, and moreover it works for you at least). |
|
@karelz good point! I just merged it. |
* more love from BenchmarkDotNet to CoreFX * describe the idea of copying extra files * don't require the users to create a dedicated config type * update the version number
* more love from BenchmarkDotNet to CoreFX * describe the idea of copying extra files * don't require the users to create a dedicated config type * update the version number Commit migrated from dotnet/corefx@969541f
To tell the long story short: I implemented new toolchain for BenchmarkDotNet which generates a self-contained app which can target ANY CoreFX and CoreCLR builds. I combined the knowledge from dogfooding docs from corefx and coreclr. dotnet cli does the job for us, it's just a matter of generating the right
.csprojandNuGet.configfiles.So you can now:
To use the new feature you need to add our CI feed to your
NuGet.configfile and download version0.10.12.433or newer.Simple example:
Comparison:
Complex example:
The output is going to contain exact CoreCLR and CoreFX versions used:
Corresponding change in BenchmarkDotNet dotnet/BenchmarkDotNet#651
@ViktorHofer @benaadams @stephentoub @JosephTremoulet @maryamariyan @AndyAyersMS
I am open to feedback and willing to improve/change the feature!