Find out why Moq's performance has become worse since version approx. 4.5

[stakx]

It was noted e.g. in #388 (comment) that Moq's performance has suffered in recent releases. It would be nice to find the reason(s) for this, and perhaps apply some optimizations to start reversing that trend before version 4.8.0 gets released.

Suggestion:

• Create some Moq test code that does something non-trivial.
• Profile that code for a couple of Moq releases, going back to approx. version 4.5.0. (Perhaps using PerfView, or another capable, freely available .NET profiling tool.)
• Report back with conclusion what has affected Moq's performance negatively in more recent releases / what are the major performance bottlenecks.

/cc @jatouma, @JohanLarsson

[toha73]

I noticed this since we're using Mock.Of<>() in alot of places in our unit tests.
Since NuGet version 4.7.99 it's extremely slow to use that method.

[stakx]

@toha73: Have you by change done any profiling and found out why that is? Do you possibly have some code / use cases you could share to be used for profiling?

[toha73]

I just did a simple script in LinqPad to test it out between different versions of Moq.

void Main()
{
	var iterations = Enumerable.Range(0, 500).ToList();
	
	var sw = Stopwatch.StartNew();
	iterations.ForEach(_ => { var x = Mock.Of<IMyInterface>(); });
	Console.WriteLine($"{sw.ElapsedMilliseconds} ms");

	sw.Restart();
	iterations.ForEach(_ => { var x = new Mock<IMyInterface>().Object; });
	Console.WriteLine($"{sw.ElapsedMilliseconds} ms");
}

public interface IMyInterface { string Property1 { get; set; } }

And the results on my machine are:

Moq 4.7.142

9760 ms
3 ms

Moq 4.7.137

9962 ms
3 ms

Moq 4.7.127

9885 ms
3 ms

Moq 4.7.99

10755 ms
4 ms

Moq 4.7.63

287 ms
4 ms

[JohanLarsson]

Ran a quick benchmark comparing with NSubstitute

BenchmarkDotNet=v0.10.9, OS=Windows 7 SP1 (6.1.7601)
Processor=Intel Xeon CPU E5-2637 v4 3.50GHzIntel Xeon CPU E5-2637 v4 3.50GHz, ProcessorCount=16
Frequency=3410107 Hz, Resolution=293.2459 ns, Timer=TSC
  [Host]     : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2116.0
  DefaultJob : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2116.0


 |                       Method |         Mean |      Error |     StdDev | Scaled | ScaledSD |   Gen 0 |  Gen 1 | Allocated |
 |----------------------------- |-------------:|-----------:|-----------:|-------:|---------:|--------:|-------:|----------:|
 |             SubstituteForFoo |     7.754 us |  0.3237 us |  0.9543 us |   1.00 |     0.00 |  1.0529 | 0.0229 |    6.5 KB |
 | SubstituteForFooValueReturns |    14.580 us |  0.2907 us |  0.8105 us |   1.91 |     0.26 |  1.7700 | 0.0305 |  10.94 KB |
 |                   MockOfIFoo |   892.996 us | 19.8067 us | 57.7770 us | 116.97 |    16.61 |  7.8125 | 3.9063 |  50.28 KB |
 |     MockOfIFooWithExpression | 2,138.479 us | 41.3511 us | 44.2452 us | 280.10 |    35.83 | 11.7188 | 3.9063 |  80.59 KB |

    public class MoqBenchmarks
    {
        public interface IFoo
        {
            int Value { get; }
        }

        [Benchmark(Baseline = true)]
        public object SubstituteForFoo()
        {
            return Substitute.For<IFoo>();
        }

        [Benchmark]
        public object SubstituteForFooValueReturns()
        {
            return Substitute.For<IFoo>().Value.Returns(1);
        }

        [Benchmark]
        public object MockOfIFoo()
        {
            return Mock.Of<IFoo>();
        }

        [Benchmark]
        public object MockOfIFooWithExpression()
        {
            return Mock.Of<IFoo>(x => x.Value == 1);
        }
    }

[stakx]

@toha73, thanks for the example. I ran your code for the same versions you mentioned, and also a slightly adapted version with BenchmarkDotNet. Below are my results for each run.

TL;DR: I do not observe that extreme jump (by more than an order of magnitude) that you saw.

Do you by chance have .NET 4.7(.1) installed? If so, are you aware of the performance problems caused by it? (See microsoft/dotnet#529 and other issues in the same repo.)

BenchmarkDotNet=v0.10.10, OS=Windows 7 SP1 (6.1.7601.0)
Processor=Intel Core i7-4770 CPU 3.40GHz (Haswell), ProcessorCount=8
Frequency=3312695 Hz, Resolution=301.8690 ns, Timer=TSC
  [Host]     : .NET Framework 4.6.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2116.0
  DefaultJob : .NET Framework 4.6.1 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2116.0

4.7.142

Using your code:

435 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 847.905 us | 9.0737 us | 8.4875 us |
 NewMockObject |   8.747 us | 0.0220 us | 0.0184 us |

4.7.99

Using your code:

467 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 727.259 us | 6.0048 us | 5.3231 us |
 NewMockObject |   8.487 us | 0.0418 us | 0.0391 us |

4.7.63

Using your code:

433 ms
3 ms

Using BenchmarkDotNet:

        Method |       Mean |     Error |    StdDev |
-------------- |-----------:|----------:|----------:|
        MockOf | 592.430 us | 1.8652 us | 1.7447 us |
 NewMockObject |   8.823 us | 0.0203 us | 0.0190 us |

[JohanLarsson]

@stakx Can we have a short private discussion, perhaps you can ping me on gitter? https://gitter.im/DotNetAnalyzers/IDisposableAnalyzers

[stakx]

@toha73 - FYI: If you upgrade to Moq 4.7.145 (or later), which is no longer affected by the mentioned .NET Framework regression (unless you opt back in), execution times should go back down to a normal level. See the changelog for details.

[stakx]

I've spent some time with PerfView. It seems that the biggest performance bottlenecks are:

1. proxy type creation (via Castle DynamicProxy)
2. expression tree compilation (via LambdaExpression.Compile)
3. source file information collection for diagnostic purposes (This has been taken care of with version 4.7.145.)

Those cannot be made more efficient, we can only try to do less of it. Notable hotspots in Moq's codebase are:

• Mock.SetupAllProperties — can cause a lot of proxy type creation
• MatcherFactory.CreateMatcher — can cause a lot of expression tree compilation

Moq's performance would likely be greatly improved by two changes:

1. Make SetupAllProperties lazy. Instead of setting up properties right away, only set them up when they're actually queried. This would reduce the amount of type creation to the level actually required.

2. Require that custom matchers (i.e. methods and properties calling Match.Create) be marked with some kind of attribute, such as [Matcher]. Moq could then rely simply on the presence of that attribute without having to compile & execute one lambda per setup parameter, just to figure out whether they're invoking Match.Create or not.

Of these two changes, (2) would definitely be a breaking change (but worth it). (1) might be done as a non-breaking change if much care is taken.