Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is EF9 slower than EF8? #35053

Closed
wallyrion opened this issue Nov 6, 2024 · 22 comments
Closed

Is EF9 slower than EF8? #35053

wallyrion opened this issue Nov 6, 2024 · 22 comments
Assignees
Labels
area-perf area-query closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported regression
Milestone

Comments

@wallyrion
Copy link

wallyrion commented Nov 6, 2024

I observed that EF 9 is running noticeably slower and is allocating twice as much memory compared to its performance with EF 8. I'm trying to determine whether this performance degradation is an isolated exception or if there might be an issue with my benchmarking setup. Any insights would be appreciated

There are some benchmarks with BenchmarkDotnet

Method Job Runtime UseSplitQuery Mean Error StdDev Ratio RatioSD Allocated Alloc Ratio
GetCustomerById .NET 8.0 .NET 8.0 False 1.688 ms 0.0368 ms 0.1038 ms 1.00 0.09 53.82 KB 1.00
GetCustomerById .NET 9.0 .NET 9.0 False 2.516 ms 0.0474 ms 0.1059 ms 1.50 0.11 196.56 KB 3.65
GetCustomerById .NET 8.0 .NET 8.0 True 2.867 ms 0.0598 ms 0.1705 ms 1.00 0.08 79.58 KB 1.00
GetCustomerById .NET 9.0 .NET 9.0 True 3.114 ms 0.0732 ms 0.2136 ms 1.09 0.10 161.05 KB 2.02

Comparing the load testing metrics with k6. Performance degradation is also noticeable.

Metric .NET 8 .NET 9
Total Requests 1,690,574 1,287,101
Requests per second 1,741.02 1,325.61
Data Received 13 GB 9.5 GB
Data Sent 220 MB 164 MB
HTTP Request Duration avg=2.29s, p(90)=4.79s, p(95)=5.51s avg=3.32s, p(90)=6.73s, p(95)=8.06s
Request Blocking avg=6.02µs, p(90)=5.01µs, p(95)=5.81µs avg=6.65µs, p(90)=5.15µs, p(95)=6.02µs
Request Receiving avg=44.14µs, p(90)=61.38µs, p(95)=75.48µs avg=42.62µs, p(90)=62.5µs, p(95)=76.01µs
Request Sending avg=14.77µs, p(90)=16.12µs, p(95)=32.88µs avg=13.8µs, p(90)=15.95µs, p(95)=29.79µs
Iteration Duration avg=3.29s, p(90)=5.79s, p(95)=6.52s avg=4.32s, p(90)=7.73s, p(95)=9.06s
Virtual Users (VUs) max=10,000 max=10,000

Query for benchmarks.

 var result =
            await customerQuery.Select(customer => new CustomerDto
                {
                    Id = customer.Id,
                    Name = customer.Name,
                    Orders = customer.Orders.Select(order => new OrderDto
                    {
                        OrderId = order.Id,
                        OrderDate = order.OrderDate,
                        OrderItems = order.OrderItems.Select(oi => new OrderItemDto
                        {
                            OrderItemId = oi.Id,
                            ProductName = oi.ProductName,
                            Quantity = oi.Quantity,
                            Price = oi.Price
                        }).ToList()
                    }).ToList()
                })
                .FirstOrDefaultAsync(c => c.Id == customerId);

Benchmarking ToListAsync() (retrieving 10000 items into memory)

We can observe 7x performance degradation for both speed and memory allocation between EF8 and EF9

Method Job Runtime Mean Error StdDev Ratio RatioSD Allocated Alloc Ratio
GetALlCustomers .NET 8.0 .NET 8.0 1.788 s 0.0194 s 0.0172 s 1.00 0.01 272.78 MB 1.00
GetALlCustomers .NET 9.0 .NET 9.0 7.926 s 0.0411 s 0.0384 s 4.43 0.05 1784.25 MB 6.54

Complete source code for the benchmarks

My repo

Include provider and version information

EF Core version:
Database provider Npgsql.EntityFrameworkCore.PostgreSQL (I tried to use SqlServer provider instead of Postgres and is shows the same results)
Version="8.0.10"
Version="9.0.0-rc.2.24474.1"

Target framework: (e.g. .NET 8.0 and .NET 9)
Operating system:
Windows 11 (for benchmarks) / Docker container (for load test)

@wallyrion wallyrion changed the title EF9 rc.2 slower than EF8? Is EF9 rc.2 slower than EF8? Nov 6, 2024
@maumar maumar self-assigned this Nov 6, 2024
@wallyrion wallyrion changed the title Is EF9 rc.2 slower than EF8? Is EF9 slower than EF8? Nov 13, 2024
@maumar
Copy link
Contributor

maumar commented Nov 14, 2024

The issue was introduced in Preview 4, specifically in this commit f0e88d4 (introduce liftable constant)

@matteomonizza
Copy link

matteomonizza commented Nov 15, 2024

In our project EF9 is more then 10x slower then previous

@maumar
Copy link
Contributor

maumar commented Nov 16, 2024

In the benchmark, considerable amount of time is now being spent in the ShaperProcessingExpressionVisitor.CompareIdentifiers (16.5% of total time, vs 0.7% in EF8). In EF8 we used to just pass a list of ValueComparers, but in EF9 we use List<Func<object, object, bool>>> which we construct manually:

        parentIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 
        outerIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 
        selfIdentifierValueComparers: [LIFTABLE Constant: Func<object, object, bool>[] { Func<object, object, bool>, Func<object, object, bool> } | Resolver: _ => new Func<object, object, bool>[]
        { 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right), 
            (left, right) => left == null ? right == null : right != null && Invoke((v1, v2) => v1 == v2, (int)left, (int)right) 
        }], 

Before the method was just doing a simple call to ValueComparer.Equals, but now we invoke complex linq processing involving Linq.Expressions.Interpreter

@matteomonizza
Copy link

In EF9, a query in SQLite that perform in EF8 in one second, now takes 12 seconds.. we are investigating..

maumar added a commit that referenced this issue Nov 17, 2024
Don't use Expression.Invoke in ValueComparer.ObjectEqualsExpression.
ValueComparer now contains the information on how to build an expression representing Equals(object, object), which uses Expression.Invoke. We found this to be a major performance problem in some scenarios (e.g. include collection navigation) where that expression is executed large number of times by the result coordinator, as it is part of the parent/outer/selfIdentifierValueComparers.
We actually know the lambda expression that is invoked in advance, so it's much more efficient to just remap the arguments and inline the lambda body into the ObjectEqualsExpression result.

Benchmark results:

ef 8

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |      Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 147.2 ms | 2.63 ms | 2.46 ms | 6.793 | 4000.0000 | 3000.0000 |  26.24 MB |
| PredicateMultipleIncludes |  True | 159.1 ms | 3.00 ms | 2.95 ms | 6.287 | 5500.0000 | 3000.0000 |  34.47 MB |

ef 9 without this change

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |       Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 322.6 ms | 0.97 ms | 0.86 ms | 3.099 | 13000.0000 | 6000.0000 |  79.48 MB |
| PredicateMultipleIncludes |  True | 344.9 ms | 6.79 ms | 6.67 ms | 2.899 | 14000.0000 | 7000.0000 |  87.72 MB |

ef 9 with this change

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |       Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 242.8 ms | 2.39 ms | 2.12 ms | 4.119 |  8000.0000 | 5000.0000 |  51.69 MB |
| PredicateMultipleIncludes |  True | 263.4 ms | 2.21 ms | 2.06 ms | 3.797 | 10000.0000 | 9000.0000 |  59.93 MB |

Benchmarks indicate that this change represents a sizable chunk of the perf regression introduced in EF9 by the AOT changes, but doesn't fully address it.

Part of #35053
@maumar
Copy link
Contributor

maumar commented Nov 17, 2024

Benchmarks after removing invoke from the ValueComparer.ObjectEqualsExpression:

ef 8

Method Async Mean Error StdDev Op/s Gen0 Gen1 Allocated
PredicateMultipleIncludes False 147.2 ms 2.63 ms 2.46 ms 6.793 4000.0000 3000.0000 26.24 MB
PredicateMultipleIncludes True 159.1 ms 3.00 ms 2.95 ms 6.287 5500.0000 3000.0000 34.47 MB

ef 9 with Invoke

Method Async Mean Error StdDev Op/s Gen0 Gen1 Allocated
PredicateMultipleIncludes False 322.6 ms 0.97 ms 0.86 ms 3.099 13000.0000 6000.0000 79.48 MB
PredicateMultipleIncludes True 344.9 ms 6.79 ms 6.67 ms 2.899 14000.0000 7000.0000 87.72 MB

ef 9 without Invoke

Method Async Mean Error StdDev Op/s Gen0 Gen1 Allocated
PredicateMultipleIncludes False 242.8 ms 2.39 ms 2.12 ms 4.119 8000.0000 5000.0000 51.69 MB
PredicateMultipleIncludes True 263.4 ms 2.21 ms 2.06 ms 3.797 10000.0000 9000.0000 59.93 MB

Significant improvement, but still doesn't fully address the problem. Looking for more...

maumar added a commit that referenced this issue Nov 18, 2024
Don't use Expression.Invoke in ValueComparer.ObjectEqualsExpression.
ValueComparer now contains the information on how to build an expression representing Equals(object, object), which uses Expression.Invoke. We found this to be a major performance problem in some scenarios (e.g. include collection navigation) where that expression is executed large number of times by the result coordinator, as it is part of the parent/outer/selfIdentifierValueComparers.
We actually know the lambda expression that is invoked in advance, so it's much more efficient to just remap the arguments and inline the lambda body into the ObjectEqualsExpression result.

Benchmark results:

ef 8

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |      Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 147.2 ms | 2.63 ms | 2.46 ms | 6.793 | 4000.0000 | 3000.0000 |  26.24 MB |
| PredicateMultipleIncludes |  True | 159.1 ms | 3.00 ms | 2.95 ms | 6.287 | 5500.0000 | 3000.0000 |  34.47 MB |

ef 9 without this change

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |       Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 322.6 ms | 0.97 ms | 0.86 ms | 3.099 | 13000.0000 | 6000.0000 |  79.48 MB |
| PredicateMultipleIncludes |  True | 344.9 ms | 6.79 ms | 6.67 ms | 2.899 | 14000.0000 | 7000.0000 |  87.72 MB |

ef 9 with this change

|                    Method | Async |     Mean |   Error |  StdDev |  Op/s |       Gen0 |      Gen1 | Allocated |
|-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:|
| PredicateMultipleIncludes | False | 242.8 ms | 2.39 ms | 2.12 ms | 4.119 |  8000.0000 | 5000.0000 |  51.69 MB |
| PredicateMultipleIncludes |  True | 263.4 ms | 2.21 ms | 2.06 ms | 3.797 | 10000.0000 | 9000.0000 |  59.93 MB |

Benchmarks indicate that this change represents a sizable chunk of the perf regression introduced in EF9 by the AOT changes, but doesn't fully address it.

Part of #35053
@zakeryooo
Copy link

zakeryooo commented Nov 25, 2024

I've just hit what I assume is this issue too. Updated to EF9 and compared to EF Core 8 when dealing with medium+ sized data sets it's very slow.

Loading a full table with 2 joins into memory takes 1.5s in EF8, it takes 9s for the same data in EF9. Saving changes to a small subset of that data in EF8 takes 0.5s, it takes 20s in EF9, same data.

Really surprised this issue hasn't had a lot more comments, it seems like a huge regression? Is the recommendation to just stay on EF8 for now?

@matteomonizza
Copy link

EF9 is killing my app.. From 1s to 10s is to much... I use DataReader and DataTable loading via DataReader, think the problem is here...

@maumar
Copy link
Contributor

maumar commented Nov 25, 2024

@zakeryooo are your queries using collection navigation? If not, could you share the query, model that is involved and the approximate size of the data in your database? Also, I'm interested in the update scenario - can you provide some more info on what you are doing (small repro code would be ideal). If you experience catastrophic perf regression on EF9 it makes sense to stay on EF8 for now. This is a high priority issue for us and we hope to have a significant improvement in the first patch release. But we really want to make sure we understand all the scenarios that are impacted - the more info users provide the better

@maumar
Copy link
Contributor

maumar commented Nov 25, 2024

@matteomonizza can you provide some more info on your scenario? What query shapes are involved and what is the size of data that you are working with? Standalone repro would be the best, but any info would be helpful - we want to make sure that root cause you encountered in the same and the one reported originally.

maumar added a commit that referenced this issue Nov 26, 2024
… in interpretation mode when the resolver itself contains a lambda

In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide.
At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda).
Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient.

Fix is to scan the resolver expression and look for nested lambdas inside - if we find some, compile the resolver in the regular mode instead.

Fixes #35208

This is part of a fix for a larger perf issue: #35053
maumar added a commit that referenced this issue Nov 26, 2024
… in interpretation mode when the resolver itself contains a lambda

In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide.
At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda).
Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient.

Fix is to use regular compilation rather than interpretation.

Fixes #35208

This is part of a fix for a larger perf issue: #35053
maumar added a commit that referenced this issue Nov 26, 2024
… in interpretation mode when the resolver itself contains a lambda (#35209)

In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide.
At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda).
Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient.

Fix is to use regular compilation rather than interpretation.

Fixes #35208

This is part of a fix for a larger perf issue: #35053
@zakeryooo
Copy link

zakeryooo commented Nov 26, 2024

@zakeryooo are your queries using collection navigation? If not, could you share the query, model that is involved and the approximate size of the data in your database? Also, I'm interested in the update scenario - can you provide some more info on what you are doing (small repro code would be ideal). If you experience catastrophic perf regression on EF9 it makes sense to stay on EF8 for now. This is a high priority issue for us and we hope to have a significant improvement in the first patch release. But we really want to make sure we understand all the scenarios that are impacted - the more info users provide the better

Thanks for the reply, yeah it's using collection navigation, I'm away at the moment so unfortunately have limited ability to provide particularly useful info but the general gist of the code is:

        var products = _ctx.Products
            .AsSplitQuery()
            .Include(p => p.ShopEntries)
                .ThenInclude(se => se.ActivePrices)
            .Include(p => p.ShopEntries)
                .ThenInclude(se => se.Shop)
                .ToList();
        foreach (var product in products)
        {
            decimal popularity = 0;
            foreach (var shopEntry in product.ShopEntries)
            {
                if (shopEntry.ActivePrices.Count == 0 || shopEntry.Shop.State == ShopState.Inactive) continue;
                if (shopEntry.ActivePrices.All(ap => ap.CurrentPrice == null)) continue;

                // Basic logic here to update the popularity value, each run usually only changes the value for < 100 records
            }
            product.ComputedPopularity = (int)popularity;
        }
        _ctx.SaveChanges();

The actual SQL queries only take ~100ms total on the DB side, 22k products, 100k shop entries, 42k active prices, 22 shops.

On the exact same data / popularity change count (recloned the DB each test iteration):

Warm cached on EF8 takes avg:
ToList Products: 2.5s - Loop Products: 20ms - SaveChanges: 0.5s

Warm cached on EF9 takes avg:
ToList Products: 8s - Loop Products: 20ms - SaveChanges: 20s.

I'm back next week and if you still need it by then can put together a small repro.

@matteomonizza
Copy link

@matteomonizza can you provide some more info on your scenario? What query shapes are involved and what is the size of data that you are working with? Standalone repro would be the best, but any info would be helpful - we want to make sure that root cause you encountered in the same and the one reported originally.

Yesterday we took time to face the problem.. The app is loading data from a SQLite database. The app is an iOS and Android app (not MAUI, we use Net-iOS and Net-Android). The Microsoft SQLite provider we use is the Microsoft.Data.Sqlite, that does not contain the DataAdapter implementation. We create out implementation and now the LoadData of the data table take s only 1s against 10s with fill from data reader (with Net 8 there was not this gap). Then, we notice that using the app in debug mode, both Android and iOS version both simulator and device, is extremely slow. For example, the same code run in 2s in release mode, in 30s in debug mode, only on .Net 9. Tried this on different machines.

@c5racing
Copy link

My situation may or may not be a bit different. We use PostgreSQL and are in the process of migrating from .NET8 to .NET9 and in doing so, changing from the Legacy POCO mapping to the newer .ToJson() Mapping.

I've attached a sample project (thrown together very quickly) using the identical data. Neither one actually modifies data. The legacy POCO mapping takes 0ms to call SaveChanges(). The newer .ToJson() takes over 2 seconds.

https://drive.google.com/file/d/1A_VoN9XgrogMEeevV7mtLh5c2s-Ds2rt/view?usp=sharing

@roji
Copy link
Member

roji commented Nov 28, 2024

@c5racing thanks, this is something that's definitely worth checking once we release some planned fixed to help the general performance problem (for 9.0.1). One thing to confirm though - are you testing both scenarios on EF 9 (i.e. both ToJson and legacy POCO mapping)?

@c5racing
Copy link

@c5racing thanks, this is something that's definitely worth checking once we release some planned fixed to help the general performance problem (for 9.0.1). One thing to confirm though - are you testing both scenarios on EF 9 (i.e. both ToJson and legacy POCO mapping)?

Yes, testing both scenarios on EF9. Calling var hasChanges = newDbContext.ChangeTracker.HasChanges(); returns false but SaveChanges() still takes 2-3 second

@roji
Copy link
Member

roji commented Nov 29, 2024

Ah, if this is about a regression in SaveChanges() performance, then can you please open a new issue for it? This issue is about regressions specifically in reading back results from queries, which is quite a different thing.

Also, if I understand correctly, you're not reporting a regression from 8 to 9, but only when moving PG legacy JSON POCO support to ToJson(), right?

@c5racing
Copy link

Ah, if this is about a regression in SaveChanges() performance, then can you please open a new issue for it? This issue is about regressions specifically in reading back results from queries, which is quite a different thing.

Also, if I understand correctly, you're not reporting a regression from 8 to 9, but only when moving PG legacy JSON POCO support to ToJson(), right?

Thanks @roji, I opened a new ticket, sorry for creating static in this one. As indicated in the new ticket, ToJson() is slower than the legacy POCO mapping in both .NET8 and .NET9; however, .ToJson() is 10x slower in .NET9 than .NET8

@f0ppa21
Copy link

f0ppa21 commented Dec 4, 2024

We are also experiencing that EF 9 are much slower than EF 8 for queries that result in large / complex datamodels. The generated Sql-query (split query) performs similar as for EF 8, its the subsequent call to ToListAsync that seems to execute much slower (x2 slower for only 2700 returned records and gets worse with more returned records).

@maumar
Copy link
Contributor

maumar commented Dec 4, 2024

@f0ppa21 we have made several improvements that will ship in 9.0.1 patch. The perf is not exactly back to EF8 levels but should be close. In the meantime, you can try our daily builds to see how the upcoming improvements affect your particular scenario. If you still see significant perf degradation using daily builds, we would appreciate the repro so we can investigate and target your specific case.

@FredSwadlingPelt8

This comment has been minimized.

@roji

This comment has been minimized.

@FredSwadlingPelt8
Copy link

@FredSwadlingPelt8 the change to translating parameterized collections to JSON was introduced in 8, not 9; the issue tracking this is #32394.

I did check, and the parameterized query issue only appeared in our queries when we updated from ef core 8 to ef core 9. I have no idea why this would be.

@maumar maumar added this to the 9.0.2 milestone Jan 14, 2025
@maumar maumar added the closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. label Jan 14, 2025
@maumar
Copy link
Contributor

maumar commented Jan 14, 2025

Added some more improvements around SaveChanges scenarios with primitive collections. This issue should now be fixed - if you still see issues after updating to EF 9.0.2 please file a new issue with details about the scenario.

@maumar maumar closed this as completed Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-perf area-query closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. customer-reported regression
Projects
None yet
Development

No branches or pull requests

9 participants