-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is EF9 slower than EF8? #35053
Comments
The issue was introduced in Preview 4, specifically in this commit f0e88d4 (introduce liftable constant) |
In our project EF9 is more then 10x slower then previous |
In the benchmark, considerable amount of time is now being spent in the ShaperProcessingExpressionVisitor.CompareIdentifiers (16.5% of total time, vs 0.7% in EF8). In EF8 we used to just pass a list of ValueComparers, but in EF9 we use
Before the method was just doing a simple call to ValueComparer.Equals, but now we invoke complex linq processing involving Linq.Expressions.Interpreter |
In EF9, a query in SQLite that perform in EF8 in one second, now takes 12 seconds.. we are investigating.. |
Don't use Expression.Invoke in ValueComparer.ObjectEqualsExpression. ValueComparer now contains the information on how to build an expression representing Equals(object, object), which uses Expression.Invoke. We found this to be a major performance problem in some scenarios (e.g. include collection navigation) where that expression is executed large number of times by the result coordinator, as it is part of the parent/outer/selfIdentifierValueComparers. We actually know the lambda expression that is invoked in advance, so it's much more efficient to just remap the arguments and inline the lambda body into the ObjectEqualsExpression result. Benchmark results: ef 8 | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|----------:|----------:|----------:| | PredicateMultipleIncludes | False | 147.2 ms | 2.63 ms | 2.46 ms | 6.793 | 4000.0000 | 3000.0000 | 26.24 MB | | PredicateMultipleIncludes | True | 159.1 ms | 3.00 ms | 2.95 ms | 6.287 | 5500.0000 | 3000.0000 | 34.47 MB | ef 9 without this change | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:| | PredicateMultipleIncludes | False | 322.6 ms | 0.97 ms | 0.86 ms | 3.099 | 13000.0000 | 6000.0000 | 79.48 MB | | PredicateMultipleIncludes | True | 344.9 ms | 6.79 ms | 6.67 ms | 2.899 | 14000.0000 | 7000.0000 | 87.72 MB | ef 9 with this change | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:| | PredicateMultipleIncludes | False | 242.8 ms | 2.39 ms | 2.12 ms | 4.119 | 8000.0000 | 5000.0000 | 51.69 MB | | PredicateMultipleIncludes | True | 263.4 ms | 2.21 ms | 2.06 ms | 3.797 | 10000.0000 | 9000.0000 | 59.93 MB | Benchmarks indicate that this change represents a sizable chunk of the perf regression introduced in EF9 by the AOT changes, but doesn't fully address it. Part of #35053
Benchmarks after removing invoke from the ValueComparer.ObjectEqualsExpression: ef 8
ef 9 with Invoke
ef 9 without Invoke
Significant improvement, but still doesn't fully address the problem. Looking for more... |
Don't use Expression.Invoke in ValueComparer.ObjectEqualsExpression. ValueComparer now contains the information on how to build an expression representing Equals(object, object), which uses Expression.Invoke. We found this to be a major performance problem in some scenarios (e.g. include collection navigation) where that expression is executed large number of times by the result coordinator, as it is part of the parent/outer/selfIdentifierValueComparers. We actually know the lambda expression that is invoked in advance, so it's much more efficient to just remap the arguments and inline the lambda body into the ObjectEqualsExpression result. Benchmark results: ef 8 | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|----------:|----------:|----------:| | PredicateMultipleIncludes | False | 147.2 ms | 2.63 ms | 2.46 ms | 6.793 | 4000.0000 | 3000.0000 | 26.24 MB | | PredicateMultipleIncludes | True | 159.1 ms | 3.00 ms | 2.95 ms | 6.287 | 5500.0000 | 3000.0000 | 34.47 MB | ef 9 without this change | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:| | PredicateMultipleIncludes | False | 322.6 ms | 0.97 ms | 0.86 ms | 3.099 | 13000.0000 | 6000.0000 | 79.48 MB | | PredicateMultipleIncludes | True | 344.9 ms | 6.79 ms | 6.67 ms | 2.899 | 14000.0000 | 7000.0000 | 87.72 MB | ef 9 with this change | Method | Async | Mean | Error | StdDev | Op/s | Gen0 | Gen1 | Allocated | |-------------------------- |------ |---------:|--------:|--------:|------:|-----------:|----------:|----------:| | PredicateMultipleIncludes | False | 242.8 ms | 2.39 ms | 2.12 ms | 4.119 | 8000.0000 | 5000.0000 | 51.69 MB | | PredicateMultipleIncludes | True | 263.4 ms | 2.21 ms | 2.06 ms | 3.797 | 10000.0000 | 9000.0000 | 59.93 MB | Benchmarks indicate that this change represents a sizable chunk of the perf regression introduced in EF9 by the AOT changes, but doesn't fully address it. Part of #35053
I've just hit what I assume is this issue too. Updated to EF9 and compared to EF Core 8 when dealing with medium+ sized data sets it's very slow. Loading a full table with 2 joins into memory takes 1.5s in EF8, it takes 9s for the same data in EF9. Saving changes to a small subset of that data in EF8 takes 0.5s, it takes 20s in EF9, same data. Really surprised this issue hasn't had a lot more comments, it seems like a huge regression? Is the recommendation to just stay on EF8 for now? |
EF9 is killing my app.. From 1s to 10s is to much... I use DataReader and DataTable loading via DataReader, think the problem is here... |
@zakeryooo are your queries using collection navigation? If not, could you share the query, model that is involved and the approximate size of the data in your database? Also, I'm interested in the update scenario - can you provide some more info on what you are doing (small repro code would be ideal). If you experience catastrophic perf regression on EF9 it makes sense to stay on EF8 for now. This is a high priority issue for us and we hope to have a significant improvement in the first patch release. But we really want to make sure we understand all the scenarios that are impacted - the more info users provide the better |
@matteomonizza can you provide some more info on your scenario? What query shapes are involved and what is the size of data that you are working with? Standalone repro would be the best, but any info would be helpful - we want to make sure that root cause you encountered in the same and the one reported originally. |
… in interpretation mode when the resolver itself contains a lambda In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide. At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda). Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient. Fix is to scan the resolver expression and look for nested lambdas inside - if we find some, compile the resolver in the regular mode instead. Fixes #35208 This is part of a fix for a larger perf issue: #35053
… in interpretation mode when the resolver itself contains a lambda In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide. At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda). Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient. Fix is to use regular compilation rather than interpretation. Fixes #35208 This is part of a fix for a larger perf issue: #35053
… in interpretation mode when the resolver itself contains a lambda (#35209) In EF9 we changed the way we generate shapers in preparation for AOT scenarios. We no longer can embed arbitrary objects into the shaper, instead we need to provide a way to construct that object in code (using LiftableConstant mechanism), or simulate the functionality it used to provide. At the end of our processing, we find all liftable constants and for the non-AOT case we compile their resolver lambdas and invoke the result with liftable context object to produce the resulting constant object we initially wanted. (in AOT case we generate code from the resolver lambda). Problem is that we are compiling the resolver lambda in the interpretation mode - if the final product is itself a delegate, that delegate will itself be in the interpreter mode and therefore less efficient. Fix is to use regular compilation rather than interpretation. Fixes #35208 This is part of a fix for a larger perf issue: #35053
Thanks for the reply, yeah it's using collection navigation, I'm away at the moment so unfortunately have limited ability to provide particularly useful info but the general gist of the code is:
The actual SQL queries only take ~100ms total on the DB side, 22k products, 100k shop entries, 42k active prices, 22 shops. On the exact same data / popularity change count (recloned the DB each test iteration): Warm cached on EF8 takes avg: Warm cached on EF9 takes avg: I'm back next week and if you still need it by then can put together a small repro. |
Yesterday we took time to face the problem.. The app is loading data from a SQLite database. The app is an iOS and Android app (not MAUI, we use Net-iOS and Net-Android). The Microsoft SQLite provider we use is the Microsoft.Data.Sqlite, that does not contain the DataAdapter implementation. We create out implementation and now the LoadData of the data table take s only 1s against 10s with fill from data reader (with Net 8 there was not this gap). Then, we notice that using the app in debug mode, both Android and iOS version both simulator and device, is extremely slow. For example, the same code run in 2s in release mode, in 30s in debug mode, only on .Net 9. Tried this on different machines. |
My situation may or may not be a bit different. We use PostgreSQL and are in the process of migrating from .NET8 to .NET9 and in doing so, changing from the Legacy POCO mapping to the newer I've attached a sample project (thrown together very quickly) using the identical data. Neither one actually modifies data. The legacy POCO mapping takes 0ms to call SaveChanges(). The newer https://drive.google.com/file/d/1A_VoN9XgrogMEeevV7mtLh5c2s-Ds2rt/view?usp=sharing |
@c5racing thanks, this is something that's definitely worth checking once we release some planned fixed to help the general performance problem (for 9.0.1). One thing to confirm though - are you testing both scenarios on EF 9 (i.e. both ToJson and legacy POCO mapping)? |
Yes, testing both scenarios on EF9. Calling |
Ah, if this is about a regression in SaveChanges() performance, then can you please open a new issue for it? This issue is about regressions specifically in reading back results from queries, which is quite a different thing. Also, if I understand correctly, you're not reporting a regression from 8 to 9, but only when moving PG legacy JSON POCO support to ToJson(), right? |
Thanks @roji, I opened a new ticket, sorry for creating static in this one. As indicated in the new ticket, ToJson() is slower than the legacy POCO mapping in both .NET8 and .NET9; however, |
We are also experiencing that EF 9 are much slower than EF 8 for queries that result in large / complex datamodels. The generated Sql-query (split query) performs similar as for EF 8, its the subsequent call to ToListAsync that seems to execute much slower (x2 slower for only 2700 returned records and gets worse with more returned records). |
@f0ppa21 we have made several improvements that will ship in 9.0.1 patch. The perf is not exactly back to EF8 levels but should be close. In the meantime, you can try our daily builds to see how the upcoming improvements affect your particular scenario. If you still see significant perf degradation using daily builds, we would appreciate the repro so we can investigate and target your specific case. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
I did check, and the parameterized query issue only appeared in our queries when we updated from ef core 8 to ef core 9. I have no idea why this would be. |
Added some more improvements around SaveChanges scenarios with primitive collections. This issue should now be fixed - if you still see issues after updating to EF 9.0.2 please file a new issue with details about the scenario. |
I observed that EF 9 is running noticeably slower and is allocating twice as much memory compared to its performance with EF 8. I'm trying to determine whether this performance degradation is an isolated exception or if there might be an issue with my benchmarking setup. Any insights would be appreciated
There are some benchmarks with BenchmarkDotnet
Comparing the load testing metrics with k6. Performance degradation is also noticeable.
Query for benchmarks.
Benchmarking ToListAsync() (retrieving 10000 items into memory)
We can observe 7x performance degradation for both speed and memory allocation between EF8 and EF9
Complete source code for the benchmarks
My repo
Include provider and version information
EF Core version:
Database provider Npgsql.EntityFrameworkCore.PostgreSQL (I tried to use SqlServer provider instead of Postgres and is shows the same results)
Version="8.0.10"
Version="9.0.0-rc.2.24474.1"
Target framework: (e.g. .NET 8.0 and .NET 9)
Operating system:
Windows 11 (for benchmarks) / Docker container (for load test)
The text was updated successfully, but these errors were encountered: