SingleOrDefaultAsync query w/ AutoMapper crash w3wp process on Azure #17617
Comments
|
The code from stacktrace no longer exist in EF Core 3.0. Please try 3.0 preview package (preview9 from nuget or daily build). |
|
@smitpatel @ajcvickers I've tried migrating to 3.0 previews earlier, but it seems so unstable that I've tried to postpone application migration until RTM. But to assist finding this bug, I made another attempt at migrating - now to the latest preview9 as you mention. However, I can't even get my tests to pass to get it through our build pipelines. 😣 I've tried to follow all available migration guides and blog posts. Still, a simple; Assert.Equal(1, _dbContext.Ads.Count());
ad = await _dbContext.Ads.SingleAsync();Renders this exception for the InMemory db. Note that the Ads.Count() returns 1 as expected by this particular test. System.NullReferenceException : Object reference not set to an instance of an object.
Stack Trace:
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.EntityProjectionExpression.BindProperty(IProperty property)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryExpressionTranslatingExpressionVisitor.BindProperty(EntityProjectionExpression entityProjectionExpression, IProperty property)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryExpressionTranslatingExpressionVisitor.BindProperty(Expression source, String propertyName, Type type)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryExpressionTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryExpressionTranslatingExpressionVisitor.Translate(Expression expression)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryProjectionBindingExpressionVisitor.Visit(Expression expression)
at Microsoft.EntityFrameworkCore.Query.IncludeExpression.VisitChildren(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.VisitExtension(Expression node)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryProjectionBindingExpressionVisitor.VisitExtension(Expression extensionExpression)
at System.Linq.Expressions.Expression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryProjectionBindingExpressionVisitor.Visit(Expression expression)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryProjectionBindingExpressionVisitor.Translate(InMemoryQueryExpression queryExpression, Expression expression)
at Microsoft.EntityFrameworkCore.InMemory.Query.Internal.InMemoryQueryableMethodTranslatingExpressionVisitor.TranslateSelect(ShapedQueryExpression source, LambdaExpression selector)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.SingleAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)How can I make my tests pass using 3.0 preview 9 ? Everything works a charm in 2.2.6. |
|
If you are using any provider other than InMemory then preview9 should work. If you are using InMemory provider then use daily builds. It will not be too much difficult if you have migrated to preview9 already. (Just setting up few feeds and restoring package). InMemory query work went in couple of days ago. |
|
@smitpatel OK, now with daily builds (3.0.0-rc1.19455.5) this particular test is running fine. Unfortunately I still have about 55 failing tests to solve before I'm able to build a release version and put on Azure to find out if the crash mentioned above for this issue is still present. Most of my failing tests seem to depend on queries using advanced expressions. We've been building this project over two years time and have a lot of heavy expression queries against EF Core. It's been challenging to make everything work, but they do all run fine with 2.2.6. Please look at the following gist (uncomment package references so that you use the daily builds): https://gist.github.com/jcemoller/312a2aaa3c27ec18318881619439e867 This particular way of building expressions does not work in Ef Core 3.0, not even the daily builds. I've tried with both SqlServer and InMemory provider, but to no success. This is the exception I get: System.InvalidOperationException
HResult=0x80131509
Message=EF.Property called with wrong property name.
Source=Microsoft.EntityFrameworkCore.Relational
StackTrace:
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
at Microsoft.EntityFrameworkCore.SqlServer.Query.Internal.SqlServerSqlTranslatingExpressionVisitor.VisitBinary(BinaryExpression binaryExpression)
at System.Linq.Expressions.BinaryExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateExpression(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateWhere(ShapedQueryExpression source, LambdaExpression predicate)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.TranslateSubquery(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.RelationalSqlTranslatingExpressionVisitor.Translate(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateExpression(Expression expression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.TranslateWhere(ShapedQueryExpression source, LambdaExpression predicate)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.CountAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
at EfCoreBugExpression.Program.<Main>d__1.MoveNext() in C:\Users\Joakim\Source\Repos\EfCoreBug\EFCoreBugExpression\Program.cs:line 30 |
|
@ajcvickers @smitpatel I'm sorry but we cannot wait for 3.1, we need to push a new release up asap. What's our options now? Our application based on 2.2.6 has recently started crashing the w3wp process, possibly due to StackOverflowException, probably due to large entity graphs. Your suggestion is to upgrade to daily builds of 3.0, but due to the mentioned regression where interfaces are not supported in expressions, we cannot even produce a release build to see if the crash is resolved in 3.0. |
|
cc: @divega |
|
@smitpatel due to the regressions in 3.0 when using expressions in queries, I spent the last 10 hours inlining all generic or interface-based expressions directly into the queries in a couple of hundred places. I'm not a fan of code duplication, but still, I managed at last to make the project build and run. On the positive side:
On the negative side:
Why are we dependent on using expressions using interfaces? For example, we have some entities that do not relate to each other but they all have a navigation collection Verifications. Previously, in 2.2.6 and earlier, I could have a generic expression based on IVerifiable that I'd tack on all entities having the Verifications collection. I could then use a generic form of IsVerified expression based on the IVerifiable interface. Now with 3.0, due to the regression mentioned above and not using generics, I have to duplicate the expression so that I have EntityAIsVerified, EntitiyBIsVerified etc. Also, on a side note. Previously using AutoMapper I did not have to use .AsNoTracking(). With 3.0 I have to add .AsNoTracking() on all AutoMapper queries otherwise I get this error message: Lastly, sorry if I've been a bit hard on the tone in this issue but I'm on a tight deadline and should have had a new release out this week. Instead I've spent four days intensively on this issue. I guess you are also under pressure now that 3.0 is closing in on release, and you're doing a great job maintaining this project. |
|
@jcemoller glad to hear that you have a workaround. Hopefully you will be able to simplify your code again when we address this limitation in 3.1. Re owned entities projected without their owner requiring AsNoTracking, last time we discussed this we came to the conclusion that switching the behavior of the query to non-tracking automatically could be too unexpected, given that the design is that owned entities are trackable objects but need to be accompanied by their owner. The other option was to auto-include the owner to be able honor tracking, but that would mean bringing more data from the database than you explicitly asked for in the query. Yet another option would be to change some of the invariants in the design and allow “free” owned entities to be tracked accompanied just with the key value of the owner. We are open to feedback on this area (as with more things). If you have an opinion on how we should change the design, feel free to create an issue specifically for this. |
|
@divega I agree that switching a supposedly tracking query to non-tracking automatically would be unexpected, but up to 3.0 all queries which do a .Select(x => new Y(x)) to project into client entities was automatically non-tracking. This is a big breaking change, since I have to find all these places and add .AsNoTracking to prevent a runtime failure. It is not documented in https://docs.microsoft.com/en-us/ef/core/what-is-new/ef-core-3.0/breaking-changes. |
|
@smitpatel @divega So, week later and I'm still investigating this issue. Turned out, that after all work migrating to 3.0 and getting our application running again, I now face major performance issues on this very same query. This query, even though it's pretty complex, used to run in a few seconds. In 2.2, after adding an additional column to an entity as reported originally for this issue, I started to get stack overflow exceptions and it crashed the entire process. In 3.0, the query now works again but very slow. I originally thought I was having SQL Server issues (see #17743) but after updating statistics using sp_updatestats I have gotten the SQL query back to run in about 1 second. This has been verified by running;
Running the actual application however, it now takes more than 1 minute to get the results back from the SingleOrDefaultAsync() method call. From investigations, it seems actually as if most of the delta between method execution time and SQL query time is spent before the actual query is sent to the SQL Server. Perhaps in some kind of query/expression tree parsing gone wild? Performance Profiler sessionThis is the query execution time reported;
This is the CPU spent in the method;
As you can see, 62% of the CPU in the entire session of loading an ASP.NET MVC page is spent in the SingleOrDefaultAsync() method! Unfortunately I cannot see further down in the call tree because I do not have the required symbols. Is this because I am running daily builds? How can I get the necessary PDB files for the daily builds? |
|
Note: Cleaned up this case by moving the unrelated expression regressions in 3.0 into separate issues. |
|
@smitpatel @divega somehow I was still on 3.0.0-rc1 branch 😫 switched to 3.0.0-rc2 and the slowdown is gone. Guess this is what happens when you're new to having to work on daily builds. Would still like to know how to access the PDB files for daily builds though. Closing this case for now. |
|
Re-opening to fix issue mentioned in #17617 (comment) For regression, we made a fix in daily builds last week only. |
|
I see #17794 covers this scenario. Closing this as dupe. |
smitpatel
added
closed-no-further-action
I'm trying to nail down a crash occurring with a particular query. The crash only occurs when running a release build on Azure App Services. Running the same query locally in Visual Studio 2019 works fine.
On Azure App Services, I've managed to create a dump file using Crash Diagnoser extension.
Steps to reproduce
I'm unable to create a small reproducible example. The query itself is quite simple, a standard AutoMapper projection. The actual query itself has been in production on the same server for several months, even though we just deployed a new version of the application using additional database columns that do affect the projected entities and the actual SQL generated.
I can provide the dump file, for size (280MB) and security reasons let me know where I can send it for further debugging.
Further technical details
EF Core version: 2.2.6
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Azure App Services
IDE: Visual Studio 2019 16.2.3
The text was updated successfully, but these errors were encountered: