Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Query/InMemory: must be reducible node during compilation for query with weak entities and Any/Contains #19742

Closed
maumar opened this issue Jan 29, 2020 · 0 comments · Fixed by #24363
Closed

Query/InMemory: must be reducible node during compilation for query with weak entities and Any/Contains #19742

maumar opened this issue Jan 29, 2020 · 0 comments · Fixed by #24363
Assignees
@smitpatel
Labels
area-in-memory area-query closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. type-bug
Milestone
6.0.0

Comments

@maumar
Copy link
Contributor

maumar commented Jan 29, 2020

query:

                ss => ss.Set<Level1>().Select(l1 => new
                {
                    l1.Name,
                    OptionalName = l1.OneToOne_Optional_FK1.Name,
                    //Contains = ss.Set<Level1>().Select(x => x.OneToOne_Optional_FK1.Name).Contains(l1.OneToOne_Optional_FK1.Name)
                    Contains = ss.Set<Level1>().Select(x => x.OneToOne_Optional_FK1.Name).Any(x => x == l1.OneToOne_Optional_FK1.Name)
                }),

expression before translation phase:

DbSet<Level1>
    .Select(l => new { 
        Name = l.Name, 
        OptionalName = EF.Property<Level2>(l, "OneToOne_Optional_FK1").Name, 
        Contains = DbSet<Level1>
            .Any(l0 => EF.Property<Level2>(l0, "OneToOne_Optional_FK1").Name == EF.Property<Level2>(l, "OneToOne_Optional_FK1").Name)
     })

subquery expression when trying to translate Any:

DbSet<Level1>
    .Any(l0 => EF.Property<Level2>(l0, "OneToOne_Optional_FK1").Name == (EntityShaperExpression: 
        EntityType: Level1.OneToOne_Required_PK1#Level2
        ValueBufferExpression: 
            (EntityProjectionExpression:
                Property: Level1.OneToOne_Required_PK1#Level2.Id (int) Required PK FK AfterSave:Throw -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 3, Property: Level1.OneToOne_Required_PK1#Level2.Id (int) Required PK FK AfterSave:Throw)
                Property: Level1.OneToOne_Required_PK1#Level2.Date (DateTime) Required -> EntityMaterializerSource.TryReadValue<Nullable<DateTime>>(valueBuffer, 4, Property: Level1.OneToOne_Required_PK1#Level2.Date (DateTime) Required)
                Property: Level1.OneToOne_Required_PK1#Level2.Level1_Optional_Id (Nullable<int>) FK Index -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 5, Property: Level1.OneToOne_Required_PK1#Level2.Level1_Optional_Id (Nullable<int>) FK Index)
                Property: Level1.OneToOne_Required_PK1#Level2.Level1_Required_Id (int) Required FK Index -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 6, Property: Level1.OneToOne_Required_PK1#Level2.Level1_Required_Id (int) Required FK Index)
                Property: Level1.OneToOne_Required_PK1#Level2.Name (string) -> EntityMaterializerSource.TryReadValue<string>(valueBuffer, 7, Property: Level1.OneToOne_Required_PK1#Level2.Name (string))
                Property: Level1.OneToOne_Required_PK1#Level2.OneToMany_Optional_Inverse2Id (no field, Nullable<int>) Shadow FK Index -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 8, Property: Level1.OneToOne_Required_PK1#Level2.OneToMany_Optional_Inverse2Id (no field, Nullable<int>) Shadow FK Index)
                Property: Level1.OneToOne_Required_PK1#Level2.OneToMany_Required_Inverse2Id (no field, int) Shadow Required FK Index -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 9, Property: Level1.OneToOne_Required_PK1#Level2.OneToMany_Required_Inverse2Id (no field, int) Shadow Required FK Index)
                Property: Level1.OneToOne_Required_PK1#Level2.OneToOne_Optional_PK_Inverse2Id (no field, Nullable<int>) Shadow FK Index -> EntityMaterializerSource.TryReadValue<Nullable<int>>(valueBuffer, 10, Property: Level1.OneToOne_Required_PK1#Level2.OneToOne_Optional_PK_Inverse2Id (no field, Nullable<int>) Shadow FK Index)
            )
        IsNullable: True
    ).Name)

exception:



  Message: 
    System.ArgumentException : must be reducible node
  Stack Trace: 
    Expression.VisitChildren(ExpressionVisitor visitor)
    ExpressionVisitor.VisitExtension(Expression node)
    Expression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    ReplacingExpressionVisitor.Visit(Expression expression) line 48
    EntityShaperExpression.VisitChildren(ExpressionVisitor visitor) line 35
    ExpressionVisitor.VisitExtension(Expression node)
    Expression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    ReplacingExpressionVisitor.Visit(Expression expression) line 48
    ReplacingExpressionVisitor.VisitMember(MemberExpression memberExpression) line 55
    MemberExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    ReplacingExpressionVisitor.Visit(Expression expression) line 48
    ExpressionVisitor.VisitBinary(BinaryExpression node)
    BinaryExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    ReplacingExpressionVisitor.Visit(Expression expression) line 48
    ReplacingExpressionVisitor.Replace(Expression original, Expression replacement, Expression tree) line 25
    InMemoryQueryableMethodTranslatingExpressionVisitor.RemapLambdaBody(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression) line 928
    InMemoryQueryableMethodTranslatingExpressionVisitor.TranslateLambdaExpression(ShapedQueryExpression shapedQueryExpression, LambdaExpression lambdaExpression, Boolean preserveType) line 917
    InMemoryQueryableMethodTranslatingExpressionVisitor.TranslateAny(ShapedQueryExpression source, LambdaExpression predicate) line 109
    QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression) line 85
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    QueryableMethodTranslatingExpressionVisitor.TranslateSubquery(Expression expression) line 557
    InMemoryExpressionTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression) line 475
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    InMemoryExpressionTranslatingExpressionVisitor.Translate(Expression expression) line 137
    InMemoryProjectionBindingExpressionVisitor.Visit(Expression expression) line 151
    InMemoryProjectionBindingExpressionVisitor.VisitNew(NewExpression newExpression) line 255
    NewExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    InMemoryProjectionBindingExpressionVisitor.Visit(Expression expression) line 168
    InMemoryProjectionBindingExpressionVisitor.Translate(InMemoryQueryExpression queryExpression, Expression expression) line 47
    InMemoryQueryableMethodTranslatingExpressionVisitor.TranslateSelect(ShapedQueryExpression source, LambdaExpression selector) line 682
    QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression) line 351
    MethodCallExpression.Accept(ExpressionVisitor visitor)
    ExpressionVisitor.Visit(Expression node)
    QueryCompilationContext.CreateQueryExecutor[TResult](Expression query) line 76
    Database.CompileQuery[TResult](Expression query, Boolean async) line 71
    QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async) line 112
    <>c__DisplayClass9_0`1.<Execute>b__0() line 96
    CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler) line 84
    CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler) line 59
    QueryCompiler.Execute[TResult](Expression query) line 92
    EntityQueryProvider.Execute[TResult](Expression expression) line 79
    EntityQueryable`1.GetEnumerator() line 95
    LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
    EnumerableHelpers.ToArray[T](IEnumerable`1 source)
    Enumerable.ToArray[TSource](IEnumerable`1 source)
@ajcvickers ajcvickers added this to the Backlog milestone Jan 31, 2020
@ajcvickers ajcvickers modified the milestones: Backlog, 6.0.0 Nov 5, 2020
@smitpatel smitpatel added the closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. label Mar 9, 2021
@smitpatel smitpatel assigned smitpatel and unassigned maumar Mar 9, 2021
smitpatel added a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression
0650384 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replaced all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934
smitpatel added a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression
2fdce5d 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replace all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934
@smitpatel smitpatel mentioned this issue Mar 10, 2021
Merged
smitpatel added a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression
3e89333 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replace all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934
Resolves #17620
Resolves #18912
smitpatel added a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression
cd69313 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replace all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934
smitpatel added a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression
9a4dcb4 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replace all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934

# Conflicts:
#	test/EFCore.InMemory.FunctionalTests/Query/NorthwindGroupByQueryInMemoryTest.cs
@ghost ghost closed this as completed in #24363 Mar 10, 2021
ghost pushed a commit that referenced this issue Mar 10, 2021
@smitpatel
InMemory: Improve indexing in server expression (#24363)
2778b76 
Implement left join as client method to reduce complexity
To resolve the indexing issue stemming from #23934
Now all the projections are applied immediately to reshape the value buffer so that all our future bindings are always read value expressions which refers to a proper index.
In order to do this, we apply select for entity type with hierarchy to avoid entity check conditional expression.
For adding projection (through ReplaceProjectionMapping),
- For client projection, we apply a select and re-generate client projection as read values
- For projection mapping, we iterate the mappings, we apply a select and re-generate the mapping as read values
- If projection mapping is empty then we add a dummy 1 so that it becomes non-range-variable
When applying projection, we generate a selector lambda to form a value buffer and replace all the expressions to read from new value buffer. Overall this solves the issue of having complex expressions to map or pull. This also removed PushDownIntoSubquery method.

In order to avoid the issue of indexes changing when generating join due to iterating projection mappings, we now also have projectionMappingExpressions which remembers the all expressions inside projectionMapping (which are all read value as we generated before). So now we don't need to iterate the mapping and we use the existing expressions directly. This keeps existing indexes.

Resolves #13561
Resolves #17539
Resolves #18194
Resolves #18435
Resolves #19344
Resolves #19469
Resolves #19667
Resolves #19742
Resolves #19967
Resolves #20359
Resolves #21677
Resolves #23360
Resolves #17537
Resolves #18394
Resolves #23934

# Conflicts:
#	test/EFCore.InMemory.FunctionalTests/Query/NorthwindGroupByQueryInMemoryTest.cs
@ajcvickers ajcvickers modified the milestones: 6.0.0, 6.0.0-preview3 Mar 25, 2021
@ajcvickers ajcvickers modified the milestones: 6.0.0-preview3, 6.0.0 Nov 8, 2021
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smitpatel smitpatel

Labels
area-in-memory area-query closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. type-bug
Projects
None yet
Milestone
6.0.0
Development

Successfully merging a pull request may close this issue.

InMemory: Improve indexing in server expression dotnet/efcore
3 participants
@ajcvickers @smitpatel @maumar