Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Named query filters #8576

Open
davidroth opened this issue May 24, 2017 · 36 comments · May be fixed by #35104
Open

Named query filters #8576

davidroth opened this issue May 24, 2017 · 36 comments · May be fixed by #35104

Comments

@davidroth
Copy link
Contributor

davidroth commented May 24, 2017

While the new query filters feature is really cool, I think it is lacking additional flexibility for configuration + usage (enable / disable).

Let me use the example posted in your announcement blog: https://blogs.msdn.microsoft.com/dotnet/2017/05/12/announcing-ef-core-2-0-preview-1/

public class BloggingContext : DbContext
{
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Post>()
            .HasQueryFilter(p => !p.IsDeleted &&
                  p.TenantId == this.TenantId );
    }
}

This example uses one filter for tenant id + isDeleted.
The only way to disable both filters is by using the following API:

Set<Post>().IgnoreQueryFilters()

Unfortunately there is no way to disable a specific part of the. It would make sense for example to disable only the IsDeleted filter, while keeping the tenant filter enabled.

This could be enabled by specifying multiple filters by key:

public class BloggingContext : DbContext
{
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Post>()
              .HasQueryFilter(p => !p.IsDeleted, key: "IsDeleted")
              .HasQueryFilter(p => p.TenantId == this.TenantId, key: "Tenant");
    }
}

Now one would have full flexibility to either disable all filters, or disable only specific filters:

Set<Post>().IgnoreQueryFilters("IsDeleted") // Only disables IsDeleted query filter, but keeps Tenant filter enabled

Set<Post>().IgnoreQueryFilters("TenantId") // Only disables TenantId query filter, but keeps is deleted filter enabled

Set<Post>().IgnoreQueryFilters() // disable all filters

Have you considered this scenario?

@ajcvickers
Copy link
Member

@davidroth Thanks for the suggestion, This is an interesting idea, but not something we are planning to implement for 2.0. We may work on it in the future, but that would likely depend on additional feedback.

We would also consider a well-written community PR for this. (If anyone does plan to send a PR, please check with us first so we can do some thinking and give some guidance/feedback on the API.)

@davidroth I suspect you could figure this out anyway, but it is possible to have a flag on the context that can be used to switch on and off various parts of the filter. Something like"

        modelBuilder.Entity<Post>()
            .HasQueryFilter(p => (!this.DeleteFilterEnabled || !p.IsDeleted)
                && p.TenantId == this.TenantId );

@ajcvickers ajcvickers added this to the Backlog milestone May 24, 2017
@ajcvickers ajcvickers added type-enhancement help wanted This issue involves technologies where we are not experts. Expert help would be appreciated. labels May 24, 2017
@popcatalin81
Copy link

popcatalin81 commented May 25, 2017

@ajcvickers @davidroth
I suggest adding a factory overload for defining the filter that allows client code create the filter as needed at query time, instead of at OnModelCreating, something like

public class BloggingContext : DbContext
{
	protected override void OnModelCreating(ModelBuilder modelBuilder)
	{
		modelBuilder.Entity<Post>()
			.HasQueryFilterFactory(ctx => 
			{ 
				if (ctx.ApplyDeleteFilter)
					return p => !p.IsDeleted && p.TenantId == ctx.TenantId
				else
					return p => p.TenantId == ctx.TenantId;
			});
	}
}

This way client code can maintain multiple situational filters without generating overly complex where clauses all the time.

@hikalkan
Copy link

I suggest to check https://github.com/zzzprojects/EntityFramework.DynamicFilters to get idea. It's widely used and makes same thing for EF6.

@f135ta
Copy link

f135ta commented Apr 17, 2018

I've had a go at this. I've managed to get it working with relatively minor changes. It involves replacing the QueryFilter lambda on the Entity with a custom dictionary of [QueryFilters]. It works fine, apart from the internal expression caching - if I call the same query twice with the filter enabled and disabled, because QueryFilter is added 'after' the query is generated, it picks the cached one instead. Does anyone have any ideas how I can clear the query cache when I switch the filter on and off?

I've put together a really simple console app to test it, and monitored the sql generated in the SQL Profiler:

public class Program
    {
        public static void Main(string[] args)
        {
            var db = new TestContext(new DbContextOptions<TestContext>());

            Console.WriteLine("Filter Enabled");

            // With Filter Enabled
            var query = db.Accounts.OrderBy(p => p.Id);
            foreach (var t in query)
            {
                Console.WriteLine(t.Name + t.IsDeleted);
            }

            Console.WriteLine("Filter Disabled");

            db.DisableQueryFilter<Account>("IsDeleted");

            // With Filter Disabled
            foreach (var t in db.Accounts.OrderBy(p => p.Name))
            {
                Console.WriteLine(t.Name + t.IsDeleted);
            }

            Console.ReadLine();
        }
    }

Notice the second query uses [p=> p.Name] for the order by expression. If I set this to the same as the first query, the filter is bypassed.

I have access to Enable and Disable methods via an extension method on the DbContext:

 /// <summary>
    /// Extension Methods for <see cref="DbContext"/>
    /// </summary>
    public static class DbContextExtensions
    {
        /// <summary>
        /// Disables the Query Filter for the specified entity
        /// </summary>
        /// <typeparam name="T">Entity Type</typeparam>
        /// <param name="context">DbContext</param>
        /// <param name="filterName">Filter Name</param>
        public static void DisableQueryFilter<T>(this DbContext context, string filterName) where T : class
        {
            var entityType = context.Model.FindEntityType(typeof(T));
            entityType.DisableQueryFilter(filterName);
        }

        /// <summary>
        /// Enables a Query Filter
        /// </summary>
        /// <typeparam name="T">Entity Type</typeparam>
        /// <param name="context">DbContext</param>
        /// <param name="filterName">Filter Name</param>
        public static void EnableQueryFilter<T>(this DbContext context, string filterName) where T : class
        {
            var entityType = context.Model.FindEntityType(typeof(T));
            entityType?.EnableQueryFilter(filterName);
        }

        /// <summary>
        /// Enables all Query Filters with the specific name
        /// </summary>
        /// <param name="context">DbContext</param>
        /// <param name="filterName">Filter Name</param>
        public static void EnableAllQueryFilters(this DbContext context, string filterName) 
        {
            var entityTypes = context.Model.GetEntityTypes();
            foreach (var entity in entityTypes)
            {
                entity?.EnableQueryFilter(filterName);
            }
        }

        /// <summary>
        /// Disables all Query Filters with the specific name
        /// </summary>
        /// <param name="context">DbContext</param>
        /// <param name="filterName">Filter Name</param>
        public static void DisableAllQueryFilters(this DbContext context, string filterName)
        {
            var entityTypes = context.Model.GetEntityTypes();
            foreach (var entity in entityTypes)
            {
                entity?.DisableQueryFilter(filterName);
            }
        }

    }

@f135ta
Copy link

f135ta commented Apr 18, 2018

If anyone is interested in helping me with this; its this method in particular that is causing my issue.

private Func<QueryContext, TFunc> GetOrAddQueryCore<TFunc>(
            object cacheKey, Func<Func<QueryContext, TFunc>> compiler)
        {
            Func<QueryContext, TFunc> compiledQuery;

            retry:
            if (!_memoryCache.TryGetValue(cacheKey, out compiledQuery))
            {
                if (!_querySyncObjects.TryAdd(cacheKey, null))
                {
                    goto retry;
                }

                try
                {
                    compiledQuery = compiler();

                    _memoryCache.Set(cacheKey, compiledQuery);
                }
                finally
                {
                    _querySyncObjects.TryRemove(cacheKey, out _);
                }
            }

            return compiledQuery;
        }

It lives in the CompiledQueryCache.cs file: Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.cs

@smitpatel
Copy link
Contributor

smitpatel commented Apr 18, 2018

@f135ta - IEntityType.EnableQueryFilter does not exist. What are you calling in that code?

And why do we want to skip cache? Why not compile just one expression which would apply all filters with disable/enable thing appropriately.

@f135ta
Copy link

f135ta commented Apr 18, 2018

Correct, its part of my changes to code base: In the EntityType.cs i've added the following code:


/// <inheritdoc />
        Dictionary<string, QueryFilterItem> IMutableEntityType.QueryFilterCollection => this.QueryFilterCollection;

        private Dictionary<string, QueryFilterItem> _queryFilterCollection;

        /// <summary>
        /// Gets or sets a collection of Filters to be executed on the entity
        /// </summary>
        public virtual Dictionary<string, QueryFilterItem> QueryFilterCollection => _queryFilterCollection ?? (_queryFilterCollection = new Dictionary<string, QueryFilterItem>());

        /// <summary>
        /// Applies a query filter to the entity
        /// </summary>
        /// <param name="filterName">Filter Name (Identifier)</param>
        /// <param name="value">Query Lamdba Expression</param>
        /// <param name="enabled">Is the filter enabled</param>
        public void AddQueryFilter(string filterName, LambdaExpression value, bool enabled)
        {
            if (value != null && (value.Parameters.Count != 1 || value.Parameters[0].Type != ClrType || value.ReturnType != typeof(bool)))
            {
                throw new InvalidOperationException(CoreStrings.BadFilterExpression(value, this.DisplayName(), ClrType));
            }

            var filterItem = new QueryFilterItem
            {
                Enabled = enabled,
                Query = value
            };

            this.QueryFilterCollection.Add(filterName, filterItem);
        }

        /// <summary>
        /// Enables a filter query
        /// </summary>
        /// <param name="filterName">Filter Name</param>
        public void EnableQueryFilter(string filterName)
        {
            if (this.QueryFilterCollection.ContainsKey(filterName))
            {
                this.QueryFilterCollection[filterName].Enabled = true;
            }
        }

        /// <summary>
        /// Disables a filter query
        /// </summary>
        /// <param name="filterName">Filter Name</param>
        public void DisableQueryFilter(string filterName)
        {
            if (this.QueryFilterCollection.ContainsKey(filterName))
            {
                this.QueryFilterCollection[filterName].Enabled = false;
            }
        }

Actually, you're correct - "And why do we want to skip cache? Why not compile just one expression which would apply all filters with disable/enable thing appropriately." I'd rather the query was cached, but it appears that its added to and retrieved from the cache 'before' the queryfilters are applied - and i cant quite work out how to switch it around

@smitpatel
Copy link
Contributor

@f135ta - We cannot mutate the IModel when running the query.

@f135ta
Copy link

f135ta commented Apr 18, 2018

Ok, so the filters would be applied during model generation in the DbContext as normal.

protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<Account>(
                entity =>
                    {
                        entity.ToTable("Account");
                        entity.Property(e => e.IsActive).HasColumnName("isActive");
                        entity.Property(e => e.IsDeleted).HasColumnName("isDeleted");
                        entity.Property(e => e.IsSystem).HasColumnName("isSystem");
                        entity.Property(e => e.Name).HasMaxLength(100);
                        **entity.HasQueryFilter("IsDeleted", p => p.IsDeleted);**
                    });
        }

Like they are currently. Except im passing in another parameter "FilterName" which is used to identify it in the QueryFilterCollection stored on the EntityType. (In place of the QueryFilter Lambda Expression that is currently there) .. The extension methods merely change a flag on the filter item in the QueryFilterCollection to enable/disable it. Would that not be correct?

@smitpatel
Copy link
Contributor

The extension methods merely change a flag on the filter item in the QueryFilterCollection to enable/disable it. Would that not be correct?

That particular bit becomes incorrect. QueryFilterCollection lives on EntityType inside the model. You cannot change the value of that flag. You would need a better way to define how to enable/disable filters. You would need a method on each query to control the behavior as suggested in first post. Similar to how include works on query roots.
We believe that if you do so then based on filter name passed into the method, there would be different ET generated hence you won't hit issue into query cache entry. (though we could optimize it and have 1 query cache entry for all variety of filters applied using boolean algebra.)

@f135ta
Copy link

f135ta commented Apr 18, 2018

Ok, so the issue is that it lives on the EntityType? It makes sense now as to why its not picked up the way I was expecting; since the EntityType is built during the model generation and sealed off. I'll take a look at the Includes functionality and have a rethink. I'd still like to keep the HasQueryFilter functionality on the onModelBuilding because it makes sense for it to be there (if possible).

Thanks for your help! I'll have another stab at it because its a feature that i'd really like to see implemented; I'll keep you updated on the progress ;-)

@smitpatel
Copy link
Contributor

@f135ta - HasQueryFilter would certainly live on OnModelCreating method. Basically, it is like tell us all the filters you would like your model to have. And then a different mechanism which has readonly access to model to enable/disable parts of it. A result operator would be easy choice to annotate to pass the info to query compiler and at the same time, differentiate the ExpressionTree in cache.

@BalassaMarton
Copy link

Not sure if I should create a new issue for this, so I'll just dump it here for now.
Background: Implementing soft delete, but the soft delete filter is just one of many global filters that I want to switch on and off even on a per-query basis.
Proposal 1: Instead of defining a single global query filter, we could use filter providers:

modelBuilder.Entity<Thing>.HasQueryFilter<SoftDeleteFilter>();

Where SoftDeleteFilter is called a filter provider, that is, something that produces a lambda for filtering.

  • HasQueryFilter can be called multiple times with different filter provider types to combine filters.
  • Filters can be removed (e.g. in a derived db context type):
modelBuilder.Entity<Thing>.RemoveQueryFilter<SoftDeleteFilter>();
  • Filter providers should be able to disable caching of the returned expressions.
  • Filters can be used/ignored on a per-query basis:
var list = db.Things.IgnoreQueryFilter<SoftDeleteFilter>().WithQueryFilter<MyCustomQueryFilter>().ToList();

Note: WithQueryFilter can be implemented today, it's just adding a Where condition with the lambda supplied by the filter provider.
Note 2: We could write extension methods to make the code more readable, like:

var list = db.Things.IncludeDeleted().WithSomeFeature().ToList();
  • At the lowest level, we should be able to add filter provider instances:
var list = db.Things.AsOf(new DateTime(DateTime.Now.Year, 1, 1)).ToList();

...which adds a filter provider under the hood:

public static IQueryable<T> AsOf<T>(this DbSet<T> source, DateTime timestamp) {
    return source.WithQueryFilter(new TemporalQueryFilterProvider<T>(timestamp));
}

Proposal 2: Thinking a little further, simple predicate-based query filters could be abstracted out into more general query rewriters:

modelBuilder.Entity<Thing>.HasQueryRewriter<MultiTenancyQueryRewriter>();
modelBuilder.Entity<Thing>.HasQueryRewriter<TemporalQueryRewriter>();

I think this last bit is already tracked in #626 although it doesn't seem like anyone was working on enabling it.

@joaopalma5
Copy link

This question have a solution? Now, how is a better way for dinamically enable/disable one Global Filter?

@ilmax
Copy link
Contributor

ilmax commented Oct 9, 2018

@f135ta are you still working on this one? I woul love to have this into effect so I'd like to help.

@f135ta
Copy link

f135ta commented Oct 10, 2018

@ilmax - Hi! Yes, its still on my list of things to work on. I've been mega busy with "real-life" so I havent had time to complete the work yet. Hopefully i'll be able to dedicate some time to it over the weekend.

@f135ta
Copy link

f135ta commented Nov 16, 2018

Quick update on my attempts with this. I've rethought my approach and instead started to look at the QueryCompiler and going down the route of intercepting the calls within there to fetch custom queryfilters and applying them before they're executed.

I believe this will mean that they are caught and processed by the CompiledQueryCache and therefore fix my original problem (noted above). Queries can still be added in the same way using the EntityTypeConfiguration. I'll then look at how we make extension methods so that queries can have an inline enable/disable function for each query.

I'm probably about 50% the way to getting a working prototype for this approach and will share my ideas in more detail when I have a proof of concept available..

@paszkowskik
Copy link

Is any chance that this feature will be implemented in future release ?

@popcatalin81
Copy link

popcatalin81 commented Jan 18, 2019

Is any chance that this feature will be implemented in future release ?

https://blogs.msdn.microsoft.com/dotnet/2018/12/04/announcing-entity-framework-core-2-2/

It's not on roadmap, so don't count on it for the next version (but there's still may be a change if EF team decides to do it)

@paszkowskik
Copy link

Thank you for the replay. I am full of hope that EF team will see a value of this feature

@vitalybibikov
Copy link

I currently have to go with RLS in MSSQL, as it's not possible to implement proper filtration on multiple conditions with EfCore at the moment.

Personally, I hate to store any kind of logic inside of a database, but this is the only option we have.

So, "more flexibility" is definitely a thing I desire to be supported.

Current solution is not usable, when in 1 app you have:

  1. Soft Delete.
  2. Multi-tenancy
  3. Segregation of users by user-groups
  4. Segregation of customers by customer-groups

@f135ta
Copy link

f135ta commented Mar 14, 2019

@vitalybibikov - I'm confused at how this function prevents you from using EF? Query Filters provide "additional" functionality to EF, in that they allow you to globally set a filter for an entity. There is absolutely nothing stopping you from coding the 4 things you have already mentioned yourself? Linq is provided for exactly that purpose; so you can shape your queries?

@vitalybibikov
Copy link

I've never said it prevents us from using Ef Core. We are using it.
What I'm saying is that we can only go with 1 option from the list at the moment.

It is possible to create a set of Extensions, though it won't be as performant as on OnModelCreating.
Currently, I'm not aware of how to do it in one place, without changes in major part of the queries in the application.

While it is possible to implement and test a proper solution, it consumes time and requires support.

@Danielku15
Copy link

Danielku15 commented Sep 5, 2019

I guess to find the ultimate approach of really implementing individual query filters, it would already help a lot to have a context with query specific properties available. With the feature set, you can add 1 or more properties to a DbContext where you manually enable-disable query filters per-entity basis. But this has the problematic limitation that it is valid for all queries you make on this context. Or simply said: you always need to update the context properties to match your query that you want to have.

Here such an example context that would not allow this flexibility:

public class BloggingContext : DbContext
{
    public bool IsPostDeleteFilterActive {get;set;}
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Post>()
            .HasQueryFilter(p => !IsPostDeleteFilterActive || !p.IsDeleted)
    }
}

var ctx = new BloggingContext();
var withDeleted = ctx.Posts;
var withoutDeleted = ctx.Posts; 
// later in the code: 
ctx.IsPostDeleteFilterActive  = true; 
var resultsWithDeleted = withDeleted.ToArray();
ctx.IsPostDeleteFilterActive  = false; 
var resultWithoutDeleted = withoutDeleted.ToArray();

It is not nice to need setting the properties correctly before you actually execute the queries. Sometimes you even cannot do this as you pass on the queryable to another library (like OData).

It would be great if we can have something like this as a first implementation on EF:

public class BloggingContext : DbContext
{
    public DbSet<Post> Posts{get;set;}
    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<Post>()
            .HasQueryFilter((p, context) => !context.Properties.Get<bool>("IsPostDeleteFilterActive", defaultValue: true) || !p.IsDeleted)
    }
}

var ctx = new BloggingContext();
var withDeleted = ctx.Posts.WithProperty("IsPostDeleteFilterActive", false);
var withoutDeleted = ctx.Posts; 
// later in the code: 
var resultsWithDeleted = withDeleted.ToArray();
var resultWithoutDeleted = withoutDeleted.ToArray();

Here the properties/flags are attached to the query and can be used within the filters.

The Properties.Get<bool>(key, defaultValue) of course would need to be translated to a constant value so that it works well together with the query compiler. I see it as main issue today, that you have no built-in way of "attaching" some metadata to the query that can be interpreted at other places. It might not be the ultimate solution to realize the full flexibility of query filters but it could build the foundation for other libraries to built a more advanced system on top.

@mhosman
Copy link

mhosman commented Jan 23, 2020

Any news in this one? Thanks!

mnijholt added a commit to mnijholt/efcore that referenced this issue May 13, 2021
- Configure multiple query filters on a entity referenced by name.
- Ignore individual filters by name.

The current implmentation of query filter is kept but when ignoring the filters using the current extension method will ignore all filters (so also the named).

Fixes dotnet#8576 dotnet#10275 dotnet#21459
@mnijholt
Copy link

I tried to implement a solution that allows for multiple query filters to be defined on a single entity and allow them to be ignored individually of each other.

When setting up a query filter you could provide a name for the query filter. This allows you to define multiple query filters on the same entity. When you call HasQueryFilter with the same name twice only the last one will be used just the way it is implemented now. When calling the HasQueryFilter method twice with the same name only the filter of the last call will be used just as it's implemented now so that it will not break current implementation.

modelBuilder.Entity<Order>()
    .HasQueryFilter(o => o.Customer != null && o.Customer.CompanyName != null);

modelBuilder.Entity<OrderDetail>()
   .HasQueryFilter("HasOrder", od => od.Order != null);

modelBuilder.Entity<OrderDetail>()
    .HasQueryFilter("Quantity", od => EF.Property<short>(od, "Quantity") > _quantity);

When it comes to ignoring query filters you can still use the current solution. This will ignore all the configured query filters (so named an not named) on all entities just like the current implementation.

context.Set<OrderDetail>()
    .Include(c => c.Orders)
    .IgnoreQueryFilters();

But if you only want to ignore a specific filter you can do so by calling IgnoreQueryFilter and specifying the name of the query filter. This will also ignore all query filters with that name for all entities. You can also disable multiple named query filters by calling the IgnoreQueryFilter method multiple times.

context.Set<OrderDetail>()
    .Include(c => c.Orders)
    .IgnoreQueryFilter("HasOrder")
    .IgnoreQueryFilter("Quantity");

This will allow you to implement solutions like soft delete and global user filtering and disable them individually context wide. But it also gives developers the freedom to use the functionality to create their own solutions.

I kept the current solution in place so calling HasQueryFilter without still adds the query filter, when calling it multiple times it will also use the last one. I also did not make any breaking changes, but I changed the internal implementation so there would not be two solution to filter the queries. I did this by using String.Empty for the name of the current implementation and not allowing the developer to specify a named query for String.Empty.

Any feedback on the current solution is welcome also any guidance to improve the solution or using a different one. I really would love this functionality to be part of efcore so I'm willing to put the time in it.

@smitpatel smitpatel removed this from the Backlog milestone May 14, 2021
@AndriySvyryd AndriySvyryd added this to the Backlog milestone May 18, 2021
@smitpatel
Copy link
Contributor

We discussed this issue in team meeting again.

Currently IgnoreQueryFilters is query-level flag which ignores all query filters. With named filter more semantics come into play that would ignore the given named filter on particular entity only or whole query level. What if you specify multiple on different with expectation that it will only apply to particular entity. How to specify ignoring them in navigation expansion - for example filtered include case. See #17347 for feedback on existing IgnoreQueryFilters API. Above API is just make problem even worse.

Also related issue - #10275 is about adding multiple query filters perhaps for ease of creation but doesn't explicitly defines goal as it to be named filters. (Named filters can be ignored individually, multiple filters is just about creating and EF Core side composing it in single lambda for users).

So above proposed design is incomplete and there are various questions which needs to be answered/decided. Further the implementation is not improving anything specific apart from just adding enumerable rather than single instance and enumerating it where instance was used. (ignoring the context where Ignore was specified. Issue with conflicting Ignore etc). We as a team don't have time to arrive at a full design for this feature or iterate over design proposals in 6.0 release timeframe.

We can continue discussing design here if desired though response from the team would be pretty much delayed till most of 6.0 work is finished.

@ntziolis
Copy link

@smitpatel The concerns about unclear scope / api is totally valid.

However, the flipside of not having this functionality is that developers have to disable the query filters globally. Which leads to manually crafting often VERY complex and error prone statements. So there is an argument to be made that not having this functionality is worst than the potential ambiguity in the api. That said I feel there should be a way to get the best of both worlds.

Some thoughts on removing ambiguity and an api proposal:

  • Not to break existing behavior if it can be avoided (IgnoreQueryFilters() = query-level flag which ignores all query filters)
  • In addition what should be possible is
    • Disable a specific query filter on query level (= query-level flag which ignores a specific query filter)
    • Disable a specific query filter on a specific entity

The above 3 scenarios could be expressed as follows:

// query-level flag which ignores ALL query filters
context.Orders
    .Include(c => c.Positions)
    .IgnoreQueryFilters();

// query-level flag which ignores A SPECIFIC query filters
context.Orders
    .Include(c => c.Positions)
    .IgnoreQueryFilters("IsDeleted");

// ignore a SPECIFIC query filter for a SPECIFIC entity
// returns all orders with isDeleted query filter active + all positions without IsDeleted query filter active
context.Orders
    .Include(c => c.Positions)
    .IgnoreQueryFilters<OrderPosition>("IsDeleted");

Scenario 1 is cannot be combined with the other scenarios as by design it disables all query filters.

However scenarios 2 + 3 can both be stacked as well as combined:

context.Customers
    .Include(c => c.Orders).ThenInclude(i => i.Positions)
    .IgnoreQueryFilters("Tenant") // disable tenant filter on query level
    .IgnoreQueryFilters<Customer>("IsDeleted") // disable is deleted filter for Customer entities
    .IgnoreQueryFilters<OrderPosition>("IsDeleted"); // disable is deleted for OrderPosition entities

In plain text the above would query:

  • data across all tenants (applies to all levels)
  • all customers (incl deleted ones)
  • their non deleted orders
  • and all positions of these orders (incl deleted ones)

This syntax removes any ambiguity I could think while keeping in line with the existing syntax as well.

Thoughts? Did I miss any use case?

@ldeluigi
Copy link

Any update on this?

@ajcvickers
Copy link
Member

@ldeluigi This issue is in the Backlog milestone. This means that it is not planned for the next release (EF Core 8.0). We will re-assess the backlog following the this release and consider this item at that time. However, keep in mind that there are many other high priority features with which it will be competing for resources. Make sure to vote (👍) for this issue if it is important to you.

@TheDusty01
Copy link

This is open since 2017 and doesn't seem very complicated. There is even a PR for this already (even though it doesn't meet the requirements?).

Anyway what are the cons or difficult design choices with this one?

@ajcvickers
Copy link
Member

@TheDusty01 I think all the discussion above outlines some of the design choices that need to be made, and shows that this is, in fact, quite complicated.

@TheDusty01
Copy link

Currently the alternative is to add a few complex statements to each call which is pretty error prone.

I do understand the thoughts and risks but there are way more complex issues which are resolved faster than this one.

So it may be better to provide any decent/slightly extensible solution instead of none at all I guess.

@Meligy
Copy link

Meligy commented Apr 8, 2024

A minimal version of this IMO is (quoting above):

// query-level flag which ignores A SPECIFIC query filters
context.Orders
    .Include(c => c.Positions)
    .IgnoreQueryFilters("IsDeleted");

It also enables entity-level disabling, as EF users can maintain the keys they used for a specific entity in a variable or something and disable them all. Maybe even add an extension method that does it.

@SimonCropp
Copy link
Contributor

SimonCropp commented May 8, 2024

so i had a slightly different requirement. i use snapshot testing when running tests against EF. so as part of that, when some code runs, any sql queries that are executed are capture into a file that is convention named based on the current test. we use QueryFilters for soft deletes, and this functionality is not relevant to the majority of tests. but our QueryFilters add significant noise to the sql snapshots. so i wanted a way of toggling QueryFilters on/off from within a test.

so even though the requirements, i suspect the approach used may help other people when customising query filters

note that none of this code is in the production app. it all exists in the test project

this is the code i came up with:

QueryFilter

a wrapper around an AsyncLocal flag

public static class QueryFilter
{
    static AsyncLocal<bool> disabled = new();

    public static void Disable() =>
        disabled.Value = true;

    public static void Enable() =>
        disabled.Value = false;

    public static bool IsEnabled => !disabled.Value;
    public static bool IsDisabled => disabled.Value;
}

QueryContextFactory

in effect calls .IgnoreQueryFilters() for any query based on QueryFilters flag.

class QueryContextFactory(QueryCompilationContextDependencies dependencies, RelationalQueryCompilationContextDependencies relationalDependencies, ISqlServerConnection connection)
    : SqlServerQueryCompilationContextFactory(dependencies, relationalDependencies, connection)
{
    [UnsafeAccessor(UnsafeAccessorKind.Field, Name = "<IgnoreQueryFilters>k__BackingField")]
    private static extern ref bool IgnoreQueryFilters(QueryCompilationContext context);

    public override QueryCompilationContext Create(bool async)
    {
        var context = base.Create(async);

        if (QueryFilter.IsDisabled)
        {
            IgnoreQueryFilters(context) = true;
        }

        return context;
    }
}

KeyGenerator

To ensure that queries that queries are cached separately based on if QueryFilters are enabled or disabled

class KeyGenerator(CompiledQueryCacheKeyGeneratorDependencies dependencies, RelationalCompiledQueryCacheKeyGeneratorDependencies relationalDependencies, ISqlServerConnection connection)
    : SqlServerCompiledQueryCacheKeyGenerator(dependencies, relationalDependencies, connection)
{
    readonly ISqlServerConnection connection = connection;

    public override object GenerateCacheKey(Expression query, bool async)
        => new SqlServerCompiledQueryCacheKey(
            GenerateCacheKeyCore(query, async),
            connection.IsMultipleActiveResultSetsEnabled,
            QueryFilter.IsEnabled);

    readonly struct SqlServerCompiledQueryCacheKey(
        RelationalCompiledQueryCacheKey relationalKey,
        bool mars,
        bool queryFilterEnabled)
        : IEquatable<SqlServerCompiledQueryCacheKey>
    {
        readonly RelationalCompiledQueryCacheKey relationalKey = relationalKey;
        readonly bool mars = mars;
        readonly bool queryFilterEnabled = queryFilterEnabled;

        public override bool Equals(object? obj)
            => obj is SqlServerCompiledQueryCacheKey key &&
               Equals(key);

        public bool Equals(SqlServerCompiledQueryCacheKey other)
            => relationalKey.Equals(other.relationalKey) &&
               mars == other.mars &&
               queryFilterEnabled == other.queryFilterEnabled;

        public override int GetHashCode()
            => HashCode.Combine(relationalKey, mars, queryFilterEnabled);
    }
}

Replace services

builder.ReplaceService<IQueryCompilationContextFactory, QueryContextFactory>();
builder.ReplaceService<ICompiledQueryCacheKeyGenerator, KeyGenerator>();

Usage

QueryFilter.Enable() or QueryFilter.Disable() can now be used in tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet