Add IN operator limit

[wolfgangcolsman]

The limit of the values per IN operator depends on the database. E.g. Oracle allows only up to 1,000 IN list values in a SQL statement.

[tooptoop4]

is this missing the change on Oracle class? Similarly Postgres also has a limit. Does the remainder part in QueryBuilder.java create separate ORs for another set of IN?

[wolfgangcolsman]

is this missing the change on Oracle class? Similarly Postgres also has a limit. Does the remainder part in QueryBuilder.java create separate ORs for another set of IN?

I don't think the Oracle and Postgres need to be changed. They would maybe need to set the correct property value.

If I understand the code correctly, the list of INs will be ORed below

[findepi]

What is the purpose of the change? At the first sight it looks redundant over domain-compaction-threshold. What am i missing?

[wolfgangcolsman]

What is the purpose of the change? At the first sight it looks redundant over domain-compaction-threshold. What am i missing?

Compaction threshold converts a list of values into a range query. This has it's issues e.g with guids.

The proposed change here addresses the situation that more than N values make it into the IN statement, when setting the compaction value > N

[findepi]

@wolfgangcolsman in our testing internally, we found our that databases typically have performance problems with large IN queries.
Also, there are PreparedStatement limits, query text limits and such, which mean that trying to push very large domains may hit other limitations.

What is the scenario you're trying to improve? What's the domain size where you see full pushdown beneficial from query performance perspective?

[wolfgangcolsman]

@findepi Our use case is large dynamic filters where we have thousands of records returned per FK in the IN clause. Full table scans are not an option, and range queries do not work because FKs use GUIDs

[colebow]

👋 @wolfgangcolsman - this PR has drawbacks that don’t align with our preferred approach, so we’re not going to proceed with this implementation. If you'd like to continue the discussion on this at any point in the future, feel free to re-open and chime in.