Improve comparison predicate pushdown when varchar column cast to date#13567
Conversation
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
2 times, most recently
from
800ebc9 to
0b0ad8d
Compare
|
Marked draft because
|
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
from
0b0ad8d to
af06f25
Compare
core/trino-main/src/test/java/io/trino/sql/query/QueryAssertions.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/sql/planner/DomainTranslator.java
Outdated
Show resolved
Hide resolved
testing/trino-testing/src/main/java/io/trino/testing/BaseConnectorTest.java
Outdated
Show resolved
Hide resolved
The case where varchar is cast to date and used in a predicate is not uncommon and it would be worth optimizing for. This involves, however, some edge cases which are easy to forget about when working on such an optimization (either within the engine, or on the connector level).
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
2 times, most recently
from
dfe10c3 to
67ef8fb
Compare
|
Ready for review now. |
findepi
requested review from
alexjo2144,
findinpath,
hashhar,
losipiuk and
martint
plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java
Outdated
Show resolved
Hide resolved
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
2 times, most recently
from
a968bbe to
c13966f
Compare
|
Added a test in TestIcebergMetadataFileOperations that shows gains for Iceberg. |
core/trino-main/src/main/java/io/trino/sql/planner/DomainTranslator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
I find this addition hidden gem.
Could we get this documented ?
We should probably add here in the test hints that the casting improvement is not for everything.
e.g. : SELECT * FROM test_varchar_as_date_predicate WHERE year(CAST(a AS date)) >= 2005 does not benefit anymore of the the improvement.
There was a problem hiding this comment.
I find this addition hidden gem.
Could we get this documented ?
"Trino does as much optimizations as possible".
I think we should not try to document all the optimizations. What for?
e.g. :
SELECT * FROM test_varchar_as_date_predicate WHERE year(CAST(a AS date)) >= 2005does not benefit anymore of the the improvement.
yes, not yet.
Out of curiosity: how would we be able to test the gains? |
We could perhaps check number of input positions (rows) retrieved from the remote database. |
core/trino-main/src/main/java/io/trino/sql/planner/DomainTranslator.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Sanity question. Char SQL type ordering uses the same order as Java char ordering? ie. if you add one to a char in java you also get the next char when it comes to SQL char sorting
There was a problem hiding this comment.
In general -- no (i think it breaks for non-BMP)
Here, we're operating within ASCII (digits and hyphen). '9' + 1 produces ':', still ASCII.
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
2 times, most recently
from
e096285 to
1ad0add
Compare
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
2 times, most recently
from
a97ecba to
422d0fb
Compare
Such a cast cannot reasonably be pruned away because the source varchar value can be any of multiple forms (surrounding whitespace, optional year sign, optional leading zeros for date components). However, a usefully narrow `Domain` can be extracted and passed over to connectors to help connectors prune the data.
findepi
force-pushed
the
findepi/predicate-cast-varchar-to-date
branch
from
422d0fb to
970a5a4
Compare
5 participants
Such a cast cannot reasonably be pruned away because the source varchar
value can be any of multiple forms (surrounding whitespace, optional
year sign, optional leading zeros for date components). However, a
usefully narrow
Domaincan be extracted and passed over to connectorsto help connectors prune the data.
Fixes #12925