Skip to content
This repository was archived by the owner on May 9, 2024. It is now read-only.

Upgrade to Calcite 1.32 #143

Merged
merged 12 commits into from
Dec 20, 2022
Merged

Upgrade to Calcite 1.32 #143

merged 12 commits into from
Dec 20, 2022

Conversation

alexbaden
Copy link
Contributor

This PR has all the machinery to upgrade to Calcite 1.32. Major changes include:

  • Remove the custom SqlToRelConverter in favor of a more targeted approach when setting up the Calcite planner. Previously, SqlToRel controlled the expansion of subqueries based on the properties of the SQL statement during conversion. Now, we control expansion based on either the SQL statement or the relational algebra composition, depending on the query. There was also some retry logic that was buried -- this logic has been pulled up into a top level try/catch (basically allows for "desugaring" patterns that can't be properly converted back to SQL after parsing). I did have to disable a few correlated subquery tests because I wasn't able to get them to work under the new scheme, but the queries there are not high priority.
  • Projected literals are (for the most part) now part of LogicalValues. This caused a few problems for our existing logical values handling, and exposed some bugs in the CAST framework. I was able to fix most issues, but we will need to address the issue of projecting BIGINT_MAX, which is our current EMPTY_KEY value. So, technically, we should restrict the range of BIGINT to be BIGINT_MIN + 1 and BIGINT_MAX - 1 (but we currently don't enforce the upper limit, at least not in logical values).

I need to make some additional cleanup passes in the Java code -- it is pretty rough. But ArrowBasedExecuteTest and CorrelatedSubQueryTest are now passing locally for me with this set of patches. So, I am putting this up to run full CI with cleanup planned shortly.

@alexbaden alexbaden force-pushed the alex/calcite_upgrade_final branch 2 times, most recently from dad07b8 to ad738ad Compare December 18, 2022 19:26
Allow expansion of IN subquery

Disable expansion of NOT IN subquery

Fix double order by in correlated sort
Replaces the resulting value with a properly typed NULL.
Avoid the Aggregate -> Scan query pattern in favor of Aggregate -> Project -> Scan
This prevents introduction of SARG, which we do not support.
Also consolidate overflow/underflow errors under standard exception class.
RelProject no longer included with RelLogicalValues
@alexbaden alexbaden force-pushed the alex/calcite_upgrade_final branch from ad738ad to 5b0c993 Compare December 18, 2022 19:36
@alexbaden alexbaden marked this pull request as ready for review December 18, 2022 20:58
@alexbaden
Copy link
Contributor Author

This has been cleaned up and is now ready for review.

@alexbaden alexbaden merged commit 9a6335a into main Dec 20, 2022
@alexbaden alexbaden deleted the alex/calcite_upgrade_final branch December 20, 2022 16:49
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant