[Windowing] Rewrite window function implementation to use real SQLite windows #169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ohaibbq
commented
Feb 15, 2024
| @@ -63,3 +63,5 @@ require ( | |||
| google.golang.org/grpc v1.54.0 // indirect | |||
| google.golang.org/protobuf v1.30.0 // indirect | |||
| ) | |||
|
|
|||
| replace github.com/mattn/go-sqlite3 => github.com/ohaibbq/go-sqlite3 v0.0.0-20240211011509-f8d4d3382d11 | |||
There was a problem hiding this comment.
This will be replaced with upstream once my PR makes it into the new release.
Codecov Report
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #169 +/- ##
==========================================
- Coverage 46.39% 46.14% -0.25%
==========================================
Files 47 47
Lines 17828 16884 -944
==========================================
- Hits 8271 7791 -480
+ Misses 8111 7684 -427
+ Partials 1446 1409 -37 |
|
First, Thank you for your contribution. I think this PR is great, but I can't merge forked go-sqlite3. I would like to reflect this after go-sqlite3 side's PR is merged. |
|
I'll need to cherry-pick Recidiviz#33 into this branch before we merge. It looks like the |
|
We will also need to cherry-pick Recidiviz#42 in order to fix #200 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @goccy @totem3, I have an exciting PR here.
I implemented the ability to register user-defined window functions in the SQLite driver at mattn/go-sqlite3#1220.
This allows us to remove the exponential complexity of formatted queries using window functions, as well as the bulk of
go-zetasqlite's custom windowing. Previously, the formatter took the query currently being processed, turned it into a newSELECT * FROM (...)clause as found here:go-zetasqlite/internal/formatter.go
Lines 1109 to 1114 in 961ce06
For one of our more complex queries, the formatted query goes from ~278 SELECT statements, with ~150 nested
SELECT .. FROM (SELECT ...)queries, down to78SELECT statements with only ~50 nested SELECT queries.This amount of nesting is still too high for SQLite to handle (#153), but the
3.46.0release of SQLite will allow the parser to dynamically grow its stack size in order to support this.Previously, the
go-zetasqlitefunctions needed to implement their own sorting and partitioning of values in the windows. This caused various issues (see #139 and #101). Now SQLite will handle ordering and partitioning for us viazetasqlite_collate.The SQLite window functions do not allow the use of
DISTINCTorIGNORE NULLS, so we still implement those via thezetasqlite_window_option_distinct()andzetasqlite_window_option_ignore_nulls()options respectively.Closes #161