Implement Window Function Spilling#14909
Conversation
Cherry-pick of trinodb/trino@17e3e09 Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
This avoids calling `Page.getRegion` on the input page and separates input management from index building. Cherry-pick of trinodb/trino@b164253 Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
Cherry-pick of trinodb/trino@7200a16 Co-authored-by: Atri Sharma <atri@linux.com>
Cherry-pick of trinodb/trino@d398d1c Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
Cherry-pick of trinodb/trino@2b821e1 Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
Cherry-pick of trinodb/trino@d76a830 Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
Cherry-pick of trinodb/trino@fcc18ff Co-authored-by: Karol Sobczak <karol.sobczak@karolsobczak.com>
|
@sachdevs : is it possible also to mention the PR in PrestoSQL that you backport from? |
|
Updated - it's the same as the ORDER BY spilling PR on prestoSQL's end. |
highker
left a comment
highker
left a comment
There was a problem hiding this comment.
"Port window operator to work processors" LGTM
highker
left a comment
highker
left a comment
There was a problem hiding this comment.
"Separate input PageSource work processor" and "Introduce revocable aggregated memory context in OperatorContext" LGTM
| } | ||
|
|
||
| // caller shouldn't close this context as it's managed by the OperatorContext | ||
| public AggregatedMemoryContext aggregateRevocableMemoryContext() |
There was a problem hiding this comment.
This looks quite general, surprised not added before when adding aggregation spilling
wenleix
left a comment
wenleix
left a comment
There was a problem hiding this comment.
First 3 commits LGTM:
- Port window operator to work processors
- Separate input PageSource work processor …
- Introduce revocable aggregated memory context in OperatorContext
wenleix
left a comment
wenleix
left a comment
There was a problem hiding this comment.
"Implement spill to disk For WindowOperator". Generally looks good. Note SpillablePagesToPagesIndexes implements some kind of external sorting by index..
| Optional<Page> currentSpillGroupRowPage; | ||
| Optional<Spiller> spiller; | ||
| ListenableFuture<?> spillInProgress = immediateFuture(null); | ||
| // Spill can be trigger by Driver, by us or both. `spillInProgress` is not empty when spill was triggered but not `finishMemoryRevoke()` yet |
There was a problem hiding this comment.
I assume "us" here means this window operator itself, especially SpillablePagesToPagesIndexes#process
wenleix
left a comment
wenleix
left a comment
There was a problem hiding this comment.
"Use OrderingCompiler in WindowOperator spilling" . LGTM.
3 participants
Original PR: trinodb/trino#228
Broken up into ORDER BY spilling (merged in #14836) and Window function spilling (this one)
Part of #14935
TODO