Disable spill for probe only grouped execution#15592
Merged
rschlussel merged 1 commit intoprestodb:masterfrom Jan 7, 2021
Merged
Disable spill for probe only grouped execution#15592rschlussel merged 1 commit intoprestodb:masterfrom
rschlussel merged 1 commit intoprestodb:masterfrom
Conversation
spill does not work for probe only grouped execution because it expects a constant number of probe operators, but instead each lifespan can be a separate operator. Previously such queries would fail with generic_internal_errors and messages like "5 probe operators finished out of 4 declared"
50ab366 to
ed5ca47
Compare
wenleix
approved these changes
Jan 7, 2021
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is to unblock safely rolling out spill for join. Spill for joins where only the probe side uses grouped execution was broken because some code relied on having a fixed number of probe operators for each build operator. These queries would fail with errors like "5 probe operators finished out of 4 declared"
A proper fix would be to remove the dependency on a fixed number of probe operators, and instead do something similar to how JoinLifecycle works, where you keep track of how many probe operator references you have, and only destroy the build table if all probe operators have finished and the LookupJoinOperatorFactory is closed. But that fix requires more rigorous testing to ensure we're able to keep unspilling and respilling the partitions of the hash table.
Test plan - Verifier. I had trouble writing a test that would reliably finish, but I ran this with verifier on queries I had previously seen failures on, and the errors went away.