[SPARK-31985][SS] Remove incomplete/undocumented stateful aggregation in continuous mode #29077
Conversation
… in continuous mode This removes the undocumented and incomplete feature of "stateful aggregation" in continuous mode. The work for the feature had been stopped for over an year and no one asked/requested for the availability of such feature in community.
HeartSaVioR
changed the title
|
Test build #125734 has finished for PR 29077 at commit
|
jose-torres
left a comment
jose-torres
left a comment
There was a problem hiding this comment.
Mostly looks good, just a question about one change that might have been accidental.
Thanks for taking care of this.
| if (operationCheckEnabled) { | ||
| UnsupportedOperationChecker.checkForContinuous(analyzedPlan, outputMode) | ||
| } | ||
| UnsupportedOperationChecker.checkForContinuous(analyzedPlan, outputMode) |
There was a problem hiding this comment.
was removing the operationCheckEnabled flag intentional?
There was a problem hiding this comment.
That's not intentional. Nice finding. Thanks! I'll revert.
|
Test build #125792 has finished for PR 29077 at commit
|
|
@jose-torres Would you mind going through another round of review? Thanks in advance! |
|
LGTM. I don't have the repo fully set up on my new computer, so I'll try to find time to set it up and merge tomorrow. (Or you can do it if you want to try out your new committer powers; congrats btw!) |
|
Thanks for the reviewing and kind words :) I'll deal with merging. |
|
@HeartSaVioR, no big deal but let's make sure to leave a comment to mention which branch this PR went through. |
|
Ah thanks for noticing. I missed it. It's merged only in master branch. |
4 participants
What changes were proposed in this pull request?
This removes the undocumented and incomplete feature of "stateful aggregation" in continuous mode, which would reduce 1100+ lines of code.
Why are the changes needed?
The work for the feature had been stopped for over an year, and no one asked/requested for the availability of such feature in community. Current state for the feature is that it only works with
coalesce(1)which force the query to read and process, and write in "a" task, which doesn't make sense in production.The remaining code increases the work on DSv2 changes as well - that's why I don't simply propose reverting relevant commits - the code path has been changed due to DSv2 evolution.
Does this PR introduce any user-facing change?
Technically no, because it's never documented and can't be used in production in current shape.
How was this patch tested?
Existing tests.