Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bugfix] fix rewrite bug after insert new partition data #20157

Merged
merged 4 commits into from
Mar 24, 2023
Merged

[Bugfix] fix rewrite bug after insert new partition data #20157

merged 4 commits into from
Mar 24, 2023

Conversation

ABingHuang
Copy link
Contributor

@ABingHuang ABingHuang commented Mar 23, 2023 •
edited
Loading

What type of PR is this：

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Which issues of this PR fixes ：

Fixes #19817

Problem Summary(Required) ：

Fix rewrite failure after inserting new partition data. The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data. And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id, which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite, which caused an invalid rewritten plan. Fix it by:

  1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
  2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr will affect users' behaviors
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto backported to target branch
    • 3.0
    • 2.5
    • 2.4
    • 2.3

@ABingHuang
fix bucket prune predicate bug
cac5be7 
Signed-off-by: ABingHuang <[email protected]>
@ABingHuang
optimize code
ca0e2cd 
Signed-off-by: ABingHuang <[email protected]>
@sonarcloud
Copy link

sonarcloud bot commented Mar 24, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 3 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

@ABingHuang ABingHuang enabled auto-merge (squash) March 24, 2023 11:24
@ABingHuang ABingHuang merged commit ce97629 into StarRocks:main Mar 24, 2023
@murphyatwork
Copy link
Contributor

@Mergifyio backport branch-2.5 branch-3.0

@mergify
Copy link
Contributor

mergify bot commented Mar 24, 2023 •
edited
Loading

backport branch-2.5 branch-3.0

✅ Backports have been created

@mergify mergify bot mentioned this pull request Mar 24, 2023
Closed
mergify bot pushed a commit that referenced this pull request Mar 24, 2023
@ABingHuang @mergify
[Bugfix] fix rewrite bug after insert new partition data (#20157)
c76db5b 
Fix rewrite failure after inserting new partition data.
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
(cherry picked from commit ce97629)
mergify bot pushed a commit that referenced this pull request Mar 24, 2023
@ABingHuang @mergify
[Bugfix] fix rewrite bug after insert new partition data (#20157)
b7b9a89 
Fix rewrite failure after inserting new partition data.
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
(cherry picked from commit ce97629)

# Conflicts:
#	fe/fe-core/src/test/java/com/starrocks/planner/MaterializedViewWithPartitionTest.java
ABingHuang added a commit to ABingHuang/starrocks that referenced this pull request Mar 27, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (StarRocks#2…
041d226 
…0157)

Fix rewrite failure after inserting new partition data.
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
ABingHuang added a commit to ABingHuang/starrocks that referenced this pull request Mar 27, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (StarRocks#2…
ad80924 
…0157)

Fix rewrite failure after inserting new partition data. 
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
ABingHuang added a commit that referenced this pull request Mar 27, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (#20157) (#2…
9f38432 
…0319)

Fix rewrite failure after inserting new partition data.
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. 
and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
ABingHuang added a commit to ABingHuang/starrocks that referenced this pull request Mar 27, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (StarRocks#2…
050b0f3 
…0157)

Fix rewrite failure after inserting new partition data.
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
ABingHuang added a commit that referenced this pull request Mar 27, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (#20157) (#2…
c812bc0 
…0323)

Fix rewrite failure after inserting new partition data. 
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1.
 and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
numbernumberone pushed a commit to numbernumberone/starrocks that referenced this pull request May 31, 2023
@ABingHuang
[Bugfix] fix rewrite bug after insert new partition data (StarRocks#2…
fabf1e2 
…0157)

Fix rewrite failure after inserting new partition data. 
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
abc982627271 pushed a commit to abc982627271/starrocks that referenced this pull request Jun 5, 2023
@ABingHuang @abc982627271
[Bugfix] fix rewrite bug after insert new partition data (StarRocks#2…
47b2d2f 
…0157)

Fix rewrite failure after inserting new partition data. 
The bug reason is that the mv plan is cached, but the plan may change after ingestion with new partitions data.
And the logic of partition predicate calculation for mv rewrite depends on scan node's selected partition id,
which may change after ingestion. So it leads to invalid compensation predicate in mv rewrite,
which caused an invalid rewritten plan. Fix it by:
1. disable partition prune rules during compile mv plan, which will keep partition predicates in mv plan, to avoid the problem of caching plan
2. remove mv partition predicate compensation logic because it is not necessary after step 1. and it also fixed the problem of redundant partition predicates after partition prune of mv.

Signed-off-by: ABingHuang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LiShuMing LiShuMing LiShuMing left review comments

@Dshadowzh Dshadowzh Dshadowzh approved these changes

@murphyatwork murphyatwork murphyatwork approved these changes

Assignees

@ABingHuang ABingHuang

Labels
title needs [type]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[materialized view] rewrite failed after insert data
4 participants
@ABingHuang @murphyatwork @Dshadowzh @LiShuMing