-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BugFix] fix project prune rule #13278
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@Mergifyio backport branch-2.4 |
✅ Backports have been created
|
kangkaisen
reviewed
Nov 11, 2022
...e/src/main/java/com/starrocks/sql/optimizer/rule/transformation/PruneProjectColumnsRule.java
Show resolved
Hide resolved
run starrocks_admit_test |
kangkaisen
previously approved these changes
Nov 11, 2022
run starrocks_admit_test |
run starrocks_admit_test |
Kudos, SonarCloud Quality Gate passed! |
run starrocks_fe_unittest |
[FE PR Coverage Check]😍 pass : 10 / 10 (100.00%) file detail
|
@Mergifyio backport branch-2.4 branch-2.5 |
✅ Backports have been created
|
✅ Backports have been created
|
kangkaisen
approved these changes
Nov 15, 2022
run pr-title-check |
HangyuanLiu
approved these changes
Nov 16, 2022
run starrocks_admit_test |
1 similar comment
run starrocks_admit_test |
mergify bot
pushed a commit
that referenced
this pull request
Nov 16, 2022
(cherry picked from commit 7e4bfc5)
mergify bot
pushed a commit
that referenced
this pull request
Nov 16, 2022
(cherry picked from commit 7e4bfc5)
wanpengfei-git
pushed a commit
that referenced
this pull request
Nov 16, 2022
(cherry picked from commit 7e4bfc5)
wanpengfei-git
pushed a commit
that referenced
this pull request
Nov 16, 2022
(cherry picked from commit 7e4bfc5)
murphyatwork
added a commit
to murphyatwork/starrocks
that referenced
this pull request
Nov 16, 2022
Squashed commit of the following: commit 5506636 Author: Murphy <[email protected]> Date: Wed Nov 16 14:52:31 2022 +0800 [Feature] infer property for mv (StarRocks#12929) commit e0a5989 Author: eyes_on_me <[email protected]> Date: Wed Nov 16 14:28:01 2022 +0800 [Feature] support big query log (StarRocks#13352) commit 63792e1 Author: leoyy0316 <[email protected]> Date: Wed Nov 16 14:22:39 2022 +0800 [Feature] Support show create external catalog table (StarRocks#13245) commit 7795735 Author: rickif <[email protected]> Date: Wed Nov 16 14:07:10 2022 +0800 [Enhancement] Add default group id for routine load (StarRocks#13160) When the consumer group id is not set by the user, the routine load coordinator BE would generate a random consumer group id for every routine load task. The kafka would retain these consumer groups until `offsets.retention.minutes` later. This PR is to avoid the creation of the consumer group on every task scheduling. It generates a random consumer group id for the routine load job whose group id is not set. All tasks of the job would share the same group id. The doc is updated by StarRocks/docs.zh-cn#2748. commit 7e4bfc5 Author: Murphy <[email protected]> Date: Wed Nov 16 12:47:30 2022 +0800 [BugFix] fix project prune rule (StarRocks#13278) commit f18128b Author: Stephen <[email protected]> Date: Wed Nov 16 12:18:33 2022 +0800 [Tool] automerge label to trigger automerge (StarRocks#13444) automerge label to trigger automerge commit 585566f Author: packy92 <[email protected]> Date: Wed Nov 16 11:07:34 2022 +0800 [Enhancement] colocate join need consider column equivalent conduction (StarRocks#13344) commit 91b82e8 Author: Stephen <[email protected]> Date: Wed Nov 16 11:06:05 2022 +0800 [Tool] new label approved to trigger admit test (StarRocks#13442) fix pr label trigger admit job bug commit 0664c10 Author: Gavin <[email protected]> Date: Wed Nov 16 11:01:11 2022 +0800 [BugFix] Fix the `show_be_version.sh` script execute error due to the lack of cachelib dependency. (StarRocks#13411) commit 2bb88ac Author: yiming <[email protected]> Date: Wed Nov 16 09:57:14 2022 +0800 [Enhancement] optimize db write lock in report handler (StarRocks#13423) Because we need to write bdb with db write lock hold, to avoid block other threads too long, we periodically release and acquire the db write lock (every MAX_DB_WLOCK_HOLDING_TIME_MS milliseconds). commit 3769636 Author: yiming <[email protected]> Date: Wed Nov 16 09:56:52 2022 +0800 [BugFix] Disable tablet migration between local disks for pk table (StarRocks#13422) Currently, primary key table doesn't support tablet migration between local disks. commit 76087b9 Author: evelyn.zhaojie <[email protected]> Date: Wed Nov 16 09:53:30 2022 +0800 add 2.2.9 (StarRocks#13429) commit 7a33ee5 Author: meegoo <[email protected]> Date: Wed Nov 16 09:48:11 2022 +0800 [Refactor] Refactor event base compaction framework (StarRocks#13043) commit 08d1f60 Author: gengjun-git <[email protected]> Date: Wed Nov 16 03:38:19 2022 +0800 [BugFix] Fix ReportHandler log Error (StarRocks#13428) commit 1dedad3 Author: Stephen <[email protected]> Date: Wed Nov 16 00:53:50 2022 +0800 Update pr-title-check.yml (StarRocks#13424) commit f845676 Author: dorianzheng <[email protected]> Date: Tue Nov 15 23:22:38 2022 +0800 [BugFix] Fix duplicate scan timer (StarRocks#13314) commit 8f979a1 Author: Smith Cruise <[email protected]> Date: Tue Nov 15 21:31:22 2022 +0800 [Enhancement][Support Struct] Support Struct subfield materialize (StarRocks#13401) This pr avoids reading redundant data in BE by determining the struct subfield that needs to be materialized in FE. commit 75ed6c1 Author: dorianzheng <[email protected]> Date: Tue Nov 15 20:26:23 2022 +0800 [BugFix] `SlotDescriptor` might be nullptr (StarRocks#13315) commit 6a4a0a8 Author: zihe.liu <[email protected]> Date: Tue Nov 15 19:50:17 2022 +0800 [BugFix] Use MockUp to mock SystemInfoService (StarRocks#13388) commit ac59e92 Author: meegoo <[email protected]> Date: Tue Nov 15 19:43:32 2022 +0800 [Refactor] update the log level of compaction tablet selection (StarRocks#13350) commit 069c66f Author: gengjun-git <[email protected]> Date: Tue Nov 15 19:23:06 2022 +0800 [BugFix] Fix the bug of slow execution of show frontends StarRocks#13340 When executing `show frontends` command, FE will get node info from bdb using ReplicationGroupAdmin. But FE only add/remove the fe host to/from ReplicationGroupAdmin in leader node. This will leave some unreachable host in non-leader node. It will cause the slow problem. To fix this bug, add/remove fe node from ReplicationGroupAdmin client in the non-leader FE. commit 7e753b6 Author: Youngwb <[email protected]> Date: Tue Nov 15 19:07:41 2022 +0800 [BugFix] Fix hive column statistics Min/Max with wrong default value (StarRocks#13371) commit 22f3619 Author: Stephen <[email protected]> Date: Tue Nov 15 19:04:18 2022 +0800 [Tool] actions info update (StarRocks#13405) Standardize action job name commit a5d7371 Author: packy92 <[email protected]> Date: Tue Nov 15 18:24:11 2022 +0800 [BugFix] fix CompoundPredicateOperator equals method (StarRocks#13271) commit 0a7fbee Author: waittting <[email protected]> Date: Tue Nov 15 16:48:52 2022 +0800 [Feature] Support cluster manage statements in new RBAC privilege framework (StarRocks#13122) PrivilegeCheckerV2 support cluster manage statements 1. `show backends`, `show frontends`, `show broker` statements. 2. `show tablet`, `show transaction`. Support upgrading the privilege of system. commit db1b8ff Author: Stephen <[email protected]> Date: Tue Nov 15 16:36:27 2022 +0800 [Tool] actions info update for readability (StarRocks#13398) actions info update for readability commit f5513e9 Author: Kevin Li <[email protected]> Date: Tue Nov 15 15:32:39 2022 +0800 [BugFix] fix JoinEliminateNulls ut in ReplyFromDumpTest (StarRocks#13394) commit a30d34f Author: Youngwb <[email protected]> Date: Tue Nov 15 15:30:20 2022 +0800 [BugFix] Adjust session variable cbo_cte_reuse_rate to enable tpcds-47 cte reuse (StarRocks#13247) commit 003c97c Author: hongli.chang <[email protected]> Date: Tue Nov 15 15:25:02 2022 +0800 [Enhancement] prune empty Range predicate (StarRocks#13323) commit b55095d Author: Stephen <[email protected]> Date: Tue Nov 15 14:55:03 2022 +0800 [Tool] actions info update (StarRocks#13361) Standardize action naming and support bug fix pr's merge into version hint information commit bf3534c Author: Sihui <[email protected]> Date: Tue Nov 15 14:36:16 2022 +0800 Update hive_catalog.md (StarRocks#13389) commit 8077f66 Author: Binglin Chang <[email protected]> Date: Tue Nov 15 13:51:35 2022 +0800 [Enhancement] load delvec when tablet scanner is openned (StarRocks#13293) This PR changes segment_iterator, get delvec when segment_iterator is created, not in _init, so delvecs are got when the query started, preventing delvec not found error when query runs for a very long time(>30min) and delvec got GCed. commit 880c49e Author: zombee0 <[email protected]> Date: Tue Nov 15 10:35:11 2022 +0800 [Feature] Support external file table (StarRocks#13064) CREATE EXTERNAL TABLE test_0(c1int NULL, c2 array<array> NULL ) ENGINE=FILE PROPERTIES ( "path"="hdfs://172.17.0.3/user/hive/warehouse/array2d_parq", "format"="parquet" ); the path can be a dir or a file, if it is a path, it must ends with '/', and we just scan the current level not its subdirectory. the table name and type must be same with the file in the directory or the file itself. support orc and parquet. then can query the hive_0: select * from hive_0 commit 17b28eb Author: Kevin Li <[email protected]> Date: Tue Nov 15 10:33:14 2022 +0800 [BugFix] Fix type bugs in Join's Eliminate nulls (StarRocks#13224) commit e8af424 Author: Xuefen Liu <[email protected]> Date: Tue Nov 15 10:26:09 2022 +0800 [BugFix] update storage_cache_ttl default 30 days (StarRocks#13228) commit 73d0575 Author: meegoo <[email protected]> Date: Tue Nov 15 10:24:03 2022 +0800 [Enhancement] Add replicated storage blacklist mechanism (StarRocks#12281) * [Enhancement] Add replicated storage blacklist mechanism commit 95cf52e Author: zombee0 <[email protected]> Date: Tue Nov 15 01:27:45 2022 +0800 [Enhancement] when use desc statement, return the keyType and valueType of MapType (StarRocks#13343) commit cddfffe Author: Sihui <[email protected]> Date: Tue Nov 15 01:26:14 2022 +0800 [Doc]Update iceberg_catalog.md (StarRocks#13363) commit 45b1d33 Author: Youngwb <[email protected]> Date: Mon Nov 14 22:07:05 2022 +0800 [Enhancement] Normalizes the naming of the glue property (StarRocks#13299) commit e9c8ddb Author: 絵空事スピリット <[email protected]> Date: Mon Nov 14 21:48:38 2022 +0800 [Doc] Add SHOW PROC (StarRocks#13345) commit 2f9711e Author: zhoukang <[email protected]> Date: Mon Nov 14 21:35:03 2022 +0800 [BugFix] Fix fe meta error when use merge condition (StarRocks#13337) commit 7ab3e6f Author: meegoo <[email protected]> Date: Mon Nov 14 20:54:34 2022 +0800 [BugFix] Fix broker load crash since merge code problem (StarRocks#13362) commit 401ced3 Author: gengjun-git <[email protected]> Date: Mon Nov 14 19:03:46 2022 +0800 [BugFix] Fix broker kerberos ticket expired bug (StarRocks#12944) Set hadoop.kerberos.keytab.login.autorenewal.enabled to true to enable auth refresh ticket. commit 6680e90 Author: packy92 <[email protected]> Date: Mon Nov 14 18:46:35 2022 +0800 [UT] ignore unstable case (StarRocks#13318) ignore unstable case commit 106d477 Author: stdpain <[email protected]> Date: Mon Nov 14 16:09:00 2022 +0800 [BugFix] Fix BE crash in streaming distinct aggregate (StarRocks#13269) commit 8a81ee5 Author: stdpain <[email protected]> Date: Mon Nov 14 16:08:39 2022 +0800 [BugFix] 'Not Found string in global dict: xxx' error in UNIQUE KEY (StarRocks#13263) commit 4fb8ef9 Author: padmejin <[email protected]> Date: Mon Nov 14 15:22:16 2022 +0800 [Feature] upgrade alter/drop/create/select on table/db/view (StarRocks#13274) upgrade alter/drop/create/select on table/db/view commit bc3e6d2 Author: Stephen <[email protected]> Date: Mon Nov 14 14:50:40 2022 +0800 [Tool] java code style check (StarRocks#13334) add java code style check commit 60c99b2 Author: stdpain <[email protected]> Date: Mon Nov 14 14:07:58 2022 +0800 [Refactor] remove some unused code in FunctionContext (StarRocks#13066)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this:
Which issues of this PR fixes :
Fixes #13205
Fixes #13260
Problem Summary(Required) :
Why the SQL in #13205 make BE crash in
NestLoopJoinOperator
?PruneProjectColumnsRule
would prune all output columns of operators under the Aggregate NodeLEFT ANTI JOIN
NestLoopJoinOperator
would failed todown_cast
column typeCrossJoinNode
does not have this issue? Cause it initializes its output chunk according to input chunk, regarding of the type info in plan. But it's not suitable for NestLoopJoin`, it must initialize the output chunk according to type info in planThe wrong plan is:
How to fix it ?
PRUNE_PROJECT
, so copy it toPRUNE_PROJECT_COLUMNS
Checklist: