[CARBONDATA-4306] Fix Query Performance issue for Spark 3.1 #4229
Conversation
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4224/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5969/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
a1e4528 to
dd685c8
Compare
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/370/ |
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4225/ |
|
retest this please |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
dd685c8 to
68bb850
Compare
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/373/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
68bb850 to
7514c8a
Compare
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4229/ |
|
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/374/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5975/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
7514c8a to
3b2d2e7
Compare
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4240/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5986/ |
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/385/ |
|
retest this please |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4242/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5988/ |
|
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/387/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
3b2d2e7 to
8e3f955
Compare
|
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4321/ |
|
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6064/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
8e3f955 to
f65ccdb
Compare
Indhumathi27
changed the title
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/454/ |
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/458/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4325/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6068/ |
|
retest this please |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
f65ccdb to
3f60c81
Compare
|
Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/461/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6071/ |
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4328/ |
Indhumathi27
force-pushed
the
tpcds_perf
branch
from
3f60c81 to
97d686b
Compare
|
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4330/ |
|
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6073/ |
|
Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/463/ |
|
LGTM |
Why is this PR needed?
Currently, with Spark 3.1, some rules are applied many times resulting in performance degrade.
What changes were proposed in this PR?
Changed Rules apply strategy from Fixed to Once and CarbonOptimizer can directly extend SparkOptimizer avoiding applying same rules many times
Does this PR introduce any user interface change?
Is any new testcase added?