[WIP][SPARK-30538][SQL] Control spark sql output small file by merge small partition

[AngersZhuuuu]

What changes were proposed in this pull request?

Add a method to control small file problem

Why are the changes needed?

Spark SQL always generate too many small files of insert type SQL result, here we add a control method to minimize the output file number automatic。

Does this PR introduce any user-facing change?

When set spark.sql.files.mergeSmallFile.enabled=false,
we will combine neighbouring small partition, then will reduce output small files

How was this patch tested?

manual tested

create table test_dy_data(id string, dt string);
create table test_dy_part(id string) partitioned by(dt string);

insert into table test_dy_data select 1, '20191201';
insert into table test_dy_data select 2, '20191201';
insert into table test_dy_data select 3, '20191201';
insert into table test_dy_data select 4, '20191201';
insert into table test_dy_data select 5, '20191201';
insert into table test_dy_data select 6, '20191201';
insert into table test_dy_data select 7, '20191201';
insert into table test_dy_data select 8, '20191201';



insert into table test_dy_data select 1, '20191202';
insert into table test_dy_data select 2, '20191202';
insert into table test_dy_data select 3, '20191202';
insert into table test_dy_data select 4, '20191202';
insert into table test_dy_data select 5, '20191202';
insert into table test_dy_data select 6, '20191202';
insert into table test_dy_data select 7, '20191202';
insert into table test_dy_data select 8, '20191202';

insert into table test_dy_data select 1, '20191203';
insert into table test_dy_data select 2, '20191203';
insert into table test_dy_data select 3, '20191203';
insert into table test_dy_data select 4, '20191203';
insert into table test_dy_data select 5, '20191203';
insert into table test_dy_data select 6, '20191203';
insert into table test_dy_data select 7, '20191203';
insert into table test_dy_data select 8, '20191203';


set hive.exec.stagingdir=/Users/yi.zhu/Documents/data/;
set hive.exec.dynamic.partition.mode=nonstrict;

insert overwrite table test_dy_part select * from test_dy_data;

spark.sql.files.mergeSmallFile.enabled=false

spark.sql.files.mergeSmallFile.enabled=true

[wangshuo128]

Maybe refer to the wrong JIRA number by "SPARK-20538", do you mean "SPARK-30538"?

[wangshuo128]

Is it too costly to trigger an action to compute all the RDDs?

[AngersZhuuuu]

Is it too costly to trigger an action to compute all the RDDs?

this is after compute rdds and coalese computed rdd.

[AngersZhuuuu]

Is it too costly to trigger an action to compute all the RDDs?

Sorry for I have make some mistake, it will re-compute last stage. but won't recompute all stage

[AngersZhuuuu]

Maybe refer to the wrong JIRA number by "SPARK-20538", do you mean "SPARK-30538"?

Yea, thanks

[AmplabJenkins]

Can one of the admins verify this patch?

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[SubhamSinghal]

Will someone be able to offer review here?