Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test][Fix](parquet-reader) Add parquet decoder unit tests and fix bugs by these tests. #49182

Merged
merged 1 commit into from
Mar 24, 2025
Merged

[Test][Fix](parquet-reader) Add parquet decoder unit tests and fix bugs by these tests. #49182

merged 1 commit into from
Mar 24, 2025

Conversation

kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Mar 18, 2025 •
edited
Loading

What problem does this PR solve?

Release note

Added Parquet decoder unit tests and fixed bugs identified through these tests.

In addition to the unit tests, the main changes include:

  • Modification of Decoder::set_data() Return Type from void to Status to capture error in the unit tests.
    Changed the return type of Decoder::set_data() from void to Status to enable checking whether the operation succeeded in unit tests.

  • Separation of Header and Implementation Files
    Moved the definitions of non-template functions from header files to corresponding .cpp files (e.g., delta_bit_pack_decoder.h, fix_length_plain_decoder.h).
    Resolved link errors caused by duplicate symbols during the build process of unit tests.
    This change optimizes the code structure, improves compilation efficiency, and avoids potential link conflicts.

  • Implements tparquet::Encoding::BIT_PACKED branch in level_decoder.cpp
    Implements tparquet::Encoding::BIT_PACKED branch in level_decoder.cpp.

  • Bug Fixes

    • Fixed the issue where BoolRLEDecoder::skip_values() was not functioning correctly.
    • Fixed the issues where FixLengthDictDecoder::read_dict_values_to_column() and FixLengthDictDecoder::convert_dict_column_to_string_column() were not functioning correctly.

Future Work:

Add more test cases for invalid scenarios in these unit tests.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from 07893b1 to 24b730d Compare March 18, 2025 05:51
@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from 24b730d to 292172b Compare March 18, 2025 06:45
@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from 292172b to 0d34f41 Compare March 18, 2025 07:56
@kaka11chen
Copy link
Contributor Author

run buildall

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from 0d34f41 to d64b5ce Compare March 18, 2025 08:17
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34235 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d64b5ce180fdadb1ebd98617c242b7a7ea2e2391, data reload: false

------ Round 1 ----------------------------------
q1	23996	5351	5324	5324
q2	2044	326	194	194
q3	10343	1401	730	730
q4	10222	1127	598	598
q5	7600	2684	2763	2684
q6	208	180	138	138
q7	1021	811	635	635
q8	9330	1526	1219	1219
q9	5170	4846	4819	4819
q10	6968	2382	1942	1942
q11	517	281	263	263
q12	363	404	232	232
q13	18004	4041	3318	3318
q14	247	256	235	235
q15	573	494	498	494
q16	645	642	597	597
q17	592	925	372	372
q18	7029	6646	6675	6646
q19	2059	1105	588	588
q20	323	353	204	204
q21	3075	2536	2048	2048
q22	1106	1088	955	955
Total cold run time: 111435 ms
Total hot run time: 34235 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5684	5530	5550	5530
q2	253	357	236	236
q3	2265	2812	2337	2337
q4	1551	2016	1499	1499
q5	4555	4531	4470	4470
q6	263	190	130	130
q7	2166	2027	1844	1844
q8	2965	2926	2837	2837
q9	7301	7274	6905	6905
q10	3246	3384	2907	2907
q11	642	516	510	510
q12	715	834	621	621
q13	3666	4065	3370	3370
q14	285	300	280	280
q15	557	497	496	496
q16	669	705	641	641
q17	1202	1766	1454	1454
q18	7942	7735	7554	7554
q19	881	886	988	886
q20	2029	2180	1835	1835
q21	5542	5137	5068	5068
q22	1163	1088	1012	1012
Total cold run time: 55542 ms
Total hot run time: 52422 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d64b5ce180fdadb1ebd98617c242b7a7ea2e2391, data reload: false

query1	1395	1057	1048	1048
query2	6184	1966	1921	1921
query3	11180	4683	4756	4683
query4	26257	24015	23573	23573
query5	5418	681	471	471
query6	307	206	200	200
query7	3990	508	309	309
query8	302	263	239	239
query9	8512	2586	2586	2586
query10	502	304	281	281
query11	15371	15160	15090	15090
query12	155	107	100	100
query13	1552	507	408	408
query14	9398	6536	6733	6536
query15	212	188	165	165
query16	7603	666	427	427
query17	1320	705	557	557
query18	1990	436	323	323
query19	185	184	163	163
query20	130	126	124	124
query21	209	125	103	103
query22	4495	4575	4387	4387
query23	34619	33727	33559	33559
query24	7528	2447	2427	2427
query25	505	471	383	383
query26	833	276	164	164
query27	2087	509	338	338
query28	4023	2476	2396	2396
query29	606	567	433	433
query30	274	227	192	192
query31	881	914	801	801
query32	73	66	66	66
query33	537	388	303	303
query34	775	870	529	529
query35	860	881	777	777
query36	982	1016	888	888
query37	129	101	84	84
query38	4366	4387	4093	4093
query39	1494	1439	1432	1432
query40	221	128	106	106
query41	61	54	55	54
query42	121	110	108	108
query43	520	521	510	510
query44	1267	817	811	811
query45	180	174	171	171
query46	941	1029	650	650
query47	1827	1855	1834	1834
query48	394	415	307	307
query49	727	534	420	420
query50	696	733	411	411
query51	4326	4346	4269	4269
query52	126	105	99	99
query53	240	261	200	200
query54	483	490	431	431
query55	81	86	82	82
query56	275	280	264	264
query57	1173	1170	1122	1122
query58	253	248	241	241
query59	2740	2940	2680	2680
query60	282	282	258	258
query61	125	121	116	116
query62	770	733	700	700
query63	224	193	184	184
query64	3230	1056	693	693
query65	4554	4442	4512	4442
query66	774	410	337	337
query67	16479	15694	15443	15443
query68	9686	878	503	503
query69	523	290	257	257
query70	1218	1135	1131	1131
query71	500	288	264	264
query72	5137	3625	3778	3625
query73	796	746	353	353
query74	9101	9275	8824	8824
query75	4440	3123	2672	2672
query76	5230	1196	747	747
query77	1011	358	280	280
query78	9865	10144	9338	9338
query79	5031	817	576	576
query80	604	515	444	444
query81	463	263	218	218
query82	248	127	93	93
query83	170	168	151	151
query84	283	95	73	73
query85	754	356	311	311
query86	329	328	287	287
query87	4465	4504	4437	4437
query88	2834	2239	2317	2239
query89	430	312	278	278
query90	2051	212	214	212
query91	138	140	109	109
query92	73	59	56	56
query93	3482	1029	571	571
query94	693	416	303	303
query95	339	265	252	252
query96	481	561	270	270
query97	3307	3413	3318	3318
query98	221	198	199	198
query99	1366	1420	1293	1293
Total cold run time: 285598 ms
Total hot run time: 193185 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d64b5ce180fdadb1ebd98617c242b7a7ea2e2391, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.11	0.11
query3	0.26	0.19	0.20
query4	1.60	0.20	0.18
query5	0.60	0.58	0.59
query6	1.19	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.53
query10	0.58	0.58	0.57
query11	0.16	0.11	0.12
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.80	2.80	2.70
query15	0.92	0.86	0.84
query16	0.37	0.40	0.39
query17	0.99	1.03	1.06
query18	0.21	0.19	0.19
query19	1.91	1.92	1.90
query20	0.02	0.01	0.02
query21	15.35	0.92	0.55
query22	0.78	1.00	0.63
query23	15.15	1.37	0.60
query24	6.98	1.73	1.31
query25	0.53	0.24	0.18
query26	0.62	0.15	0.14
query27	0.05	0.04	0.06
query28	10.37	0.86	0.43
query29	12.55	3.89	3.26
query30	0.25	0.08	0.06
query31	2.83	0.60	0.38
query32	3.23	0.54	0.46
query33	2.96	3.06	3.02
query34	15.78	5.18	4.53
query35	4.57	4.59	4.55
query36	0.65	0.51	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.3 s
Total hot run time: 31.76 s

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from d64b5ce to ab6b680 Compare March 18, 2025 13:29
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32271 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ab6b6802607e565d122c493dc83efc45c1ad7265, data reload: false

------ Round 1 ----------------------------------
q1	24211	5067	5007	5007
q2	2045	312	194	194
q3	10357	1213	680	680
q4	10221	1058	557	557
q5	7496	2379	2330	2330
q6	188	159	131	131
q7	913	724	616	616
q8	9315	1348	1042	1042
q9	5002	4650	4796	4650
q10	6852	2321	1911	1911
q11	477	278	257	257
q12	350	352	217	217
q13	17782	3679	3114	3114
q14	229	230	206	206
q15	539	485	482	482
q16	618	605	584	584
q17	584	869	331	331
q18	6785	6460	6275	6275
q19	1229	955	559	559
q20	320	330	206	206
q21	2940	2203	1944	1944
q22	997	1020	978	978
Total cold run time: 109450 ms
Total hot run time: 32271 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5461	5147	5136	5136
q2	239	320	229	229
q3	2119	2687	2321	2321
q4	1443	1840	1381	1381
q5	4271	4135	4373	4135
q6	213	174	131	131
q7	2014	1870	1748	1748
q8	2637	2668	2590	2590
q9	7169	7036	7183	7036
q10	2999	3190	2696	2696
q11	585	487	490	487
q12	691	788	612	612
q13	3590	3930	3301	3301
q14	284	315	276	276
q15	539	489	513	489
q16	651	701	655	655
q17	1150	1606	1377	1377
q18	7611	7646	7594	7594
q19	853	836	896	836
q20	1990	2051	1878	1878
q21	5477	4812	4829	4812
q22	1064	1066	994	994
Total cold run time: 53050 ms
Total hot run time: 50714 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192303 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ab6b6802607e565d122c493dc83efc45c1ad7265, data reload: false

query1	1405	1084	1060	1060
query2	6150	1907	1886	1886
query3	11008	4427	4348	4348
query4	55296	25520	23477	23477
query5	5083	582	480	480
query6	375	208	185	185
query7	5038	504	292	292
query8	348	248	233	233
query9	6470	2622	2636	2622
query10	423	331	261	261
query11	15289	15284	14892	14892
query12	161	112	104	104
query13	1144	520	400	400
query14	10977	6391	6868	6391
query15	196	193	203	193
query16	7039	651	454	454
query17	1073	703	577	577
query18	1506	396	316	316
query19	190	210	166	166
query20	134	127	119	119
query21	211	120	111	111
query22	4526	4404	4499	4404
query23	33872	33324	33286	33286
query24	5646	2513	2453	2453
query25	471	471	417	417
query26	699	273	159	159
query27	1801	504	344	344
query28	2873	2437	2461	2437
query29	560	567	440	440
query30	278	224	187	187
query31	888	906	796	796
query32	78	68	63	63
query33	463	387	303	303
query34	764	842	519	519
query35	819	833	777	777
query36	965	996	911	911
query37	128	99	75	75
query38	4274	4206	4193	4193
query39	1481	1467	1420	1420
query40	220	124	102	102
query41	51	52	49	49
query42	119	107	111	107
query43	508	508	499	499
query44	1313	804	800	800
query45	183	170	166	166
query46	834	1039	667	667
query47	1890	1873	1787	1787
query48	390	423	311	311
query49	706	494	418	418
query50	702	773	437	437
query51	4319	4225	4267	4225
query52	106	112	99	99
query53	257	269	199	199
query54	502	504	427	427
query55	81	80	83	80
query56	281	274	270	270
query57	1162	1164	1098	1098
query58	255	248	246	246
query59	2878	2854	2651	2651
query60	292	289	298	289
query61	125	120	123	120
query62	744	754	701	701
query63	240	194	196	194
query64	1875	1061	726	726
query65	4579	4448	4426	4426
query66	738	417	297	297
query67	15739	15457	15397	15397
query68	5431	827	515	515
query69	518	304	265	265
query70	1236	1120	1151	1120
query71	448	321	273	273
query72	6097	3867	4018	3867
query73	1196	750	357	357
query74	9030	8884	8683	8683
query75	3226	3185	2732	2732
query76	3766	1159	737	737
query77	550	374	276	276
query78	10083	10235	9247	9247
query79	2900	814	596	596
query80	675	531	458	458
query81	492	258	232	232
query82	740	129	96	96
query83	176	162	154	154
query84	290	96	83	83
query85	783	364	308	308
query86	383	296	284	284
query87	4471	4465	4312	4312
query88	3678	2264	2263	2263
query89	426	315	282	282
query90	1777	210	210	210
query91	137	146	103	103
query92	71	60	57	57
query93	2651	1069	593	593
query94	692	395	305	305
query95	344	270	257	257
query96	488	569	275	275
query97	3301	3365	3323	3323
query98	247	219	189	189
query99	1349	1388	1312	1312
Total cold run time: 298843 ms
Total hot run time: 192303 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ab6b6802607e565d122c493dc83efc45c1ad7265, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.19	0.11
query5	0.56	0.56	0.55
query6	1.22	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.52	0.52
query10	0.61	0.59	0.59
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.79	2.78	2.80
query15	0.92	0.84	0.84
query16	0.38	0.39	0.39
query17	1.04	1.04	1.02
query18	0.21	0.19	0.20
query19	2.06	1.82	1.84
query20	0.01	0.00	0.01
query21	15.36	0.93	0.54
query22	0.77	1.26	0.74
query23	14.79	1.39	0.61
query24	7.26	1.16	0.94
query25	0.50	0.31	0.06
query26	0.46	0.16	0.14
query27	0.05	0.05	0.05
query28	9.70	0.86	0.42
query29	12.52	3.98	3.27
query30	0.24	0.09	0.07
query31	2.82	0.58	0.39
query32	3.25	0.55	0.46
query33	2.96	3.04	3.08
query34	15.73	5.10	4.49
query35	4.55	4.50	4.51
query36	0.67	0.50	0.48
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.45 s
Total hot run time: 31.22 s

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from ab6b680 to 19f8160 Compare March 18, 2025 15:50
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32117 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19f8160ebb5228e651c7858a1fb92c545b32a09f, data reload: false

------ Round 1 ----------------------------------
q1	24222	5092	5033	5033
q2	2044	295	202	202
q3	10369	1206	665	665
q4	10226	987	545	545
q5	7556	2299	2347	2299
q6	187	159	132	132
q7	902	760	610	610
q8	9330	1287	1108	1108
q9	5090	4657	4832	4657
q10	6855	2306	1883	1883
q11	486	273	259	259
q12	368	351	220	220
q13	17796	3669	3078	3078
q14	224	226	210	210
q15	530	478	474	474
q16	614	630	590	590
q17	583	869	329	329
q18	6855	6451	6154	6154
q19	2238	1353	564	564
q20	300	319	201	201
q21	2680	2145	1901	1901
q22	1047	1016	1003	1003
Total cold run time: 110502 ms
Total hot run time: 32117 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5322	5206	5175	5175
q2	238	321	231	231
q3	2156	2664	2253	2253
q4	1415	1803	1364	1364
q5	4234	4122	4406	4122
q6	216	174	133	133
q7	1999	1935	1755	1755
q8	2664	2709	2621	2621
q9	7231	7140	7189	7140
q10	3014	3142	2668	2668
q11	564	512	514	512
q12	678	773	632	632
q13	3532	3939	3335	3335
q14	296	290	273	273
q15	536	511	498	498
q16	639	713	658	658
q17	1138	1634	1335	1335
q18	7709	7554	7545	7545
q19	844	855	987	855
q20	1995	2054	1914	1914
q21	5388	4778	4925	4778
q22	1091	1040	1036	1036
Total cold run time: 52899 ms
Total hot run time: 50833 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191648 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19f8160ebb5228e651c7858a1fb92c545b32a09f, data reload: false

query1	1397	1076	1047	1047
query2	6071	1898	1856	1856
query3	11003	4599	4459	4459
query4	54206	24799	23321	23321
query5	5248	536	466	466
query6	405	204	184	184
query7	5173	512	291	291
query8	341	254	239	239
query9	6890	2600	2620	2600
query10	426	308	260	260
query11	15374	15134	14816	14816
query12	164	117	104	104
query13	1198	514	411	411
query14	10165	6365	6946	6365
query15	203	208	193	193
query16	7025	662	463	463
query17	1041	704	548	548
query18	1523	430	322	322
query19	190	180	162	162
query20	127	122	130	122
query21	209	128	102	102
query22	4355	4567	4353	4353
query23	34123	33355	33368	33355
query24	5882	2455	2442	2442
query25	457	470	422	422
query26	748	285	157	157
query27	1845	490	331	331
query28	3077	2495	2457	2457
query29	605	580	438	438
query30	266	223	190	190
query31	869	882	790	790
query32	72	64	63	63
query33	470	371	317	317
query34	759	861	508	508
query35	817	836	749	749
query36	944	990	866	866
query37	122	102	81	81
query38	4317	4239	4163	4163
query39	1520	1425	1401	1401
query40	213	118	105	105
query41	63	60	49	49
query42	116	109	110	109
query43	488	513	481	481
query44	1358	824	810	810
query45	176	173	163	163
query46	848	1047	670	670
query47	1818	1863	1793	1793
query48	393	423	319	319
query49	703	531	424	424
query50	722	752	420	420
query51	4273	4360	4295	4295
query52	104	111	98	98
query53	239	259	204	204
query54	509	523	459	459
query55	93	87	87	87
query56	302	287	270	270
query57	1189	1202	1126	1126
query58	267	255	290	255
query59	2691	2729	2699	2699
query60	304	279	254	254
query61	123	119	117	117
query62	742	736	694	694
query63	234	197	195	195
query64	1994	1059	686	686
query65	4508	4443	4410	4410
query66	786	398	294	294
query67	15739	15533	15191	15191
query68	6985	886	503	503
query69	537	304	271	271
query70	1154	1086	1034	1034
query71	510	288	270	270
query72	5997	3601	3826	3601
query73	1225	745	349	349
query74	9395	9201	8948	8948
query75	3691	3161	2703	2703
query76	4287	1206	757	757
query77	614	446	275	275
query78	10161	10177	9271	9271
query79	2387	945	607	607
query80	590	516	454	454
query81	469	256	226	226
query82	559	126	91	91
query83	174	166	158	158
query84	283	97	69	69
query85	782	359	308	308
query86	409	309	276	276
query87	4479	4540	4247	4247
query88	3554	2280	2300	2280
query89	407	316	281	281
query90	1851	221	221	221
query91	143	144	112	112
query92	77	59	58	58
query93	1837	1067	577	577
query94	665	389	304	304
query95	353	274	263	263
query96	491	572	280	280
query97	3371	3406	3285	3285
query98	233	202	206	202
query99	1400	1385	1276	1276
Total cold run time: 299692 ms
Total hot run time: 191648 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19f8160ebb5228e651c7858a1fb92c545b32a09f, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.10	0.10
query3	0.24	0.18	0.19
query4	1.59	0.19	0.18
query5	0.60	0.58	0.59
query6	1.19	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.53	0.53
query10	0.57	0.60	0.57
query11	0.16	0.11	0.10
query12	0.16	0.11	0.12
query13	0.62	0.60	0.60
query14	2.69	2.82	2.68
query15	0.92	0.86	0.84
query16	0.36	0.39	0.37
query17	1.00	1.07	1.02
query18	0.21	0.20	0.20
query19	1.93	1.90	1.89
query20	0.01	0.01	0.01
query21	15.35	0.93	0.54
query22	0.76	1.30	0.72
query23	14.73	1.37	0.62
query24	6.60	1.72	1.01
query25	0.59	0.18	0.14
query26	0.60	0.15	0.14
query27	0.05	0.05	0.04
query28	10.44	0.88	0.42
query29	12.54	4.03	3.30
query30	0.25	0.09	0.06
query31	2.82	0.61	0.38
query32	3.22	0.55	0.46
query33	3.08	2.96	2.99
query34	15.76	5.09	4.47
query35	4.55	4.53	4.48
query36	0.66	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.55 s
Total hot run time: 31.31 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 82.18% (166/202) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 49.10% (13150/26780)
Line Coverage 38.66% (113496/293569)
Region Coverage 37.49% (57792/154155)
Branch Coverage 32.55% (29046/89238)

@kaka11chen kaka11chen force-pushed the add_parquet_decoder_ut branch from 19f8160 to edbbd65 Compare March 19, 2025 08:42
@kaka11chen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31946 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edbbd650498a5ecad48c6c6c72ade29c88731e9e, data reload: false

------ Round 1 ----------------------------------
q1	24032	5019	4998	4998
q2	2045	286	168	168
q3	10407	1218	671	671
q4	10224	996	526	526
q5	7569	2275	2355	2275
q6	180	161	128	128
q7	902	745	605	605
q8	9298	1277	1113	1113
q9	4834	4751	4577	4577
q10	6824	2261	1879	1879
q11	464	269	254	254
q12	346	355	213	213
q13	17765	3633	3057	3057
q14	229	221	212	212
q15	516	479	474	474
q16	608	632	587	587
q17	574	863	346	346
q18	6689	6348	6289	6289
q19	1228	943	524	524
q20	309	327	197	197
q21	2762	2141	1906	1906
q22	1024	1014	947	947
Total cold run time: 108829 ms
Total hot run time: 31946 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5062	5146	5126	5126
q2	237	321	227	227
q3	2103	2667	2313	2313
q4	1413	1782	1380	1380
q5	4225	4118	4208	4118
q6	206	165	130	130
q7	1956	1905	1749	1749
q8	2590	2671	2552	2552
q9	7348	7116	7201	7116
q10	3013	3186	2761	2761
q11	554	502	497	497
q12	653	733	660	660
q13	3562	3856	3222	3222
q14	274	304	285	285
q15	521	483	465	465
q16	641	694	650	650
q17	1146	1588	1348	1348
q18	7714	7705	7548	7548
q19	810	812	909	812
q20	1960	2038	1854	1854
q21	5410	4968	4683	4683
q22	1085	1087	1020	1020
Total cold run time: 52483 ms
Total hot run time: 50516 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192623 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edbbd650498a5ecad48c6c6c72ade29c88731e9e, data reload: false

query1	1374	1037	1034	1034
query2	6282	1994	2012	1994
query3	10997	4539	4462	4462
query4	25700	23787	23445	23445
query5	4841	649	484	484
query6	304	202	186	186
query7	3991	504	292	292
query8	298	243	248	243
query9	8508	2645	2612	2612
query10	461	298	252	252
query11	15798	15115	15020	15020
query12	183	109	104	104
query13	1576	527	404	404
query14	10086	6679	6360	6360
query15	210	190	169	169
query16	7607	662	478	478
query17	1203	744	580	580
query18	2007	409	331	331
query19	191	193	158	158
query20	128	121	115	115
query21	201	122	107	107
query22	4398	4566	4362	4362
query23	34735	33565	33315	33315
query24	7344	2487	2428	2428
query25	498	482	394	394
query26	1006	274	160	160
query27	2184	508	330	330
query28	3977	2470	2410	2410
query29	638	566	440	440
query30	283	220	193	193
query31	916	899	790	790
query32	74	63	60	60
query33	558	353	313	313
query34	804	875	512	512
query35	825	895	784	784
query36	984	1013	890	890
query37	122	102	76	76
query38	4315	4216	4232	4216
query39	1469	1432	1469	1432
query40	206	118	104	104
query41	52	52	50	50
query42	123	104	103	103
query43	517	522	510	510
query44	1320	809	796	796
query45	184	174	172	172
query46	851	1051	661	661
query47	1843	1887	1779	1779
query48	386	408	303	303
query49	728	509	440	440
query50	705	745	419	419
query51	4302	4368	4228	4228
query52	107	101	99	99
query53	235	263	203	203
query54	499	509	417	417
query55	80	83	88	83
query56	276	271	259	259
query57	1190	1178	1125	1125
query58	266	263	238	238
query59	2831	3014	2785	2785
query60	285	278	263	263
query61	128	126	160	126
query62	790	725	657	657
query63	239	198	197	197
query64	3256	1041	675	675
query65	4600	4466	4443	4443
query66	794	396	319	319
query67	16123	15545	15469	15469
query68	8557	873	505	505
query69	495	309	293	293
query70	1189	1104	1107	1104
query71	417	298	257	257
query72	5130	3520	3756	3520
query73	757	750	348	348
query74	9074	8874	8941	8874
query75	3286	3165	2782	2782
query76	3265	1194	751	751
query77	593	364	283	283
query78	9973	10223	9372	9372
query79	1780	863	587	587
query80	731	517	450	450
query81	498	267	222	222
query82	424	124	90	90
query83	173	167	153	153
query84	244	91	75	75
query85	788	351	306	306
query86	321	312	271	271
query87	4353	4603	4467	4467
query88	2832	2250	2222	2222
query89	393	305	282	282
query90	1717	208	208	208
query91	146	138	110	110
query92	65	65	57	57
query93	1287	1044	580	580
query94	609	419	293	293
query95	346	269	258	258
query96	479	550	273	273
query97	3314	3335	3288	3288
query98	223	202	200	200
query99	1354	1396	1284	1284
Total cold run time: 274805 ms
Total hot run time: 192623 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edbbd650498a5ecad48c6c6c72ade29c88731e9e, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.11	0.11
query3	0.23	0.20	0.19
query4	1.60	0.19	0.20
query5	0.59	0.58	0.58
query6	1.19	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.51
query10	0.58	0.59	0.56
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.62	0.60	0.62
query14	2.81	2.71	2.83
query15	0.89	0.84	0.86
query16	0.38	0.38	0.40
query17	0.99	1.02	1.07
query18	0.21	0.19	0.19
query19	1.94	2.01	1.82
query20	0.01	0.02	0.01
query21	15.40	0.90	0.58
query22	0.77	1.19	0.75
query23	14.80	1.37	0.66
query24	7.88	0.79	1.36
query25	0.52	0.29	0.09
query26	0.57	0.16	0.13
query27	0.05	0.05	0.04
query28	8.69	0.86	0.42
query29	12.61	3.93	3.31
query30	0.25	0.08	0.06
query31	2.84	0.60	0.39
query32	3.22	0.54	0.46
query33	3.07	3.00	3.06
query34	15.80	5.10	4.47
query35	4.51	4.57	4.47
query36	0.67	0.51	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.3 s
Total hot run time: 31.11 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 82.18% (166/202) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 49.06% (13141/26784)
Line Coverage 38.63% (113457/293717)
Region Coverage 37.42% (57716/154240)
Branch Coverage 32.52% (29040/89286)

@kaka11chen kaka11chen mentioned this pull request Mar 20, 2025
Closed
16 tasks
morningman
morningman approved these changes Mar 20, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 20, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 6ce45c4 into apache:master Mar 24, 2025
29 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/2.1.x-conflict dev/3.0.x dev/3.0.x-conflict reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@kaka11chen @Thearas @doris-robot @hello-stephen @morningman @CalvinKirs @wuwenchi @hubgeter