Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](hudi) replace non thread safe SimpleDateFormat #48923

Merged
merged 4 commits into from
Mar 13, 2025
Merged

[fix](hudi) replace non thread safe SimpleDateFormat #48923

merged 4 commits into from
Mar 13, 2025

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Mar 11, 2025 •
edited
Loading

What problem does this PR solve?

SimpleDateFormat is not thread safe, replace it with DateTimeFormatter

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32381 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76cafbf6f0fd3e62bd1e85bc3297926cd980d2e8, data reload: false

------ Round 1 ----------------------------------
q1	17599	5197	5055	5055
q2	2057	295	167	167
q3	10398	1257	693	693
q4	10212	1015	528	528
q5	7514	2413	2284	2284
q6	186	168	138	138
q7	917	735	617	617
q8	9293	1242	1050	1050
q9	4865	4712	4878	4712
q10	6822	2292	1887	1887
q11	462	274	248	248
q12	346	347	211	211
q13	17771	3677	3079	3079
q14	233	226	225	225
q15	529	483	481	481
q16	641	604	572	572
q17	576	852	338	338
q18	7133	6466	6373	6373
q19	1390	952	552	552
q20	317	327	198	198
q21	2701	2108	1996	1996
q22	1095	1023	977	977
Total cold run time: 103057 ms
Total hot run time: 32381 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5153	5106	5117	5106
q2	236	334	240	240
q3	2182	2622	2249	2249
q4	1433	1820	1402	1402
q5	4204	4088	4150	4088
q6	203	161	124	124
q7	1861	1846	1819	1819
q8	2560	2591	2562	2562
q9	7273	7110	7178	7110
q10	2952	3173	2756	2756
q11	583	506	479	479
q12	665	771	614	614
q13	3568	3869	3227	3227
q14	281	315	262	262
q15	513	471	491	471
q16	650	681	648	648
q17	1141	1612	1324	1324
q18	7717	7597	7479	7479
q19	807	802	918	802
q20	2012	1993	1821	1821
q21	5426	4884	4815	4815
q22	1070	1045	1010	1010
Total cold run time: 52490 ms
Total hot run time: 50408 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76cafbf6f0fd3e62bd1e85bc3297926cd980d2e8, data reload: false

query1	999	398	384	384
query2	6517	1887	1877	1877
query3	6804	211	220	211
query4	26281	23249	23325	23249
query5	4348	677	495	495
query6	309	199	184	184
query7	4606	496	297	297
query8	298	247	244	244
query9	8627	2638	2640	2638
query10	464	350	254	254
query11	15834	15080	14978	14978
query12	160	112	104	104
query13	1656	529	423	423
query14	10327	6802	6947	6802
query15	205	203	171	171
query16	7669	619	474	474
query17	1449	722	570	570
query18	1997	411	317	317
query19	200	192	160	160
query20	122	113	112	112
query21	207	123	105	105
query22	4381	4553	4366	4366
query23	33933	32939	32868	32868
query24	7707	2408	2376	2376
query25	541	448	380	380
query26	1236	265	156	156
query27	2092	483	330	330
query28	3886	2431	2385	2385
query29	720	554	421	421
query30	286	239	178	178
query31	945	852	748	748
query32	71	63	63	63
query33	550	356	326	326
query34	773	866	501	501
query35	793	841	732	732
query36	946	995	908	908
query37	112	96	76	76
query38	4101	4306	4017	4017
query39	1432	1388	1424	1388
query40	209	115	105	105
query41	54	52	50	50
query42	119	103	105	103
query43	490	510	492	492
query44	1276	801	781	781
query45	179	192	160	160
query46	817	1010	620	620
query47	1722	1803	1704	1704
query48	381	419	317	317
query49	793	485	411	411
query50	693	722	394	394
query51	4180	4197	4124	4124
query52	105	112	99	99
query53	242	271	190	190
query54	491	508	417	417
query55	81	84	80	80
query56	264	256	261	256
query57	1148	1129	1091	1091
query58	250	237	233	233
query59	2498	2822	2736	2736
query60	290	280	260	260
query61	125	120	121	120
query62	811	723	676	676
query63	233	190	193	190
query64	4322	1008	667	667
query65	4433	4351	4324	4324
query66	1050	405	301	301
query67	15721	15665	15499	15499
query68	9233	889	500	500
query69	462	300	259	259
query70	1152	1150	1099	1099
query71	442	372	273	273
query72	5146	3492	3718	3492
query73	782	728	351	351
query74	9116	9076	8657	8657
query75	4125	3164	2681	2681
query76	3605	1178	723	723
query77	820	374	287	287
query78	10036	10122	9336	9336
query79	2307	828	584	584
query80	718	553	435	435
query81	473	258	220	220
query82	409	128	100	100
query83	215	168	155	155
query84	278	93	75	75
query85	741	362	309	309
query86	332	325	283	283
query87	4412	4658	4470	4470
query88	2924	2273	2285	2273
query89	391	326	286	286
query90	2079	218	219	218
query91	147	137	110	110
query92	74	63	56	56
query93	1222	1086	588	588
query94	657	424	302	302
query95	355	276	263	263
query96	494	562	278	278
query97	3278	3452	3358	3358
query98	222	206	217	206
query99	1420	1408	1300	1300
Total cold run time: 274628 ms
Total hot run time: 186195 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76cafbf6f0fd3e62bd1e85bc3297926cd980d2e8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.57	0.57	0.55
query6	1.18	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.52	0.51
query10	0.57	0.62	0.59
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.68	2.70	2.71
query15	0.92	0.86	0.84
query16	0.39	0.39	0.38
query17	1.07	1.02	0.99
query18	0.21	0.20	0.19
query19	1.87	1.79	2.01
query20	0.01	0.01	0.01
query21	15.37	0.91	0.55
query22	0.76	1.20	0.90
query23	14.69	1.36	0.63
query24	6.90	2.15	1.31
query25	0.48	0.15	0.25
query26	0.58	0.16	0.13
query27	0.05	0.04	0.05
query28	10.00	0.82	0.42
query29	12.55	4.04	3.36
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.54	0.47
query33	3.05	3.01	3.04
query34	15.74	5.18	4.56
query35	4.54	4.56	4.57
query36	0.67	0.50	0.48
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.14
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.16 s
Total hot run time: 31.69 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

wangbo
wangbo previously approved these changes Mar 12, 2025
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 12, 2025
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c2de487855b1fcf577d82481e5db4b27dd98b770, data reload: false

------ Round 1 ----------------------------------
q1	17617	5234	5144	5144
q2	2039	284	165	165
q3	10426	1321	693	693
q4	10219	1022	528	528
q5	7513	2403	2387	2387
q6	189	163	133	133
q7	894	751	610	610
q8	9301	1289	1190	1190
q9	4990	4844	4820	4820
q10	6817	2317	1871	1871
q11	472	269	253	253
q12	342	355	222	222
q13	17751	3655	3111	3111
q14	226	222	207	207
q15	525	486	472	472
q16	623	605	590	590
q17	581	898	361	361
q18	6729	6419	6334	6334
q19	1085	956	551	551
q20	326	336	188	188
q21	2874	2192	1912	1912
q22	1047	1017	991	991
Total cold run time: 102586 ms
Total hot run time: 32733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5203	5226	5199	5199
q2	231	326	225	225
q3	2162	2677	2306	2306
q4	1449	1799	1376	1376
q5	4254	4133	4192	4133
q6	213	165	127	127
q7	1874	1914	1808	1808
q8	2624	2669	2563	2563
q9	7313	7124	7217	7124
q10	3040	3179	2746	2746
q11	573	509	503	503
q12	721	801	625	625
q13	3528	3892	3253	3253
q14	299	300	270	270
q15	529	489	489	489
q16	643	686	640	640
q17	1140	1583	1372	1372
q18	7816	7579	7487	7487
q19	826	917	1023	917
q20	1963	1979	1947	1947
q21	5525	4985	4904	4904
q22	1093	1038	1025	1025
Total cold run time: 53019 ms
Total hot run time: 51039 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185276 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2de487855b1fcf577d82481e5db4b27dd98b770, data reload: false

query1	992	402	375	375
query2	6539	1845	1854	1845
query3	6800	220	218	218
query4	26777	23783	23125	23125
query5	4307	656	484	484
query6	309	203	193	193
query7	4599	498	308	308
query8	298	246	239	239
query9	8608	2637	2639	2637
query10	468	306	264	264
query11	15455	15151	14875	14875
query12	162	112	105	105
query13	1666	511	395	395
query14	9587	6309	6937	6309
query15	212	197	181	181
query16	7714	636	453	453
query17	1595	736	566	566
query18	1991	414	308	308
query19	194	181	159	159
query20	123	122	118	118
query21	214	123	105	105
query22	4112	4309	4283	4283
query23	33943	33048	33142	33048
query24	7389	2387	2407	2387
query25	508	443	412	412
query26	1215	273	154	154
query27	2361	475	320	320
query28	4177	2413	2390	2390
query29	702	592	445	445
query30	285	219	192	192
query31	984	844	768	768
query32	75	65	62	62
query33	557	367	298	298
query34	784	836	503	503
query35	809	811	736	736
query36	958	979	839	839
query37	110	98	79	79
query38	4099	4151	4140	4140
query39	1466	1391	1427	1391
query40	234	118	105	105
query41	59	52	51	51
query42	117	98	100	98
query43	492	505	477	477
query44	1265	784	782	782
query45	174	167	164	164
query46	841	1029	615	615
query47	1773	1773	1724	1724
query48	381	415	296	296
query49	782	492	415	415
query50	684	727	400	400
query51	4203	4197	4143	4143
query52	104	103	96	96
query53	237	257	182	182
query54	483	478	405	405
query55	80	75	83	75
query56	281	254	260	254
query57	1117	1145	1090	1090
query58	250	235	229	229
query59	2612	2733	2613	2613
query60	271	275	260	260
query61	122	138	123	123
query62	888	757	668	668
query63	222	191	194	191
query64	4231	1006	727	727
query65	4414	4322	4335	4322
query66	1065	407	302	302
query67	15658	15289	15401	15289
query68	8021	868	498	498
query69	462	304	262	262
query70	1242	1069	1115	1069
query71	482	298	265	265
query72	5350	3551	3785	3551
query73	834	728	357	357
query74	9186	9170	8636	8636
query75	3799	3158	2732	2732
query76	3719	1206	756	756
query77	779	358	282	282
query78	10231	10176	9358	9358
query79	2786	816	590	590
query80	646	512	426	426
query81	492	261	219	219
query82	681	129	96	96
query83	181	170	152	152
query84	248	102	76	76
query85	809	378	309	309
query86	336	325	264	264
query87	4491	4569	4436	4436
query88	3526	2279	2271	2271
query89	388	313	287	287
query90	1904	212	210	210
query91	140	141	108	108
query92	72	63	55	55
query93	1613	1089	571	571
query94	686	417	311	311
query95	346	272	259	259
query96	490	555	280	280
query97	3350	3404	3303	3303
query98	274	214	206	206
query99	1441	1397	1276	1276
Total cold run time: 275189 ms
Total hot run time: 185276 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2de487855b1fcf577d82481e5db4b27dd98b770, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.56	0.55	0.57
query6	1.20	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.60	0.53	0.52
query10	0.59	0.60	0.58
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.62	0.60
query14	2.69	2.71	2.80
query15	0.92	0.85	0.87
query16	0.38	0.37	0.38
query17	1.02	1.01	1.04
query18	0.21	0.19	0.19
query19	1.94	1.78	1.98
query20	0.01	0.02	0.01
query21	15.40	0.88	0.54
query22	0.76	1.36	0.68
query23	14.80	1.37	0.60
query24	6.68	1.82	1.08
query25	0.44	0.22	0.14
query26	0.62	0.16	0.14
query27	0.05	0.05	0.05
query28	8.98	0.90	0.42
query29	12.54	4.02	3.34
query30	0.29	0.10	0.07
query31	3.38	0.59	0.38
query32	3.52	0.55	0.46
query33	2.99	2.99	3.13
query34	15.97	5.14	4.47
query35	4.63	4.56	4.54
query36	0.64	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.24 s
Total hot run time: 31.07 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 481867d into apache:master Mar 13, 2025
25 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 13, 2025
@morningman
[fix](hudi) replace non thread safe SimpleDateFormat (#48923)
5050a0a 
### What problem does this PR solve?

`SimpleDateFormat` is not thread safe, replace it with
`DateTimeFormatter`
@github-actions github-actions bot mentioned this pull request Mar 13, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Mar 13, 2025
@morningman
[fix](hudi) replace non thread safe SimpleDateFormat (#48923)
ce20e21 
### What problem does this PR solve?

`SimpleDateFormat` is not thread safe, replace it with
`DateTimeFormatter`
@github-actions github-actions bot mentioned this pull request Mar 13, 2025
Merged
yiguolei pushed a commit that referenced this pull request Mar 14, 2025
@github-actions @morningman
branch-2.1: [fix](hudi) replace non thread safe SimpleDateFormat #48923…
b5c85e0 
… (#49022)

Cherry-picked from #48923

Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
dataroaring pushed a commit that referenced this pull request Mar 22, 2025
@github-actions @morningman
branch-3.0: [fix](hudi) replace non thread safe SimpleDateFormat #48923…
ea8d0d8 
… (#49021)

Cherry-picked from #48923

Co-authored-by: Mingyu Chen (Rayner) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

@wangbo wangbo wangbo left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_test reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@morningman @hello-stephen @doris-robot @wangbo @CalvinKirs @wuwenchi @suxiaogang223 @yiguolei @dataroaring