Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](deps) upgrade hadoop to 3.3.6.6 #49181

Merged
merged 1 commit into from
Mar 18, 2025
Merged

[feat](deps) upgrade hadoop to 3.3.6.6 #49181

merged 1 commit into from
Mar 18, 2025

Conversation

morningman
Copy link
Contributor

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

run buildall

@morningman morningman changed the title [deps] upgrade hadoop to 3.3.6.6 [feat](deps) upgrade hadoop to 3.3.6.6 Mar 18, 2025
@morningman morningman mentioned this pull request Mar 18, 2025
Merged
16 tasks
@doris-robot
Copy link

TPC-H: Total hot run time: 32273 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c792eeceb202d022eb5d3faa3741c92e5a6717b7, data reload: false

------ Round 1 ----------------------------------
q1	24143	5042	5069	5042
q2	2044	306	186	186
q3	10366	1201	696	696
q4	10217	1046	552	552
q5	7538	2385	2369	2369
q6	190	163	133	133
q7	928	754	614	614
q8	9309	1317	1068	1068
q9	4891	4796	4583	4583
q10	6822	2323	1881	1881
q11	482	291	261	261
q12	364	377	224	224
q13	17775	3719	3104	3104
q14	236	225	208	208
q15	537	480	477	477
q16	628	608	578	578
q17	573	854	333	333
q18	6740	6350	6239	6239
q19	1512	941	568	568
q20	331	352	207	207
q21	2756	2331	1970	1970
q22	1037	995	980	980
Total cold run time: 109419 ms
Total hot run time: 32273 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5186	5202	5169	5169
q2	235	323	235	235
q3	2139	2669	2279	2279
q4	1436	1783	1396	1396
q5	4235	4093	4395	4093
q6	217	166	127	127
q7	1975	1889	1783	1783
q8	2574	2579	2568	2568
q9	7176	7074	7135	7074
q10	2967	3201	2740	2740
q11	564	496	467	467
q12	678	768	629	629
q13	3558	3998	3377	3377
q14	283	295	279	279
q15	513	491	469	469
q16	644	699	661	661
q17	1166	1555	1382	1382
q18	7784	7635	7566	7566
q19	833	875	985	875
q20	1967	2058	1872	1872
q21	5553	4788	5012	4788
q22	1127	1043	1013	1013
Total cold run time: 52810 ms
Total hot run time: 50842 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191114 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c792eeceb202d022eb5d3faa3741c92e5a6717b7, data reload: false

query1	1391	1072	1027	1027
query2	6110	1942	1854	1854
query3	10979	4522	4439	4439
query4	53430	25844	23020	23020
query5	5295	539	482	482
query6	416	205	190	190
query7	5346	505	293	293
query8	323	241	244	241
query9	7063	2515	2523	2515
query10	421	308	258	258
query11	15444	15793	14791	14791
query12	156	112	101	101
query13	1263	527	377	377
query14	9906	6358	6377	6358
query15	196	202	173	173
query16	6957	675	490	490
query17	1064	698	556	556
query18	1540	408	327	327
query19	200	191	174	174
query20	139	128	122	122
query21	206	127	121	121
query22	4550	4422	4362	4362
query23	34140	33328	33448	33328
query24	6568	2463	2441	2441
query25	482	474	430	430
query26	751	293	214	214
query27	2178	485	324	324
query28	3138	2478	2426	2426
query29	589	594	427	427
query30	280	224	193	193
query31	867	847	806	806
query32	71	62	64	62
query33	457	378	302	302
query34	741	866	502	502
query35	803	839	785	785
query36	964	990	892	892
query37	119	100	77	77
query38	4282	4327	4144	4144
query39	1489	1449	1468	1449
query40	209	116	106	106
query41	53	52	52	52
query42	127	106	105	105
query43	489	502	512	502
query44	1311	814	811	811
query45	184	176	168	168
query46	871	1039	643	643
query47	1816	1876	1810	1810
query48	388	445	313	313
query49	754	527	412	412
query50	731	743	414	414
query51	4312	4307	4178	4178
query52	111	108	102	102
query53	238	259	199	199
query54	480	499	433	433
query55	83	81	81	81
query56	270	270	268	268
query57	1154	1171	1100	1100
query58	261	265	248	248
query59	2582	2833	2719	2719
query60	279	288	260	260
query61	126	125	122	122
query62	759	737	690	690
query63	230	226	192	192
query64	1795	1064	688	688
query65	4574	4464	4349	4349
query66	802	395	290	290
query67	15751	15672	15180	15180
query68	7297	830	499	499
query69	596	306	264	264
query70	1224	1110	1077	1077
query71	498	296	265	265
query72	5796	3654	3752	3654
query73	1073	738	338	338
query74	9035	8968	8883	8883
query75	3679	3133	2779	2779
query76	4312	1175	756	756
query77	639	381	274	274
query78	10143	9961	9332	9332
query79	5358	816	557	557
query80	679	536	444	444
query81	502	251	226	226
query82	678	131	94	94
query83	320	167	154	154
query84	287	95	74	74
query85	807	421	307	307
query86	413	313	276	276
query87	4531	4519	4288	4288
query88	3513	2233	2233	2233
query89	455	317	288	288
query90	1883	215	214	214
query91	140	138	110	110
query92	79	58	59	58
query93	3022	1046	572	572
query94	695	418	305	305
query95	366	276	290	276
query96	479	555	280	280
query97	3294	3449	3245	3245
query98	259	211	200	200
query99	1431	1391	1284	1284
Total cold run time: 304537 ms
Total hot run time: 191114 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c792eeceb202d022eb5d3faa3741c92e5a6717b7, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.11	0.11
query3	0.25	0.19	0.19
query4	1.60	0.19	0.20
query5	0.59	0.57	0.59
query6	1.18	0.72	0.72
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.59	0.52	0.53
query10	0.57	0.61	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.62
query14	2.66	2.73	2.80
query15	0.94	0.85	0.84
query16	0.38	0.38	0.39
query17	1.03	1.04	1.04
query18	0.21	0.20	0.20
query19	1.86	1.88	1.87
query20	0.02	0.01	0.01
query21	15.36	0.89	0.55
query22	0.75	1.19	1.05
query23	14.69	1.41	0.64
query24	6.91	2.24	0.42
query25	0.51	0.18	0.20
query26	0.58	0.16	0.13
query27	0.05	0.05	0.05
query28	9.24	0.85	0.45
query29	12.59	4.07	3.38
query30	0.25	0.08	0.06
query31	2.82	0.59	0.39
query32	3.22	0.55	0.48
query33	2.94	3.00	3.05
query34	15.91	5.08	4.49
query35	4.47	4.52	4.52
query36	0.66	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.47 s
Total hot run time: 31.41 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.88% (13092/26784)
Line Coverage 38.44% (112832/293523)
Region Coverage 37.26% (57396/154023)
Branch Coverage 32.34% (28837/89164)

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 18, 2025
@morningman morningman merged commit 39598c0 into apache:master Mar 18, 2025
28 of 30 checks passed
morningman added a commit that referenced this pull request Mar 19, 2025
@morningman
[fix](kerberos) Back to use principal and keytab to login kerberos in…
c505e47 
…stead of using kerberos ticket cache. (#48655)

### What problem does this PR solve?

Related PR: #47299, #49181

This PR mainly changes:

1. Back to use principal and keytab to login kerberos instead of using
kerberos ticket cache.
Discard what I did in #47299. It looks like there are a lot of issue
when using ticket cache in multi-kerberos env.
    So I abandoned that logic. 

2. Config's default value
    Change the default value of related to hdfs file handle cache

    1. `max_hdfs_file_handle_cache_num`: from 1000 to 20000
    2. `max_hdfs_file_handle_cache_time_sec`: from 3600 to 28800

3. Fix a bug the cleanup thread of `FileHandleCache` is not working
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@morningman @hello-stephen @doris-robot @CalvinKirs @wuwenchi @hubgeter