Skip to content

[enhance](auth)ranger support storage vault and compute group#47925

Merged
morrySnow merged 4 commits intoapache:masterfrom
zddr:ranger_compute_group
Apr 8, 2025
Merged

[enhance](auth)ranger support storage vault and compute group#47925
morrySnow merged 4 commits intoapache:masterfrom
zddr:ranger_compute_group

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Feb 14, 2025 •
edited
Loading

What problem does this PR solve?

ranger support storage vault and compute group
ranger pr: morningman/ranger#2
doc pr: apache/doris-website#2277
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
ranger support storage vault and compute group

Release note

ranger support storage vault and compute group

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Feb 14, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Feb 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

TPC-H: Total hot run time: 31359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04e75c40a02d179481c25f4c1cce5d966fd91658, data reload: false

------ Round 1 ----------------------------------
q1	17578	5172	5026	5026
q2	2052	315	170	170
q3	10541	1240	752	752
q4	10218	990	518	518
q5	7544	2358	2298	2298
q6	191	167	131	131
q7	897	769	596	596
q8	9290	1257	1046	1046
q9	4779	4703	4653	4653
q10	6847	2297	1873	1873
q11	468	278	256	256
q12	349	350	218	218
q13	17784	3679	3039	3039
q14	234	231	205	205
q15	525	467	460	460
q16	611	623	568	568
q17	571	864	341	341
q18	6769	6228	6196	6196
q19	1216	934	531	531
q20	309	333	199	199
q21	2793	2138	1974	1974
q22	368	325	309	309
Total cold run time: 101934 ms
Total hot run time: 31359 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5090	5053	5081	5053
q2	230	323	227	227
q3	2201	2657	2304	2304
q4	1446	1850	1366	1366
q5	4212	4127	4159	4127
q6	202	163	123	123
q7	1867	1830	1660	1660
q8	2552	2647	2540	2540
q9	7211	7150	7106	7106
q10	3016	3184	2786	2786
q11	561	513	508	508
q12	707	782	611	611
q13	3619	3831	3337	3337
q14	286	298	257	257
q15	506	461	456	456
q16	642	684	628	628
q17	1128	1573	1346	1346
q18	7548	7328	7247	7247
q19	803	998	1138	998
q20	1996	1991	1900	1900
q21	5406	4888	4889	4888
q22	617	566	546	546
Total cold run time: 51846 ms
Total hot run time: 50014 ms

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

TPC-DS: Total hot run time: 189822 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04e75c40a02d179481c25f4c1cce5d966fd91658, data reload: false

query1	1316	973	921	921
query2	6244	1827	1842	1827
query3	11024	4480	4230	4230
query4	52752	24666	22877	22877
query5	5408	605	472	472
query6	345	201	213	201
query7	4942	502	291	291
query8	318	241	232	232
query9	5957	2538	2525	2525
query10	434	308	254	254
query11	15173	15136	14882	14882
query12	162	103	100	100
query13	1046	499	380	380
query14	10562	6793	6469	6469
query15	200	198	187	187
query16	7071	657	503	503
query17	1072	712	569	569
query18	1591	420	335	335
query19	203	234	159	159
query20	127	125	125	125
query21	210	122	116	116
query22	4376	4414	4299	4299
query23	33995	33421	33404	33404
query24	5621	2466	2460	2460
query25	448	476	408	408
query26	700	279	156	156
query27	1826	495	354	354
query28	2809	2462	2446	2446
query29	623	624	462	462
query30	248	187	155	155
query31	912	873	789	789
query32	73	68	62	62
query33	437	337	305	305
query34	752	845	515	515
query35	805	834	764	764
query36	958	1003	950	950
query37	133	102	75	75
query38	4192	4313	4318	4313
query39	1504	1502	1439	1439
query40	204	115	107	107
query41	52	49	49	49
query42	122	107	107	107
query43	495	520	491	491
query44	1286	837	811	811
query45	180	172	163	163
query46	877	1056	670	670
query47	1834	1868	1801	1801
query48	393	421	323	323
query49	691	530	415	415
query50	708	759	432	432
query51	4290	4335	4270	4270
query52	109	111	105	105
query53	246	256	197	197
query54	502	522	442	442
query55	85	80	85	80
query56	286	296	303	296
query57	1166	1177	1137	1137
query58	254	255	256	255
query59	2914	2875	2620	2620
query60	310	300	277	277
query61	142	142	155	142
query62	748	784	744	744
query63	245	192	201	192
query64	1438	1035	668	668
query65	3278	3229	3123	3123
query66	775	392	296	296
query67	15771	15437	15442	15437
query68	5587	775	501	501
query69	540	303	262	262
query70	1236	1151	1131	1131
query71	434	304	265	265
query72	6280	3706	3819	3706
query73	1289	734	346	346
query74	8973	9064	9027	9027
query75	3191	3145	2686	2686
query76	3970	1178	741	741
query77	502	378	287	287
query78	10049	10117	9270	9270
query79	2049	803	591	591
query80	735	519	436	436
query81	511	275	235	235
query82	407	129	97	97
query83	177	167	159	159
query84	292	87	83	83
query85	729	343	293	293
query86	359	317	311	311
query87	4405	4439	4311	4311
query88	3065	2193	2178	2178
query89	397	322	331	322
query90	1683	189	186	186
query91	135	143	107	107
query92	72	59	56	56
query93	2681	989	567	567
query94	686	383	291	291
query95	351	265	261	261
query96	485	565	270	270
query97	2748	2859	2716	2716
query98	234	207	203	203
query99	1305	1411	1266	1266
Total cold run time: 291527 ms
Total hot run time: 189822 ms

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04e75c40a02d179481c25f4c1cce5d966fd91658, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.08	0.06
query4	1.61	0.09	0.11
query5	0.42	0.41	0.39
query6	1.18	0.67	0.65
query7	0.02	0.02	0.01
query8	0.04	0.02	0.03
query9	0.61	0.52	0.53
query10	0.57	0.58	0.58
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.69	2.70	2.71
query15	0.92	0.84	0.86
query16	0.36	0.39	0.38
query17	1.00	1.04	1.04
query18	0.22	0.19	0.20
query19	1.95	1.79	2.02
query20	0.01	0.02	0.02
query21	15.42	0.91	0.55
query22	0.76	1.18	0.66
query23	14.96	1.38	0.62
query24	8.56	3.38	0.45
query25	0.45	0.18	0.18
query26	0.82	0.18	0.15
query27	0.05	0.05	0.06
query28	7.90	0.76	0.42
query29	12.61	4.00	3.30
query30	0.24	0.10	0.06
query31	2.82	0.58	0.38
query32	3.22	0.54	0.48
query33	2.97	3.03	3.00
query34	15.64	5.06	4.49
query35	4.52	4.50	4.53
query36	0.67	0.51	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.02	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.92 s
Total hot run time: 30.24 s

@zddr
Copy link
Contributor Author

zddr commented Feb 14, 2025

run p0

@zddr zddr marked this pull request as draft February 19, 2025 09:57
@zddr zddr marked this pull request as ready for review March 3, 2025 10:59
@zddr
Copy link
Contributor Author

zddr commented Mar 3, 2025

run buildall

morrySnow
morrySnow previously approved these changes Mar 10, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 10, 2025

PR approved by anyone and no changes requested.

@zddr
Copy link
Contributor Author

zddr commented Mar 27, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 27, 2025
@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 34122 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8fe5efdb8ff3060340e19789732ab776a2d5c4f, data reload: false

------ Round 1 ----------------------------------
q1	26145	5034	5024	5024
q2	2073	296	163	163
q3	10622	1224	682	682
q4	10238	984	521	521
q5	7810	2383	2346	2346
q6	192	164	134	134
q7	896	736	614	614
q8	9299	1265	1071	1071
q9	6751	5074	5176	5074
q10	6819	2323	1884	1884
q11	481	268	271	268
q12	344	350	214	214
q13	17768	3642	3129	3129
q14	233	225	213	213
q15	524	485	492	485
q16	630	627	610	610
q17	585	863	341	341
q18	7824	7224	7213	7213
q19	1233	944	535	535
q20	331	336	191	191
q21	3822	2593	2445	2445
q22	1054	1016	965	965
Total cold run time: 115674 ms
Total hot run time: 34122 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5125	5163	5192	5163
q2	236	327	228	228
q3	2200	2647	2303	2303
q4	1422	1777	1396	1396
q5	4520	4456	4420	4420
q6	213	164	130	130
q7	1977	1867	1764	1764
q8	2597	2586	2503	2503
q9	7262	7222	7209	7209
q10	2963	3200	2753	2753
q11	571	506	499	499
q12	707	769	588	588
q13	3518	3938	3317	3317
q14	276	310	275	275
q15	526	500	479	479
q16	672	685	672	672
q17	1130	1543	1444	1444
q18	7853	7578	7452	7452
q19	811	802	891	802
q20	2009	1955	1845	1845
q21	5140	4746	4505	4505
q22	1094	1019	1005	1005
Total cold run time: 52822 ms
Total hot run time: 50752 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 186634 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8fe5efdb8ff3060340e19789732ab776a2d5c4f, data reload: false

query1	994	482	480	480
query2	6568	2007	2012	2007
query3	6823	226	220	220
query4	27261	23919	23065	23065
query5	4341	648	460	460
query6	278	197	206	197
query7	4594	480	281	281
query8	285	230	230	230
query9	8640	2573	2545	2545
query10	446	319	250	250
query11	15287	15109	14831	14831
query12	154	110	115	110
query13	1645	525	396	396
query14	8672	6187	6497	6187
query15	210	189	168	168
query16	7118	633	473	473
query17	927	701	541	541
query18	1962	398	300	300
query19	189	182	153	153
query20	120	113	114	113
query21	206	122	101	101
query22	4089	4160	4258	4160
query23	33959	33178	33047	33047
query24	8203	2429	2374	2374
query25	548	451	387	387
query26	1229	264	141	141
query27	2736	482	325	325
query28	4409	2408	2377	2377
query29	779	555	418	418
query30	285	222	193	193
query31	929	861	783	783
query32	73	66	64	64
query33	558	355	315	315
query34	775	856	495	495
query35	837	840	751	751
query36	978	1035	901	901
query37	117	105	85	85
query38	4148	4157	4089	4089
query39	1473	1404	1386	1386
query40	210	121	109	109
query41	69	59	58	58
query42	127	109	112	109
query43	508	531	496	496
query44	1295	819	816	816
query45	179	181	178	178
query46	841	1039	643	643
query47	1761	1791	1710	1710
query48	382	403	308	308
query49	821	532	443	443
query50	675	735	432	432
query51	4190	4222	4126	4126
query52	120	105	104	104
query53	232	256	183	183
query54	496	509	421	421
query55	86	81	84	81
query56	305	281	326	281
query57	1123	1145	1082	1082
query58	242	247	244	244
query59	2654	2840	2825	2825
query60	297	279	266	266
query61	126	127	130	127
query62	827	729	637	637
query63	217	182	183	182
query64	4428	1079	696	696
query65	4401	4249	4240	4240
query66	1130	401	304	304
query67	15872	15766	15446	15446
query68	9286	872	525	525
query69	470	296	265	265
query70	1244	1085	1124	1085
query71	455	292	278	278
query72	5783	4838	4857	4838
query73	756	670	350	350
query74	8999	8949	8722	8722
query75	4230	3240	2719	2719
query76	3856	1191	778	778
query77	1012	363	286	286
query78	9970	10162	9308	9308
query79	5572	809	552	552
query80	674	511	444	444
query81	483	274	226	226
query82	714	124	145	124
query83	219	178	162	162
query84	283	103	77	77
query85	774	357	313	313
query86	335	319	283	283
query87	4380	4357	4550	4357
query88	2847	2244	2238	2238
query89	453	320	282	282
query90	1920	210	217	210
query91	134	140	114	114
query92	84	60	59	59
query93	2780	1035	583	583
query94	692	399	305	305
query95	362	264	275	264
query96	476	583	275	275
query97	3212	3231	3189	3189
query98	225	217	202	202
query99	1447	1381	1275	1275
Total cold run time: 280586 ms
Total hot run time: 186634 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8fe5efdb8ff3060340e19789732ab776a2d5c4f, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.24	0.19	0.19
query4	1.59	0.19	0.19
query5	0.61	0.58	0.60
query6	1.18	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.54	0.51
query10	0.63	0.59	0.58
query11	0.16	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.71	2.80	2.70
query15	0.93	0.84	0.87
query16	0.38	0.40	0.39
query17	1.01	1.05	1.03
query18	0.22	0.21	0.20
query19	1.87	2.05	1.78
query20	0.01	0.01	0.02
query21	15.37	0.94	0.55
query22	0.74	1.21	0.71
query23	14.89	1.38	0.63
query24	6.96	1.69	0.87
query25	0.47	0.25	0.08
query26	0.64	0.15	0.12
query27	0.05	0.06	0.05
query28	9.27	0.87	0.43
query29	12.53	3.92	3.30
query30	0.25	0.09	0.06
query31	2.81	0.60	0.39
query32	3.23	0.55	0.47
query33	3.01	3.03	3.07
query34	15.73	5.09	4.51
query35	4.51	4.52	4.50
query36	0.66	0.51	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.15	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.72 s
Total hot run time: 31.21 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

PR approved by at least one committer and no changes requested.

@swjtu-zhanglei
Copy link
Contributor

swjtu-zhanglei commented Apr 7, 2025 •
edited
Loading

do we need regression test with cloud_p0、vault_p0、 p0 for this feature?

@morrySnow morrySnow merged commit 2aa1734 into apache:master Apr 8, 2025
26 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request May 27, 2025
@zddr
[enhance](auth)ranger support storage vault and compute group (#47925)
2cc4a10
@github-actions github-actions bot mentioned this pull request May 27, 2025
Merged
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@zddr
[enhance](auth)ranger support storage vault and compute group (apache…
41f55c8 
…#47925)
github-actions bot pushed a commit that referenced this pull request Jun 20, 2025
@zddr
[enhance](auth)ranger support storage vault and compute group (#47925)
ed0d241
@github-actions github-actions bot mentioned this pull request Jun 20, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 23, 2025
@github-actions @zddr
branch-3.1: [enhance](auth)ranger support storage vault and compute g…
4336c73 
…roup #47925 (#52086)

Cherry-picked from #47925

Co-authored-by: zhangdong <zhangdong@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jul 14, 2025
@github-actions @zddr
branch-3.0: [enhance](auth)ranger support storage vault and compute g…
7838ed9 
…roup #47925 (#51288)

Cherry-picked from #47925

Co-authored-by: zhangdong <zhangdong@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zddr @hello-stephen @doris-robot @swjtu-zhanglei @morningman @morrySnow @dataroaring