Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)Optimize MTMV lock logic #44967

Merged
merged 1 commit into from
Dec 5, 2024
Merged

[enhance](mtmv)Optimize MTMV lock logic #44967

merged 1 commit into from
Dec 5, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Dec 4, 2024 •
edited
Loading

What problem does this PR solve?

  • org.apache.doris.mtmv.MTMVPartitionUtil#getPartitionsIdsByNames If the lock is no longer obtained and the partition does not exist, just throw an exception
  • show create materialized view Narrowing down the scope of the lock
  • OlapTable adds a method called getAndCopyPartitionItemWithoutLock that does not require obtaining a lock, for use in external scenarios where it already exists

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Optimize MTMV lock logic

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Dec 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40394 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 23f43b917d3427f9b69d9895f164467312aafbdc, data reload: false

------ Round 1 ----------------------------------
q1	17586	7553	7641	7553
q2	2052	172	169	169
q3	10755	1134	1214	1134
q4	10489	771	693	693
q5	7631	2726	2769	2726
q6	236	147	145	145
q7	1011	650	608	608
q8	9266	1868	1928	1868
q9	6637	6532	6533	6532
q10	7037	2277	2291	2277
q11	460	277	255	255
q12	441	216	222	216
q13	17765	2994	3004	2994
q14	242	204	212	204
q15	577	523	518	518
q16	651	601	579	579
q17	979	534	539	534
q18	7389	6837	6644	6644
q19	1330	987	973	973
q20	470	180	181	180
q21	4003	3292	3284	3284
q22	385	312	308	308
Total cold run time: 107392 ms
Total hot run time: 40394 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7234	7241	7269	7241
q2	334	231	231	231
q3	2913	2847	2950	2847
q4	2028	1863	1853	1853
q5	5652	5723	5683	5683
q6	228	148	145	145
q7	2304	1825	1807	1807
q8	3422	3560	3513	3513
q9	9031	9060	9064	9060
q10	3579	3554	3575	3554
q11	609	501	510	501
q12	807	631	616	616
q13	11158	3196	3231	3196
q14	302	279	273	273
q15	573	535	531	531
q16	689	643	644	643
q17	1854	1652	1642	1642
q18	8335	7897	7617	7617
q19	1705	1670	1515	1515
q20	2107	1890	1881	1881
q21	5595	5453	5478	5453
q22	654	596	598	596
Total cold run time: 71113 ms
Total hot run time: 60398 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196973 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 23f43b917d3427f9b69d9895f164467312aafbdc, data reload: false

query1	1258	953	930	930
query2	6239	2081	1992	1992
query3	11005	4391	4484	4391
query4	33399	23555	23619	23555
query5	3553	475	447	447
query6	275	185	165	165
query7	3989	293	322	293
query8	287	226	234	226
query9	9586	2721	2710	2710
query10	461	246	250	246
query11	17881	15238	15275	15238
query12	145	104	97	97
query13	1543	407	405	405
query14	9704	6654	6701	6654
query15	248	204	189	189
query16	8067	498	472	472
query17	1563	592	604	592
query18	2203	317	330	317
query19	262	168	165	165
query20	128	111	113	111
query21	198	105	103	103
query22	4907	4605	4457	4457
query23	35377	36587	34403	34403
query24	10716	2584	2537	2537
query25	541	382	406	382
query26	783	155	158	155
query27	2844	274	297	274
query28	8199	2522	2506	2506
query29	671	413	470	413
query30	233	143	157	143
query31	1074	866	866	866
query32	95	57	52	52
query33	769	298	295	295
query34	1129	514	537	514
query35	936	797	752	752
query36	1119	972	984	972
query37	128	79	74	74
query38	4516	4511	4449	4449
query39	1512	1461	1485	1461
query40	257	103	104	103
query41	46	45	56	45
query42	114	98	98	98
query43	528	499	492	492
query44	1278	851	849	849
query45	188	171	174	171
query46	1175	723	726	723
query47	2025	1980	1971	1971
query48	426	316	321	316
query49	998	406	442	406
query50	832	412	392	392
query51	7443	7289	7262	7262
query52	103	94	95	94
query53	254	186	179	179
query54	1092	415	408	408
query55	87	81	79	79
query56	271	237	240	237
query57	1298	1171	1165	1165
query58	238	209	212	209
query59	3217	2936	3019	2936
query60	264	242	237	237
query61	113	103	107	103
query62	888	671	684	671
query63	213	187	182	182
query64	3973	673	661	661
query65	3279	3199	3218	3199
query66	954	360	294	294
query67	15854	15590	15754	15590
query68	4818	570	578	570
query69	438	250	259	250
query70	1167	1152	1045	1045
query71	334	248	245	245
query72	6025	4151	4329	4151
query73	773	356	365	356
query74	10532	9126	8960	8960
query75	3408	2629	2706	2629
query76	2697	1017	1113	1017
query77	374	274	313	274
query78	10285	9458	9387	9387
query79	1687	587	609	587
query80	967	425	427	425
query81	554	237	228	228
query82	860	125	119	119
query83	239	143	142	142
query84	232	73	73	73
query85	1297	302	290	290
query86	371	311	302	302
query87	4651	4500	4539	4500
query88	3415	2199	2178	2178
query89	432	287	294	287
query90	2016	187	183	183
query91	137	105	104	104
query92	58	50	50	50
query93	1739	551	546	546
query94	927	293	296	293
query95	353	282	249	249
query96	615	282	280	280
query97	2831	2678	2655	2655
query98	220	199	190	190
query99	1552	1333	1344	1333
Total cold run time: 300955 ms
Total hot run time: 196973 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 23f43b917d3427f9b69d9895f164467312aafbdc, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.42	0.41	0.41
query6	1.14	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.49
query10	0.57	0.56	0.57
query11	0.13	0.10	0.10
query12	0.15	0.10	0.12
query13	0.60	0.60	0.58
query14	2.73	2.74	2.78
query15	0.91	0.82	0.83
query16	0.38	0.37	0.39
query17	1.01	1.06	1.06
query18	0.22	0.21	0.21
query19	1.94	1.91	2.05
query20	0.02	0.01	0.02
query21	15.36	0.58	0.59
query22	2.64	2.12	1.90
query23	17.07	0.81	0.74
query24	3.00	1.42	1.40
query25	0.29	0.13	0.04
query26	0.49	0.13	0.14
query27	0.04	0.04	0.04
query28	10.41	1.09	1.07
query29	12.57	3.27	3.22
query30	0.26	0.06	0.06
query31	2.87	0.37	0.37
query32	3.28	0.47	0.45
query33	2.98	3.12	3.01
query34	17.08	4.52	4.50
query35	4.50	4.47	4.50
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.06	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.81 s
Total hot run time: 33.05 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 4, 2024

PR approved by anyone and no changes requested.

CalvinKirs
CalvinKirs approved these changes Dec 4, 2024
View reviewed changes
Copy link
Member

@CalvinKirs CalvinKirs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env files LGTM

@morrySnow morrySnow merged commit ab29448 into apache:master Dec 5, 2024
33 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 5, 2024
@zddr
[enhance](mtmv)Optimize MTMV lock logic (#44967)
5c5f18c 
- `org.apache.doris.mtmv.MTMVPartitionUtil#getPartitionsIdsByNames ` If
the lock is no longer obtained and the partition does not exist, just
throw an exception
- `show create materialized view` Narrowing down the scope of the lock
- `OlapTable `adds a method called `getAndCopyPartitionItemWithoutLock`
that does not require obtaining a lock, for use in external scenarios
where it already exists
@github-actions github-actions bot mentioned this pull request Dec 5, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Dec 5, 2024
@zddr
[enhance](mtmv)Optimize MTMV lock logic (#44967)
40c912a 
- `org.apache.doris.mtmv.MTMVPartitionUtil#getPartitionsIdsByNames ` If
the lock is no longer obtained and the partition does not exist, just
throw an exception
- `show create materialized view` Narrowing down the scope of the lock
- `OlapTable `adds a method called `getAndCopyPartitionItemWithoutLock`
that does not require obtaining a lock, for use in external scenarios
where it already exists
@github-actions github-actions bot mentioned this pull request Dec 5, 2024
Closed
zddr added a commit to zddr/incubator-doris that referenced this pull request Dec 6, 2024
@zddr
[enhance](mtmv)Optimize MTMV lock logic (apache#44967)
36d9cc8 
- `org.apache.doris.mtmv.MTMVPartitionUtil#getPartitionsIdsByNames ` If
the lock is no longer obtained and the partition does not exist, just
throw an exception
- `show create materialized view` Narrowing down the scope of the lock
- `OlapTable `adds a method called `getAndCopyPartitionItemWithoutLock`
that does not require obtaining a lock, for use in external scenarios
where it already exists
@zddr zddr mentioned this pull request Dec 6, 2024
Merged
yiguolei pushed a commit that referenced this pull request Dec 6, 2024
@zddr
[enhance](mtmv)Optimize MTMV lock logic (#44967) (#45076)
4f45fef 
pick: #44967
morningman pushed a commit that referenced this pull request Dec 8, 2024
@github-actions @zddr
branch-3.0: [enhance](mtmv)Optimize MTMV lock logic #44967 (#45014)
abe8f22 
Cherry-picked from #44967

Co-authored-by: zhangdong <[email protected]>
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@morrySnow morrySnow morrySnow approved these changes

@seawinde seawinde seawinde approved these changes

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zddr @doris-robot @CalvinKirs @morrySnow @seawinde @morningman @yiguolei