[fix](routine load) reset Kafka progress cache when routine load job topic change #38474

sollhui · 2024-07-29T08:48:43Z

When change routine load job topic from test_topic_before to test_topic_after by

ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after");

(test_topic_before has 5 rows and test_topic_after has 1 rows)

Exception happened, which cannot consume any data:

2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615
2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of
fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615

It is necessary to reset Kafka progress cache when routine load job topic change.

doris-robot · 2024-07-29T08:48:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

sollhui · 2024-07-29T08:48:56Z

run buildall

doris-robot · 2024-07-29T09:07:15Z

TPC-H: Total hot run time: 41673 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee7ad73126d55d25a44fe5964893aff5c7974349, data reload: false

------ Round 1 ----------------------------------
q1	17705	4103	4033	4033
q2	2023	208	206	206
q3	10526	1323	1411	1323
q4	10850	838	990	838
q5	8194	3018	3006	3006
q6	228	142	140	140
q7	1057	630	608	608
q8	11292	1834	1989	1834
q9	8628	6628	6635	6628
q10	8732	3832	3857	3832
q11	429	249	254	249
q12	408	228	228	228
q13	17765	2959	2936	2936
q14	269	243	245	243
q15	532	488	488	488
q16	497	400	387	387
q17	966	918	916	916
q18	7978	7226	7228	7226
q19	1384	1204	1220	1204
q20	587	340	337	337
q21	5312	4730	4725	4725
q22	352	286	291	286
Total cold run time: 115714 ms
Total hot run time: 41673 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4034	4004	3997	3997
q2	335	222	219	219
q3	2990	3169	3149	3149
q4	2035	2020	1988	1988
q5	5489	5472	5375	5375
q6	217	130	131	130
q7	2191	1797	1874	1797
q8	3370	3422	3347	3347
q9	8713	8497	8647	8497
q10	3917	4022	3996	3996
q11	558	459	466	459
q12	738	600	610	600
q13	16814	3118	3165	3118
q14	301	266	265	265
q15	526	505	502	502
q16	469	407	432	407
q17	1748	1748	1730	1730
q18	8347	7900	7696	7696
q19	1709	1722	1695	1695
q20	1972	1779	1756	1756
q21	5674	5417	5446	5417
q22	530	500	464	464
Total cold run time: 72677 ms
Total hot run time: 56604 ms

doris-robot · 2024-07-29T09:17:54Z

TPC-DS: Total hot run time: 169656 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee7ad73126d55d25a44fe5964893aff5c7974349, data reload: false

query1	910	365	364	364
query2	6303	1770	1671	1671
query3	6663	212	219	212
query4	20457	17447	17373	17373
query5	3668	503	519	503
query6	285	209	163	163
query7	4594	309	285	285
query8	250	188	199	188
query9	8512	2448	2376	2376
query10	440	281	261	261
query11	10522	10037	10079	10037
query12	119	87	88	87
query13	1618	375	381	375
query14	9640	7144	7508	7144
query15	195	162	165	162
query16	7063	462	434	434
query17	944	575	588	575
query18	1906	295	282	282
query19	191	140	142	140
query20	90	86	85	85
query21	205	100	100	100
query22	4095	4051	4015	4015
query23	34040	33558	33269	33269
query24	10366	3142	3098	3098
query25	673	438	408	408
query26	1765	153	164	153
query27	3103	305	285	285
query28	7456	2033	2028	2028
query29	1223	438	438	438
query30	233	150	151	150
query31	943	767	783	767
query32	98	55	55	55
query33	677	325	329	325
query34	920	501	505	501
query35	864	771	759	759
query36	1047	869	859	859
query37	285	89	89	89
query38	2942	2828	2788	2788
query39	854	804	807	804
query40	257	110	112	110
query41	46	43	43	43
query42	123	95	106	95
query43	477	437	424	424
query44	1139	717	728	717
query45	207	177	175	175
query46	1074	800	786	786
query47	1818	1730	1724	1724
query48	367	291	287	287
query49	905	410	405	405
query50	912	430	433	430
query51	6813	6697	6699	6697
query52	100	91	87	87
query53	249	178	178	178
query54	611	454	450	450
query55	74	73	73	73
query56	280	260	270	260
query57	1136	1027	1033	1027
query58	260	252	264	252
query59	2531	2453	2255	2255
query60	296	268	271	268
query61	99	97	95	95
query62	872	664	705	664
query63	210	176	180	176
query64	5626	1903	1910	1903
query65	3146	3127	3114	3114
query66	1299	337	331	331
query67	15209	14798	14769	14769
query68	4291	556	574	556
query69	446	300	290	290
query70	1116	1053	1012	1012
query71	419	270	277	270
query72	7041	2696	2510	2510
query73	772	326	329	326
query74	6061	5655	5606	5606
query75	3387	2758	2727	2727
query76	2133	1357	1394	1357
query77	468	309	299	299
query78	9377	8966	8985	8966
query79	1564	541	526	526
query80	956	504	512	504
query81	552	224	219	219
query82	1089	127	126	126
query83	259	165	169	165
query84	261	79	80	79
query85	1156	343	307	307
query86	387	303	299	299
query87	3269	3099	3069	3069
query88	2939	2478	2462	2462
query89	376	293	291	291
query90	1691	188	189	188
query91	129	102	100	100
query92	59	49	51	49
query93	1393	632	619	619
query94	763	298	305	298
query95	389	276	356	276
query96	595	292	278	278
query97	3206	3005	3036	3005
query98	227	211	206	206
query99	1652	1268	1286	1268
Total cold run time: 260718 ms
Total hot run time: 169656 ms

doris-robot · 2024-07-29T09:23:22Z

ClickBench: Total hot run time: 30.18 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee7ad73126d55d25a44fe5964893aff5c7974349, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.48	0.47	0.48
query6	1.14	0.73	0.71
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.56	0.50	0.53
query10	0.56	0.55	0.57
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.61	0.60	0.60
query14	0.77	0.78	0.80
query15	0.88	0.86	0.86
query16	0.38	0.36	0.36
query17	1.06	1.00	1.00
query18	0.22	0.20	0.22
query19	1.80	1.71	1.71
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	3.76	7.19	1.57
query23	17.95	1.37	1.22
query24	2.29	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.22	0.21
query27	0.46	0.24	0.23
query28	13.14	0.99	0.95
query29	12.57	3.35	3.34
query30	0.26	0.06	0.05
query31	2.86	0.41	0.42
query32	3.24	0.48	0.48
query33	2.92	2.92	2.97
query34	15.44	4.24	4.23
query35	4.30	4.29	4.29
query36	0.69	0.49	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.13	0.14
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.37 s
Total hot run time: 30.18 s

dataroaring

LGTM

github-actions · 2024-07-29T14:06:12Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-29T14:06:15Z

PR approved by anyone and no changes requested.

sollhui · 2024-07-30T02:44:58Z

run buildall

doris-robot · 2024-07-30T03:45:24Z

TPC-H: Total hot run time: 41818 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0bfb8693ace43b9c78522cf93ca8c5ba9213345, data reload: false

------ Round 1 ----------------------------------
q1	17617	4124	4095	4095
q2	2029	211	203	203
q3	10438	1403	1381	1381
q4	10167	831	976	831
q5	7622	3076	2992	2992
q6	216	136	140	136
q7	1045	613	610	610
q8	9448	1717	1967	1717
q9	8613	6659	6681	6659
q10	8758	3813	3877	3813
q11	433	251	254	251
q12	422	231	225	225
q13	17758	2989	2941	2941
q14	269	242	246	242
q15	537	504	492	492
q16	527	388	397	388
q17	986	944	920	920
q18	8070	7311	7341	7311
q19	1487	1226	1215	1215
q20	577	334	340	334
q21	5450	4957	4777	4777
q22	361	286	285	285
Total cold run time: 112830 ms
Total hot run time: 41818 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4057	4024	4024	4024
q2	324	223	218	218
q3	3001	3037	3162	3037
q4	2003	2049	1949	1949
q5	5655	5517	5433	5433
q6	223	132	126	126
q7	2125	1810	1858	1810
q8	3374	3397	3390	3390
q9	8715	8711	8870	8711
q10	3957	4087	3910	3910
q11	560	453	475	453
q12	784	601	612	601
q13	16347	3122	3097	3097
q14	301	289	272	272
q15	532	488	480	480
q16	458	422	429	422
q17	1768	1741	1717	1717
q18	8248	7750	7795	7750
q19	1702	1743	1718	1718
q20	2081	1857	1821	1821
q21	5793	5544	5288	5288
q22	513	465	481	465
Total cold run time: 72521 ms
Total hot run time: 56692 ms

doris-robot · 2024-07-30T03:56:07Z

TPC-DS: Total hot run time: 169771 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0bfb8693ace43b9c78522cf93ca8c5ba9213345, data reload: false

query1	921	404	387	387
query2	6457	1678	1699	1678
query3	6642	208	225	208
query4	20139	17439	17268	17268
query5	3653	517	525	517
query6	293	176	192	176
query7	4602	307	302	302
query8	257	202	196	196
query9	8514	2407	2386	2386
query10	433	277	266	266
query11	10560	10029	10187	10029
query12	123	88	89	88
query13	1691	374	373	373
query14	8654	6956	8539	6956
query15	201	168	165	165
query16	6810	478	466	466
query17	946	560	565	560
query18	1920	282	279	279
query19	188	138	136	136
query20	89	86	85	85
query21	203	102	97	97
query22	4574	4108	4077	4077
query23	33845	33547	33446	33446
query24	10343	3079	3197	3079
query25	685	424	415	415
query26	1775	153	150	150
query27	2972	286	285	285
query28	7673	2065	2020	2020
query29	1345	433	463	433
query30	244	161	152	152
query31	965	775	743	743
query32	101	57	54	54
query33	682	335	331	331
query34	929	491	494	491
query35	877	765	773	765
query36	1022	895	874	874
query37	300	83	78	78
query38	2998	2859	2778	2778
query39	872	797	822	797
query40	260	115	110	110
query41	45	44	40	40
query42	123	101	105	101
query43	483	419	420	419
query44	1189	725	720	720
query45	211	176	180	176
query46	1088	811	778	778
query47	1774	1712	1706	1706
query48	375	293	296	293
query49	951	439	447	439
query50	915	434	431	431
query51	6787	6730	6643	6643
query52	103	88	87	87
query53	252	187	181	181
query54	621	455	456	455
query55	74	73	72	72
query56	266	254	252	252
query57	1157	1051	1067	1051
query58	294	267	267	267
query59	2675	2277	2518	2277
query60	305	272	277	272
query61	99	102	97	97
query62	892	650	676	650
query63	205	180	181	180
query64	5650	1939	1894	1894
query65	3161	3079	3103	3079
query66	1311	331	327	327
query67	15228	15356	14760	14760
query68	4306	580	581	580
query69	438	295	302	295
query70	1159	1024	1109	1024
query71	405	284	285	284
query72	7111	2715	2541	2541
query73	766	338	338	338
query74	6057	5611	5634	5611
query75	3389	2746	2747	2746
query76	2427	1328	1416	1328
query77	431	312	297	297
query78	9454	8898	8890	8890
query79	1957	527	525	525
query80	1189	511	503	503
query81	557	228	225	225
query82	1020	128	127	127
query83	234	170	165	165
query84	276	81	79	79
query85	1323	318	309	309
query86	443	295	313	295
query87	3280	3296	3185	3185
query88	2956	2475	2464	2464
query89	387	294	299	294
query90	1717	190	191	190
query91	124	104	101	101
query92	69	48	50	48
query93	1563	625	624	624
query94	849	288	289	288
query95	380	275	274	274
query96	596	279	277	277
query97	3235	3040	3092	3040
query98	223	197	196	196
query99	1661	1285	1290	1285
Total cold run time: 261974 ms
Total hot run time: 169771 ms

doris-robot · 2024-07-30T04:01:38Z

ClickBench: Total hot run time: 30.21 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0bfb8693ace43b9c78522cf93ca8c5ba9213345, data reload: false

query1	0.05	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.49	0.49
query6	1.15	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.57	0.51	0.51
query10	0.57	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.61	0.61	0.60
query14	0.77	0.80	0.79
query15	0.89	0.86	0.85
query16	0.35	0.35	0.35
query17	1.02	1.01	0.98
query18	0.22	0.21	0.21
query19	1.82	1.69	1.79
query20	0.01	0.00	0.00
query21	15.41	0.77	0.66
query22	4.37	8.50	1.50
query23	17.93	1.35	1.33
query24	2.26	0.23	0.23
query25	0.19	0.08	0.08
query26	0.31	0.22	0.21
query27	0.46	0.23	0.23
query28	13.17	1.00	0.97
query29	12.54	3.40	3.39
query30	0.26	0.06	0.06
query31	2.88	0.41	0.39
query32	3.24	0.47	0.47
query33	2.89	2.97	2.90
query34	15.45	4.22	4.26
query35	4.32	4.29	4.27
query36	0.66	0.48	0.47
query37	0.21	0.18	0.17
query38	0.17	0.15	0.16
query39	0.04	0.04	0.03
query40	0.16	0.14	0.13
query41	0.11	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108 s
Total hot run time: 30.21 s

liaoxin01

LGTM

github-actions · 2024-07-30T06:47:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-30T10:18:27Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-07-30T12:11:58Z

run buildall

github-actions · 2024-07-30T12:17:57Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring

LGTM

github-actions · 2024-07-31T03:48:51Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-05T08:22:24Z

PR approved by at least one committer and no changes requested.

sollhui · 2024-08-05T10:54:25Z

run buildall

github-actions · 2024-08-05T11:01:25Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-08-06T02:28:58Z

run buildall

github-actions · 2024-08-06T02:35:20Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-06T03:30:40Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-08-06T03:36:17Z

TPC-H: Total hot run time: 41893 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce5bd49094ab2d14eb76abda055e5a234c141ec2, data reload: false

------ Round 1 ----------------------------------
q1	17696	4244	4068	4068
q2	2042	208	207	207
q3	10495	1332	1344	1332
q4	10262	809	973	809
q5	7665	3009	3001	3001
q6	228	140	142	140
q7	1073	636	622	622
q8	9431	1980	1989	1980
q9	8458	6607	6604	6604
q10	8784	3884	3874	3874
q11	436	250	257	250
q12	418	247	236	236
q13	17770	2961	2948	2948
q14	272	248	250	248
q15	533	491	496	491
q16	524	399	398	398
q17	991	943	911	911
q18	8093	7345	7210	7210
q19	1515	1258	1237	1237
q20	551	325	346	325
q21	5445	4721	4892	4721
q22	352	281	291	281
Total cold run time: 113034 ms
Total hot run time: 41893 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4302	4054	4058	4054
q2	327	216	227	216
q3	3014	3059	3148	3059
q4	2061	2044	2025	2025
q5	5681	5503	5458	5458
q6	223	136	137	136
q7	2150	1822	1819	1819
q8	3312	3360	3372	3360
q9	8605	8708	8750	8708
q10	3960	4075	3951	3951
q11	565	467	459	459
q12	784	583	605	583
q13	13841	3113	3081	3081
q14	312	275	282	275
q15	548	481	500	481
q16	486	434	416	416
q17	1785	1781	1733	1733
q18	8135	7732	7646	7646
q19	4155	1784	1741	1741
q20	2109	1900	1854	1854
q21	5813	5552	5319	5319
q22	529	461	475	461
Total cold run time: 72697 ms
Total hot run time: 56835 ms

doris-robot · 2024-08-06T03:46:49Z

TPC-DS: Total hot run time: 169717 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce5bd49094ab2d14eb76abda055e5a234c141ec2, data reload: false

query1	933	389	370	370
query2	6480	1740	1727	1727
query3	6663	211	229	211
query4	20224	17605	17529	17529
query5	3656	532	546	532
query6	285	173	184	173
query7	4604	308	289	289
query8	265	204	211	204
query9	8507	2398	2386	2386
query10	441	278	273	273
query11	10432	9937	9836	9836
query12	138	98	88	88
query13	1607	365	368	365
query14	9140	7489	7088	7088
query15	205	160	167	160
query16	6851	493	452	452
query17	932	550	559	550
query18	1904	275	294	275
query19	197	148	149	148
query20	91	85	84	84
query21	210	99	94	94
query22	4165	4074	4102	4074
query23	33856	33762	33484	33484
query24	9065	3105	3007	3007
query25	633	423	398	398
query26	741	154	152	152
query27	2394	283	288	283
query28	5813	2041	2017	2017
query29	920	462	435	435
query30	246	153	154	153
query31	938	750	788	750
query32	100	58	63	58
query33	601	326	332	326
query34	912	496	521	496
query35	896	765	780	765
query36	1036	901	903	901
query37	197	88	87	87
query38	2915	2856	2796	2796
query39	846	809	829	809
query40	202	112	112	112
query41	46	49	45	45
query42	121	102	102	102
query43	458	455	413	413
query44	1140	733	741	733
query45	210	184	179	179
query46	1106	804	801	801
query47	1825	1704	1707	1704
query48	360	286	287	286
query49	839	431	431	431
query50	910	440	443	440
query51	6835	6776	6758	6758
query52	105	93	88	88
query53	262	190	184	184
query54	634	453	452	452
query55	79	80	74	74
query56	278	256	257	256
query57	1116	1037	1015	1015
query58	269	273	269	269
query59	2521	2390	2312	2312
query60	283	277	264	264
query61	97	95	95	95
query62	884	663	650	650
query63	210	199	186	186
query64	4299	1912	1868	1868
query65	3164	3068	3083	3068
query66	848	335	331	331
query67	15105	14739	14701	14701
query68	4313	576	573	573
query69	653	363	313	313
query70	1122	1010	1070	1010
query71	388	302	280	280
query72	7063	2733	2482	2482
query73	768	330	329	329
query74	6060	5648	5587	5587
query75	3368	2737	2720	2720
query76	2179	1241	1298	1241
query77	517	315	324	315
query78	9404	9010	8864	8864
query79	1893	550	561	550
query80	1003	498	581	498
query81	578	228	229	228
query82	1030	135	132	132
query83	261	177	177	177
query84	275	80	79	79
query85	1006	323	308	308
query86	374	310	293	293
query87	3270	3094	3046	3046
query88	2938	2417	2421	2417
query89	390	299	288	288
query90	1651	202	204	202
query91	125	101	101	101
query92	54	50	53	50
query93	1542	631	643	631
query94	761	298	302	298
query95	390	273	275	273
query96	596	284	284	284
query97	3206	3103	3031	3031
query98	230	208	193	193
query99	1623	1282	1295	1282
Total cold run time: 253276 ms
Total hot run time: 169717 ms

doris-robot · 2024-08-06T03:52:19Z

ClickBench: Total hot run time: 29.98 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce5bd49094ab2d14eb76abda055e5a234c141ec2, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.06	0.06
query5	0.49	0.48	0.48
query6	1.15	0.72	0.72
query7	0.03	0.01	0.01
query8	0.04	0.04	0.04
query9	0.57	0.51	0.50
query10	0.56	0.57	0.57
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.63	0.61	0.60
query14	0.77	0.80	0.79
query15	0.87	0.88	0.85
query16	0.35	0.36	0.36
query17	0.96	1.01	0.97
query18	0.21	0.23	0.21
query19	1.86	1.75	1.73
query20	0.01	0.00	0.01
query21	15.40	0.75	0.65
query22	4.73	7.00	1.34
query23	17.94	1.39	1.30
query24	2.24	0.21	0.22
query25	0.18	0.07	0.08
query26	0.32	0.22	0.21
query27	0.47	0.24	0.23
query28	13.16	0.99	0.97
query29	12.51	3.25	3.25
query30	0.24	0.06	0.06
query31	2.89	0.41	0.40
query32	3.26	0.49	0.47
query33	2.92	2.98	2.97
query34	15.45	4.25	4.28
query35	4.25	4.30	4.26
query36	0.69	0.48	0.48
query37	0.18	0.16	0.17
query38	0.16	0.16	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 108.22 s
Total hot run time: 29.98 s

XuJianxu

LGTM

…topic change (#38474) (#39181) pick (#38474) When change routine load job topic from test_topic_before to test_topic_after by ``` ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after"); ``` (test_topic_before has 5 rows and test_topic_after has 1 rows) Exception happened, which cannot consume any data: ``` 2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 ``` It is necessary to reset Kafka progress cache when routine load job topic change.

…topic change (#38474) When change routine load job topic from test_topic_before to test_topic_after by ``` ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after"); ``` (test_topic_before has 5 rows and test_topic_after has 1 rows) Exception happened, which cannot consume any data: ``` 2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 ``` It is necessary to reset Kafka progress cache when routine load job topic change.

…topic change (#38474) (#39528) pick (#38474) When change routine load job topic from test_topic_before to test_topic_after by ``` ALTER ROUTINE LOAD FOR test_topic_change FROM KAFKA("kafka_topic" = "test_topic_after"); ``` (test_topic_before has 5 rows and test_topic_after has 1 rows) Exception happened, which cannot consume any data: ``` 2024-07-29 15:57:28,122 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,123 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,125 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,126 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,128 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,129 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,131 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,133 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,134 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,136 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 2024-07-29 15:57:28,137 WARN (Routine load task scheduler|55) [KafkaRoutineLoadJob.hasMoreDataToConsume():792] Kafka offset fallback. partition: 0, cache offset: 5 get latest of fset: 1, task 16656914-ba0a-465d-8e79-8252b423b0fc, job 16615 ``` It is necessary to reset Kafka progress cache when routine load job topic change.

dataroaring previously approved these changes Jul 29, 2024

View reviewed changes

dataroaring added dev/2.1.x dev/2.0.x labels Jul 29, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024

github-actions bot added the reviewed label Jul 29, 2024

sollhui dismissed dataroaring’s stale review via d0bfb86 July 30, 2024 02:44

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 30, 2024

liaoxin01 previously approved these changes Jul 30, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024

sollhui marked this pull request as draft July 30, 2024 07:09

sollhui dismissed liaoxin01’s stale review via 3e4bf96 July 30, 2024 10:12

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 30, 2024

sollhui marked this pull request as ready for review July 30, 2024 12:11

dataroaring previously approved these changes Jul 31, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2024

reset Kafka progress cache when routine load job topic change

ce5bd49

sollhui dismissed liaoxin01’s stale review via ce5bd49 August 6, 2024 02:28

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 6, 2024

liaoxin01 approved these changes Aug 6, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024

XuJianxu approved these changes Aug 6, 2024

View reviewed changes

dataroaring merged commit 1f2f74d into apache:master Aug 6, 2024
29 of 31 checks passed

sollhui mentioned this pull request Aug 10, 2024

[fix](routine load) reset Kafka progress cache when routine load job topic change (#38474) #39181

Merged

yiguolei added dev/2.1.6-merged and removed dev/2.1.x labels Aug 10, 2024

dataroaring added the dev/3.0.2-merged label Aug 12, 2024

sollhui mentioned this pull request Aug 18, 2024

[fix](routine load) reset Kafka progress cache when routine load job topic change (#38474) #39528

Merged

dataroaring added dev/2.0.15-merged and removed dev/2.0.x labels Aug 26, 2024

yiguolei mentioned this pull request Sep 5, 2024

Release Note 2.1.6 #40443

Open

3 tasks

gavinchou mentioned this pull request Oct 13, 2024

Release Note 3.0.2 #41558

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](routine load) reset Kafka progress cache when routine load job topic change #38474

[fix](routine load) reset Kafka progress cache when routine load job topic change #38474

sollhui commented Jul 29, 2024

doris-robot commented Jul 29, 2024

sollhui commented Jul 29, 2024

doris-robot commented Jul 29, 2024

doris-robot commented Jul 29, 2024

doris-robot commented Jul 29, 2024

dataroaring left a comment

github-actions bot commented Jul 29, 2024

github-actions bot commented Jul 29, 2024

sollhui commented Jul 30, 2024

doris-robot commented Jul 30, 2024

doris-robot commented Jul 30, 2024

doris-robot commented Jul 30, 2024

liaoxin01 left a comment

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

sollhui commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

dataroaring left a comment

github-actions bot commented Jul 31, 2024

github-actions bot commented Aug 5, 2024

sollhui commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

sollhui commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

XuJianxu left a comment

[fix](routine load) reset Kafka progress cache when routine load job topic change #38474

[fix](routine load) reset Kafka progress cache when routine load job topic change #38474

Conversation

sollhui commented Jul 29, 2024

doris-robot commented Jul 29, 2024

sollhui commented Jul 29, 2024

doris-robot commented Jul 29, 2024

doris-robot commented Jul 29, 2024

doris-robot commented Jul 29, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 29, 2024

github-actions bot commented Jul 29, 2024

sollhui commented Jul 30, 2024

doris-robot commented Jul 30, 2024

doris-robot commented Jul 30, 2024

doris-robot commented Jul 30, 2024

liaoxin01 left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

sollhui commented Jul 30, 2024

github-actions bot commented Jul 30, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 31, 2024

github-actions bot commented Aug 5, 2024

sollhui commented Aug 5, 2024

github-actions bot commented Aug 5, 2024

sollhui commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

github-actions bot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

doris-robot commented Aug 6, 2024

XuJianxu left a comment

Choose a reason for hiding this comment