add index status was always running and can not be cancelled due to dead lock

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb_enable_dist_task='on'
1、run workload
2、add index for one table
3、inject one of tikv network partition

case config
tag: "ha-test-add-index"
workloads:
- type: sysbench
storage: s3://benchmark/sysbench_64_7000w
db: sysbench_64_7000w
tableNum: 64
tableSize: 70000000
threads: 64
subType: oltp_read_write
ignoreErrors: "all"
cases:
- name: "ha_tikv(random)_to_all_network_partition"
warmUpTime: 1m
faultType: network_partition
selector: "tikv(random)_to_all"
period: "@every 5m"
faultDuration: 3m
faultTotalRunTime: 30m
ddl:
addIndex:
- db: "sysbench_64_7000w"
table: "sbtest1"
timeout: 60m
times: 5
index_col_name: "c"
job_type: "ingest"
admin_check: false
count_check: false

2. What did you expect to see? (Required)

add index can success

3. What did you see instead (Required)

add index status was always running due to dead lock

the status of ddl job is not synced after 1h0m0s (now: 2023-11-04 09:14:24, jobId: 475, job type: add index /* ingest /, state: running)
operatorLogs:
[2023-11-04 08:14:10] ###### start adding index
alter table sbtest1 add index index_test_1699056850827 (c)
[2023-11-04 08:14:10] ###### wait for ddl job finish
[2023-11-04 09:14:24] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1699056850827 (c)'
jobId: 475, job type: add index / ingest */, state: running

the status of ddl job is not synced or running (now: 2023-11-04 09:15:32, jobId: 476, job type: add index, state: queueing)
operatorLogs:
[2023-11-04 09:15:02] ###### start adding index
alter table sbtest1 add index index_test_1699060502481 (c)
[2023-11-04 09:15:02] ###### wait for ddl job finish
[2023-11-04 09:15:32] ###### wait for import job to finish failed
select job_id, job_type, state from information_schema.ddl_jobs where query = 'alter table sbtest1 add index index_test_1699060502481 (c)'
jobId: 476, job type: add index, state: queueing

4. What is your TiDB version? (Required)

git hash：aa930910eb2316bec085ca235cb9f11fe8c7ccfa

[Lily2025]

/assign ywqzzy

[ywqzzy]

[ywqzzy]

also related to #48352

[ywqzzy]

also related to #46321

[Lily2025]

/remove-severity major
/severity critical

[Lily2025]

/remove-label may-affects-6.5
/remove-label may-affects-7.1
/remove-label may-affects-6.1
/remove-label may-affects-5.4
/remove-label may-affects-5.3