lightning import failed after inject pdleader io delay 1s

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、lightning import
- storage: s3://benchmark/location-dumpling
s3Endpoint: http://minio.pingcap.net:9000
db: location
options: --enable-checkpoint false
config: |-
[lightning]
max-error = 9999999999
[tikv-importer]
duplicate-resolution = "record"
incremental-import = true
crc32Checksum: 2645404411
adminCheck:
- table: AreaMeshLake
indexes:
- areameshlakeid
backend: local
sortedKvDir: /tmp/sorted-kv-dir
- storage: s3://brie/lightning-mhy-100G-csv
s3Endpoint: http://minio.pingcap.net:9001
db: sysbench
backend: local
sortedKvDir: /tmp/sorted-kv-dir
options: "--enable-checkpoint false"
config: "[[mydumper.files]]\npattern = "sysbench-schema-create.sql"\nschema = "sysbench"\ntype = "schema-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1-schema.sql"\nschema = "sysbench"\ntable = "user_data1"\ntype = "table-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1.000000[0-3][0-9][0-9].csv"\nschema = "sysbench"\ntable = "user_data1"\ntype = "csv""
crc32Checksum: 619903217
2、inject pdleader io delay 1s
- name: ha_pdleader_io_delay(1s)
faultType: iochaos_io_delay(volumePath:/var/lib/pd,path:/var/lib/pd/data/**/*,delay:1s,percent:100)
selector: pdleader
warmUpTime: 1m
period: "@every 3m"
faultDuration: 1m
faultTotalRunTime: 20m
lightningList: *id001

2. What did you expect to see? (Required)

lightning import can success

3. What did you see instead (Required)

lightning import failed after inject pdleader io delay 1s

tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table sysbench.user_data1 failed: Post "http://tc-pd:2379/pd/api/v1/config/region-label/rule": http: ContentLength=274 with Body length 0

github.com/pingcap/errors.AddStack
	/go/pkg/mod/github.com/pingcap/[email protected]/errors.go:174
github.com/pingcap/errors.Trace
	/go/pkg/mod/github.com/pingcap/[email protected]/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/pdutil.pdRequestWithCode
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:203
github.com/pingcap/tidb/br/pkg/pdutil.pdRequest
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:154
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).CreateOrUpdateRegionLabelRule
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:983
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).pauseSchedulerByKeyRangeWithTTL
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1048
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).PauseSchedulersByKeyRange
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1027
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).ImportEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1523
github.com/pingcap/tidb/br/pkg/lightning/backend.(*ClosedEngine).Import
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/backend.go:365
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importKV
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:1242
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:891
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngines.func3
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:504
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.5.0-alpha
Edition: Community
Git Commit Hash: af7b32c
Git Branch: heads/refs/tags/v7.5.0-alpha
UTC Build Time: 2023-10-19 11:43:11
GoVersion: go1.21.3
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-10-20T08:01:59.525+0800

[Lily2025]

/severity major
/assign lance6716

[lance6716]

I guess it's caused by

tidb/br/pkg/pdutil/pd.go

Lines 167 to 174 in 05b50a1

	req, err := http.NewRequestWithContext(ctx, method, reqURL, body)
	if err != nil {
	return 0, nil, errors.Trace(err)
	}
	var resp *http.Response
	count := 0
	for {
	resp, err = cli.Do(req) //nolint:bodyclose

client.Do will close req.Body, so in second retry body size is 0.