when dumpling to s3, failed to close file causes data file lost

[D3Hunter]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[2023/07/12 16:37:37.070 +00:00] [WARN] [writer_util.go:507] ["fail to close file"] [path=s3://xxx/xxx.0003288560000.sql] [error="RequestError: send request failed
caused by: Put \"https://bucket.xxxxx.s3.ap-northeast-1.vpce.amazonaws.com/this-is-the-redacted-file-name.sql?partNumber=19......": read tcp 10.25.5.0:38834->10.25.116.244:443: read: connection reset by peer"] [errorVerbose="RequestError: send request failed
caused by: Put \"https://bucket.xxxxx.s3.ap-northeast-1.vpce.amazonaws.com/this-is-the-redacted-file-name.sql?partNumber=19......": read tcp 10.25.5.0:38834->10.25.116.244:443: read: connection reset by peer
github.com/pingcap/errors.AddStack
    github.com/pingcap/[email protected]/errors.go:174
github.com/pingcap/errors.Trace
    github.com/pingcap/[email protected]/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/storage.(*S3Uploader).Write
    github.com/pingcap/tidb/br/pkg/storage/s3.go:110
github.com/pingcap/tidb/br/pkg/storage.(*bufferedWriter).uploadChunk
    github.com/pingcap/tidb/br/pkg/storage/writer.go:214
github.com/pingcap/tidb/br/pkg/storage.(*bufferedWriter).Close
    github.com/pingcap/tidb/br/pkg/storage/writer.go:220
github.com/pingcap/tidb/dumpling/export.buildInterceptFileWriter.func2
    github.com/pingcap/tidb/dumpling/export/writer_util.go:505
github.com/pingcap/tidb/dumpling/export.(*Writer).tryToWriteTableData
    github.com/pingcap/tidb/dumpling/export/writer.go:240
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData.func1
    github.com/pingcap/tidb/dumpling/export/writer.go:224
github.com/pingcap/tidb/br/pkg/utils.WithRetry
    github.com/pingcap/tidb/br/pkg/utils/retry.go:53
github.com/pingcap/tidb/dumpling/export.(*Writer).WriteTableData
    github.com/pingcap/tidb/dumpling/export/writer.go:192
github.com/pingcap/tidb/dumpling/export.(*Writer).handleTask
    github.com/pingcap/tidb/dumpling/export/writer.go:115
github.com/pingcap/tidb/dumpling/export.(*Writer).run
    github.com/pingcap/tidb/dumpling/export/writer.go:93
github.com/pingcap/tidb/dumpling/export.(*Dumper).startWriters.func4
    github.com/pingcap/tidb/dumpling/export/dump.go:356
golang.org/x/sync/errgroup.(*Group).Go.func1
    golang.org/x/[email protected]/errgroup/errgroup.go:75
runtime.goexit
    runtime/asm_amd64.s:1594"]

in here, we didn't retry on close err, cause the file lost

tidb/dumpling/export/writer.go

Line 244 in 2af9a67

tearDown(tctx)

2. What did you expect to see? (Required)

all data file dumped to s3

3. What did you see instead (Required)

2 file lost

4. What is your TiDB version? (Required)

6.5.2