global sort import 1T data fail for "[0]runtime error: makeslice: len out of range"

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. set global tidb_cloud_storage_uri="s3://xxx"
2. import into globalsort.item_core_2 FROM "s3://xxx&force-path-style=false&region=Beijing&provider=ks" with thread=16,detached;

2. What did you expect to see? (Required)

import successfully

3. What did you see instead (Required)

+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+----------+-----------+--------+------------------+---------------+-----------------------------------------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                                                                                                                                     | Target_Table               | Table_ID | Phase     | Status | Source_File_Size | Imported_Rows | Result_Message                                | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+----------+-----------+--------+------------------+---------------+-----------------------------------------------+----------------------------+----------------------------+----------------------------+------------+
| 150001 | s3://qe-testing/xxx/10T/data3/test.item_core.*.csv?access-key=xxxxxx&endpoint=xxx&force-path-style=false&provider=ks&region=Beijing&secret-access-key=xxxxxx | `globalsort`.`item_core_2` |      163 | importing | failed | 1.044TB          |          NULL | [0]runtime error: makeslice: len out of range | 2023-10-12 10:54:49.935207 | 2023-10-12 10:54:51.070273 | 2023-10-12 13:49:36.077075 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+----------+-----------+--------+------------------+---------------+-----------------------------------------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (0.01 sec)

MySQL [(none)]> 

4. What is your TiDB version? (Required)

3eadac4

[D3Hunter]

panic stack, seems the length is too long or negative
related pr #47288, the hash is the last commit of it(except merge master), so seems not fixed

[2023/10/12 13:49:35.694 +08:00] [ERROR] [scheduler.go:107] ["BaseScheduler panicked"] [recover="runtime error: makeslice: len out of range"] [stack="github.com/pingcap/tidb/disttask/framework/scheduler.(*BaseScheduler).Run.func1
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/scheduler.go:107
runtime.gopanic
    /usr/local/go/src/runtime/panic.go:914
runtime.panicmakeslicelen
    /usr/local/go/src/runtime/slice.go:29
runtime.makeslicecopy
    /usr/local/go/src/runtime/slice.go:44
github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*byteReader).readNBytes
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/byte_reader.go:143
github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*kvReader).nextKV
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/kv_reader.go:59
github.com/pingcap/tidb/br/pkg/lightning/backend/external.kvReaderProxy.next
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:269
github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*mergeIter[...]).next
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:219
github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*MergeKVIter).Next
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/iter.go:323
github.com/pingcap/tidb/br/pkg/lightning/backend/external.(*Engine).LoadIngestData
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/external/engine.go:128
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateAndSendJob
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/local/local.go:1195
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).prepareAndSendJob
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/local/local.go:1148
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).doImport
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/local/local.go:1675
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).ImportEngine
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/br/pkg/lightning/backend/local/local.go:1550
github.com/pingcap/tidb/disttask/importinto.(*writeAndIngestStepExecutor).RunSubtask
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/importinto/scheduler.go:389
github.com/pingcap/tidb/disttask/framework/scheduler.(*BaseScheduler).runSubtask
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/scheduler.go:237
github.com/pingcap/tidb/disttask/framework/scheduler.(*BaseScheduler).run
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/scheduler.go:231
github.com/pingcap/tidb/disttask/framework/scheduler.(*BaseScheduler).Run
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/scheduler.go:115
github.com/pingcap/tidb/disttask/importinto.(*importScheduler).Run
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/importinto/scheduler.go:466
github.com/pingcap/tidb/disttask/framework/scheduler.(*Manager).onRunnableTask
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/manager.go:342
github.com/pingcap/tidb/disttask/framework/scheduler.(*Manager).onRunnableTasks.func1
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/disttask/framework/scheduler/manager.go:211
github.com/pingcap/tidb/resourcemanager/pool/spool.(*Pool).run.func1
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tidb/resourcemanager/pool/spool/spool.go:145"]