LOAD DATA LOCAL INFILE fails when row size exceeds 64K (65535 bytes) limit

[TianyuZhang1214]

Bug Description

When executing the following command:

mysql> LOAD DATA LOCAL INFILE 't_b.csv' INTO TABLE t_b FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (id, real_sql);
ERROR 1105 (HY000): bufio.Scanner: token too long

An error occurs when the row length exceeds 64K (65535 bytes). The error appears to be related to the use of bufio.Scanner in the Go MySQL server implementation.

---

Steps to Reproduce

1. Create the table using the following schema:

   CREATE DATABASE t_d;
   USE t_d;
   CREATE TABLE `t_b` (
     `id` bigint NOT NULL,
     `real_sql` text NOT NULL,
     PRIMARY KEY (`id`)
   );

2. Prepare a CSV file (t_b.csv) with content similar to this:

   1,AAAAAA (where the length of 'AAAAAA' exceeds 65535 characters)
   

3. Execute the LOAD DATA LOCAL INFILE statement:

   LOAD DATA LOCAL INFILE 't_b.csv' INTO TABLE t_b FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (id, real_sql);

---

Possible Cause
The issue appears to be caused by the use of bufio.Scanner, which imposes a token size limit of 64K, as seen in the Go MySQL server code, particularly in ddl_iters.go.

[timsehn]

@jycor will look at this today.

[jycor]

Hey @TianyuZhang1214, thanks for reporting this issue.
We've increased the scanner token limit to 4GB (the max size of a LongText); hopefully, you don't have any rows larger than that.

Also, Text type does not hold 65535 bytes, you'll want to change the type to LongText.

The fix has been merged to gms, and is making its wait to dolt main. Expect a dolt release early next week.

[TianyuZhang1214]

Hey @TianyuZhang1214, thanks for reporting this issue.

We've increased the scanner token limit to 4GB (the max size of a LongText); hopefully, you don't have any rows larger than that.

Also, Text type does not hold 65535 bytes, you'll want to change the type to LongText.

The fix has been merged to gms, and is making its wait to dolt main. Expect a dolt release early next week.

Thank you for addressing the issue so promptly! I apologize for overlooking the text length limit in the reported case. In our original table, we are using two columns with the TEXT data type.