readline sometimes produces lines with \r in them

[alecmev]

Version

19.3.0

Platform

6.1.1-arch1-1 x86_64 GNU/Linux

Subsystem

readline

What steps will reproduce the bug?

Unfortunately, the files readline has trouble with are proprietary and I can't share them. I have some ideas on how to craft one, but it would take some time, so I'd like to see first if it will be needed at all.

How often does it reproduce? Is there a required condition?

Consistently. Isn't present in 19.2.0 and earlier.

What is the expected behavior?

No \r in any of the lines.

What do you see instead?

Some of the files (not all) produce 1-3 lines with an \r in it. In one file it happens 21870 lines in (109243 preceding characters), in another 75432, in another 120396. I see no pattern, but it's deterministic.

Additional information

Here are a few lines from one of the files:

  0
VERTEX
  8
14
 10
22.0936
 20
12.9141
  0
VERTEX
  8
14
 10
22.8017
 20
12.4141
  0
VERTEX
  8
14
 10
23.6255
 20
11.9128
  0

The lines are \r\n-delimited. Could be caused by #45614? Not sure why only a couple of lines get affected.

[aduh95]

Do you set crlfDelay option? The recommended value for files is Infinity as shown in the doc example. FWIW if you use filehandle.readLines(), you won't have to worry about this.

[alecmev]

Thanks for looking into this! I do have crlfDelay set to Infinity. I'm using createInterface because I don't want to restrict the library to just files. The source is a simple createReadStream.

[aduh95]

Can you open a PR to add a failing test case to test/known_issues? Or if you’re able to fix the issue yourself that’s even better of course, but having a test would already be a big help :)

[alecmev]

Failing test added: #46075

[aduh95]

Interestingly, it "just" works if the input file has 0x3333 lines, but fails as soon as it has 0x3334 or more. I'm investigating.

The lines are each 5-byte long in the test case, and 0x3333*5 = 0xffff. If I change the test to use 4-byte long lines, it no longer reproduce. My guess is that having a carriage return on position 0xffff is problematic. 0xffff = (64 * 1024) - 1, and 64*1024 is the default highWaterMark according to https://nodejs.org/api/fs.html#fscreatereadstreampath-options.

By using fs.createReadStream(filepath, { highWaterMark: 6 }), I'm able to reproduce the issue from the first second line.

EDIT: I've simplified the test case to be minimal, and opened #46306 with a fix.