Node read stream hangs on specific file, base64 encoding

[jan-j]

• Version: 7.9.0/8.9.3/9.7.1
• Platform: Windows/Linux/MacOS

I have following code working for every file except the one that keeps hanging without emitting end or error events (I tried other stream events too).

const fs = require('fs');

const rs = fs.createReadStream(filePath, {
    encoding: 'base64',
});

rs.on('data', () => {
    console.log('data');
});
rs.on('end', () => {
    console.log('end');
});
rs.on('error', e => {
    console.log('error', e);
});

If I move read point with start option to 1 instead of 0 it works properly. Same if highWaterMark is set to value other than default. It doesn't really help as it seems it can fail with other "corrupted" file.

It seems like Node bug, but maybe there's something I'm missing here.

Here's file to recreate the issue:
http://s3.eu-west-1.amazonaws.com/jjapitest/file

Here's interactive demo of the issue:
https://repl.it/repls/AnimatedDisguisedNumerator

[addaleax]

Fwiw, this seems to work on master but not on the other release lines, so maybe it’s just about figuring out which change fixed this.

[marcosc90]

It will fail If highWaterMark is file.length - 1, and encoding base64, at least on v8.x.

In the following snippet, not even data is emitted.

const fs = require('fs');
const invalid = '/tmp/invalid';
fs.writeFileSync(invalid, '1234');

function readFileBase64(filePath) {
  const rs = fs.createReadStream(filePath, {
      encoding: 'base64',
      highWaterMark: 2 // with 3 data is called, but end no
  });

  rs.on('data', (data) => {
      console.log(filePath, 'data', Buffer.from(data, 'base64').toString());
      // Nothing for hwm == 2
      // 123 for hwm == 3
      // 123 & 4 for hwm == 4 == file.length
  });
  rs.on('end', () => {
      console.log(filePath, 'end');
  });
  rs.on('error', e => {
      console.log(filePath, 'error', e);
  });
}

readFileBase64(invalid);

This works fine on v10.1.0

I'm willing to track down the changes on master to fix this issue.

[marcosc90]

@addaleax I found the issue, and already solved it for v8.x. I haven't got the time to go through all the code in _stream_readable.js, so if someone with more expertise in the subject care to take a look at my code, to make sure nothing else would break, even though all tests are passing, that would be great.

[mcollina]

cc @nodejs/streams

[mcollina]

I fear this might not be fixable in Node 6, 8, 9 without backporting a semver-major change: #18994

[antsmartian]

Are we planning to backport?

[mcollina]

@antsmartian I would not recommend it, as it could actually break things.

[mcollina]

This is fixed in Node 10, I'm closing this.

[antsmartian]

Thanks for the update @mcollina