Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

readline unexpected behavior #42581

Closed
batrudinych opened this issue Apr 2, 2022 · 7 comments
Closed

readline unexpected behavior #42581

batrudinych opened this issue Apr 2, 2022 · 7 comments
Labels
readline Issues and PRs related to the built-in readline module.

Comments

@batrudinych
Copy link

batrudinych commented Apr 2, 2022

Version

10.24.1, 12.22.10, 14.19.1

Platform

Linux l1 5.15.25-1-MANJARO #1 SMP PREEMPT Wed Feb 23 14:44:03 UTC 2022 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

  1. create in.txt file with the following contents:
1
  1. create a test.js file in the same directory and paste the following code snippet:
'use strict'

const fs = require('fs')
const readline = require('readline')

;(async function () {
  const inputFileName = './in.txt'
  const outputFileName = './out.txt'
  const inputReadStream = fs.createReadStream(inputFileName)
  const rl = readline.createInterface({
    input: inputReadStream,
    crlfDelay: Infinity
  })

  fs.open(outputFileName, 'w', (err, outFd) => {
    fs.appendFile(outFd, '0', async () => {
      for await (const line of rl) {
        fs.appendFileSync(outFd, line)
      }
      fs.close(outFd, () => {
        console.log('closed')
      })
    })
  })
})()
  1. execute the snippet: node test.js
  2. open out.txt file and verify the contents

How often does it reproduce? Is there a required condition?

always

What is the expected behavior?

the out.txt should contain 01

What do you see instead?

the out.txt file contains 0

Additional information

Script produces expected result with Node.js 16.14.2

Script returns expected results in case if synchronous versions of fs module functions are used

Additionally, if you modify the script in the following way, you'll get the expected result:

'use strict'

const fs = require('fs')
const readline = require('readline')

;(async function () {
  const inputFileName = './in.txt'
  const outputFileName = './out.txt'
  const inputReadStream = fs.createReadStream(inputFileName)

  fs.open(outputFileName, 'w', (err, outFd) => {
    fs.appendFile(outFd, '0', async () => {
      const rl = readline.createInterface({
        input: inputReadStream,
        crlfDelay: Infinity
      })
      for await (const line of rl) {
        fs.appendFileSync(outFd, line)
      }
      fs.close(outFd, () => {
        console.log('closed')
      })
    })
  })
})()

readline interface is created after the append operation.
resulting contents of out.txt:

01
@VoltrexKeyva VoltrexKeyva added the readline Issues and PRs related to the built-in readline module. label Apr 3, 2022
@aduh95
Copy link
Contributor

aduh95 commented Apr 3, 2022

The docs say:

node/doc/api/readline.md

Lines 466 to 468 in 059b890

`readline.createInterface()` will start to consume the input stream once
invoked. Having asynchronous operations between interface creation and
asynchronous iteration may result in missed lines.

The reason your original snippet doesn't work is that the stream has already completed once you start iterating over the readline interface.
FWIW there's no need to await fs.appendFileSync, as its name implies it's synchronous, so you should get better result without the await.
You might also be interested in the FS promise API, which may improve the readability of your code:

'use strict'

const fs = require('node:fs/promises')
const readline = require('node:readline')

;(async function () {
  const inputFileName = './in.txt'
  const outputFileName = './out.txt'

  let inFh, outFh
  try {
    inFh = await fs.open(inputFileName)
    outFh = await fs.open(outputFileName, 'w')

    await outFh.appendFile('0')

    const rl = readline.createInterface({
        input: inFh.createReadStream(),
        crlfDelay: Infinity,
    })
    await outFh.appendFile(rl)
  } finally {
    await Promise.all([inFh?.close(), outFh?.close()])
    console.log('closed')
  }
})()

@batrudinych
Copy link
Author

hey @aduh05

@batrudinych
Copy link
Author

hmm @aduh95 why does this work just fine in node v16.x then? Any ideas?

@aduh95
Copy link
Contributor

aduh95 commented Apr 3, 2022

hmm @aduh95 why does this work just fine in node v16.x then? Any ideas?

Not sure I understand your question, what do you mean by "this"?

@batrudinych
Copy link
Author

Not sure I understand your question, what do you mean by "this"?

I mean that the code snippet given in the description of this issue produces expected result in node v16 but fails in v10 - v14. However, v16 docs have the same statement:

`readline.createInterface()` will start to consume the input stream once 
 invoked. Having asynchronous operations between interface creation and 
 asynchronous iteration may result in missed lines.

@aduh95
Copy link
Contributor

aduh95 commented Apr 3, 2022

On v15.x the FS stream implementation has changed quite significantly, which probably explains the behavior change. But you shouldn't rely on this behavior, it's just a race condition, if you add a await process.nextTick before the fs.open call, the readline interface will have the time to consume the stream and you get the same behavior as on v14.x.

@batrudinych
Copy link
Author

On v15.x the FS stream implementation has changed quite significantly, which probably explains the behavior change. But you shouldn't rely on this behavior, it's just a race condition, if you add a await process.nextTick before the fs.open call, the readline interface will have the time to consume the stream and you get the same behavior as on v14.x.

I see, thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
readline Issues and PRs related to the built-in readline module.
Projects
None yet
Development

No branches or pull requests

3 participants