repl: fix .load infinite loop caused by shared use of lineEnding RegExp

[Theo-Steiner]

repl: fix .load infinite loop caused by shared use of lineEnding RegExp

Since the lineEnding is declared on the module scope of lib/internal/readline/interface.js,
recursive invocations of its [kTtyWrite] method share one instance of this Regular Expression.
Since the state of a RegExp is managed by instance, alternately calling RegExpPrototypeExec
with the same RegExp on different strings can lead to the state changing unexpectedly.
This was the root cause of this infinite loop bug when calling .load on javascript files of certain shapes.

Fixes: #46731

I attempted to write a test for this, mimicking test/parallel/test-repl-load-multiline.js.
However, I failed reproducing the infinite loop bug from the programatic REPL, as the problematic indentation preservation is not executed for some reason I have yet to understand. I thought setting terminal: true might produce the intended result, but I had no luck with it for now.
If anyone has ideas for how to test for this, please let me know!

Edit 4: I found a solution to the core issue and updated this PR accordingly
Edit 5: also found a way to test this

[Theo-Steiner]

To be honest, I am not quite sure, why calling .load puts the REPL into editorMode in the first place.
If that could be safely removed, that might be an even better option.

[tniessen]

cc @nodejs/repl

[Theo-Steiner]

#46731 (comment)
I found that the actual problem that triggers this is with reusing the lineEnding RegExp.
Do you think that using String.prototype.matchAll with this RegExp might be a better solution than disabling the editor mode?

EDIT: Turns out avoiding using methods that depend on the RegExp prototype was what introduced this bug in the first place. #36593 -> #45614

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/49875/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/49901/

[aduh95]

Hi, thanks for investigating that and sending a PR. It seems that the added test already passes on main, which makes me think that it's not able to reproduce the issue. Could you have a look?

[Theo-Steiner]

It seems that the added test already passes on main, which makes me think that it's not able to reproduce the issue. Could you have a look?

Yes, I did not manage to reproduce the infinite loop in a test. The problem is that the infinite loop only occurs if the code block starting on this line

node/lib/repl.js

Line 858 in b85b5ba

if (self.editorMode) {

is entered. For some reason the test does not enter this block. Probably there is some difference between the programmatic REPL and the interactive one at play here?
I've tried passing {terminal: true} to the REPL, but that did not do it either.
Do you have an idea for what could be the difference between programmatic and interactive REPL?
I won't be able to further investigate this until Monday, I hope that is alright.

[aduh95]

I won't be able to further investigate this until Monday, I hope that is alright.

Of course it's alright, take all the time you need! Node.js is striving from free contributions from contributors like you and me, the only expectations we set is to comply to the Code of Conduct, we don't have any expectations regarding the quantity or the rapidity of work :)

[aduh95]

I'm not sure if it's going to perform better, but the bug can be resolved with a more minimal diff:

diff --git a/lib/internal/readline/interface.js b/lib/internal/readline/interface.js
index 45524612ae..f69d6b14b0 100644
--- a/lib/internal/readline/interface.js
+++ b/lib/internal/readline/interface.js
@@ -1327,9 +1327,11 @@ class Interface extends InterfaceConstructor {
               this[kInsertString](StringPrototypeSlice(s, 0, nextMatch.index));
               let { lastIndex } = lineEnding;
               while ((nextMatch = RegExpPrototypeExec(lineEnding, s)) !== null) {
+                const previousLastIndex = lineEnding.lastIndex;
                 this[kLine]();
                 this[kInsertString](StringPrototypeSlice(s, lastIndex, nextMatch.index));
-                ({ lastIndex } = lineEnding);
+                lastIndex = previousLastIndex;
+                lineEnding.lastIndex = previousLastIndex;
               }
               if (lastIndex === s.length) this[kLine]();
             } else {

When it comes to the repro, you may have more luck with trying to reproduce the bug in node:readline/promises rather than node:repl, REPL adds quite a bit of complexity that you can avoid by hitting directly the source of the bug. Let me know if you need more help with the issue.

[Theo-Steiner]

@aduh95 thank you for your advice.
Using it I managed to write a test using readline's interface that shows the underlying problem!
It fails for everything after 18.12.0 and is fixed by this PR.

I intentionally changed the implementation to use a separate instance of the RegExp, since I thought a similar problem could occur in the future if we fix it by just shuffling the variable declarations around.
The side effects of RegExpPrototypeExec are very counterintuitive to me, so I thought other developers might overlook them too. That's why I refactored the code to be (1) more obvious in what it does and (2) prevent errors caused by alternately calling RegExpPrototypeExec with the same RegExp and a different string.

If keeping diffs small is desirable however, I totally understand and will make the changes as suggested by you. Just let me know how to proceed 🥳

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/50055/

[Theo-Steiner]

I added a line to reset the RegExp's lastIndex property to 0 before commencing the search.
I think now all tests should pass!

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/50125/

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/46742
✔  Done loading data for nodejs/node/pull/46742
----------------------------------- PR info ------------------------------------
Title      repl: fix .load infinite loop caused by shared use of lineEnding RegExp (#46742)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     Theo-Steiner:fix/repl-infinite-loop -> nodejs:main
Labels     readline, repl, author ready, needs-ci
Commits    12
 - repl: fix .load infinite loop caused by shared use of lineEnding RegExp
 - test: add test that fails without the changes of #46742
 - chore: remove unrelated comment
 - chore: fix linting errors and rename test
 - remove magic number from test
 - fix: refactor to use module scope instance of RegExp safely for perfo…
 - chore: fix linting errors
 - docs: update comments
 - chore: make finishing touches on comments
 - test: fix test failing due to difference in terminal codes
 - Revert "test: fix test failing due to difference in terminal codes"
 - fix: erase lastIndex state for previous searches
Committers 2
 - Theodor Steiner 
 - GitHub 
PR-URL: https://github.com/nodejs/node/pull/46742
Fixes: https://github.com/nodejs/node/issues/46731
Reviewed-By: Kohei Ueno 
Reviewed-By: Antoine du Hamel 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/46742
Fixes: https://github.com/nodejs/node/issues/46731
Reviewed-By: Kohei Ueno 
Reviewed-By: Antoine du Hamel 
--------------------------------------------------------------------------------
   ℹ  This PR was created on Mon, 20 Feb 2023 11:54:47 GMT
   ✔  Approvals: 2
   ✔  - Kohei Ueno (@cola119): https://github.com/nodejs/node/pull/46742#pullrequestreview-1319782434
   ✔  - Antoine du Hamel (@aduh95) (TSC): https://github.com/nodejs/node/pull/46742#pullrequestreview-1319512073
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2023-03-01T08:33:26Z: https://ci.nodejs.org/job/node-test-pull-request/50125/
   ℹ  Last Benchmark CI on 2023-02-27T19:56:39Z: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1299/
- Querying data for job/node-test-pull-request/50125/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  No git cherry-pick in progress
   ✔  No git am in progress
   ✔  No git rebase in progress
--------------------------------------------------------------------------------
- Bringing origin/main up to date...
From https://github.com/nodejs/node
 * branch                  main       -> FETCH_HEAD
✔  origin/main is now up-to-date
- Downloading patch for 46742
From https://github.com/nodejs/node
 * branch                  refs/pull/46742/merge -> FETCH_HEAD
✔  Fetched commits as da0bc6db98ce..395054215dc3
--------------------------------------------------------------------------------
[main b230fffbcf] repl: fix .load infinite loop caused by shared use of lineEnding RegExp
 Author: Theodor Steiner 
 Date: Mon Feb 20 20:47:33 2023 +0900
 3 files changed, 66 insertions(+), 10 deletions(-)
 create mode 100644 test/parallel/test-repl-load-multiline-function.js
[main 645a7b54da] test: add test that fails without the changes of #46742
 Author: Theodor Steiner 
 Date: Mon Feb 27 17:23:13 2023 +0900
 2 files changed, 31 insertions(+), 47 deletions(-)
 create mode 100644 test/parallel/test-readline-recursive-writes.js
 delete mode 100644 test/parallel/test-repl-load-multiline-function.js
[main 12cec6217c] chore: remove unrelated comment
 Author: Theodor Steiner 
 Date: Mon Feb 27 17:29:53 2023 +0900
 1 file changed, 1 deletion(-)
[main ac24d34df7] chore: fix linting errors and rename test
 Author: Theodor Steiner 
 Date: Mon Feb 27 17:33:58 2023 +0900
 1 file changed, 4 insertions(+), 5 deletions(-)
 rename test/parallel/{test-readline-recursive-writes.js => test-readline-interface-recursive-writes.js} (86%)
[main 8029f97d09] remove magic number from test
 Author: Theodor Steiner <[email protected]>
 Date: Tue Feb 28 09:39:32 2023 +0900
 1 file changed, 8 insertions(+), 5 deletions(-)
[main 8ec5286f41] fix: refactor to use module scope instance of RegExp safely for performance considerations
 Author: Theodor Steiner 
 Date: Tue Feb 28 15:16:02 2023 +0900
 1 file changed, 3 insertions(+), 4 deletions(-)
[main 7871c71f83] chore: fix linting errors
 Author: Theodor Steiner 
 Date: Tue Feb 28 15:17:54 2023 +0900
 1 file changed, 1 insertion(+), 2 deletions(-)
[main ace780692a] docs: update comments
 Author: Theodor Steiner 
 Date: Tue Feb 28 15:52:02 2023 +0900
 1 file changed, 1 insertion(+), 5 deletions(-)
[main fdbc00369a] chore: make finishing touches on comments
 Author: Theodor Steiner <[email protected]>
 Date: Tue Feb 28 22:55:35 2023 +0900
 1 file changed, 3 insertions(+), 3 deletions(-)
[main e8d71d9d6a] test: fix test failing due to difference in terminal codes
 Author: Theodor Steiner 
 Date: Wed Mar 1 11:49:30 2023 +0900
 1 file changed, 5 insertions(+), 2 deletions(-)
[main 7e6509411b] Revert "test: fix test failing due to difference in terminal codes"
 Author: Theodor Steiner 
 Date: Wed Mar 1 15:54:34 2023 +0900
 1 file changed, 2 insertions(+), 5 deletions(-)
[main 5c29f7d92d] fix: erase lastIndex state for previous searches
 Author: Theodor Steiner 
 Date: Wed Mar 1 15:56:00 2023 +0900
 1 file changed, 2 insertions(+)
   ✔  Patches applied
There are 12 commits in the PR. Attempting autorebase.
Rebasing (2/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

repl: fix .load infinite loop caused by shared use of lineEnding RegExp
Since the lineEnding Regular Expression is declared on the module scope,

recursive invocations of its [kTtyWrite] method share one instance of

this Regular Expression.

Since the state of a RegExp is managed by instance,

alternately calling RegExpPrototypeExec with the same RegExp on

different strings can lead to the state changing unexpectedly.

This is the root cause of this infinite loop bug when calling .load on

javascript files of certain shapes.
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 74d5e0d47b] repl: fix .load infinite loop caused by shared use of lineEnding RegExp

Author: Theodor Steiner [email protected]

Date: Mon Feb 20 20:47:33 2023 +0900

3 files changed, 66 insertions(+), 10 deletions(-)

create mode 100644 test/parallel/test-repl-load-multiline-function.js

Rebasing (3/24)

Rebasing (4/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: add test that fails without the changes of #46742
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD bb55df9366] test: add test that fails without the changes of #46742

Author: Theodor Steiner [email protected]

Date: Mon Feb 27 17:23:13 2023 +0900

2 files changed, 31 insertions(+), 47 deletions(-)

create mode 100644 test/parallel/test-readline-recursive-writes.js

delete mode 100644 test/parallel/test-repl-load-multiline-function.js

Rebasing (5/24)

Rebasing (6/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

chore: remove unrelated comment
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 34391c9429] chore: remove unrelated comment

Author: Theodor Steiner [email protected]

Date: Mon Feb 27 17:29:53 2023 +0900

1 file changed, 1 deletion(-)

Rebasing (7/24)

Rebasing (8/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

chore: fix linting errors and rename test
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 06ecce247c] chore: fix linting errors and rename test

Author: Theodor Steiner [email protected]

Date: Mon Feb 27 17:33:58 2023 +0900

1 file changed, 4 insertions(+), 5 deletions(-)

rename test/parallel/{test-readline-recursive-writes.js => test-readline-interface-recursive-writes.js} (86%)

Rebasing (9/24)

Rebasing (10/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

remove magic number from test
Co-authored-by: Antoine du Hamel [email protected]

PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 85c32ca998] remove magic number from test

Author: Theodor Steiner [email protected]

Date: Tue Feb 28 09:39:32 2023 +0900

1 file changed, 8 insertions(+), 5 deletions(-)

Rebasing (11/24)

Rebasing (12/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fix: refactor to use module scope instance of RegExp safely for performance considerations
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 191f742311] fix: refactor to use module scope instance of RegExp safely for performance considerations

Author: Theodor Steiner [email protected]

Date: Tue Feb 28 15:16:02 2023 +0900

1 file changed, 3 insertions(+), 4 deletions(-)

Rebasing (13/24)

Rebasing (14/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

chore: fix linting errors
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD abd94f85b4] chore: fix linting errors

Author: Theodor Steiner [email protected]

Date: Tue Feb 28 15:17:54 2023 +0900

1 file changed, 1 insertion(+), 2 deletions(-)

Rebasing (15/24)

Rebasing (16/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

docs: update comments
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD f68814e734] docs: update comments

Author: Theodor Steiner [email protected]

Date: Tue Feb 28 15:52:02 2023 +0900

1 file changed, 1 insertion(+), 5 deletions(-)

Rebasing (17/24)

Rebasing (18/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

chore: make finishing touches on comments
Co-authored-by: Antoine du Hamel [email protected]

PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 27b4146827] chore: make finishing touches on comments

Author: Theodor Steiner [email protected]

Date: Tue Feb 28 22:55:35 2023 +0900

1 file changed, 3 insertions(+), 3 deletions(-)

Rebasing (19/24)

Rebasing (20/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

test: fix test failing due to difference in terminal codes
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD d6950dbf57] test: fix test failing due to difference in terminal codes

Author: Theodor Steiner [email protected]

Date: Wed Mar 1 11:49:30 2023 +0900

1 file changed, 5 insertions(+), 2 deletions(-)

Rebasing (21/24)

Rebasing (22/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

Revert "test: fix test failing due to difference in terminal codes"
This reverts commit 6699359c7c15b92256c1c3c9813bd50f05a5606f.
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD e80778e35c] Revert "test: fix test failing due to difference in terminal codes"

Author: Theodor Steiner [email protected]

Date: Wed Mar 1 15:54:34 2023 +0900

1 file changed, 2 insertions(+), 5 deletions(-)

Rebasing (23/24)

Rebasing (24/24)
Executing: git node land --amend --yes

--------------------------------- New Message ----------------------------------

fix: erase lastIndex state for previous searches
PR-URL: #46742

Fixes: #46731

Reviewed-By: Kohei Ueno [email protected]

Reviewed-By: Antoine du Hamel [email protected]
[detached HEAD 2f274c0c73] fix: erase lastIndex state for previous searches

Author: Theodor Steiner [email protected]

Date: Wed Mar 1 15:56:00 2023 +0900

1 file changed, 2 insertions(+)
Successfully rebased and updated refs/heads/main.
ℹ  Add commit-queue-squash label to land the PR as one commit, or commit-queue-rebase to land as separate commits.

https://github.com/nodejs/node/actions/runs/4303874198

[nodejs-github-bot]

Landed in 333aff0

[aduh95]

Thanks a lot for your work here, and congrats for your first merged PR for Node.js 🎉

[starball5]

I see that this fix is in the v19 changelog, but I don't see it in the v18 changelog. As mentioned earlier, this bug exists starting after 18.12.0.

Considering that v18 is currently in active LTS status, can we get the fix in v18 as well please?

Related Stack Overflow questions:

• Why does using the .load special command in NodeJS v18 result in an infinite loop of loading the first line of the specified file?
• .load Command Goes In An Infinite Loop When Trying To Load A File In Node.js REPL