Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node 20.3.0 images give error /usr/bin/env: 'node': Text file busy #1912

Open
culli opened this issue Jun 12, 2023 · 88 comments
Open

Node 20.3.0 images give error /usr/bin/env: 'node': Text file busy #1912

culli opened this issue Jun 12, 2023 · 88 comments

Comments

@culli
Copy link

culli commented Jun 12, 2023

Environment

  • Platform: macos 13.4 M1
  • Docker Version: 24.0.2
  • Node.js Version: 20.3.0
  • Image Tag: node:20-alpine3.17

Expected Behavior

yarn tsc should run correctly

Current Behavior

 > [service 5/5] RUN yarn build:
#0 0.298 yarn run v1.22.19
#0 0.314 $ tsc
#0 0.325 /usr/bin/env: 'node': Text file busy
#0 0.334 error Command failed with exit code 126.

Possible Solution

downgrade to node:20-alpine3.16 (or node:20.2.0-bullseye-slim)

Steps to Reproduce

dockerfile:

FROM node:20-alpine3.17 as install
WORKDIR /service
COPY ./service ./
RUN yarn install --frozen-lockfile
RUN yarn build

yarn build creates a directory, compiles some protobuf, then execs tsc (bin from typescript dependency)

Additional Information

Things work correctly on 3.16, but give the text file busy error with 3.17 and 3.18. Also tried bullseye-slim and node:20.3.0-bullseye-slim gives the same error, but 20.2.0 works correctly.

Maybe this is an architecture mismatch or file permissions problem?

@culli
Copy link
Author

culli commented Jun 12, 2023

I can provide a more complete repro node project if that is needed.

@danlordan
Copy link

+1, getting an issue on this as well, had to switch to node:18 to get builds to run, instead of node:latest

@Eeemil
Copy link

Eeemil commented Jun 14, 2023

A colleague of mine running a Mac M1 has the same issue, but it works for me on PC (Ubuntu). Might be an issue for Mac M1 only.

@egl74
Copy link

egl74 commented Jun 14, 2023

Can (temporarily) be fixed by using OrbStack instead of Docker for Mac

@mfn
Copy link

mfn commented Jun 14, 2023

Same here with node:20.3-slim running in Docker Desktop 4.20.1 on macOS 13.4 Apple Silicon.

Reverting to node:20.2-slim worked.

Only affected shell/bash scripts trying to start node (e.g. yarn start in a NextJS project), when I ran it manually it worked.

Only saw this on local dev and not pro container (but prod != ARM, so… 🤷🏼 )

@yann1731
Copy link

A colleague of mine running a Mac M1 has the same issue, but it works for me on PC (Ubuntu). Might be an issue for Mac M1 only.

Can confirm I got this issue on M1 mac and Pop!_os

@bjonnh-work
Copy link

This is breaking in docker on x86 as well as M1 for us, and we get the same error on containers that start yarn.

@book000
Copy link

book000 commented Jun 14, 2023

This issue was also observed in the following environments:

  • Windows 10 22H2 (Build 19045.2965)
  • CPU: AMD Ryzen 9 5900X (x86_64)
  • Docker Desktop 4.19.0 (106363)
  • Image tag: node:20-alpine
  • Package manager: Yarn

It is not limited to a specific Node.js project, but occurs in all projects that use node:20-alpine. I have confirmed that replacing it with node:18-alpine works.


However, this problem does not occur in the following environments:

  • Ubuntu 22.04.2 LTS
  • CPU: Intel Core i7-6700 (x86_64)
  • Docker 24.0.2

@jerlendds
Copy link

jerlendds commented Jun 15, 2023

This can be recreated in my project from commit b76bc9b with the frontend Dockerfile but I have since pushed this version: node:18-alpine.

Seemingly related: Docker forums
My related issue with more details can be found here: jerlendds/osintbuddy#49

@evanw
Copy link

evanw commented Jun 15, 2023

FYI there is another case of people hitting this problem here: evanw/esbuild#3156.

@JoostK
Copy link

JoostK commented Jun 15, 2023

Sharing an additional data point here as well (from evanw/esbuild#3156 (comment)): the issue only reproduces for me using Docker Engine 24, whereas the image works fine using Docker Engine 20. Tested on M1 Max.

@GabeDuarteM
Copy link

This was also happening with me and all my coworkers that use mac for development.

The "solution" we found for now was to use OrbStack instead, it's a drop-in replacement for docker desktop, and seems to solve this problem. While not ideal, it did unblock us while we wait for the official solution.

@ahamlinman
Copy link

ahamlinman commented Jun 16, 2023

I started looking more closely at the libuv v1.45.0 upgrade after @JoostK mentioned it in the esbuild thread, particularly some of the changes around io_uring support. There is an environment variable (undocumented and unstable, intended for debugging purposes only) to disable libuv's use of io_uring, and with UV_USE_IO_URING=0 set in my container's environment I can no longer reproduce the issue during esbuild installs with any of the Node.js v20.3.0 image variants I've tested (M1 Max, Docker Desktop v4.20.1, Docker Engine v24.0.2).

I also have an Arch Linux ARM virtual machine on my Mac separate from Docker Desktop, and tried building and installing libuv v1.45.0 there. In that case, I couldn't reproduce the same issue during an esbuild install, even though I could confirm with strace that Node.js was dispatching io_uring operations. That makes me wonder if the kernel version is partly at play (6.3.7 on my Arch VM vs. 5.15.49-linuxkit-pr in the Docker VM). (edit: I set up a separate system with a separate build of Linux 5.15.49, and couldn't reproduce either on the system or with the Docker images. Looking at kernel versions alone may not be a useful lead.)

Given @evanw's note in the original thread about possible changes to fs.renameSync, it might be notable that rename is apparently one of the operations now backed by io_uring, per libuv/libuv#4012 (my guess is that the *Sync functions in Node.js still use libuv under the hood, but I'm not certain). If that really is where this comes from, the discussion may ultimately belong with libuv, or even with Node.js if it's somehow particular to Node's usage of libuv. (edit: This entire part is incorrect, please see below. Thank you @santigimeno for the correction, I apologize for the error.)

@harvey-appleton
Copy link

Experiencing same issue here, but downgrading to node:20-alpine3.16 seems to have fixed it. Seems the issue reappears for us in 20-alpine3.17 (which includes Node 20.3.0) - perhaps (guessing here) it's related to a line in the changelog for Node v20.3.0 which says:

[bfcb3d1d9a] - deps: upgrade to libuv 1.45.0, including significant performance improvements to file system operations on Linux (Santiago Gimeno) nodejs/node#48078

(which does backup what @ahamlinman mentioned above)

@mgattei
Copy link

mgattei commented Jun 16, 2023

Same here for a pipeline that run semantic-release. Gitlab - node:alpine

- yarn global add handpick
- handpick --target=devDependencies --manager=yarn
- yarn semantic-release

Error:

$ yarn global add handpick
yarn global v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Installed "[email protected]" with binaries:
      - handpick
Done in 0.96s.
$ handpick --target=devDependencies --manager=yarn
Picking DIRTY devDependencies via YARN
Done 854 packages in 71.30 seconds
$ yarn semantic-release
yarn run v1.22.19
$ /builds/project/node_modules/.bin/semantic-release
env: can't execute 'node': Text file busy
error Command failed with exit code 126.

By retrying several times, it seems to work after a while.... I solved the problem temporarily by downgrading to an older node image.

@jkuchar
Copy link

jkuchar commented Jun 16, 2023

Not sure what causes this, but this is what I have found:

  • problem happens on version 20.3.0 on ubuntu 22.04 (manually installed), Debian 10, 11, 12 (node images from docker hub)
  • problem does NOT happen with node 20.2.0 or node 19

So I would guess this is some bug in node itself?

@santigimeno
Copy link
Member

santigimeno commented Jun 16, 2023

Given @evanw's note in the original thread about possible changes to fs.renameSync, it might be notable that rename is apparently one of the operations now backed by io_uring, per libuv/libuv#4012 (my guess is that the *Sync functions in Node.js still use libuv under the hood, but I'm not certain). If that really is where this comes from, the discussion may ultimately belong with libuv, or even with Node.js if it's somehow particular to Node's usage of libuv.

FWIW, that specific libuv commit is not yet in any release so it didn't get to Node.js. Also, the fs sync operations aren't io_uring backed.

@santigimeno
Copy link
Member

  • problem happens on version 20.3.0 on ubuntu 22.04 (manually installed),

@jkuchar can you elaborate exactly what steps to follow? I'd like to have a reproducer in that environment. Thanks!

@jkuchar
Copy link

jkuchar commented Jun 18, 2023

This was my test scenario environment:

FROM ubuntu:22.04

# NodeJS & Chromium for tests
RUN curl -sL https://deb.nodesource.com/setup_20.x | bash -
RUN apt install -y nodejs

@danopia
Copy link

danopia commented Jun 26, 2023

I'm seeing this within a CircleCI linux environment when attempting to bump node from 20.2.0-alpine3.17 to 20.3.1-alpine3.17.

This is the docker server info from CircleCI:

Server Engine Details:
  Version:          20.10.18
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.6
  Git commit:       e42327a
  Built:            2022-09-08T23:09:30.000000000+00:00
  OS/Arch:          linux/amd64
  Experimental:     false
The error happened during `docker build`, when yarn tried running `react-scripts build`:
Step 15/23 : RUN yarn build && yarn test -- --watchAll=false
 ---> Running in e344e5cf9846
yarn run v1.22.19
$ react-scripts build
env: can't execute 'node': Text file busy
error Command failed with exit code 126.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
The command '/bin/sh -c yarn build && yarn test -- --watchAll=false' returned a non-zero code: 126

@FernandoKGA
Copy link

FernandoKGA commented Jun 27, 2023

Not sure what causes this, but this is what I have found:

* problem happens on version 20.3.0 on ubuntu 22.04 (manually installed), Debian 10, 11, 12 (node images from docker hub)

* problem does NOT happen with node 20.2.0 or node 19

So I would guess this is some bug in node itself?

I could not reproduce in my system which is a Ubuntu 22.04 that was upgraded automatically from the Ubuntu 20.04.

I notice a difference between my installed packages and my colleagues packages:

 @ docker-buildx-plugin : 0.10.5-1~ubuntu.20.04~focal -> 0.10.5-1~ubuntu.22.04~jammy
 @ docker-ce : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-ce-cli : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-ce-rootless-extras : 5:24.0.2-1~ubuntu.20.04~focal -> 5:24.0.2-1~ubuntu.22.04~jammy
 @ docker-compose-plugin : 2.18.1-1~ubuntu.20.04~focal -> 2.18.1-1~ubuntu.22.04~jammy

My Docker packages are 20.04~focal, although I have the same version as him, the OS package is different.

We have the same versions for Docker: 24.0.2, build cb74dfc and the uname -a show pretty much the same kernel and configs: 5.19.0-45-generic #46~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 7 15:06:04 UTC 20 x86_64 x86_64 x86_64 GNU/Linux.

The docker compose was different, I was on v2.2.3 and him at v2.15.1, I upgraded to the latest v2.19.0, but still couldn't reproduce.

We will try to see if the environment variable that @ahamlinman mentioned (UV_USE_IO_URING=0) can allow him to build the image. Just read @ahamlinman comment again, this won't work anyway by now.

The production builds at our company didn't fail with the upgrade to node:20-alpine, but it is using another version of Docker to build the images.

gregtyler added a commit to ministryofjustice/opg-sirius-lpa-frontend that referenced this issue Jun 28, 2023
Node 20.3.x doesn't work with our local versions of Docker so we must lock to 20.2.x for now

[Issue on nodejs/docker-node](nodejs/docker-node#1912)

#patch
@malix0
Copy link

malix0 commented Jan 11, 2024

I just add my experience. I had an old Docusaurus image built FROM node:20-alpine3.17 that worked without problem, the node version here was node 20.2.0 I had to rebuild this image than I switch it to node:20-alpine so I start getting this error because node was updated to latest 20.10.0 version and searching on internet I reached this issue. I solved the error switching back to node 20.2.0 (FROM node:20.2.0-alpine3.17)

Edit: excuse me, I just had no time to read all the thread, I read it now and maybe this post is not so useful, but I want to add that my host is Ubuntu 22.04.1 with kernel 5.19.0-32, so as wrote by @santigimeno maybe this kernel version will never receive fix that can solve this, so at the moment my solution is to use node 20.2.0

mpyne-navy added a commit to mpyne-navy/proj-waterfall-backend that referenced this issue Mar 30, 2024
The new version of chai pulled in as part of the update requires ESM
only, which forces the test case to use ESM, which then forces the
app.js to also use ESM.

These changes make the test suite work with both "npm test" and "yarn
test", though note that currently I need to set the environment variable
"UV_USE_IO_URING=0" to avoid an error about "text file busy".  I
discovered this trick from a Github issue comment [1].

Hopefully this also works on AWS Lambda but we'll see.

[1]: nodejs/docker-node#1912 (comment)
@kode54
Copy link

kode54 commented May 18, 2024

Experiencing this on Arch Linux (x86_64) as of updating to kernel 6.9.1. Failing to build AUR thelounge package with these Text file busy errors. nodejs 22.0.0-1 from Arch repos.

Edit: Dang, I thought this was the nodejs repo, my bad.

@MartinX3
Copy link

@kode54 Thank you for the hint.
I also updated to kernel 6.9.1 and if I run husky via pnpm it results into the 126 bad interpreter: cannot execute or modify file (busy) error.
Starting husky directly from cmd doesn't make this error.

So it must be caused by the 6.9.x kernel.

@luisbc92
Copy link

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.

Switching back to 6.8.x (lts kernel) fixes it.

@thanhquang1988
Copy link

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.

Switching back to 6.8.x (lts kernel) fixes it.

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

@hoaithing
Copy link

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

I have upgraded to 20.13 but the issue still be remaining.

@remisiki
Copy link

Just adding another datapoint, node fails in various different places with bad interpreter: Text file busy on Arch Linux 6.9.1 kernel.
Switching back to 6.8.x (lts kernel) fixes it.

I have same issue with Arch 6.9, but after upgrade nodejs from 20.9 => 20.13 problem fixed also

same issue here on arch 6.9.1 and node 22.2.0. npm works without errors and only yarn fails.

@FLX-0x00
Copy link

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

@is0ly
Copy link

is0ly commented May 26, 2024

yarn set version stable
yarn install

from yarn site - fixed this wired behavior without switching kernel version

@felixonmars
Copy link

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

@FLX-0x00
Copy link

FLX-0x00 commented May 26, 2024

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

Still the same error even with new kernel after reboot. Switched to the linux-lts kernel.

@MartinX3
Copy link

At my side it got fixed with the upgrade from pnpm 9.1.2 to 9.1.3.
And I also use the 6.9.2 kernel now.

I don't know if the fix is in any of these commits or it is just coincidence:
pnpm/pnpm@v9.1.2...v9.1.3

@zguig52
Copy link

zguig52 commented May 27, 2024

Workaround to avoid downgrading the kernel:
UV_USE_IO_URING=0 your_command_using_node

Ex: UV_USE_IO_URING=0 yarn dev

@ZulluBalti
Copy link

At my side it got fixed with the upgrade from pnpm 9.1.2 to 9.1.3. And I also use the 6.9.2 kernel now.

I don't know if the fix is in any of these commits or it is just coincidence: pnpm/[email protected]

I think it's a coincident as I"m also using pnpm version 9.1.3 with kernel 6.9.2-arch1-1 and still getting this error

@ZulluBalti
Copy link

Cant get this working with arch linux 6.9.1-arch. tried nodejs-lts-hydrogen 18.20, nodejs-lts-iron 20.13 or nodejs 22.2

Please try Arch Linux's new kernel 6.9.2-1 and see if it makes a difference.

Nope, it doesn't make any difference still getting this error with 6.9.2-1

@MartinX3
Copy link

@ZulluBalti woopsi.
You're right.
After a computer restart the issue reappeared at my site as well.

@skabber
Copy link

skabber commented May 28, 2024

Same error /usr/bin/env: ‘node’: Text file busy using nixos kernel 6.9.2. Backing out to 6.8.11 fixes the issue.

@mzndr
Copy link

mzndr commented May 29, 2024

For anyone wondering on how to downgrade your arch kernel, you can downgrade to previous installations (of any package) from chache:
pacman -U /var/cache/pacman/pkg/linux-<your-version>.pkg.tar.zst

@skabber
Copy link

skabber commented May 29, 2024

Here is the issue tracked on nodejs/node nodejs/node#53051

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests