🐛 Bug Report: System limit for number of file watchers reached

[myplaybooks]

📜 Description

I am getting the following issue in the build pipeline
The docker image I am using 18.14.1-alpine

yarn run v1.22.19
$ tsc
Done in 31.32s.
yarn run v1.22.19
$ backstage-cli package build
Building app separately because it is a bundled package
app: [2K
app: [1G[2m$ backstage-cli package build[22m
app:
app:
app: [31mError: ENOSPC: System limit for number of file watchers reached, watch '/app/app-config.yaml'[39m
app:
app:
app: [2K
app: [1G[31merror[39m Command failed with exit code 1.
app:
app: [2K[1G[34minfo[39m Visit [1mhttps://yarnpkg.com/en/docs/cli/run[22m for documentation about this command.
app:
Command 'yarn' exited with code 1

👍 Expected behavior

The prod pipeline should have any file watchers.

👎 Actual Behavior with Screenshots

�[36mINFO�[0m[1198] Running: [/bin/sh -c yarn tsc && yarn --cwd packages/backend build]
yarn run v1.22.19
$ tsc
Done in 31.32s.
yarn run v1.22.19
$ backstage-cli package build
Building app separately because it is a bundled package
app: �[2K
app: �[1G�[2m$ backstage-cli package build�[22m
app:
app:
app: �[31mError: ENOSPC: System limit for number of file watchers reached, watch '/app/app-config.yaml'�[39m
app:
app:
app: �[2K
app: �[1G�[31merror�[39m Command failed with exit code 1.
app:
app: �[2K�[1G�[34minfo�[39m Visit �[1mhttps://yarnpkg.com/en/docs/cli/run�[22m for documentation about this command.
app:

Command 'yarn' exited with code 1

👟 Reproduction steps

The build pipeline produces the above error.

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Linux 5.15.96-0-virt - linux/x64
node: v18.14.1
yarn: 1.22.19
cli: 0.22.9 (installed)
backstage: N/A

Dependencies:
@backstage/app-defaults 1.4.1
@backstage/backend-app-api 0.4.5
@backstage/backend-common 0.19.1
@backstage/backend-dev-utils 0.1.1
@backstage/backend-plugin-api 0.5.4
@backstage/backend-tasks 0.5.4
@backstage/catalog-client 1.4.3
@backstage/catalog-model 1.4.1
@backstage/cli-common 0.1.12
@backstage/cli-node 0.1.2
@backstage/cli 0.22.9
@backstage/config-loader 1.3.2
@backstage/config 1.0.8
@backstage/core-app-api 1.9.0
@backstage/core-components 0.13.3
@backstage/core-plugin-api 1.5.3
@backstage/dev-utils 1.0.17
@backstage/errors 1.2.1
@backstage/eslint-plugin 0.1.3
@backstage/integration-aws-node 0.1.5
@backstage/integration-react 1.1.15
@backstage/integration 1.5.1
@backstage/plugin-analytics-module-ga 0.1.31
@backstage/plugin-api-docs 0.9.6
@backstage/plugin-app-backend 0.3.47
@backstage/plugin-auth-backend 0.18.5
@backstage/plugin-auth-node 0.2.16
@backstage/plugin-catalog-backend-module-bitbucket-server 0.1.12
@backstage/plugin-catalog-backend 1.11.0
@backstage/plugin-catalog-common 1.0.15
@backstage/plugin-catalog-graph 0.2.32
@backstage/plugin-catalog-import 0.9.10
@backstage/plugin-catalog-node 1.4.0
@backstage/plugin-catalog-react 1.8.0
@backstage/plugin-catalog 1.12.0
@backstage/plugin-events-node 0.2.8
@backstage/plugin-github-actions 0.6.1
@backstage/plugin-home-react 0.1.1
@backstage/plugin-home 0.5.4
@backstage/plugin-org 0.6.10
@backstage/plugin-permission-common 0.7.7
@backstage/plugin-permission-node 0.7.10
@backstage/plugin-permission-react 0.4.14
@backstage/plugin-proxy-backend 0.2.41
@backstage/plugin-scaffolder-backend 1.15.1
@backstage/plugin-scaffolder-common 1.3.2
@backstage/plugin-scaffolder-node 0.1.5
@backstage/plugin-scaffolder-react 1.5.1
@backstage/plugin-scaffolder 1.14.1
@backstage/plugin-search-backend-module-catalog 0.1.3
@backstage/plugin-search-backend-module-pg 0.5.8
@backstage/plugin-search-backend-module-techdocs 0.1.3
@backstage/plugin-search-backend-node 1.2.3
@backstage/plugin-search-backend 1.3.3
@backstage/plugin-search-common 1.2.5
@backstage/plugin-search-react 1.6.3
@backstage/plugin-search 1.3.3
@backstage/plugin-sonarqube-backend 0.2.1
@backstage/plugin-sonarqube-react 0.1.7
@backstage/plugin-sonarqube 0.7.1
@backstage/plugin-tech-radar 0.6.6
@backstage/plugin-techdocs-backend 1.6.4
@backstage/plugin-techdocs-module-addons-contrib 1.0.15
@backstage/plugin-techdocs-node 1.7.3
@backstage/plugin-techdocs-react 1.1.8
@backstage/plugin-techdocs 1.6.5
@backstage/plugin-user-settings 0.7.5
@backstage/release-manifests 0.0.9
@backstage/test-utils 1.4.1
@backstage/theme 0.4.1
@backstage/types 1.1.0
@backstage/version-bridge 1.0.4

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

No, I don't have time to work on this right now

[myplaybooks]

I am not sure if it is an environmental issue or not. However, I am interested to know the purpose of the file watcher in the production environment.

[awanlin]

Hi @myplaybooks, would it be possible to share your Dockerfile, please?

[myplaybooks]

@awanlin
This is my Dockerfile, its based on the https://backstage.io/docs/deployment/docker/#multi-stage-build

I am facing issue at: yarn --cwd packages/backend build

My Dockerfile
FROM my-private-repo/docker.io/node:18.14.1-alpine AS build
ARG CACHEBUST=1
WORKDIR /app
COPY package.json yarn.lock .yarnrc generate_npmrc.sh ./
COPY packages packages
COPY plugins plugins
RUN find packages ! -name "package.json" -mindepth 2 -maxdepth 2 -exec rm -rf {} +
RUN apk update &&
apk add -q --no-cache sqlite-dev~=3.40 python3~=3.10 g++=12.2 build-base=0.5 &&
yarn config set python /usr/bin/python3 &&
yarn config set strict-ssl false
RUN yarn install --frozen-lockfile --network-timeout 600000 && rm -rf "$(yarn cache dir)"
COPY . .
RUN yarn tsc &&
yarn --cwd packages/backend build

FROM my-private-repo/docker.io/node:18.14.1-alpine
RUN npm update -g npm &&
npm install -g [email protected] &&
apk update &&
apk add -q --no-cache sqlite-dev~=3.40 python3~=3.10 g++=12.2 build-base=0.5 py3-pipcommunity~=22.3 binutils~=2.40 &&
yarn config set python /usr/bin/python3 &&
yarn config set strict-ssl false
WORKDIR /app

COPY --from=build /app/yarn.lock /app/package.json /app/packages/backend/dist/skeleton.tar.gz ./
COPY --from=build /app/.yarnrc ./
RUN tar xzf skeleton.tar.gz && rm skeleton.tar.gz && yarn config set strict-ssl false
RUN yarn install --frozen-lockfile --production --network-timeout 600000 && rm -rf "$(yarn cache dir)"
COPY --from=build /app/packages/backend/dist/bundle.tar.gz .
RUN tar xzf bundle.tar.gz && rm bundle.tar.gz
COPY app-config.ist.yaml ./
RUN chown -R node:node /app
USER node
CMD ["node", "packages/backend", "--config", "app-config.ist.yaml"]

[awanlin]

Thanks @myplaybooks, can you tell me where are you running this when you hit the error? Just locally? When you build this in maybe a CI/CD pipeline?

Also, host custom is this: my-private-repo/docker.io/node:18.14.1-alpine?

[myplaybooks]

Yeah, its happening in the production CI/CD pipeline. I couldn't reproduce the error locally.
The Alpine image is from my-private-repo/docker.io, which is our private repository, I just masked the company URL.

I think ENOSPC: System limit for number of file watchers reached error is due to fs.inotify.max_user_watches value.

I need to know why we have file watchers? can we disable it in the production environment?

[camilaibs]

Hi @myplaybooks, could you try increasing the fs.inotify.max_user_watches?

[awanlin]

Was just going to follow up with what @camilaibs suggested, also if possible reboot as that was another solution I saw on the Backstage Discord server that worked.

Some more ideas might in this issue: gatsbyjs/gatsby#11406

Also, a bit more background, I'm still digging into this myself as a learning opportunity: from my reading react seems to use watchman on Linux which then uses inotify, but I'm still not sure why that gets used for this command by Backstage, it could just be some underlying part of react that is using it. https://reactnative.dev/docs/troubleshooting.html#unable-to-start-react-native-package-manager-on-linux

[myplaybooks]

@camilaibs @awanlin
I have tried to increase the max_user_watches through dockerfile file, but when I try to reload the configuration after I set the value, I got sysctl: error setting key 'fs.inotify.max_user_watches': Read-only file system error.

After some research, I got to know that Docker doesn't let the sysctl to be modified during the build for images because it takes it from the host. To increase max_user_watches in the host, the DevOps team is asking for the reason for watchers in the production environment: https://shorturl.at/emzQ5.

Do we know why we need watchers? Hard to get approval to increase the number of file watchers without knowing the reason.

[Rugvip]

@myplaybooks the build system requires watchers to be set up for each configuration file during the build. You'll only need a small number of available watchers.

In case anyone wants to contribute a fix for this I think the best path forward would be an option for FileConfigSource that makes it possible to disable watching, and using that when loading the config for building in the CLI

[antoniomuso]

I have the same issue during application startup.

[Rugvip]

@antoniomuso that'd be the same solution as #19619 (comment)

[myplaybooks]

@Rugvip
I reproduced the issue locally by reducing fs.inotify.max_user_watches < 300 and got the above error.

My doubt is are we using ~300 file watchers?? Will the above PR fix that?

[jhaals]

Is the 300ish file watcher happening during yarn build or yarn start? If it's yarn start locally we expect a lot of file watchers as we monitor source files for hot reload

[myplaybooks]

during backstage-cli package build stage

[freben]

Alright, thanks for reporting back. That does sound unexpected. Maybe the webpack build spins up watchers and needs to have that explicitly disabled, or similar? Opening up for more investigation and contributions 🙏

[myplaybooks]

@freben
PR added disable config watching flag is already in progress. I have tested the changes in the PR in our environment and it worked fine.
Thanks for the help @jhaals @Rugvip @freben