Use npm instead of yarn to build Theia

[tsmaeder]

What it does

This PR rewrites the Theia build process to use npm in stead of yarn. This required a couple of changes:

Contrary to yarn, npm does not invoke the prepare scripts of workspace packages in the order of their dependencies (npm/cli#3034). So instead of relying on yarn install to invoke the necessary build scripts, we have to invoke the build process explicitly after the npm install. The build process is built on this logic:

1. Each package has a script called "build". The goal of that target is to fully build all artifacts this repo produces. For regular Theia packages, this is just incrementally compiling the typescript sources of the package an it's dependencies, but for applications, for example, it means creating the application bundle. So calling "build" in an application package should be all that's needed in regular development
2. To build the repo initially after check-out, there are a couple of additinal steps necesary:
   • compile build tools (re-exports package)
   • generate re-exports where applicable
   • compile all packages
   • build alll application packages
     This can be achieved by invoking npm run build in the repo root.

Fixes #13948

Contributed on behalf of STMicroelectronics

How to test

Make sure we can build Theia as described above and the apps work fine.

Follow-ups

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[dankeboy36]

Fixes #13948

Possible related: #775

[tsmaeder]

@msujew @sdirix the PR checks now pass and it's ready to be reviewed minus doc.

[sdirix]

So far it works very well.

I used npm ci to install. I tested all applications and was able to build and start all of them. I also tested watching for the browser application. Tests also completed successfully.

Linting is currently broken but this is already mentioned in the current PR description.

@theia/request: > @theia/[email protected] lint
@theia/request: > theiaext lint
@theia/request: $ eslint --cache=true --no-error-on-unmatched-pattern=true "{src,test}/**/*.{ts,tsx}"
@theia/request: /bin/sh: 1: node: not found
@theia/request: Oops! Something went wrong! :(
@theia/request: ESLint: 8.57.1
@theia/request: SyntaxError: Failed to load plugin '@theia' declared in '.eslintrc.js » ../../configs/build.eslintrc.json » ./base.eslintrc.json': Unexpected end of JSON input
@theia/request:     at JSON.parse (<anonymous>)
@theia/request:     at PackageReExports.FromPackageSync (/home/user/git/theia/dev-packages/private-re-exports/lib/package-re-exports.js:127:47)
@theia/request:     at Object.<anonymous> (/home/user/git/theia/dev-packages/private-eslint-plugin/rules/shared-dependencies.js:24:40)

[sdirix]

Can we keep this? Why remove it?

[tsmaeder]

Because we now have the preference "files.insertFinalNewline" which acomplishes the same thing.

[msujew]

Works pretty well, but there are still a few calls to yarn littered throughout the build scripts:

• theia/scripts/run-reverse-topo.js

  Line 34 in 37d8330

  const YARN_WORKSPACES = JSON.parse(cp.execSync('yarn --silent workspaces info').toString());

• theia/scripts/translation-update.js

  Line 42 in 37d8330

  const childProcess = cp.spawnSync('yarn', [

There might be more. See also below.

[tsmaeder]

@JonasHelming in response to your question elsewhere, the documentation on https://theia-ide.org/ mostly uses the yeoman generator output and the theia-ide repository as examples. While the output of the generator and theia-ide still use yarn, it makes no sense to update the documentation.

[tsmaeder]

@msujew @sdirix could I get some love for this one, please? If we merge it, we should merge it early in the release cycle.

[msujew]

Looks good to me know. Everything seems to work as expected without relying on yarn.