Upgrade Framework to Yarn v3

[thedavidprice]

related to #2907

cc @larixer

Overview

This PR is now ready for review.

Yarn v3 is a significant improvement over v1. Performance is snappier and resolutions are greatly improved. The new features like doctor have identified previously unknown issues.

Additionally, this resolves the "connection time out" error we were seeing in CI checks, which means overall CI time is about 30-50% reduced (depending on cache-hit status).

We can proceed with using Yarn v3 for the Framework and Yarn v1 for Projects. Once we are comfortable with Yarn v3 (and have tested upgrading Projects via canary releases), we can suggest projects migrate to v3. Assuming the migration is smooth, I see us moving the CRWA template to Yarn v3 sooner than later because of the performance and stability gains.

Reference:

• Migration guide
• Blog Post v3 Announcement
• v3 Changelog

Current Status

This upgrade was significantly more feasible than my previous attempt to upgrade to Yarn v2. 🚀 Overall, I'm excited about the bug-fixes and potential performance improvements (including CI E2E). This is looking solid.

Note: using v3 on the Framework does not require upgrading Redwood Projects to use v3. (However, we might determine this is a simple + helpful recommendation in the future.) Also, the v3 binary is installed in the project directory itself, which means you can have v1 installed globally and only use v3 when in the context of the Framework project. Slick...

New Yarn utilities

yarn add --interactive
Reuse the specified package from other workspaces in the project. Example:

yarn workspace create-redwood-app add -i rimraf

yarn workspaces foreach ...
This is a command from the workspaces plugin. Runs the command across all workspaces. Example:

yarn workspaces foreach -i  -v some-package

-v: outputs the package name the command is currently running against

Added to CI: dedupe and constraints

yarn dedupe --check

Duplicates are defined as descriptors with overlapping ranges being resolved and
locked to different locators. They are a natural consequence of Yarn's
deterministic installs, but they can sometimes pile up and unnecessarily
increase the size of your project.
This command dedupes dependencies in the current project using different
strategies (only one is implemented at the moment):

yarn constraints
See new file constraints.pro for repo config

• https://yarnpkg.com/features/constraints
• Reference from Babel project: https://github.com/babel/babel/blob/main/constraints.pro

ToDo

• upgrade and config; set tasks/framework-tools to use binaries+plugins in root
• add workspaces plugin
• use v2 for tasks/all-contributors
• use v2 for tasks/e2e
• use v2 for tasks/test-project
• clean up core and testing package dependencies
• BUG: fix yarn lint eslint error
• BUG: yarn install does not run the script postinstall. However, I can run yarn postinstall successfully
  • confirming that postinstall does run after git clean -fxd. Apparently it doesn't run unless changes/updates have occured.
• run through all scripts and tasks/ commands
• test yarn rwfw project:sync with a local Redwood project against this branch (Framework using yarn v3)
• WINDOWS: test yarn lint (cross-platform compatibility when passing Env)
• test commands within individual packages
• publishing: canary, rc, and latest
• confirm canary version of this PR is compatible with project using Yarn v1

Notes

"Binary hoisting" is no longer allowed

Example:

The main problems you are facing with Yarn 3 is that Yarn is now strict about binaries you run from your workspaces. You are not allowed to run rimraf on behalf of the root workspace if it is not in the dependencies of root package.json. You are not allowed to run babel-node on behalf of create-redwood-app workspace if @babel/node is not in its dependencies.

This is a significant change for us — for example, to run build and test within individual packages, we had to install babel, typescript, and jest within each workspace package.json.

Installing with the add -i ... interactive option ensures same dependency version cross-workspaces. We also now have dedupe and constraints running in CI to help manage.

Will need to take more care when upgrading dependencies for consistency across packages.

Specifying Yarn v1 binary (when necessary)

Yarn v1 is installed globally and Yarn v3 is specified locally, which allows us to use both at the same time. However, in the case a script is called via yarn run [script] (e.g. yarn build:test-project), the process defaults to using the Yarn v3 binary for all yarn commands. This will break the yarn commands run within the new test-project directory, for example. So we ended up learning a lot of tricks to manage which yarn binary is used.

Use yarn dlx

"This command will install a package within a temporary environment, and run its binary script if it contains any. The binary will run within the current cwd."

Example: run yarn v1 install

yarn dlx yarn install

We also ended up needing to set the packageLocator, which is null in Yarn v3 and the value would be passed down to Yarn v1 executed command. Note: would need to use cross-env or otherwise for multi-platform compatibility.

Example:

YARN_NODE_LINKER="node-modules" yarn dlx yarn install

Set YARN_IGNORE_PATH=1
This ignores the local yarn version settings. Note: would need to use cross-env or otherwise for multi-platform compatibility.

Example:

YARN_IGNORE_PATH=1 yarn install

** Set version in directory**
This adds a .yarnrc file that explicitly sets v1. Note: this is what we ended up using here.

Example:

yarn set version classic

postinstall

The postinstall script (in package.json) was not always executing. I found this explanation, which explained what I was experiencing:

The 2.x will not print the logs for successful builds (just like 1.x), but it will also not run the postinstall scripts unless the dependency tree changed. The 1.x tried to have a similar behaviour (for example the bypass previously mentioned caused the postinstall scripts to not be run on every install) but was fairly buggy and sometimes unpredictable in this aspect.

[thedavidprice]

re: Cypress --> missing dependencies, e.g. command not found: rimraf. Need to diagnose why install isn't running correctly.

[larixer]

re: Cypress --> missing dependencies, e.g. command not found: rimraf. Need to diagnose why install isn't running correctly.

The main problems you are facing with Yarn 3 is that Yarn is now strict about binaries you run from your workspaces. You are not allowed to run rimraf on behalf of the root workspace if it is not in the dependencies of root package.json. You are not allowed to run babel-node on behalf of create-redwood-app workspace if @babel/node is not in its dependencies.

If its okay to add these dependencies you can do this via:
yarn add -i rimraf
and
yarn workspace create-redwood-app add -i @babel/node
-i here is to interactively reuse existing versions of rimraf and @babel/node used within the project.

[thedavidprice]

I believe build, lint, and test are all working correctly. I do have a build:clean warning to diagnose:

• [A: no] does the build:clean Prisma warning indicate cwd is not correct or being applied?

$ yarn build:clean
prisma:warn The postinstall script automatically ran `prisma generate` and did not find your `prisma/schema.prisma`.
If you have a Prisma schema file in a custom path, you will need to run
`prisma generate --schema=./path/to/your/schema.prisma` to generate Prisma Client.
If you do not have a Prisma schema file yet, you can ignore this message.

Cypress E2E

The tasks/run-e2e succesfully build + syncs a temp project, starts the dev server, and then starts Cypress. However, running the tests results in all failing due to errors like cy.exec not in lockfile... (or something to that effect; I did not copy before terminating).

• the temp Redwood project is using Yarn v1; I don't believe this should be a problem given how the dependencies are installed separately and the dist/ contents copied. Need to diagnose.
• Why would Cypress start but the test runners all fail at each step to run the cy commands?

[thedavidprice]

🤯

---

Still can't run tasks/run-e2e locally. But this is a suprisingly great result!

[merceyz]

Seems some install-state.gz files got committed by accident, they should be gitignored as well

[thedavidprice]

🚀

This is ready to merge.

To Do:

• move this comment into a new Issue

[peterp]

Can we jump on a call to discuss this PR. It appears to break my mental model of "framework-tools" and I just want to discuss what the constraints and workarounds are.

[peterp]

Happy for this to go in after our discussion.

[thedavidprice]

Continued in #3516