Fix support for Yarn Plug'n Play in the plugins execution

[ehmicky]

@netlify/build runs yarn on two instances, which are both opt-in behavior:

• When the @netlify/plugin-functions-install-core plugin is used, we run yarn in the functions directory
• When the @netlify/plugin-local-install-core plugin is used, we run yarn in any local plugins directory

In both cases, we detect yarn by looking for the presence of a yarn.lock. This does not work in many cases, which is why we might want to also check for the presence of a USE_YARN environment, as suggested in netlify/build-image#440

Also, the command we are running is yarn install --no-progress --non-interactive --cache-folder=/opt/buildhome/.yarn_cache. According to netlify/build-image#440, this has few issues:

• --cache-folder was replaced with the environment variable YARN_GLOBAL_FOLDER in Yarn v2
• --no-progress and --non-interactive do not exist anymore

[Embraser01]

Thanks for looking at this!

A few notes I forgot to mention yesterday:

• YARN_GLOBAL_FOLDER is the folder where everything global to yarn 2 is stored. For now I think it is only a cache folder so not really a problem.
• You can force disable progress bars with this option (YARN_ENABLE_PROGRESS_BARS=false) even though it is already disabled on CI env
• Omitting the --non-interactive option should not cause any problems as yarn install should never ask for anything
• Even though caching $YARN_GLOBAL_FOLDER/cache might be enough, I'd like to note that some install could have a "local" cache folder (aka Offline mirrors in v1) that is or isn't committed in the repo. It is used mainly for PnP installs (the default in v2) and caching this folder (when not committed in vcs) instead of the global one should be more efficient. node_modules installs should use enableGlobalCache: true so it shouldn't matter much for those.

Also, how could we detect the usage of Yarn v2? Another env var?

Actually the easiest and more reliable way would be too run yarn --version and parse result to check version

[ehmicky]

Thanks a lot for this information, this is very helpful! ❤️

[gregberge]

I am using yarn v2 on xstyled project, it works fine but it looks like caching is not really effective. See this build: https://app.netlify.com/sites/xstyled/deploys/60584152ce7ae70008522e1f

I saw that I can configure a custom cache folder using environment variables but I think dependencies caching should work out of the box for common package managers like yarn.

[mosesoak]

Hi @gregberge and the Netlify team. We were considering trying out Yarn 2, which was released at the beginning of 2020, but are concerned about how little support Netlify seems to have for it.

Wanted to draw your attention to the fact that there's been a few posts to your forums over the last year without any actual movement. One of them claims that adding an extra path configuration file to the repo makes it at least buildable with Yarn 2. https://answers.netlify.com/t/using-the-new-yarn-release-2-0-0-berry/8270

But the issue notes above indicate that there's v1-specific yarn install code baked into Netlify, so I sort of doubt those workarounds do all that much. (I don't even know if you're supposed to run yarn install in v2, since the idea with it is that deps are pre-bundled in the repo.)

All of these things make our team wary of even trying it out while using Netlify, although we'd like to -- Yarn 2 is a major version of one of the most popular package managers after all. Any progress to report here? Thanks!

[panmona]

@mosesoak
I agree with your general points just wanted to add something to the following:

One of them claims that adding an extra path configuration file to the repo makes it at least buildable with Yarn 2.

I was able to verify that this works for one of my projects.

---

I don't even know if you're supposed to run yarn install in v2, since the idea with it is that deps are pre-bundled in the repo.

You're probably referring to zero installs here and that is something that is completely optional (see more here). Netlify should probably verify if a project is using zero installs and then don't execute yarn install.

[jgerigmeyer]

Any progress on this? It seems that build plugins installed via package.json also break in Yarn v2/3 (with PnP): https://app.netlify.com/sites/oddleventy/deploys/614a0a6b6bee0100bad715d3

[kachkaev]

Another confused user here 😥

I am switching a Yarn 3 project to Netlify, but can’t get cache working no matter what I try. The most rational move I could come up was setting these variables in Netlify UI and then trying to run the deploy twice:

YARN_CACHE_FOLDER=/opt/buildhome/cache/.yarn_cache
YARN_ENABLE_GLOBAL_CACHE=true

Regardless of what I do, I keep getting:

10:56:02 PM: ➤ YN0000: ┌ Fetch step
10:58:21 PM: ➤ YN0013: │ 14 packages were already cached, 730 had to be fetched
10:58:21 PM: ➤ YN0000: └ Completed in 2m 19s

which means more than two minutes are wasted on every build.

I am using Next and my build setup on Netlify is pretty basic:

Base directory	Not set
Build command	next build
Publish directory	.next
Deploy log visibility	Logs are public
Builds	Active

# netlify.toml

[build]
publish = ".next"

[[plugins]]
package = "@netlify/plugin-nextjs"

# .yarnrc.yml

nodeLinker: node-modules

yarnPath: .yarn/releases/yarn-3.1.1.cjs

I have even tried setting YARN_CACHE_FOLDER=.next/cache/yarn in Netlify UI. Despite that Next keeps its own cache, Yarn cache is not becoming a part of it, as if the environment variable is not there.

The oddest thing is this:

11:47:01 PM: Starting to download cache of 289.4MB 👈 👈 👈 👈 👈
11:47:03 PM: Finished downloading cache in 1.838382038s
11:47:03 PM: Starting to extract cache
11:47:12 PM: Finished extracting cache in 9.146324277s
11:47:12 PM: Finished fetching cache in 11.039018404s
11:47:12 PM: Starting to prepare the repo for build

Something is being cached somewhere and it’s clearly bigger than Next.js cache (≈50MB).

I am giving up after wasting the whole evening 😭 Does anyone have any working solution for Yarn 3? If you do, please share 🙏

[ZotyoNegentropics]

I am giving up after wasting the whole evening 😭 Does anyone have any working solution for Yarn 3? If you do, please share 🙏

Hi @kachkaev ,

We don't use Next, we use React App with Create React App (4.0.3) and the yarn cache is working (but it wasn't easy to find the "rigth" setup)

Our netlify.toml:

[build.environment]
   NODE_VERSION = "16"
   YARN_VERSION = "3.1.1"
   YARN_CACHE_FOLDER = "/opt/buildhome/.yarn_cache"

We have .yarnrc and .yarnrc.yml with the same content:
(The .yarnrc the official yarn config file and we need it for local development and netlify uses the .yarnrc.yml file)

nodeLinker: node-modules

plugins:
  - path: .yarn/plugins/@yarnpkg/plugin-interactive-tools.cjs
    spec: "@yarnpkg/plugin-interactive-tools"

yarnPath: .yarn/releases/yarn-3.1.1.cjs

Our netlify build log:

12:49:01 PM: Build ready to start
12:49:03 PM: build-image version: 122b31996ccaffd45d820a452d6227f8312110cc (focal)
12:49:03 PM: build-image tag: v4.5.3
12:49:03 PM: buildbot version: 44e4d661fad7f5ef0b9bffed6dfdedaa2c1ad008
12:49:03 PM: Fetching cached dependencies
12:49:03 PM: Starting to download cache of 730.3MB
12:49:08 PM: Finished downloading cache in 4.930520996s
12:49:08 PM: Starting to extract cache
12:49:31 PM: Finished extracting cache in 22.562380332s
12:49:31 PM: Finished fetching cache in 27.65318033s
...
12:49:40 PM: Started restoring cached yarn cache
12:49:41 PM: Finished restoring cached yarn cache
12:49:42 PM: No yarn workspaces detected
12:49:42 PM: Started restoring cached node modules
12:49:42 PM: Finished restoring cached node modules
12:49:43 PM: Installing NPM modules using Yarn version 3.1.1
12:49:44 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
...

12:49:45 PM: ➤ YN0000: ┌ Fetch step
12:49:46 PM: ➤ YN0013: │ 1780 packages were already cached
12:49:46 PM: ➤ YN0000: └ Completed in 0s 957ms

We are getting this warning, so I don't know how long it will work.
12:49:44 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead

I hope this will help you!

Ps.: I checked the Netlify build-image repo to find out how the netlify cache works and it helped a lot.

[kachkaev]

Thanks a lot for your reply @ZotyoNegentropics – your solution worked! All I had to do in my case was adding these lines to netlify.toml:

  [build]
  publish = ".next"

+ ## Yarn 3 cache does not work out of the box as of Jan 2022. Context:
+ ## https://github.com/netlify/build/issues/1535#issuecomment-1021947989
+ [build.environment]
+ YARN_CACHE_FOLDER = "/opt/buildhome/.yarn_cache"
+ YARN_VERSION = "3.1.1"
+
  [[plugins]]
  package = "@netlify/plugin-nextjs"

Here is the result:

9:47:51 AM: ➤ YN0000: ┌ Fetch step
9:47:52 AM: ➤ YN0013: │ 740 packages were already cached
9:47:52 AM: ➤ YN0000: └ Completed in 0s 866ms

🎉

I think I tried both YARN_CACHE_FOLDER and YARN_VERSION in my earlier experiments. Maybe I did not set both correct values at the same tome or there is some difference between Netlify UI and the toml file here.

Thanks again – and I hope that Netlify will fix this permanently soon!

[ineffyble]

Just tried to set up a new Netlify site with a Netlify Build Plugin defined in netlify.toml and hit this. Disabling Yarn PnP by using nodeLinkers: node-modules worked but is non-ideal.

[lukasholzer]

@ineffyble do you have a reproducible example? (with the plugin inside the netlify.tom) So that I can troubleshoot whats happening here?

[ineffyble]

@lukasholzer Have a look at https://github.com/ineffyble/effy.space/tree/broken - I've just reverted the change in this branch so that you can see the netlify build failure.

[lukasholzer]

@ineffyble I see yarn Plug'n Play is failing for you inside the plugins installation for local plugins:

https://app.netlify.com/sites/bespoke-meerkat-b0b937/deploys/6377551993b59a000932e6ba

[ineffyble]

@lukasholzer I'm not quite sure what you mean. Yarn is happy, but Netlify believes the plugin isn't installed.

[lukasholzer]

This happens locally as well for me when running netlify build with the CLI

[Quentin-Danjou]

Hello @lukasholzer! I just happen to have found this thread and was wondering where are we in the support of Yarn PnP ( and also Corepack)?

Also will Netlify support Yarn v4 that landed few month ago?

The current documentation says that PnP is not supported (https://docs.netlify.com/configure-builds/manage-dependencies/#yarn)