feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags #28334

KyleAMathews · 2020-11-27T22:47:50Z

We're investigating ways to be smarter about clearing caches.

Currently we delete the entire cache when your gatsby-node.js, gatsby-config.js, or package versions change or you run gatsby clean. This is unnecessarily aggressive — changing these things don't invalidate most caches. The current cache clearing behavior means very expensive operations like downloading and processing images have to be repeated quite often which makes iterating on the website a lot more frustrating than it should.

This PR adds two new flags:

PRESERVE_FILE_DOWNLOAD_CACHE prevents file downloads from being deleted during cache clearing events (other than gatsby clean which still deletes the entire cache).
PRESERVE_WEBPACK_CACHE prevents the webpack cache from deleted with the same caveat for gatsby clean

We will continue to clear other plugin caches e.g. those created by source & transformer plugins. We're looking at ways to safely making more selective clearing those caches as well.

To enable the new cache behavior, add to your gatsby-config.js:

module.exports = {
  flags: {
    PRESERVE_FILE_DOWNLOAD_CACHE: true,
    PRESERVE_WEBPACK_CACHE: true
  },
  plugins: [...]
}

These will also be enabled if you add the FAST_DEV flag which enables a number of WIP ways to speed up the dev server.

KyleAMathews · 2020-11-28T22:43:02Z

Hmmm is this even a breaking change? Why not just release this now? Not behind a flag?

I think I started with this being a flag in mind as I also want to change the behavior of gatsby clean to let people clean only parts of the cache and steer them away from clearing expensive parts of the cache that aren't probably causing trouble — this would be a breaking change but what this PR ended up being doesn't seem like it is.

wardpeet · 2020-11-30T13:12:53Z

Technically this can end up with issues. Unsure if cache keys of webpack & babel are correct. If we change gatsby-source-filesystem we could end up in a weird state. Do I think this is likely, no but better safe than sorry.

… to updated versions of source-filesystem

KyleAMathews · 2020-11-30T18:54:00Z

Some updates from a conversation between @pieh, @wardpeet, and me earlier

this isn't a breaking change per se but we need more testing to be sure we're covering our basis before shipping this to everyone so keep it behind a flag
we need to fix caching for realz for createRemoteFileNode with a new jobs API
add semver check for invite to try file downloads flag so we don't keep inviting people after we've fixed this in gatsby-source-filesystem

packages/gatsby/src/services/initialize.ts

pieh · 2020-12-01T22:24:53Z

packages/gatsby/src/services/initialize.ts

-      // Attempt to empty dir if remove fails,
-      // like when directory is mount point
-      await fs.remove(cacheDirectory).catch(() => fs.emptyDir(cacheDirectory))
+      // Comment out inviet until we can test perf impact


Suggested change

// Comment out inviet until we can test perf impact

// Comment out invite until we can test perf impact

;)

…loads behind flags (#28334) * feat(gatsby): Add new caching behavior for v3 behind a flag * change to CACHE_CLEAR * cleanup * Split into two flags * Update invite wording + do semver check so we don't keep showing this to updated versions of source-filesystem * Add invite for webpack flag * First try grabbing the plugin from and if that fails, try requiring * Don't use del or readdir until measure perf more * Remove now unused imports * Nest ifs (cherry picked from commit f57d590)

…loads behind flags (#28334) (#28425) * feat(gatsby): Add new caching behavior for v3 behind a flag * change to CACHE_CLEAR * cleanup * Split into two flags * Update invite wording + do semver check so we don't keep showing this to updated versions of source-filesystem * Add invite for webpack flag * First try grabbing the plugin from and if that fails, try requiring * Don't use del or readdir until measure perf more * Remove now unused imports * Nest ifs (cherry picked from commit f57d590) Co-authored-by: Kyle Mathews <[email protected]>

…atsby#28334)

SirJosh1987

Like this

gatsbot bot added the status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer label Nov 27, 2020

feat(gatsby): Add new caching behavior for v3 behind a flag

a1df5c4

KyleAMathews force-pushed the v3-cache branch from 7403aa1 to a1df5c4 Compare November 27, 2020 22:49

KyleAMathews added impact: high and removed status: triage needed Issue or pull request that need to be triaged and assigned to a reviewer labels Nov 27, 2020

KyleAMathews added 2 commits November 27, 2020 15:10

change to CACHE_CLEAR

2853ca0

cleanup

a81c184

KyleAMathews changed the title ~~feat(gatsby): Add new caching behavior for v3 behind a flag~~ feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags Nov 30, 2020

KyleAMathews added 2 commits November 30, 2020 09:53

Split into two flags

037dfa2

Update invite wording + do semver check so we don't keep showing this…

24d6c6e

… to updated versions of source-filesystem

Add invite for webpack flag

aa25c92

pieh reviewed Nov 30, 2020

View reviewed changes

packages/gatsby/src/services/initialize.ts Outdated Show resolved Hide resolved

KyleAMathews added 3 commits November 30, 2020 11:24

First try grabbing the plugin from and if that fails, try requiring

6226466

Don't use del or readdir until measure perf more

18db44a

Remove now unused imports

8b5979c

pieh reviewed Dec 1, 2020

View reviewed changes

packages/gatsby/src/services/initialize.ts Outdated Show resolved Hide resolved

KyleAMathews added 2 commits December 1, 2020 14:08

Merge branch 'master' into v3-cache

11de7b5

Nest ifs

1af631b

pieh reviewed Dec 1, 2020

View reviewed changes

pieh approved these changes Dec 1, 2020

View reviewed changes

pieh merged commit f57d590 into master Dec 1, 2020

delete-merged-branch bot deleted the v3-cache branch December 1, 2020 22:44

gatsbybot mentioned this pull request Dec 1, 2020

feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags (#28334) #28425

Merged

fbuireu added a commit to fbuireu/biancafiore that referenced this pull request Dec 22, 2020

Experimental Gatsby parallel sourcing flag added (umbrella gatsbyjs/g…

ca78913

…atsby#28334)

fbuireu added a commit to fbuireu/biancafiore that referenced this pull request Dec 22, 2020

Experimental Gatsby parallel sourcing flag added (umbrella gatsbyjs/g…

3d47c18

…atsby#28334)

SirJosh1987 reviewed Jun 9, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags #28334

feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags #28334

KyleAMathews commented Nov 27, 2020 •

edited

Loading

KyleAMathews commented Nov 28, 2020 •

edited

Loading

wardpeet commented Nov 30, 2020

KyleAMathews commented Nov 30, 2020

pieh Dec 1, 2020

SirJosh1987 Jun 9, 2021

SirJosh1987 Jun 9, 2021

SirJosh1987 left a comment

	// Comment out inviet until we can test perf impact
	// Comment out invite until we can test perf impact

feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags #28334

feat(gatsby): Add new caching clearing behavior for webpack/file downloads behind flags #28334

Conversation

KyleAMathews commented Nov 27, 2020 • edited Loading

KyleAMathews commented Nov 28, 2020 • edited Loading

wardpeet commented Nov 30, 2020

KyleAMathews commented Nov 30, 2020

pieh Dec 1, 2020

Choose a reason for hiding this comment

SirJosh1987 Jun 9, 2021

Choose a reason for hiding this comment

SirJosh1987 Jun 9, 2021

Choose a reason for hiding this comment

SirJosh1987 left a comment

Choose a reason for hiding this comment

KyleAMathews commented Nov 27, 2020 •

edited

Loading

KyleAMathews commented Nov 28, 2020 •

edited

Loading