Loading chunks while a new release is deployed

[antoinerousseau]

Description

I think there is a structural issue with the lazy loading of page chunks.
I came across various ChunkLoadErrors in Sentry reports, and just now it happened to me in Chrome, and it happened when I had loaded a page, then deployed a new version (via Netlify), and then tried to navigate to another page without a full page reload: the new deployed chunks have different paths, so the Gatsby Links can't find the ones it knew... throwing 404s and resulting in not being able to navigate!

Steps to reproduce

You can easily reproduce this by deploying, going to the website, changing code & deploying again, and then trying to navigate to another page.

Otherwise, let me know and I'll setup a test website.

Expected result

The navigation should always work.

Actual result

Nothing happens when the visitor tries to navigate to another page.

Possible fixes

A fix could be to fallback to normal navigation (full page reload), instead of letting the visitor on the same page without any reaction...

Now, I read on other issues that it's supposed to fail because maybe the user if offline and we can't detect it: I disagree, there is a difference between a failed fetch because of network issues) and a 404, and it can be detected in Javascript (and maybe use NavigatorOnLine if available, too).

Environment

  System:
    OS: macOS 10.14.6
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.19.0 - /usr/local/bin/yarn
    npm: 6.10.3 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Languages:
    Python: 2.7.16 - /usr/local/bin/python
  Browsers:
    Chrome: 77.0.3865.120
    Firefox: 69.0
    Safari: 13.0.2
  npmPackages:
    gatsby: ^2.15.29 => 2.15.29
    gatsby-image: ^2.2.24 => 2.2.24
    gatsby-plugin-google-analytics: ^2.1.20 => 2.1.20
    gatsby-plugin-manifest: ^2.2.20 => 2.2.20
    gatsby-plugin-netlify: ^2.1.17 => 2.1.17
    gatsby-plugin-offline: ^3.0.11 => 3.0.11
    gatsby-plugin-react-helmet: ^3.1.10 => 3.1.10
    gatsby-plugin-sharp: ^2.2.28 => 2.2.28
    gatsby-plugin-sitemap: ^2.2.16 => 2.2.16
    gatsby-plugin-styled-components: ^3.1.8 => 3.1.8
    gatsby-plugin-typescript: ^2.1.11 => 2.1.11
    gatsby-source-contentful: ^2.1.45 => 2.1.45
    gatsby-source-filesystem: ^2.1.29 => 2.1.29
    gatsby-transformer-inline-svg: ^0.0.1 => 0.0.1
    gatsby-transformer-sharp: ^2.2.20 => 2.2.20

Related issues

#11194
#2954

[thenovelnomad]

We are having a similar issue, although different cause. We are running behind an authentication proxy, and when the users session expires, navigation fails due to ChunkLoadErrors.

Is this possible related to a regression from this change?#2996

[GooBall]

Same issue here.

To replicate (as of v2.16.1):

1. Clone the base Gatsby starter.
2. Run gatsby build.
3. Run gatsby serve.
4. Go to localhost:9000/, open network tools and hover over the "Go to page 2" link and see the network requests get made for component---src-pages-page-2-js-[HASH].js.
5. Go to your ./public/ folder and delete component---src-pages-page-2-js-[HASH].js.
6. Go back to localhost:9000/ do not refresh and hover over the "Go to page 2" link. This time you will see a 404 for loading the chunk.

The above steps are what a user would experience if they were navigating a site as a new build was deployed. The user will go to click something and nothing will happen - only a page refresh will fix the issue in those cases.

I would be happy to help fix but might need pointing in the right direction as I haven't poked around in the Gatsby code to much.

[KyleAMathews]

/cc @pieh

[jjang16]

I've also got this issue since upgrading gatsby from v2.15.34 to v2.17.1
Rolling back to v2.15.37 works with no problem :)
In my case, the gatsby target in package.json was ^2.15.28.
Calling yarn upgrade has updated the package to v2.17.1, resulting in this issue.

[JustFly1984]

Having same issues with netlify and AWS Cloudfront.

[DevSide]

loadComponent is never caught

gatsby/packages/gatsby/cache-dir/loader.js

Line 195 in 90f0401

return this.loadComponent(componentChunkName).then(component => {

I guess we don't even need to call loadComponent if the current hash compilation has changed.

I'm not sure navigation should keep comparing hash, instead having this information from the loadPage call.

gatsby/packages/gatsby/cache-dir/navigation.js

Line 99 in 90f0401

pageResources.page.webpackCompilationHash !==

The e2e test is skipped 🤔

gatsby/e2e-tests/production-runtime/cypress/integration/compilation-hash.js

Line 37 in 90f0401

it.skip(`should reload page if build occurs in background`, () => {

@pieh, Do you need help to move forward ?

[johndaskovsky]

We're seeing this same issue as well.

[manutdkid77]

Looks like I'm facing a similar issue as this as well, running gatsby 2.15.33 and hosted on Netlify.

Following are the logs in the console from Firefox:

Error: Loading CSS chunk 1 failed.
(/styles.372f89c282666a74bcb5.css) webpack-runtime-a06d2033887a729e229a.js:1:1913
Source map error: Error: request failed with status 404
Resource URL: https://www.websitename.com/webpack-runtime-a06d2033887a729e229a.js
Source Map URL: webpack-runtime-a06d2033887a729e229a.js.map

Edit:
Found a similar issue while browsing starters on Gatsby website:

[JustFly1984]

This is major bug, which contradicts the main goal of CI/CD flow.

[PeterDekkers]

Can anyone identify a version of Gatsby that doesn't exhibit this bug? Rolling back to 2.15.37 as suggested by @jjang16 didn't seem to work for us. In fact we're running 2.15.18 which is also affected.

Or rather is this an issue with one of Gatsby's dependencies?

Sorry, I don't have much more to contribute at this stage apart from questions.

[ryami333]

Strange that the tests were disabled because they were failing, isn't that the idea of tests?? 😅I'd have thought that a regression which breaks a test (disabled or not) would necessitate a pretty hasty hotfix! @pieh I see you're assigned to this one - do you have the context to address it or do you need some help getting to the bottom of it?

[JustFly1984]

@pieh @ryami333 @editkid I can only guess at this point, but I think it could be issue with gatsby-plugin-offline

[johndaskovsky]

@JustFly1984 we have the issue but don't have gatsby-plugin-offline enabled

[JustFly1984]

@johndaskovsky thank you for clarification, this is even worst than I thought.

[ryami333]

Yeah, no gatsby-plugin-offline for us either - it's not that.

[JustFly1984]

@Sever1an can you check if you have same issue?

[mitchellbutler]

We're running an e-commerce site with Gatsby on Netlify and have noticed a number of these ChunkLoadError logs in Sentry whenever we deploy. I've tested this with the official Gatsby starter and it seems to have the same behavior.

I wonder if more people are having this issue and simply aren't noticing because they aren't monitoring client errors or don't have a lot of active sessions during deployment. For now, we're watching real-time sessions and deploying at odd hours to avoid the issue but it would be great not to have to worry about it.

Minimal Steps to Reproduce

I've uploaded a screen recording of the process here.
edit: video no longer available

1. Create a new gatsby site e.g., gatsby new gatsby-issue-18866
2. Publish it to Netlify
3. Navigate to the Netlify link, but do not hover over or click the "Go to page 2" link
4. Keep the window open and deploy some changes to ./src/pages/page-2
5. Wait for Netlify to process the build.
6. Now hover or click the "Go to page 2" link

Result

The link does not function and throws a ChunkLoadError on each hover and click.

The output looks similar to this:

webpack-runtime-fcfa123cba71df322157.js:1 GET https://cocky-jackson-37ee43.netlify.com/component---src-pages-page-2-js-9091273db8b77923343d.js net::ERR_ABORTED 404
webpack-runtime-fcfa123cba71df322157.js:1 Uncaught (in promise) ChunkLoadError: Loading chunk 5 failed.
(error: https://cocky-jackson-37ee43.netlify.com/component---src-pages-page-2-js-9091273db8b77923343d.js)
    at Function.u.e (https://cocky-jackson-37ee43.netlify.com/webpack-runtime-fcfa123cba71df322157.js:1:2172)
    at Object.component---src-pages-page-2-js (https://cocky-jackson-37ee43.netlify.com/app-41cbafd5610a9d24601a.js:1:87364)
    at r.loadComponent (https://cocky-jackson-37ee43.netlify.com/app-41cbafd5610a9d24601a.js:1:78493)
    at https://cocky-jackson-37ee43.netlify.com/app-41cbafd5610a9d24601a.js:1:76405

[PeterDekkers]

I wonder if more people are having this issue and simply aren't noticing because they aren't monitoring client errors.

I think so, yes.

We've noticed ChunkLoadError happen many days after a deploy, likely due to people leaving our website open in their browser for prolonged periods. It's especially problematic when we deploy backwards-incompatible changes to our server-side API and need all our clients to refresh.

[christerwpersson]

We are also running an e-commerce with Gatsby on Netlify (which is a lovely experience overall i must say!) but we too, have been running into a bunch of not so nice experiences for customers who has been in the middle of the checkout flow, as it relies on the user being navigated between different pages, which breaks when we do a new deploy (and was not built with this issue in mind)

Since there are users in the checkout flow basically all the time, there is no 'safe' window where we can do a deploy without putting ourselves in a really bad spot. We've basically had to revert to avoid doing deploys as often, which is obviously a not so nice experience.

I'm not sure this is all on Gatsby though, e.g. if Netlify would avoid deleting all old files (ie. the chunks would still be there) on new deploys this wouldn't be an issue i believe? But it feels like Gatsby should have a safe fallback for this?

[JustFly1984]

We have same issues both on netlify and AWS Cloudfront/S3.

[GooBall]

The only fix I have been able to implement has been to remove all use of gatsby-link and just use anchor tags. Full page reloads are not nice but nothing can be done until this is fixed.

[JustFly1984]

@GooBall in our case page reload is not acceptable, cos we have authenticated state. and Safari drops cookies for cross-domain requests.

[PeterDekkers]

Our site's design relies heavily on page transitions hence removing gatsby-link is not an option for us either.

[smbkr]

I've done a bit of investigation into this myself as it's causing a lot of alerts for us when we deploy, due to active user on our website, and sometimes days later, which I assume is users who had opened the site before a deploy and continued to use it some time later like @editkid describes.

Here's the PR that I believe introduced this bug: #16686

If you look at the CI logs for this PR, the disabled test in e2e-tests/production-runtime/cypress/integration/compilation-hash.js#L37 that @DevSide has pointed out was skipped, so I guess this bug was missed.

The most recent change to packages/gatsby/cache-dir/loader.js prior to the one from the PR links was run against the now disabled test, and passed.

Unfortunately I don't know enough about this side of WebPack or Gatsby to be of much further use, but hopefully this is of some help @pieh

E: we've rolled back to Gatsby ~2.16.0 and it appears that we're no longer experiencing this issue. The issue persists.

[ryami333]

Sorry to "at" you @KyleAMathews, but there hasn't been any comment or activity from @pieh on this since it was assigned to him a couple of months ago, so I'm worried it's just flying under the radar because of the "assigned" status. Is there someone else who might be able to check this out? There seems to be a reasonable number of people affected by this, and it's easily and reliably reproducible with the latest version of Gatsby (including any given "starter").

[ceiphr]

I would love if someone who regularly sees this problem could test out the fix and report back!

Thank you for this! I believe it is working for me. I refactored a few pages on my site and when I clicked a gatsby-router link after deployment, the page refreshed instead of causing a navigation failure.

The only caveat is that errors are still logged by Sentry, so I added ignoreErrors: ["ChunkLoadError"] to my gatsby-config.js. It's probably not the best approach to ignore an error, but in this context it seems like your fix is working.

[antoinerousseau]

Where is the failing Promise anyway? Can we not perform the fallback in its .catch()?

Something like

navigateTo(destination).catch((error) => {
  if (error instanceof ChunkLoadError) {
    window.location.pathname = destination
  } else {
    throw error
  }
})

[KyleAMathews]

I don't think we can catch the error because it's happening inside webpack. So unhandledrejection is our only bet here.

[AlexanderBich]

Has anyone already tested the proposed workound?
It would be great to see some feedback
@philschonholzer - any updates on your side?

[philschonholzer]

@AlexanderBich It does just what it should. Since we added the workaround to gatsby-browser.js the page refreshes automatically after a new deployment and the error disappeared from sentry.io.

@KyleAMathews This should really be part of every Gatsby installation.

[lloydh]

@AlexanderBich It does just what it should. Since we added the workaround to gatsby-browser.js the page refreshes automatically after a new deployment and the error disappeared from sentry.io.

@KyleAMathews This should really be part of every Gatsby installation.

Seconded

[stuckj]

Yep, it's working for us as well. If this gets added in some default way to gatsby, please update this thread so we can remove our patch. :)

[AlexanderBich]

This is happening thousands of times a week for me with the new gatsby-plugin-image since it does dynamic imports on intersection-observer and lazy-hydate causing them to be bundle split. Those assets do exist, so no idea why they randomly fail to load. However, my issue is unrelated to the topic which is geared toward while its being deployed.

gatsby-plugin-image still throws chunk errors even after fix.
But this is maybe due to a different issue and even possibly with service worker of gatsby-plugin-offline and presumably can be fixed by disabling skipWaiting option @mmclafferty

[mmclafferty]

This is happening thousands of times a week for me with the new gatsby-plugin-image since it does dynamic imports on intersection-observer and lazy-hydate causing them to be bundle split. Those assets do exist, so no idea why they randomly fail to load. However, my issue is unrelated to the topic which is geared toward while its being deployed.

gatsby-plugin-image still throws chunk errors even after fix.
But this is maybe due to a different issue and even possibly with service worker of gatsby-plugin-offline and presumably can be fixed by disabling skipWaiting option @mmclafferty

I was able to work around this issue by adding this to my webpack config. It forces their dynamic imports into the main app bundle.

  if (stage === `build-javascript`) {
    config.optimization.splitChunks.cacheGroups.images = {
      name: 'app',
      test(module) {
        return /node_modules[/\\]gatsby-plugin-image[/\\]/.test(
          module.identifier()
        );
      },
    };
  }

[ryami333]

I think this discussion about gatsby-plugin-image and gatsby-plugin-offline is becoming a little bit distracting. As I mentioned above, this issue is easily and reliably reproducible with the latest version of Gatsby (including any given "starter"). Any specific problems that arise from the inclusion of those plugins should be raised in separate issues. We're already starved for the attention of core developers here, so we should be mindful not just to create more noise for them!

[laustdeleuran]

We're running into this as well on Netlify + Gatsby. Would love to hear from the Gatsby folks if there's any plan to bridge this?

[Elyx0]

@KyleAMathews What should be done when the Javascript and CSS step of npx check-gatsby-caching expects something different. (Gatsby 3.1.2+Netlify), getting Chunks errors day and night.

HTML
https://app.charged.fi
✅ headers are correct!


app-data.json
https://app.charged.fi/page-data/app-data.json
✅ headers are correct!


Page Data
https://app.charged.fi/page-data/index/page-data.json
✅ headers are correct!


JavaScript and CSS
https://app.charged.fi/webpack-runtime.js
❌

- Expected
+ Received

  Object {
-   "immutable": true,
-   "max-age": 31536000,
+   "max-age": 0,
+   "must-revalidate": true,
    "public": true,
  }

[KyleAMathews]

@Elyx0 — the last error is fine — immutable is preferred as it's faster but must-revalidate is perfectly safe. This should ideally be a warning not an error — the tool isn't smart enough to differentiate between ideal and less-than-ideal

One thing that could help is adding #18866 (comment) to your gatsby-browser.js

[KyleAMathews]

BTW, I opened up a PR to add my code snippet to the framework #33032

[seanflahertyT4]

It would be awesome to backport this to gatsby 3... I know its not supported anymore but having this bug in the last release of it is quite frustrating for devs who are stuck on gatsby 3 for production sites. https://stackoverflow.com/questions/71506592/gatsby-development-server-stops-on-every-syntax-error