[v2] Hulksmash build slowdowns on larger sites #6226

KyleAMathews · 2018-06-29T03:44:01Z

Spent the afternoon and evening going through critical path for creating pages.

Currently a 5000 page site can build in ~37 seconds. A very significant
improvement over the current.

This does necessitate one breaking change namely creating nodes for each page. This frankly
was more of academic interest and for debugging purposes and given that it adds a
large amount of slowdowns and should be rarely if ever used in sites, we should
hopefully be finish letting it go. In any case, if people are using it, there's
far better ways of querying the same data.

TODO

Evaluate why "write out pages data" gets really slow at 25k
Do node profiling on adding thousands of pages
remove profile calls
Calculate how much faster large sites are with this PR vs. current beta.
Do memory profiling on same site.
test freecodecamp
add benchmarking folder and first two sites — minimal createPages in memory and larger one that writes out markdown files
Add progress indicator for longer-running things like graphql queries + page building w/ final status at end
Remove extra packages from benchmark sites

KyleAMathews · 2018-06-29T03:48:58Z

Deploy preview for using-glamor failed.

Built with commit fa32e5d

https://app.netlify.com/sites/using-glamor/deploys/5b457017c6aed64e9461ef06

KyleAMathews · 2018-06-29T03:56:39Z

Hmm and a 25,000 page site builds in ~7.5 minutes. The file for pages metadata is now enormous (~4mb) so we'll need to fix that but this is all very promising.

KyleAMathews · 2018-06-29T06:02:35Z

Hmmm, gatsbyjs.org isn't building since StaticQueries aren't being run during builds. Is this a known issue? Been heads down last few days.

m-allanson · 2018-06-29T08:25:46Z

packages/gatsby/src/internal-plugins/query-runner/query-runner.js

  }

+  // Delete internal data from pageContext
+  delete result.pageContext.jsonName


Refs: #5096

pieh · 2018-06-29T09:36:26Z

This does necessitate one breaking change namely creating nodes for each page. This frankly
was more of academic interest and for debugging purposes and given that it adds a
large amount of slowdowns and should be rarely if ever used in sites, we should
hopefully be finish letting it go. In any case, if people are using it, there's
far better ways of querying the same data.

This would break current implementation of sitemap plugin I think - but we can get that info for redux store instead of querying for it. This also would allow us to skip updating schema part (which is needed to add SitePage related queries to schema right now)

Switching nodes reducer to map and mutating it instead of creating new state for every CREATE_NODE action will defenitely help here too

Hmmm, gatsbyjs.org isn't building since StaticQueries aren't being run during builds. Is this a known issue? Been heads down last few days.

I'm not seeing this here

m-allanson · 2018-06-29T09:40:16Z

Hmmm, gatsbyjs.org isn't building since StaticQueries aren't being run during builds. Is this a known issue? Been heads down last few days.

I'm not seeing this here

Me neither. Seems like nice speedups for .org. Unscientific tests running gatsby build a few times (with warm cache):

current beta
✨  Done in 130.25s.
✨  Done in 97.00s.
✨  Done in 163.13s.

this PR
✨  Done in 72.71s.
✨  Done in 70.16s.
✨  Done in 74.66s.

Edit: oh yeah, gatsby develop doesn't seem quite right. Content changes take a long time and StaticQuery doesn't always update.

KyleAMathews · 2018-06-29T15:24:57Z

Switching nodes reducer to map and mutating it instead of creating new state for every CREATE_NODE action will defenitely help here too

Oh, great point. That would speed up creating the SitePage nodes a ton. I'll try that too in this PR before declaring SitePage nodes dead.

KyleAMathews · 2018-06-29T15:25:38Z

I'm not seeing this here

Hrrmmm... weird. I deleted node_modules and yarn.lock. Will keep poking at this.

KyleAMathews · 2018-06-29T15:26:52Z

This also would allow us to skip updating schema part (which is needed to add SitePage related queries to schema right now)

This would be nice still. Though it seems your changes have been making the schema generation a lot faster.

jquense · 2018-06-29T18:24:24Z

packages/gatsby/src/commands/build-html.js

@@ -13,7 +13,7 @@ module.exports = async (program: any) => {

  debug(`generating static HTML`)
  // Reduce pages objects to an array of paths.
-  const pages = store.getState().pages.map(page => page.path)
+  const pages = [...store.getState().pages.values()].map(page => page.path)


Array.from(store.getState().pages.values(), page => page.path) maybe?

Oh haha — didn't know this was part of Array.from. Yeah, totally makes sense.

i just learned it myself!

jquense · 2018-06-29T18:25:11Z

packages/gatsby/src/internal-plugins/query-runner/page-query-runner.js

@@ -81,7 +90,7 @@ const findIdsWithoutDataDependencies = () => {
  // paths.
  const notTrackedIds = _.difference(
    [
-      ...state.pages.map(p => p.path),
+      ...[...state.pages.values()].map(p => p.path),
      ...[...state.staticQueryComponents.values()].map(c => c.jsonName),


we should do Array.from(set, mapFn) here probably as well

KyleAMathews · 2018-06-30T01:11:54Z

Finally got gatsbyjs.org to work — had deleted public/static/d without also deleting the .cache directory 🤦‍♂️

Builds are taking anywhere between ~55-80 seconds on a warm cache. Not bad!

gatsbybot · 2018-06-30T02:50:05Z

Deploy preview for using-drupal ready!

Built with commit b647bb1

https://deploy-preview-6226--using-drupal.netlify.com

gatsbybot · 2018-06-30T02:51:28Z

Deploy preview for gatsbygram ready!

Built with commit b647bb1

https://deploy-preview-6226--gatsbygram.netlify.com

KyleAMathews · 2018-07-04T01:48:41Z

This is coming along!

Just built a 25k page site in a bit over 2 minutes

success open and validate gatsby-config — 0.082 s
success onPreBootstrap — 0.071 s
success delete html and css files from previous builds — 0.007 s
success copy gatsby files — 0.043 s
success source and transform nodes — 0.019 s
success building schema — 0.133 s
success createPages — 33.258 s
success createPagesStatefully — 0.184 s
success onPreExtractQueries — 0.000 s
success update schema — 0.056 s
success extract queries from components — 0.075 s
success run graphql queries — 21.329 s
success write out page data — 32.011 s
success write out redirect data — 0.001 s
success onPostBootstrap — 0.004 s

info bootstrap finished - 91.063 s

success Building production JavaScript and CSS bundles — 13.668 s
success Building static HTML for pages — 29.937 s
info Done building in 134.79 sec

Creating pages can still probably be made a lot faster. Writing out page data is getting weirdly slow w/ large number of pages but progress.

For building a 25k page site, it reduced the time spent writing out page data from 32 seconds to 0.32 seconds 😱

KyleAMathews · 2018-07-06T01:41:02Z

Aaaandddd dropped a 25k build site another 75% to 32 seconds :-D

/p/t/my-hello-world gatsby build
success open and validate gatsby-config — 0.007 s
success onPreBootstrap — 0.027 s
success delete html and css files from previous builds — 0.005 s
success copy gatsby files — 0.036 s
success source and transform nodes — 0.015 s
success building schema — 0.088 s
success createPages — 4.843 s
success createPagesStatefully — 0.111 s
success onPreExtractQueries — 0.003 s
success update schema — 0.055 s
success extract queries from components — 0.076 s
success run graphql queries — 8.749 s
success write out page data — 0.328 s
success write out redirect data — 0.002 s
success onPostBootstrap — 0.001 s

info bootstrap finished - 16.599 s

success Building production JavaScript and CSS bundles — 3.022 s
success Building static HTML for pages — 12.919 s
info Done building in 32.619 sec

A 10k page site builds in ~18 seconds and a 100k page site builds in 175 seconds.

Not really loving all the nested caching I'm adding... but it's necessary. Will revisit this Monday to see if there's a cleaner way to cache things.

zachgibson · 2018-07-08T23:02:28Z

This frankly was more of academic interest and for debugging purposes and given that it adds a large amount of slowdowns and should be rarely if ever used in sites, we should hopefully be finish letting it go.

Is this still the case, or will this be a viable solution? I’m looking at using Gatsby for a 400k page site, but not sure if things would just explode trying to do this.

KyleAMathews · 2018-07-09T04:25:05Z

@zachgibson not sure I understand what you mean?

zachgibson · 2018-07-09T16:09:04Z

@KyleAMathews It seemed your comment was saying this PR was an experiment. I was wondering if the techniques you ended up finding in this work would end up getting merged into core Gatsby.

m-allanson · 2018-07-09T16:15:14Z

@zachgibson the plan is to merge this in to Gatsby core as soon as we can!

tradziej · 2018-07-17T14:09:59Z

@KyleAMathews I don't know how this works but is there a need to keep all those empty directories (public/static/d/*) after the build (gatsby build) was finished?

gatsby/packages/gatsby/src/bootstrap/index.js

Line 185 in d863bcb

fs.ensureDir(`${program.directory}/public/static/d/${i}`)

KyleAMathews · 2018-07-17T19:53:07Z

@tradziej yeah :-) I'm going to put up a PR in a sec to not pre-create the folders.

stoltzrobin · 2018-08-16T07:38:25Z

@KyleAMathews Intresting to see your benchmarks. I'm at the moment trying out building ~12k pages and it takes around 22 minutes. Our bottleneck is the graphQL queries that takes around 1000s-1500s to complete. What type website did you build with this timings? As I see in your output the GraphQL queries take not that long time to run, how many queries do you run?

KyleAMathews · 2018-08-16T11:16:59Z

The benchmark sites https://github.com/gatsbyjs/gatsby/tree/master/benchmarks

Would love to hear more details about your site in another issue to see if we can figure out how to optimize it!

stoltzrobin · 2018-08-16T11:20:23Z

@KyleAMathews thanks, will check that out.

Will post another issue if I can't find out any other ways to do it.

I've created an issue for this now #7373

Hulksmash slowdowns on larger sites

ac4a874

m-allanson reviewed Jun 29, 2018

View reviewed changes

jquense reviewed Jun 29, 2018

View reviewed changes

KyleAMathews added 2 commits June 29, 2018 19:23

Switch more existsSync to use caching version

cb33713

Convert node reducer to use Map

253e353

KyleAMathews added 2 commits June 29, 2018 20:52

Fix most lint errors

84c516a

Merge branch 'master' into speed-large-site

b647bb1

KyleAMathews added 3 commits July 3, 2018 19:53

Split writing page data json files into 999 folders to speed writes

266f5ae

Use forEach instead of reduce when prepping page data

110b99e

For building a 25k page site, it reduced the time spent writing out page data from 32 seconds to 0.32 seconds 😱

Profiled code and fixed hot functions

710d9d8

KyleAMathews added 2 commits July 6, 2018 15:17

Merge remote-tracking branch 'origin/master' into speed-large-site

b039a64

WIP commit to dramatically speed up graphql queries

6a8e9a0

Not really loving all the nested caching I'm adding... but it's necessary. Will revisit this Monday to see if there's a cleaner way to cache things.

KyleAMathews added 15 commits July 10, 2018 16:14

Disable profiling

fe4125b

Remove extra dependencies

d3a95a0

Format + fix linting

d6eefc5

remove console.log and unused profile

b729410

Remove another profile

34db11f

Fix tests (hopefully)

daf690f

Merge remote-tracking branch 'origin/master' into speed-large-site

db319af

Check if nodes exists, sometimes a test fails otherwise

d3acb9e

debugging...

145a34b

Work around...

c85d41e

Don't need to run bootstrap on tests as jest compiles code

2cb28c7

Lodash uses Object.entries which fails on node 6... :-"

b7ebd8e

Ok, do need bootstrap to link packages

a4a1391

Try try again

aa47043

Ugh, problem was Object.entries isn't in Node 6

fa32e5d

KyleAMathews merged commit 27c644b into master Jul 11, 2018

KyleAMathews deleted the speed-large-site branch July 11, 2018 02:56

porfirioribeiro mentioned this pull request Jul 11, 2018

[v2] Page hot reload error #6382

Closed

m-allanson mentioned this pull request Jul 11, 2018

Fix gatsby-transformer-screenshot #6390

Merged

KyleAMathews mentioned this pull request Jul 14, 2018

Upgrade to latest gatsby + use id for lots of speedups freeCodeCamp/guide#8274

Merged

3 tasks

pieh mentioned this pull request Jul 19, 2018

[v2] Evaluate fields added to "pageContext" and eliminate ones that seem very unlikely to be used e.g. internally keys #5096

Closed

KyleAMathews mentioned this pull request Jul 27, 2018

Stack size limit on generation of more then 10000 pages #3341

Closed

KyleAMathews mentioned this pull request Sep 10, 2018

Gatsby build slow (800 pages) #3790

Closed

kakadiadarpan mentioned this pull request Oct 30, 2018

Gatsby Limit? #9542

Closed

swyxio mentioned this pull request Jan 17, 2022

Bad Ways to Get Data swyxio/swyxdotio#111

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[v2] Hulksmash build slowdowns on larger sites #6226

[v2] Hulksmash build slowdowns on larger sites #6226

KyleAMathews commented Jun 29, 2018 •

edited by hu0p

Loading

KyleAMathews commented Jun 29, 2018 •

edited

Loading

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

m-allanson Jun 29, 2018

pieh commented Jun 29, 2018 •

edited

Loading

m-allanson commented Jun 29, 2018 •

edited

Loading

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

jquense Jun 29, 2018

KyleAMathews Jun 29, 2018

jquense Jul 2, 2018

jquense Jun 29, 2018

KyleAMathews commented Jun 30, 2018

gatsbybot commented Jun 30, 2018 •

edited

Loading

gatsbybot commented Jun 30, 2018 •

edited

Loading

KyleAMathews commented Jul 4, 2018

KyleAMathews commented Jul 6, 2018

zachgibson commented Jul 8, 2018

KyleAMathews commented Jul 9, 2018

zachgibson commented Jul 9, 2018

m-allanson commented Jul 9, 2018

tradziej commented Jul 17, 2018

KyleAMathews commented Jul 17, 2018

stoltzrobin commented Aug 16, 2018

KyleAMathews commented Aug 16, 2018

stoltzrobin commented Aug 16, 2018 •

edited

Loading

[v2] Hulksmash build slowdowns on larger sites #6226

[v2] Hulksmash build slowdowns on larger sites #6226

Conversation

KyleAMathews commented Jun 29, 2018 • edited by hu0p Loading

TODO

KyleAMathews commented Jun 29, 2018 • edited Loading

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

m-allanson Jun 29, 2018

Choose a reason for hiding this comment

pieh commented Jun 29, 2018 • edited Loading

m-allanson commented Jun 29, 2018 • edited Loading

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

KyleAMathews commented Jun 29, 2018

jquense Jun 29, 2018

Choose a reason for hiding this comment

KyleAMathews Jun 29, 2018

Choose a reason for hiding this comment

jquense Jul 2, 2018

Choose a reason for hiding this comment

jquense Jun 29, 2018

Choose a reason for hiding this comment

KyleAMathews commented Jun 30, 2018

gatsbybot commented Jun 30, 2018 • edited Loading

gatsbybot commented Jun 30, 2018 • edited Loading

KyleAMathews commented Jul 4, 2018

KyleAMathews commented Jul 6, 2018

zachgibson commented Jul 8, 2018

KyleAMathews commented Jul 9, 2018

zachgibson commented Jul 9, 2018

m-allanson commented Jul 9, 2018

tradziej commented Jul 17, 2018

KyleAMathews commented Jul 17, 2018

stoltzrobin commented Aug 16, 2018

KyleAMathews commented Aug 16, 2018

stoltzrobin commented Aug 16, 2018 • edited Loading

KyleAMathews commented Jun 29, 2018 •

edited by hu0p

Loading

KyleAMathews commented Jun 29, 2018 •

edited

Loading

pieh commented Jun 29, 2018 •

edited

Loading

m-allanson commented Jun 29, 2018 •

edited

Loading

gatsbybot commented Jun 30, 2018 •

edited

Loading

gatsbybot commented Jun 30, 2018 •

edited

Loading

stoltzrobin commented Aug 16, 2018 •

edited

Loading