Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add batching to Github API requests #2295

Merged
merged 6 commits into from
Jan 26, 2021
Merged

Conversation

samajammin
Copy link
Member

@samajammin samajammin commented Jan 21, 2021

Description

Batch GraphQL requests (10 at a time) to Github API to avoid rate limits & broken builds ☠️

Context for the curious:
https://www.gatsbyjs.com/plugins/gatsby-source-graphql/#performance-tuning

Related Issue

#2160

@samajammin samajammin marked this pull request as draft January 21, 2021 21:41
@github-actions github-actions bot added dependencies 📦 Changes related to project dependencies Type: Internal labels Jan 21, 2021
@samajammin samajammin marked this pull request as ready for review January 23, 2021 01:23
@samajammin samajammin requested a review from wackerow January 23, 2021 01:23
@samajammin
Copy link
Member Author

@wackerow @ryancreatescopy I think I finally got this working 😵

Mind checking this branch out locally & seeing if things work for you?

  1. Check out this branch: git checkout gatsby-source-graphql-fixes
  2. Add ENV variable to .env: GATSBY_EXPERIMENTAL_QUERY_CONCURRENCY=20
  3. Run yarn build (once, or twice, or thrice)

Does anything break!?

@samajammin samajammin changed the title [WIP] Refactor Github API to use Apollo Link Refactor Github API to use Apollo Link Jan 23, 2021
@samajammin samajammin mentioned this pull request Jan 23, 2021
@github-actions github-actions bot added the documentation 📖 Change or add documentation label Jan 23, 2021
@samajammin samajammin changed the title Refactor Github API to use Apollo Link Add batching to Github API requests Jan 23, 2021
@samajammin samajammin force-pushed the gatsby-source-graphql-fixes branch from 849d1e9 to 1dcfe52 Compare January 23, 2021 02:03
Copy link
Member

@wackerow wackerow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Working great on my end! No breaks after running a few builds in a row. 🎉

Copy link
Contributor

@ryancreatescopy ryancreatescopy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it built the first time but had the following... (not sure if that's a problem at this point).
image
image
image
image

@ryancreatescopy
Copy link
Contributor

Second time it built without the errors but still lots of warnings about missing custom ID on headers etc...

@ryancreatescopy
Copy link
Contributor

Same for the thrice time 👍

@samajammin
Copy link
Member Author

Groovy! Seems this is working given all builds appear to work. I'm going to merge this.

@ryancreatescopy re: those .gif errors - I received the same thing originally. Mind running a gatsby clean, then a yarn build & see if those errors persist? Thanks!

@samajammin samajammin merged commit 42ee407 into dev Jan 26, 2021
@samajammin samajammin deleted the gatsby-source-graphql-fixes branch January 26, 2021 02:43
This was referenced Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies 📦 Changes related to project dependencies documentation 📖 Change or add documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants