perf(gatsby-plugin-mdx): Stop clobbering the same file over and over again #27974
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gatsbot
bot
added
the
status: triage needed
pvdz
added
topic: MDX
topic: performance
|
How does memory usage look like? We probably keep many strings in memory, what does gatsbyjs.com say? |
|
No idea. Will have to test this later. Currently have everything wired up for tracing. |
|
Danger run resulted in 1 fail and 1 markdown; to find out more, see the checks page. Generated by 🚫 dangerJS |
vladar
pushed a commit
that referenced
this pull request
Jan 5, 2021
…again (#27974) * perf(gatsby-plugin-mdx): Stop clobbering the same file over and over again * Undo the await part, can be separate PR Co-authored-by: gatsbybot <[email protected]> (cherry picked from commit 3d100e3)
gatsbybot
pushed a commit
that referenced
this pull request
Jan 5, 2021
…again (#27974) (#28874) * perf(gatsby-plugin-mdx): Stop clobbering the same file over and over again * Undo the await part, can be separate PR Co-authored-by: gatsbybot <[email protected]> (cherry picked from commit 3d100e3) Co-authored-by: Peter van der Zee <[email protected]>
This was referenced Mar 6, 2021
This was referenced Nov 5, 2022
This was referenced Nov 5, 2022
This was referenced Dec 25, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This function (
cacheScope) takes a bunch of imports and exports, generates a snippet of JS, conditionally runs it through Babel, and writes the resulting snippet out to a file. The filename is a hash based on the resulting data.This means that if
cacheScopereceives the same inputs, it will have the same behavior. Writing the same output to the same file, over and over again. This PR stops that.It uses a global set to use as session cache which I think is safe since these files should not be deleted during a build (not even
develop).Tested this on
gabe-fs-mdxat 100k pages.The sourcing step went down from 260s to 200s, roughly a 25% perf win.
(Unfortunately the page query step is still 50 minutes)
This work is roughly based on #25808, which I abandoned after some time (I forgot why). I feel pretty confident about this PR.