Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement fs searching in rust #5481

Merged
merged 55 commits into from
Jan 2, 2021
Merged

Implement fs searching in rust #5481

merged 55 commits into from
Jan 2, 2021

Conversation

devongovett
Copy link
Member

@devongovett devongovett commented Dec 15, 2020

This adds a new package, @parcel/fs-search, which implements some very common filesystem searching methods natively in Rust, with a fallback to a JS implementation. These include finding a file by searching up a directory tree, finding a node_modules directory, and checking a number of extensions. Using the native implementation instead of JS is much faster because it avoids crossing the JS -> C++ barrier for every FS call and instead only crosses it once per search.

rust 46104.62 opts/sec (mean: 0.022ms, stddev: 0.016ms, 2500 samples)
js 4051.36 opts/sec (mean: 0.247ms, stddev: 0.185ms, 2500 samples)

This results in ~15% faster builds overall.

TODO

  • Figure out how to build and publish native modules as part of the monorepo
  • Fix the circular dependency which breaks the tests due to import not being lazy (NodeFS -> utils -> fs-search -> NodeFS)

@height
Copy link

height bot commented Dec 15, 2020

Link Height tasks by mentioning a task ID in the pull request title or description, commit messages, or comments.

💡Tip: You can also use "Close T-X" to automatically close a task when the pull request is merged.

@parcel-benchmark
Copy link

parcel-benchmark commented Dec 15, 2020

Benchmark Results

Kitchen Sink ✅

Timings

Description Time Difference
Cold 4.39s -33.00ms
Cached 833.00ms -16.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/legacy/parcel.d5807e82.webp 102.94kb +0.00b 79.00ms +35.00ms ⚠️
dist/modern/parcel.d5807e82.webp 102.94kb +0.00b 48.00ms -28.00ms 🚀
dist/legacy/index.5acd07fb.js 1.07kb +0.00b 1.57s +89.00ms ⚠️
dist/legacy/index.html 701.00b +0.00b 1.57s +84.00ms ⚠️
dist/legacy/index.5859ab37.css 77.00b +0.00b 1.57s +84.00ms ⚠️

Cached Bundles

Bundle Size Difference Time Difference
dist/legacy/parcel.d5807e82.webp 102.94kb +0.00b 59.00ms -32.00ms 🚀
dist/modern/parcel.d5807e82.webp 102.94kb +0.00b 84.00ms +20.00ms ⚠️
dist/legacy/index.5acd07fb.js 1.07kb +0.00b 85.00ms -12.00ms 🚀
dist/legacy/index.html 701.00b +0.00b 83.00ms -10.00ms 🚀
dist/legacy/index.5859ab37.css 77.00b +0.00b 84.00ms -9.00ms 🚀

React HackerNews ✅

Timings

Description Time Difference
Cold 23.98s -1.50s 🚀
Cached 14.53s -7.47s 🚀

Cold Bundles

Bundle Size Difference Time Difference
dist/index.js 479.93kb +0.00b 9.34s -764.00ms 🚀
dist/PermalinkedComment.52a3fc6f.js 4.22kb +0.00b 3.98s +2.80s ⚠️
dist/UserProfile.42c44c22.js 1.70kb +0.00b 3.98s +2.80s ⚠️
dist/logo.24c8bf9e.png 274.00b +0.00b 61.00ms -40.00ms 🚀

Cached Bundles

Bundle Size Difference Time Difference
dist/index.js 479.93kb +0.00b 79.00ms -9.02s 🚀
dist/PermalinkedComment.52a3fc6f.js 4.22kb +0.00b 79.00ms -40.00ms 🚀
dist/UserProfile.42c44c22.js 1.70kb +0.00b 79.00ms -40.00ms 🚀
dist/NotFound.984005f8.js 532.00b +0.00b 43.00ms -6.00ms 🚀
dist/logo.24c8bf9e.png 274.00b +0.00b 44.00ms +13.00ms ⚠️

AtlasKit Editor ✅

Timings

Description Time Difference
Cold 2.06m -3.98s
Cached 2.98s -10.00ms

Cold Bundles

Bundle Size Difference Time Difference
dist/card.f3741dfa.js 53.76kb +0.00b 37.06s -38.62s 🚀
dist/Modal.34634fc9.js 39.03kb +0.00b 5.47s +378.00ms ⚠️
dist/component.088d2ae3.js 30.87kb +0.00b 2.11s -363.00ms 🚀
dist/component.22035ce4.js 22.40kb +0.00b 5.48s +379.00ms ⚠️
dist/DatePicker.a992c084.js 21.16kb +0.00b 6.91s -3.85s 🚀
dist/dropzone.374e8cb6.js 16.03kb +0.00b 24.09s -22.32s 🚀
dist/ui.078d3b2e.js 14.11kb +0.00b 10.01s -1.09m 🚀
dist/workerHasher.83104f73.js 11.90kb +0.00b 11.13s +1.93s ⚠️
dist/card.c2c3e64d.js 5.71kb +0.00b 1.23m +1.02m ⚠️
dist/dropzone.c5e4485d.js 3.44kb +0.00b 15.84s -8.34s 🚀
dist/Modal.420db102.js 3.15kb +0.00b 2.33s -1.41s 🚀
dist/16.93e3596b.js 2.49kb +0.00b 2.08s -385.00ms 🚀
dist/ResourcedEmojiComponent.557284bd.js 2.15kb +0.00b 9.27s -1.11m 🚀
dist/card.09f4d8e9.js 2.15kb +0.00b 13.59s -1.37s 🚀
dist/images.c0aa934a.js 1.90kb +0.00b 7.10s +1.16s ⚠️
dist/feedback.0e55f851.js 1.86kb +0.00b 6.91s -3.85s 🚀
dist/workerHasher.245b987a.js 1.75kb +0.00b 11.13s +1.94s ⚠️
dist/workerHasher.f5a304ed.js 1.75kb +0.00b 1.23m +26.32s ⚠️
dist/list-number.41f28a70.js 1.68kb +0.00b 6.02s -1.07s 🚀
dist/status.eb15cda1.js 1.68kb +0.00b 7.30s +1.07s ⚠️
dist/link.a9025e77.js 1.53kb +0.00b 6.02s -1.07s 🚀
dist/heading6.053111a7.js 1.53kb +0.00b 6.91s +578.00ms ⚠️
dist/emoji.d5ab3e45.js 1.45kb +0.00b 7.10s +1.16s ⚠️
dist/16.7fc8af6d.js 1.45kb +0.00b 2.08s -384.00ms 🚀
dist/heading5.731efbe4.js 1.40kb +0.00b 6.91s +578.00ms ⚠️
dist/expand.7a2a1b06.js 1.38kb +0.00b 6.91s -3.85s 🚀
dist/16.a4fa1a69.js 1.34kb +0.00b 2.08s -384.00ms 🚀
dist/media-card-analytics-error-boundary.5ce95433.js 1.30kb +0.00b 1.23m +1.02m ⚠️
dist/16.55b14beb.js 1.27kb +0.00b 2.08s -385.00ms 🚀
dist/component.242a45f9.js 1.23kb +0.00b 2.31s -1.44s 🚀
dist/16.f1c68cda.js 1.23kb +0.00b 2.31s -1.44s 🚀
dist/panel-warning.7dcb372a.js 1.21kb +0.00b 6.12s -1.05s 🚀
dist/media-picker-analytics-error-boundary.607a1e52.js 1.05kb +0.00b 24.09s -22.33s 🚀
dist/media-viewer-analytics-error-boundary.cceb3bd6.js 995.00b +0.00b 13.59s -1.37s 🚀
dist/simpleHasher.26a74474.js 755.00b +0.00b 11.13s +1.94s ⚠️
dist/index.html 119.00b +0.00b 3.21s -565.00ms 🚀

Cached Bundles

Bundle Size Difference Time Difference
dist/index.a3370519.js 2.31mb -5.00b 🚀 598.00ms +79.00ms ⚠️
dist/pdfRenderer.f8527c1c.js 1.11mb +0.00b 679.00ms +80.00ms ⚠️
dist/editorView.4de8e34d.js 587.39kb +0.00b 702.00ms +70.00ms ⚠️
dist/popup.5d067d62.js 169.21kb +0.00b 697.00ms +77.00ms ⚠️
dist/EmojiPickerComponent.13c2889c.js 139.08kb +0.00b 643.00ms +53.00ms ⚠️
dist/Toolbar.7ab8152a.js 99.13kb +0.00b 702.00ms +70.00ms ⚠️
dist/media-viewer.336ad151.js 72.60kb +0.00b 679.00ms +80.00ms ⚠️
dist/card.f7371915.js 53.89kb +0.00b 680.00ms +90.00ms ⚠️
dist/card.d41d0ab9.js 51.65kb +0.00b 690.00ms +69.00ms ⚠️
dist/Modal.fcb6df16.js 38.80kb +0.00b 513.00ms +46.00ms ⚠️
dist/esm.2811a427.js 27.71kb +0.00b 643.00ms +53.00ms ⚠️
dist/component.22035ce4.js 22.40kb +0.00b 522.00ms +55.00ms ⚠️
dist/DatePicker.5e8e3524.js 21.03kb +0.00b 643.00ms +64.00ms ⚠️
dist/smartMediaEditor.54e2469a.js 16.69kb +0.00b 702.00ms +70.00ms ⚠️
dist/js.0a808d22.js 16.55kb +0.00b 513.00ms +46.00ms ⚠️
dist/dropzone.5effed87.js 15.96kb +0.00b 697.00ms +88.00ms ⚠️
dist/ui.078d3b2e.js 14.11kb +0.00b 680.00ms +90.00ms ⚠️
dist/workerHasher.83104f73.js 11.90kb +0.00b 664.00ms +74.00ms ⚠️
dist/component.1880fd07.js 6.27kb +0.00b 528.00ms +61.00ms ⚠️
dist/card.c2c3e64d.js 5.71kb +0.00b 680.00ms +82.00ms ⚠️
dist/media-viewer.083ce2a1.js 4.01kb +0.00b 679.00ms +59.00ms ⚠️
dist/EmojiPickerComponent.55e1f602.js 3.56kb +0.00b 664.00ms +74.00ms ⚠️
dist/index.10f02a5e.css 3.46kb +0.00b 707.00ms +71.00ms ⚠️
dist/dropzone.c5e4485d.js 3.44kb +0.00b 697.00ms +77.00ms ⚠️
dist/Modal.420db102.js 3.15kb +0.00b 604.00ms +155.00ms ⚠️
dist/clipboard.a67b66af.js 2.97kb +0.00b 697.00ms +88.00ms ⚠️
dist/16.93e3596b.js 2.49kb +0.00b 500.00ms +51.00ms ⚠️
dist/ResourcedEmojiComponent.557284bd.js 2.15kb +0.00b 664.00ms +74.00ms ⚠️
dist/card.09f4d8e9.js 2.15kb +0.00b 680.00ms +60.00ms ⚠️
dist/date.c50d5e8c.js 1.96kb +0.00b 590.00ms +59.00ms ⚠️
dist/images.c0aa934a.js 1.90kb +0.00b 590.00ms +40.00ms ⚠️
dist/feedback.0e55f851.js 1.86kb +0.00b 642.00ms +65.00ms ⚠️
dist/16.8ab9c9cf.js 1.86kb +0.00b 570.00ms +54.00ms ⚠️
dist/browser.dfefe24a.js 1.82kb +0.00b 690.00ms +81.00ms ⚠️
dist/16.aa2b1e8c.js 1.79kb +0.00b 614.00ms +110.00ms ⚠️
dist/workerHasher.245b987a.js 1.75kb +0.00b 664.00ms +74.00ms ⚠️
dist/workerHasher.f5a304ed.js 1.75kb +0.00b 690.00ms +58.00ms ⚠️
dist/list-number.41f28a70.js 1.68kb +0.00b 597.00ms +47.00ms ⚠️
dist/status.eb15cda1.js 1.68kb +0.00b 614.00ms +38.00ms ⚠️
dist/code.d63c0558.js 1.61kb +0.00b 584.00ms +53.00ms ⚠️
dist/link.a9025e77.js 1.53kb +0.00b 597.00ms +47.00ms ⚠️
dist/heading6.053111a7.js 1.53kb +0.00b 634.00ms +58.00ms ⚠️
dist/heading3.9115aad6.js 1.51kb +0.00b 634.00ms +67.00ms ⚠️
dist/16.b19b33ca.js 1.51kb +0.00b 549.00ms +33.00ms ⚠️
dist/16.fe7e09db.js 1.46kb +0.00b 547.00ms +43.00ms ⚠️
dist/emoji.d5ab3e45.js 1.45kb +0.00b 590.00ms +40.00ms ⚠️
dist/16.7fc8af6d.js 1.45kb +0.00b 499.00ms +48.00ms ⚠️
dist/16.474c8276.js 1.44kb +0.00b 584.00ms +54.00ms ⚠️
dist/16.d49a0bb7.js 1.40kb +0.00b 584.00ms +54.00ms ⚠️
dist/heading5.731efbe4.js 1.40kb +0.00b 634.00ms +58.00ms ⚠️
dist/expand.7a2a1b06.js 1.38kb +0.00b 642.00ms +65.00ms ⚠️
dist/16.c4941d0d.js 1.36kb +0.00b 572.00ms +42.00ms ⚠️
dist/16.a4fa1a69.js 1.34kb +0.00b 503.00ms +54.00ms ⚠️
dist/heading2.de347970.js 1.33kb +0.00b 634.00ms +67.00ms ⚠️
dist/16.ad29438e.js 1.32kb +0.00b 572.00ms +42.00ms ⚠️
dist/16.3d7a389d.js 1.31kb +0.00b 584.00ms +53.00ms ⚠️
dist/media-card-analytics-error-boundary.5ce95433.js 1.30kb +0.00b 680.00ms +82.00ms ⚠️
dist/mention.1703453a.js 1.29kb +0.00b 597.00ms +45.00ms ⚠️
dist/heading4.5089aec7.js 1.29kb +0.00b 634.00ms +65.00ms ⚠️
dist/Modal.ae6f5f7a.js 1.28kb +0.00b 513.00ms +46.00ms ⚠️
dist/16.55b14beb.js 1.27kb +0.00b 499.00ms +48.00ms ⚠️
dist/layout.32cc11b1.js 1.27kb +0.00b 597.00ms +47.00ms ⚠️
dist/16.45835899.js 1.27kb +0.00b 570.00ms +54.00ms ⚠️
dist/16.e83a0923.js 1.26kb +0.00b 547.00ms +43.00ms ⚠️
dist/16.d315628f.js 1.26kb +0.00b 549.00ms +33.00ms ⚠️
dist/16.cc8ae77e.js 1.26kb +0.00b 555.00ms +39.00ms ⚠️
dist/divider.6cc4f21a.js 1.25kb +0.00b 590.00ms +40.00ms ⚠️
dist/quote.b905d023.js 1.25kb +0.00b 614.00ms +38.00ms ⚠️
dist/component.242a45f9.js 1.23kb +0.00b 598.00ms +79.00ms ⚠️
dist/action.3a733320.js 1.23kb +0.00b 584.00ms +53.00ms ⚠️
dist/16.f1c68cda.js 1.23kb +0.00b 598.00ms +79.00ms ⚠️
dist/decision.1f5dd6ad.js 1.21kb +0.00b 590.00ms +49.00ms ⚠️
dist/panel-warning.7dcb372a.js 1.21kb +0.00b 614.00ms +38.00ms ⚠️
dist/16.8cd85281.js 1.18kb +0.00b 572.00ms +42.00ms ⚠️
dist/list.7d80b06a.js 1.18kb +0.00b 597.00ms +47.00ms ⚠️
dist/heading1.c1474f93.js 1.18kb +0.00b 614.00ms +47.00ms ⚠️
dist/panel-error.4dcbeea1.js 1.11kb +0.00b 614.00ms +62.00ms ⚠️
dist/panel.9407e91b.js 1.10kb +0.00b 614.00ms +38.00ms ⚠️
dist/table.547bc9c1.js 1.09kb +0.00b 615.00ms +49.00ms ⚠️
dist/media-picker-analytics-error-boundary.607a1e52.js 1.05kb +0.00b 697.00ms +92.00ms ⚠️
dist/panel-success.23208184.js 1.05kb +0.00b 614.00ms +39.00ms ⚠️
dist/media-card-analytics-error-boundary.f92fc998.js 1.05kb +0.00b 680.00ms +90.00ms ⚠️
dist/media-card-analytics-error-boundary.36e013e9.js 1.05kb +0.00b 690.00ms +68.00ms ⚠️
dist/panel-note.7a985368.js 1.05kb +0.00b 614.00ms +58.00ms ⚠️
dist/media-viewer-analytics-error-boundary.cceb3bd6.js 995.00b +0.00b 680.00ms +60.00ms ⚠️
dist/simpleHasher.26a74474.js 755.00b +0.00b 664.00ms +73.00ms ⚠️
dist/simpleHasher.042b95c1.js 755.00b +0.00b 690.00ms +58.00ms ⚠️
dist/index.html 119.00b +0.00b 522.00ms +72.00ms ⚠️

Three.js ✅

Timings

Description Time Difference
Cold 15.59s -215.00ms
Cached 697.00ms +10.00ms

Cold Bundles

No bundle changes detected.

Cached Bundles

Bundle Size Difference Time Difference
dist/Three.js 580.68kb +0.00b 101.00ms +8.00ms ⚠️

Click here to view a detailed benchmark overview.

scripts/link-native.js Outdated Show resolved Hide resolved
Only used by original RPi 2 Model B, backward compatible with armv6, process.arch doesn't distinguish between them
@devongovett devongovett marked this pull request as ready for review December 20, 2020 18:08

return resolveConfig(fs, filepath, filenames, opts);
): Promise<?FilePath> {
// TODO: realpath
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear about this: it doesn't break any tests at the moment to remove it. However, I think it will change again soon anyway though perhaps slightly different (realpath the result rather than the input). See #5183.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will be unnecessary once #5508 is merged since all asset file paths (and therefore where we are searching for config from) will be realpaths. Remove the TODO comment in that PR.

Copy link
Member

@DeMoorJasper DeMoorJasper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

* Use inputFS.realpathSync get realpath of module

* Add test for symlink structure

* add symlink file

* dynamically create symlinks for test

* Update packages/utils/node-resolver-core/src/NodeResolver.js

Co-authored-by: Amin Yahyaabadi <[email protected]>

* Update NodeResolver.js

* Update packages/utils/node-resolver-core/src/NodeResolver.js

Co-authored-by: Amin Yahyaabadi <[email protected]>

* Update packages/utils/node-resolver-core/src/NodeResolver.js

Co-authored-by: Amin Yahyaabadi <[email protected]>

* use realpath

* Tests use overlayFS

* Update packages/utils/node-resolver-core/src/NodeResolver.js

Co-authored-by: Amin Yahyaabadi <[email protected]>

* Realpath at the end of resolution

* Remove realpaths that are now unnecessary

* Fix sourcemaps tests

* Add tests for symlink resolution

* Remove realpath comment

* prettier

Co-authored-by: dishuostec <[email protected]>
Co-authored-by: Niklas Mischkulnig <[email protected]>
Co-authored-by: Jasper De Moor <[email protected]>
Co-authored-by: Amin Yahyaabadi <[email protected]>
Co-authored-by: dishuostec <[email protected]>
Co-authored-by: Will Binns-Smith <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants