Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eliminate node_modules rewriting #1435

Merged
merged 78 commits into from
Jun 28, 2023
Merged

Eliminate node_modules rewriting #1435

merged 78 commits into from
Jun 28, 2023

Conversation

ef4
Copy link
Contributor

@ef4 ef4 commented May 18, 2023

This is picking up the work explored in #1374. I'm going to probably land those same code changes, but in a different order so we can more easily keep the tests green through the refactor.

My plan here:

  1. Extract reusable API for explicitly asking for rewritten packages, as in aac1c75.
  2. Use that API explicitly everywhere that wants to see rewritten packages.
  3. This eliminates the need for MovedPackageCache and lets all package caches get unified as in d7c17fe.
  4. Then it should be easier to land the simplified stage1 as in cc3185e

ef4 added 4 commits May 18, 2023 12:00
It was unused because the code to emit the index doesn't exist, but I'm about to add it, and I'd like to focus on one thihng at a time.
@ef4 ef4 force-pushed the done-rewrite-node-modules2 branch from 11f723b to 8bccbd7 Compare May 19, 2023 14:11
It existed primarily to power our node modules rewriting.
@ef4 ef4 force-pushed the done-rewrite-node-modules2 branch from bc32c93 to 9425b06 Compare May 22, 2023 16:10
@mansona mansona force-pushed the done-rewrite-node-modules2 branch from ec433a3 to 09efde9 Compare June 8, 2023 13:30
ef4 added a commit that referenced this pull request Jun 22, 2023
This is a side-quest for #1435. The engines-related tests fail there because we still have app-tree-merging support for dumping addon files into engines, which requires us to manage two copies of each engine. Since our module-resolver can now handle all the app-tree-merging virtually, we can drop that whole part of AppDiffer (and maybe *all* of AppDiffer) to solve the problem.

And reforming AppDiffer makes this a good time to remove the Stage concept..

I'm doing this on a separate PR because this is going to take a step backard in terms of passing tests before it moves forward again. I expect the first commit breaks some fastboot-specific features in apps (we already handle those same features in addons via the module-resolver).
@ef4
Copy link
Contributor Author

ef4 commented Jun 28, 2023

This is getting really close. The remaining failures are because the new uses of tree-sync that are introduced turn out to be too aggressive (tree-sync wants to control its entire output directory and will delete anything it doesn't recognize, whereas we're trying to blend several outputs into the same directory). Things like the public assets are getting deleted, and the test are picking up on that.

@ef4 ef4 marked this pull request as ready for review June 28, 2023 12:03
@ef4 ef4 merged commit 3030376 into main Jun 28, 2023
@ef4 ef4 deleted the done-rewrite-node-modules2 branch June 28, 2023 12:03
@NullVoxPopuli
Copy link
Collaborator

Congrats! Must feel great to finally have this done!

@ef4 ef4 added the enhancement New feature or request label Jun 28, 2023
@ef4 ef4 changed the title Working to eliminate node_modules rewriting Eliminate node_modules rewriting Jun 28, 2023
@jrjohnson
Copy link
Contributor

This afternoon after the 3.1.0 release we started seeing Cannot find module './node_modules/@percy/sdk-utils/dist/bundle.js' from '/home/runner/work/common/common/node_modules/.embroider/rewritten-packages/@embroider/synthesized-vendor' in our embroider test runs in an addon. I haven't dug into it yet, but wanted to mention it quickly in case it helps and it seems like from limited context it is probably coming from this change.

@Techn1x
Copy link

Techn1x commented Jun 29, 2023

@jrjohnson I was about to comment the same thing, after updating to embroider 3.1.0 - same @percy/sdk-utils package error for me too.

I'm using pnpm if that matters - I get the error even after a fresh pnpm - pnpm store prune && pnpm install --force

@ef4
Copy link
Contributor Author

ef4 commented Jun 29, 2023

@jrjohnson @Techn1x we debugged this today and @mansona is going to prepare the fix. Thanks for reporting.

@jrjohnson
Copy link
Contributor

That's fantastic, thanks @ef4 and @mansona!

@Techn1x
Copy link

Techn1x commented Jun 30, 2023

Oh boy, when I consume the updates I get past that issue with percy/sdk-utils (thanks so much!!), but end up with a whole different error on build.. (note: using pnpm)

Looks like this issue is related to ember-modal-dialog somehow. When I remove that addon from my app, and all references to it, I get past the error and my app builds and runs.

I'm not sure if maybe I have embroider configured incorrectly or someting - it looks very similar to the error stacks encountered here #1371 (comment) and here #1483 but they could be a red herring.

ERROR in ../rewritten-packages/ember-modal-dialog.fc5b408b/_app_/components/modal-dialog.js 1:0-137
Module not found: Error: Can't resolve '../../../../../.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat' in '/home/boverton/Blake/teacher-ui-client/node_modules/.embroider/rewritten-app/components'
resolve '../../../../../.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat' in '/home/boverton/Blake/teacher-ui-client/node_modules/.embroider/rewritten-app/components'
  using description file: /home/boverton/Blake/teacher-ui-client/node_modules/.embroider/rewritten-app/package.json (relative path: ./components)
    Field 'browser' doesn't contain a valid alias configuration
    No description file found in /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon or above
    no extension
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat doesn't exist
    .wasm
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.wasm doesn't exist
    .mjs
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.mjs doesn't exist
    .js
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.js doesn't exist
    .json
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.json doesn't exist
    .ts
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.ts doesn't exist
    .hbs
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.hbs doesn't exist
    .hbs.js
      Field 'browser' doesn't contain a valid alias configuration
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat.hbs.js doesn't exist
    as directory
      /home/boverton/Blake/.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat doesn't exist
 @ ./assets/teacher-ui-client.js 5276:13-68

webpack 5.88.1 compiled with 1 error in 23875 ms
cleaning up...
Build Error (PackagerRunner) in ../rewritten-packages/ember-modal-dialog.fc5b408b/_app_/components/modal-dialog.js

Module not found: Error: Can't resolve '../../../../../.pnpm/@[email protected]_@[email protected]/node_modules/@embroider/macros/src/addon/es-compat' in '/home/boverton/Blake/teacher-ui-client/node_modules/.embroider/rewritten-app/components'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants