[Feature] pack/publish non-workspace folder

[bgotink]

• I'd be willing to implement this feature
• This feature can already be implemented through a plugin

Describe the user story

As a developer I have a build step in my package but I want to publish a flat package.

For example, my package's typescript sources are in a src subfolder. In the published package I don't want there to be a dist or lib folder with the resulting javascript files. I want the javascript files to be in the root of the published package. See also: the angular package format (spec)
On the other hand, I don't want to have these javascript files dirty my workspace. Our current approach is to have a dist folder which contains the built javascript files, the package manifest and any assets, which then gets published instead of publishing the workspace folder.

Describe the solution you'd like

I'd like to be able to use yarn npm publish and yarn pack to publish folders that are not a workspace.

I can see two ways for this to work at first glance:

• Like in yarn v1 make the yarn pack and yarn npm publish commands work in any folder, regardless of whether it's a workspace
• Add a property to tell yarn that we run pack/publish in a workspace, it should actually run in a different folder. This could be
  • a command parameter, but we'd have to pass it with every execution which is annoying and error-prone
  • a property in package.json, but then we'd be adding custom properties to the package.json…
  • a configuration variable, but this folder isn't necessarily the same for every package (e.g. we've currently got <repo root>/dist/@scope/package as output folder)

Describe the drawbacks of your solution

• Extra complexity in the pack/publish commands
• Allowing pack/publish everywhere is another way for people to make mistakes + it changes what happens if you run yarn pack in a subfolder of a package

Describe alternatives you've considered

• Not using yarn 2 (e.g. npm) to pack & publish
• Making the build output the package folder, but that looks messy and it breaks our setup where we've got an index.js that loads ts-node and then re-exports the src/index.ts file to allow us to run our workspace packages from source within the workspace.
• Publishing non-flat packages. This is actually what we did in our original project setup four years ago, and it led to issues with secondary entrypoints in packages (@scope/package/entry-point)
• Implement this as a plugin. This is the most viable alternative imo, but also leads to duplicating a lot of yarn logic

[arcanis]

I'm not super enthusiast about this - I feel like fixing #650 would solve the same use case and more, wdyt? (instead of flattening the package, you'd just remap it to expose the lib folder at the root - possibly using something like publishConfig.exports)

[bgotink]

Let's start with: I fully understand the lack of enthusiasm.

The package exports proposal looks like it covers a lot of our use case, but…

1. We run some tests on our packages as if we're a consumer of the package. Using these package exports I'm not sure how we could do that without breaking our development flow where we use the typescript source of our packages
2. We would prefer to generate these exports, because angular packages have a fixed structure and writing them by hand is error-prone & exposes an implementation detail of the builder to every package that it builds. Extra downside: generating content at build time in a file that's version controlled is generally not a pleasant experience.
3. We need to have a bunch of extra properties in the published package.json (next to main we need typings, module, fesm5, fesm2015 etc) that don't need to be present in the repo. We would prefer to generate these instead of write by hand because history teaches us mistakes happen way too often. Of course this could move to package exports as well, but that's up to angular's devkit and I assume typescript'll have to support this first as well.
4. We've also got some extra properties in our manifest that we would need to change, e.g. the schematics property points to a JSON file pointing to javascript files in the package. We'd need to replace the path to the JSON file in the published package.
5. For non-flat packages it's harder to follow the angular package format.

We can cover most of these "but"s via a yarn plugin:

1. ??? I've thought of a couple of things but they all have issues / downsides:
   • Could we "pack" to a folder (instead of a tarball) so we can point tsconfig paths to these folders? But how would we be able to load dependencies in this folder in a pnp context? I'd be running into this exact same problem in our current setup if I had gotten that far yet in testing.
   • Could we change resolution somehow to e.g. load from dist if some environment variable is set? Not without complicating the pnp file a lot
   • Could we point the manifest's main/typings/... to the (nonexistent) built files and overrule that via tsconfig paths to load from source instead? That's actually what we've done in a previous iteration of our project setup, but it lead to way too much lost time because of user error (wrong tsconfigs for some packages, paths in manifest are version controlled instead of generated, …) so we moved away from that.
2. A yarn plugin with a beforeWorkspacePacking hook can handle this
3. A yarn plugin with a beforeWorkspacePacking hook can handle this
4. A yarn plugin with a beforeWorkspacePacking hook can handle this
5. It's harder, not impossible.

That being said, I'm not sure if the same is true for some other build tools that output into a different folder, e.g. Google's bazel (which currently uses npm to publish).

In any case I'll be trying out some stuff with a plugin, but I'm not happy with the possible solutions so far for point 1.

[AndyClausen]

For our use case, we have a root level dist folder for a monorepo where all workspaces are compiled to. This issue is keeping us from publishing packages right now. @bgotink did you figure out a workaround with a plugin or something?

[bgotink]

Yeah, @AndyClausen, I'm using a homebrew plugin to publish packages from another folder.

The version I'm using daily at work is propietary, but it is more or less identical in idea to this one I've created in my spare time: https://github.com/snuggery/snuggery/blob/main/packages/yarn-plugin-snuggery-workspace/src/commands/publish.ts

Note that this plugin makes the assumption that yarn pack (or rather the alternative pack command in the same plugin, because yarn's own pack command also doesn't support packing other folders) is executed separately, which matches this specific use case. You might want to run pack as part of your publish command depending on your usage.

Also note I'm not sure how well (if at all) pre/post pack/publish lifecycle scripts are handled by the plugin. We use the angular CLI to publish via ng deploy, so our "pre-pack script" is made part of that command rather than a real prepack script in the workspace.

[AndyClausen]

Thank you so much for sharing, and for the elaborate explanation!

Also, anyone publishing compiled files from berry workspaces will most likely meet this issue. I expect this to become a big problem once more people start using berry.

[wawyed]

I just want to mention that I have the exact same problem with using an angular cli project to manage multiple libraries. The way angular cli builds the packages it creates a root level dist folder with all the libraries ready to publish in there. Which is separate to the workspaces folder. Would be great to get a way to publish those even though they are not part of the workspace.

[AndyClausen]

I haven't managed to fix this besides an ugly workaround where I change the workspace root to the build folder.
This is still a big problem for monorepos with generated package.jsons in the build folder - or any other project with build files outside the source root.

There is no reason for this blocker to prevent publishing from a separate folder, is there?

Could we get a yes or no on a solution for this, so we know whether to wait and keep using workarounds or to find a more permanent solution ourselves? It's completely understandable if this is just the intended behavior, I just want to know where to put my time and effort.

[AndyClausen]

For NX users, I've written this plugin. Hope it helps.

[mycroes]

Just want to add my 2cts after spending a day on the exports suggestion...

As @arcanis points out exports should be able to handle this. It also shouldn't be that hard to programmatically generate exports for every directory in src/, for instance. I can also imagine that building to a different folder has major impact on yarn internals. For instance I'm dynamically generating the package version using GitVersion, so if I'd only have that version in the build-directory of my package (which would then not be a workspace), how would yarn figure the inter-workspace dependencies?

That said, I tried using exports in package.json yesterday and I just can't seem to make it work. I'm not sure if there's anything I'm missing setting-wise, but I'd love to see a package with multiple entry points (like f.i. MUI, where you can do import Button from '@mui/material/Button';) that actually works using exports. I was searching my node_modules for packages making use of the exports field and there's barely any packages using it, especially with multiple entry points.

I noticed @eslint/eslintrc actually is using it:

  "type": "module",
  "main": "./dist/eslintrc.cjs",
  "exports": {
    ".": {
      "import": "./lib/index.js",
      "require": "./dist/eslintrc.cjs"
    },
    "./package.json": "./package.json",
    "./universal": {
      "import": "./lib/index-universal.js",
      "require": "./dist/eslintrc-universal.cjs"
    }
  },

So I thought, let's just try to import @eslint/eslintrc/universal and see if it works, which it does. However, it seems it only works because the package actually also has universal.js at the actual import path! I guess the contents of that file go to show just how well exports are supported:

// Jest (and probably some other runtimes with custom implementations of
// `require`) doesn't support `exports` in `package.json`, so this file is here
// to help them load this module. Note that it is also `.js` and not `.cjs` for
// the same reason - `cjs` files requires to be loaded with an extension, but
// since Jest doesn't respect `module` outside of ESM mode it still works in
// this case (and the `require` in _this_ file does specify the extension).

// eslint-disable-next-line no-undef
module.exports = require("./dist/eslintrc-universal.cjs");

This all brings us back to the original question of permitting publish from within another directory, because this compatibility approach still requires all export paths under the package root instead of in a subdirectory.

After analyzing all this I'm contemplating life's choices. Sometimes I think I should stop using NPM, Yarn, TypeScript and JavaScript altogether. Professionally I'm a C# developer (although our function descriptions do list TypeScript these days and I'm the very person who introduced it to the company), I consider myself a polyglot, but I just hate how hard it is to get anything built in this ecosystem (and don't get me wrong, I think Yarn is way better than NPM if we're talking quality of life for developers). But since we've got projects to deliver and I do hope for a future where this all gets easier, I see the following options:

1. Build the package to a separate root, publish using NPM
2. Build the package to the package root, publish using Yarn

As for option 1, I can be quite short: It doesn't work for me either. If the npm publish would actually work, this could be an option, but I guess the problems with inter-workspace dependencies still hold.

So option 2 it is then, with all the possible drawbacks of that choice. Since my packages get build on a CI system that starts from scratch for every commit I don't really need to worry about the pollution that much. I don't need to build during development, although that means I'm never actually testing if the build works until it hits CI (and even then I currently have no guarantee that the built package is actually usable, that's why I ended up here in the first place). I can probably gitignore all the artifacts as well, since the sources are in the src/ folder, so basically ignoring any other folder and index.js should get me going. I still don't like this though, so I'd love to hear @bgotink and @arcanis perspective on this.

[michaelfaith]

For instance I'm dynamically generating the package version using GitVersion, so if I'd only have that version in the build-directory of my package (which would then not be a workspace), how would yarn figure the inter-workspace dependencies?

Does it need to? I feel like yarn is trying to be too much. It shouldn't force a version workflow. It should potentially enable a version workflow, but why should my package manager force how I version my packages in this way, especially if my use case isn't exactly covered in how they've imagined it. It's overreaching, imho.

1. Build the package to a separate root, publish using NPM

This is what I've (begrudgingly) resorted to. And it works fine, despite the "dirtiness" of having to use two package managers to do something that one should (and used to) be capable of. What issues are you having with this flow?

[mycroes]

For instance I'm dynamically generating the package version using GitVersion, so if I'd only have that version in the build-directory of my package (which would then not be a workspace), how would yarn figure the inter-workspace dependencies?

Does it need to? I feel like yarn is trying to be too much. It shouldn't force a version workflow. It should potentially enable a version workflow, but why should my package manager force how I version my packages in this way, especially if my use case isn't exactly covered in how they've imagined it. It's overreaching, imho.

I'm sorry but you totally missed this point. Yarn isn't forcing a version workflow, I'm actually using GitVersion instead of whatever Yarn approach you could take. What is important though is that if I have 5 packages in their own workspaces which have dependencies between them that those are kept in sync when I publish my packages (because a dependency version of workspace:^ is useless in a published package). Other than that I guess they also did a pretty good job with what yarn version can do for you, it just doesn't fit (nor hinder) my current workflow...

1. Build the package to a separate root, publish using NPM

This is what I've (begrudgingly) resorted to. And it works fine, despite the "dirtiness" of having to use two package managers to do something that one should (and used to) be capable of. What issues are you having with this flow?

See the linked issue 😉

[mycroes]

2. Build the package to the package root, publish using Yarn

This sounds great of course, but I can't actually build from ./src to ./ with TypeScript because setting outDir and declarationDir to ./ will actually prevent it from processing ./src. Not Yarn related, but yet another reason why it's hard to publish the workspace itself. Of course I can work around this as well, but as my colleague expressed it this feels like a hack stacked on a hack, stacked on a hack, stacked on a hack... Contemplating life again, figuring if perhaps option 3: copy the entire repository structure is the solution... Help is greatly appreciated!

[michaelfaith]

For instance I'm dynamically generating the package version using GitVersion, so if I'd only have that version in the build-directory of my package (which would then not be a workspace), how would yarn figure the inter-workspace dependencies?

Does it need to? I feel like yarn is trying to be too much. It shouldn't force a version workflow. It should potentially enable a version workflow, but why should my package manager force how I version my packages in this way, especially if my use case isn't exactly covered in how they've imagined it. It's overreaching, imho.

I'm sorry but you totally missed this point. Yarn isn't forcing a version workflow, I'm actually using GitVersion instead of whatever Yarn approach you could take. What is important though is that if I have 5 packages in their own workspaces which have dependencies between them that those are kept in sync when I publish my packages (because a dependency version of workspace:^ is useless in a published package). Other than that I guess they also did a pretty good job with what yarn version can do for you, it just doesn't fit (nor hinder) my current workflow...

I totally get what you're saying. You have peer projects that depend on each other and yarn keeps them in sync. What I'm saying, is I shouldn't be forced to keep them in sync. What if I just want to use a version range (e.g. ^ or ~) or manually sync them. I can't because yarn forces their version syncing, which is what i meant by forcing a version workflow. I should be able to opt out of that if i want to sync my peer package deps differently, which would free all these commands to be able to operate outside of the workspace like yarn 1.x does.

[michaelfaith]

Just to illustrate the point. Let's say I have packages/pkg-a and packages/pkg-b in my workspace.

pkg-a/package.json

{
  "name": "pkg-a"
  "version": "1.0.0"
...
}

pkg-b/package.json

{
  "name": "pkg-b"
  "version": "1.0.0"
  "dependencies": {
    "pkg-a": "^1.0.0"
  }
}

If I bump the version of pkg-a to 1.1.0 and publish it's going to increment the dependency in pkg-b's package.json to ^1.1.0, which perhaps I don't want that. What if I want to support a larger range of versions, or manually adjust that dependency. yarn forcing a version/release workflow with no opt-out, to me, is beyond the scope of what a package manager should force. Great to have it for people whose use-case matches that flow, but it becomes a problem when your workflow doesn't match, and you have to start thinking about how to re-architect your project / pipeline just to use a package manager you otherwise like and appreciate.

[mycroes]

@michaelfaith I think you're wrong and maybe Workspaces | Yarn can help you out. More importantly, let's stop this discussion in this thread. I'd love to think along, but let's keep the issues clean and focused. Feel free to mention me or email me if you want to continue this discussion with me.

[michaelfaith]

Well, definitely not wrong. I encountered this issue, which is why I'm even here. But I agree, let's keep this issue on task. Happy to discuss with you off-thread.

[mycroes]

OK, I want to reiterate on my previous messages and at my new findings. I can't stand when something (allegedly) doesn't work as documented, so I started with a small sample. Turns out that after all, it does work.

I started with a handcrafted package with CommonJS and ES module, in cjs and esm directories. I exported using conditional exports (require and import). All my files were just named .js, so in order for node to understand that the esm .js files I added are esm, I added a package.json with just {"type":"module"} to the esm directory. At this point I could actually either require my package from a .cjs file or import from a .mjs file, both executed with node to test the basic functionality.

Adding Vite to the mix I first struggled with npm run build, but at some point noticed that npm run dev (which in turn invokes vite) was actually working. Turns out the build script was running tsc first, and there was the next problem. TypeScript actually does support modules, but not without further configuration. You need to set moduleResolution to either node16 or nodenext for it to parse exports from package.json. While this works fine, another consequence of setting this moduleResolution is that you can no longer perform extensionless imports within the tree you're building with TypeScript (error TS2835). This error actually suggests using .js as extionsion for the import, because if you use the actual extension (ts or tsx) you'd get an error (TS2691) stating that an import path cannot end in said extension.

So there we are, this now works, but updating my package to satisfy these constraints will then require me to update my tsconfig.json in consumers. After doing that I'll have to change every import to include the .js extension in order for the import to work again. Also, for index.ts imports using just the directory path you also have to specify the full name now. Also, in case anyone is wondering if this is just something temporary, no, TypeScript will not add support to transform imports to include the .js extension in any way. I think this is fair, valid reasons are given, still I dislike the fact that it seems like I need to do another overhaul to keep the stuff working that was working before.

In relation to the issue at hand, @arcanis suggestion to use exports to overcome the issue of not being able to publish from a non-workspace directory just won't work. It doesn't fix the problem we're having. It opens up a whole new set of problems. Maybe we'll have to face those problems some day anyway, but I'd appreciate if today's not the day.

[mycroes]

Final comment on the current situation from me. I resorted to using prepack scripts to copy the CommonJS files and type definitions into the workspace, and then postpack to clean this up again (sampling the cjs and types directories the files were copied from to see what to clean up). This seems to work well, I have run into occassional issues where the files are in use when they're getting cleaned up. I'm in doubt if I'll just ignore those errors in the cleanup script, but for now it's not bothering me much. I don't actually have exports set at this point and because I do have multiple entry points my packages are kinda worthless as ESM modules, but that's no different than it was before. Theoretically that's fixable by using exports, but then I'll also have to fix the local imports to include file extensions. All in all I haven't made my mind up about that part yet.

I still hope there's an opportunity to improve this situation and that my travel notes here will help others in the same situation.

[Pokang]

Unfortunately having the same issue in an angular monorepo where we build and publish libraries. I had to resort to making my built dist folders listed as workspaces, which I feel is a bad workaround.
Publishing sources is not an option for us so we can't use the files protocol in our package.json to whitelist the built files, because it creates a bad folder structure for the package and breaks exports. It would also make the process unnecessarily more complicated.

Either I'm missing something, and I'm not seeing the recommended way to publish built packages with Yarn, or everyone who wants to publish built packages will run into this issue, or have to make complicated third-party tools to deal with this new restriction.

This was not an issue when we used Yarn Classic, and has made migrating a very tedious process. We could also resort to using npm to publish but we'd of course rather use Yarn everywhere for consistency because we appreciate it.