Esbuild override

[konsumer]

It would be handy to have a built-in override for esbuild config, in remix.config.js. Currently I use remix-esbuild-override which messes with your esbuild config (as it's installed in node_modules.) It works ok, if you run it before every build, but it seems flakey & hacky, and it fails to work if I don't setup a prebuild script for it. It seems like it would work better to put it in remix, so we have a 1st-class way to override how esbuild works.

Here is an example config, I am currently using, that injects my override to add buffer into global on cloudflare-workers (so mantine will work) and a couple other things.

/**
 * @type {import('@remix-run/dev').AppConfig}
 */

const nodepoly = require('@esbuild-plugins/node-globals-polyfill').default

module.exports = {
  serverBuildTarget: 'cloudflare-workers',
  server: './server.js',
  devServerBroadcastDelay: 1000,
  ignoredRouteFiles: ['.*'],

  // this is used by remix-esbuild-override
  esbuildOverride: (option, { isServer, isDev }) => {
    // allow import of gql text files for GraphQL
    option.loader = option.loader || {}
    option.loader['.gql'] = 'text'

    // add Buffer to global, so mantine CSS works right
    option.plugins = option.plugins || []
    option.plugins.push(nodepoly({ buffer: true }))

    // make globalThis work in code that assumes global is set
    option.define = option.define || {}
    option.define.global = 'globalThis'

    return option
  }
}

This sets up fake node ES6 stuff for the worker (so like global and Buffer is defined) and lets me import gql files (so I can keep my graphql queries in separate files.)

Related: mantine/issues#1033

[vixalien]

@mjackson @ryanflorence very important!!

(sorry for the pings)

[kiliman]

At the moment, I don't believe there are any plans to open the Remix compiler to customize the configuration. For now, you will need to use patch-package or other techniques to customize your build.

[konsumer]

For now, you will need to use patch-package

As I said, the above works, but it could be a nice feature for remix. Esbuild has such a huge ecosystem of modifier plugins for how it builds, that it would open up a lot of easy & free features for remix.

I looked at it a bit closer here and it appears they move the the esbuild dir, and overwrite how it works, which probably accounts for why it's so flakey. If there is interest in setting up some kind of override options in the builders I can work on a PR, it doesn't seem too terribly complicated. I'd rather not work on it though, if there is no hope of it being merged. Like it works ok for me, now.

@kiliman are you a maintainer? Who makes these kinds of decisions for remix?

[kiliman]

No. I'm not. Just someone that's been using Remix for over a year. The comments are based on discussions with Ryan and Michael on Discord.

Their main concern is that once they enable customizing the compiler, they're stuck supporting that forever. They want the flexibility to change the compiler behavior as needed without worrying about breaking projects.

[konsumer]

First, I think it is a bit funny to speak for other developers, but I will reach out to them myself, on discord. Second, I'm not sure how familiar you are with how it works now, but it is very hardcoded to use esbuild. This change would only allow the user to setup stuff (which is already setup right now, with hardcoded objects in those builders I linked to.) It's not really a loss of anything, it's just a gain, in the sense that it could be configured, but it's not like it further locks remix into esbuild or anything.

[sergiodxa]

Before Remix used esbuild they used to use Rollup and the code also looked very hardcoded to Rollup, yet they were able to do the release without braking anyone code.

I know they want to still be able to do that, and if something better than esbuild is released on the future or if they build their own tool they want to be able to change esbuild without breaking anyone code.

By exposing esbuild they create a hard dependency on it, now people will build their apps around esbuild plugins, they may even write their own slow plugins making Remix build feels slow, so far that’s the reason they decided to don’t expose it.

If you search here on GitHub for previous issues, discussions and pull request about this you will find a few comments from Ryan, Michael, and the rest of the team, explaining this.

[konsumer]

I think I understand that perspective. From that point of view, it seems liek we could say my code is broken now. Just like with rollup -> esbuild, now remix doesn't use rollup, so if you want to use the latest version of it, you have to do it the esbuild way. I don't understand how using the same defaults that are hardcoded now, but allowing config-overrides is any worse of a situation. Like say we switch to a new build system tomorrow, it will have defaults that work the same, and equally need a way to override it's settings. That syntax might be different (because it's another build tool) but it doesn't really change anything.

When I look through recent issues relate to esbuild config,. I see a whole bunch of "I need to change how this one thing works in esbuild" and sometimes PRs associated with new hard-coded values. This suggestion would do away with that entire class of needs, because people could override whatever it is they want.

[vixalien]

It's kinda important, especially for loading some CSS (specifically Stylus) files and other assets. It's not unleashing the power of Esbuild.

[sergiodxa]

You don’t need this to use Stylus, run the Stylus compiler together with Remix and import the output CSS files in your app.

Similar to how Tailwind is used in Remix and documented in the docs.

[vixalien]

I know I could do that but it's kinda nice to have esbuild automatically build it for me instead of running a different terminal and have to manage the output CSS (I would like it to be in the same folder as the components that use them) and make sure they are not committed to git.

[nadiTime]

also being able to use esbuild support for css sourcefiles

[aiji42]

Hi, I am the owner of remix-esbuild-override. I happened to find this discussion.
I too hope that remix will support esbuild override as @konsumer says. I previously created a PR to make it feasible but it was rejected for the same reason as this comment. I understand the team's policy and the risk of providing an esbuild override, but I think that as long as Remix supports multiple runtimes, compiler extensions are essential and should be made available at the user's own risk.
While I am the owner of remix-esbuild-override, I hope that this abominable flakey & hacky library will not be needed if at all possible.

[konsumer]

I hope that this abominable flakey & hacky library will not be needed if at all possible.

I hope there is no offense taken. As I said, it got me running, so I appreciate it, but I agree it would be cool if there was a less-flakey way to do it.

[aiji42]

I am currently working on some examples of solutions for libraries that don't work in cloudflare.
This process has made me realize that most of the errors are caused by libraries that are Buffer dependent. In particular, many of the styling libraries depend on Buffer.

I believe that instead of overriding esbuild, Remix should be supported to automatically polyfill Buffer when cloudflare is selected at runtime.

[aiji42]

I would like to know if anyone knows, the compiler uses @esbuild-plugins/node-modules-polyfill but not @esbuild-plugins/node-globals-polyfill.
https://github.com/remix-run/remix/blob/main/packages/remix-dev/compiler.ts#L7

Since many libraries use global Buffer and process, I think it would be better to use both plugins, but is there any reason why Remix compiler does not use node-globals-polyfill in its plugins?

[aiji42]

If there is no reason why this should be denied, I would like to do Issue and PR creation.

[nicksrandall]

I would LOVE to see this as well. It would be awesome if Remix would expose an API similar to what is described here but mark it as unstable so that it would be clear to users that it is "use at your own risk".

[konsumer]

I made a new issue that tracks all this specific to the instructions for mantine, but generally I think Buffer polyfill built-in would be a good idea. Ideally, I would be able to just override ebsuild how I normally do that, but it seems like that is not an option.

[aiji42]

I won't go into details here because it would defeat the purpose of this discussion, but as far as I can tell, the combination of mantine, cloudflare, and remix-esbuild-override is working fine. (Remix v1.3.5)
https://github.com/aiji42/remix-esbuild-override/tree/main/examples/mantine-cloudflare

[konsumer]

As I said in the issue, you can just follow the instructions on the mantine site, for working with remix, and it will fail, if you use cloudflare workers. It may be some other combination of libraries or tools I was using, but it stopped working for me on a fresh install of an existing project, too, when it used to work fine. The lack of configurability is a huge problem for me in general, and things that are easy to fix with a little config are just broken with no solid path forward.

For my own project, I ended up replacing it with a standard graphql server (on a worker) and a fronted managed with vite (faster dev, with better hot-reloading and very fast builds) and deployed to worker pages. I used miniflare and vite proxy config to pass requests to the mock-backend, and I have a pretty nice dev-server that hot-reloads immediately every time (this was a re-occurring issue with remix, for me) and can be configured however I need to (for example using remote DO with local KV.)

For this project it works much better! We don't have any SSR requirements, and I can use preact on frontend, which makes a much smaller bundle. I can just use the edge-worker API in graphql resolvers, which my team is already pretty familiar with. It can do graphql subscriptions (via websockets) and all works really reliably, with no hacks.

Even though I'm not going to use remix on this project, I will keep an eye on it's progress, and I still agree that a node-worker-polyfill is probably a really good idea for working with regular node-modules. Even just Buffer is pretty basic, and would help.

[silence48]

I happened to come across this thread, i know it's pretty old but you can polyfill buffer in the global on cloudflare VERY easy.

In your npm dependencies, alias buffer to buffer-polyfill,
"buffer-polyfill": "npm:buffer@^6.0.3",
in server.entry.js

import { Buffer } from "buffer-polyfill";

// Polyfill Buffer on the server
globalThis.Buffer = Buffer as unknown as BufferConstructor;

[jdnichollsc]

Hey folks, check my example to fix this issue https://gist.github.com/jdnichollsc/b73eca1f34bdd1e3a95283774fda2212

[amitt-pop]

We used @jdnichollsc's patch above and also the esbuild-override lib. Both fixed our issues with a missing process polyfill locally but when we tried to get our app deploying to vercel, we ran into all sorts of issues due to the way vercel handles Remix apps (they compile and deploy their own version of Remix that can't be patched):

Given this we're faced with a trade-off matrix because of the lack of esbuild config built-into the app.

1. keep remix, keep full 3rd party lib support that requires polyfills, ditch vercel
2. keep remix, ditch full 3rd party lib support that requires polyfills, keep vercel
3. ditch remix, keep full 3rd party lib support that requires polyfills, keep vercel
4. Any other ideas? Maybe we missed something else?

Unfortunately for us, to unblock our business and keep moving quickly, we're going to probably go with Option 3 of switching off of Remix. I'm writing this comment with the hope that someone from the Remix team sees it and considers reversing the decision to not allow esbuild configurability (or at least polyfill common libraries (buffer, processing, crypto, etc.)) out of the box as it is too limiting b/c of not being able to work with the rest of the JS ecosystem (& cloud providers like Vercel where we have other apps deployed)

[jdnichollsc]

Let's ask them from Twitter! <3 https://twitter.com/jdnichollsc/status/1673819710415142912

[cha0s]

As a newcomer to the ecosystem it does seem a little odd that there are multiple APIs exposed with the unstable_ prefix, but building is excluded. Isn't the entire point of that prefix to absolve devs of responsibility for breakage in client code?

[kiliman]

I think it was a good decision to not open up the compiler. This enabled them to move to Vite and get out of the compiler business altogether.

As for the unstable prefix, that's for early adopters who want to try out new APIs while they are still being developed.