Allow users to customize wrangler build step

[ashleymichal]

Today, Wrangler is rather prescriptive about what happens during the build stage; this works relatively well for Rust projects in particular, but begins to cause headaches for more complex javascript/webpack work flows. We should provide and document some way for users to customize their build step.

Related issues:

• allow for wrangler publish without npm install #778
• Suggest re-building node_modules on failed builds where appropriate #428

[ashleymichal]

Feature request from #778:

💡 Feature request

Overview and problem statement

Livepeer has a Yarn monorepo and running npm install in it kind of mucks things up. I've already got a completely compiled worker.js suitable for uploading as a Cloudflare worker, as well as a static HTML Next.js site that I'd like uploaded to Workers Sites. I don't need any of the npm automation provided by wranglerjs, but it'd still be nice to be able to use wrangler publish.

Here's the wrangler.toml I'm using.

Basic example

Seems like it'd be a configuration flag in wrangler.toml?

do_install = false

[ashleymichal]

bug report from #428

We assume that the existence of node_modules in a webpack project means the dependencies are installed and we don't need to run npm install again. However, if the user does ctrl-c during wrangler build, wrangler publish or wrangler preview while the npm install stage is running, it is possible to have a node_modules folder with invalid contents.

[mrw]

This would be super helpful for my use case as well. I proposed a "pre-publish" command in #1224 but customizing the build process in general would definitely accomplish the same end-goal:

Overview and problem statement

I think it'd be helpful for us Workers Sites users if wrangler.toml allowed a "pre-publish" command to be set.

In my case I'm using Jekyll to generate the static files prior to publishing with wrangler. When running jekyll build for production, we'll often include different command line variables or Jekyll config than we do locally/on staging so the output is optimized for production. It'd be great if wrangler allowed a "pre-publish" command to be set so it automatically runs the proper Jekyll build command before publishing.

Basic example

In wrangler.toml, something like this:

[site]
bucket = "./_site"
entry-point = "workers-site"
pre-publish = "jekyll build --config ./jekyll-production-config.yml"

Thank you!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[erickskrauch]

Bump.

[jamesarosen]

Our use-case: we have a monorepo containing a number of different workers plus packages with shared worker code:

my-repo/
  packages/
    worker-library-1/
      package.json
      index.js
    worker-library-2/
      package.json
      index.js
    worker-a/
      package.json
      index.js
      wrangler.toml
    worker-b/
      package.json
      index.js
      wrangler.toml

We use a yarn workspace and yarn install to install the dependencies. (pnpm would work just as well for our case.)

We can't use this suggestion to include an empty package.json in the worker directories because those package.json files declare real dependencies to other packages in the workspace.

[thefill]

Our use-case: we have a monorepo containing a number of different workers plus packages with shared worker code:

my-repo/
  packages/
    worker-library-1/
      package.json
      index.js
    worker-library-2/
      package.json
      index.js
    worker-a/
      package.json
      index.js
      wrangler.toml
    worker-b/
      package.json
      index.js
      wrangler.toml

We use a yarn workspace and yarn install to install the dependencies. (pnpm would work just as well for our case.)

We can't use this suggestion to include an empty package.json in the worker directories because those package.json files declare real dependencies to other packages in the workspace.

You can get this working good ol' "hacky way":

• in package.json:

{
   "scripts": {
     "build:a": "cd packages/worker-library-1 && mkdir node_modules && wrangler dev",
     "build:b": "cd packages/worker-library-2 && mkdir node_modules && wrangler dev",
     "cleanup": "rm packages/worker-library-1/node_modules packages/worker-library-1/node_modules"
}

• execute bash yarn build:a or bash yarn build:b
• be happy (obv execute cleanup as a git prehook or something ;-) )

If you take a look at src/wranglerjs/mod.rs devs are running rather naive check if !dir.join("node_modules").exists() {} that is causing a bit of problems for executing in custom dir. If you trick it wrangler works like a charm.

[thefill]

@nataliescottdavidson you are aware you can make a load of devs happy and resolve most of the monorepo issues (especially for nx users) by simply introducing in the src/wranglerjs/mod.rs line 235 simple bottom-up fs search for closes node_modules?

[jamesarosen]

If you take a look at src/wranglerjs/mod.rs devs are running rather naive check if !dir.join("node_modules").exists() {} that is causing a bit of problems for executing in custom dir. If you trick it wrangler works like a charm.

In that case, I could probably also do this:

$ touch packages/worker-library-1/node_modules/.gitkeep
$ touch packages/worker-library-2/node_modules/.gitkeep
$ git add -A
$ git commit -m "Keep packages/*/node_modules/ for Wrangler

See: https://github.com/cloudflare/wrangler/issues/820"

[thefill]

If you take a look at src/wranglerjs/mod.rs devs are running rather naive check if !dir.join("node_modules").exists() {} that is causing a bit of problems for executing in custom dir. If you trick it wrangler works like a charm.

In that case, I could probably also do this:

$ touch packages/worker-library-1/node_modules/.gitkeep
$ touch packages/worker-library-2/node_modules/.gitkeep
$ git add -A
$ git commit -m "Keep packages/*/node_modules/ for Wrangler

See: https://github.com/cloudflare/wrangler/issues/820"

yeah sure, you can persist this change.

I would not recommend that as this can confuse other devs but maybe that is just me ;-)

[osdiab]

looking forward to #1748, would allow me to share code between my web app, API, and the cloudflare Workers Site project that hosts my company's web app! We use a yarn workspaces monorepo so this should work fine, the root node_modules has each subpackage symlinked from there. Definitely wouldn't work with yarn berry/pnp, but that's not a show-stopper... for now

[thefill]

looking forward to #1748, would allow me to share code between my web app, API, and the cloudflare Workers Site project that hosts my company's web app! We use a yarn workspaces monorepo so this should work fine, the root node_modules has each subpackage symlinked from there. Definitely wouldn't work with yarn berry/pnp, but that's not a show-stopper... for now

Keep in mind that there are some limitations to this PR. You need to have "webpack.config.json" file present in the dir you want to build a worker. There is still a bit of work to be done to make this CLI "monorepo friendly" but its a step in the right direction.

I hope it will work for you - if you encounter any problems you can always PM me and will try to progress this further.

[xortive]

closing since this is addressed by #1677, which will be going out as part of an RC in the near future