The future of Wrangler configuration

[penalosa]

Following on from #1929 —it's probably worthwhile having a discussion about the future of configuration for Wrangler, pulling together the various discussions there have been at different points. The linked PR shows one approach—wholesale migrating from TOML->JSON—but I know there are different opinions to be heard here. From my own thinking (to kick this off):

1. It's useful for Wrangler to have a configuration file
2. TOML is rarely used in the Javascript ecosystem, and has various footguns that lead to hard to understand errors (array closing being the primary one that comes to mind)
3. JSON (or JSONC/JSON5) is more easily understandable, and is harder to write incorrectly (and we could leverage things like JSON schema to provide schema error checking in people's IDEs)
4. The API-ification of Wrangler is another possible way forward here, with something like a wrangler.ts or wrangler.js file that describes the configuration of your Worker (but see point 1, this can't be the only way)

The outcome of this discussion could well be that we decide to stick with TOML, but I thought it was worth airing all the various thoughts on this in one place, rather than spread across different GitHub issues.

[zaynehz]

I think #3 is a very important point to make and principle to follow - using wrangler should be as easy as possible and provide the best possible developer experience. JSON is more easily readable, more difficult to write incorrectly, and more familiar in the javascript ecosystem; ergo, a better developer experience.

(Disclaimer: this comment is brought to you by a couple of hours of trying to debug an issue with wrangler that turned out to be two brackets in the wrong place in my TOML file)

[danielrs]

Agree that JSON is the most familiar to the JS ecosystem. Another good thing about JSON is that it's a subset of YAML, so any developers that want to use YAML can just convert it to JSON. On another note, I'm not sure what the official plans are but it would be nice to have some sort of modular configuration so different resources can be defined in "blocks", similar to what Kubernetes does with its resources type. This will allow us to:

• Define all the resources for a project in file
• Or define all resources in separate file
• Easier to automate or generate the files using templating tools
  Just my two cents.

[ndisidore]

Sharing from an internal discussion:

Something like cuelang could be interesting.
It is pretty dang straightforward in the way its syntax works, supports typing, has built in validation, is scriptable, has a built in formatter (no need to worry about whitespace) and has transforms to accept json & yaml as both input or output (for multi style support)
There are wasm builds for cuelang that could be used to integrate with wrangler.

However, it is a newer config language that has a bit of its own learning curve, and for that reason alone it may be more of a fun though experiment then something practical.

[petebacondarwin]

While Cue looks very interesting, I think the bigger driving force here is to make the configuration accessible to the majority of Workers developers, who are mostly used to the JS ecosystem, which is what gives JSON(C/5)/YAML a clear headstart.

[cmackenzie1]

Adding a concrete example to @zaynehz 's comment.

When referencing the docs for wrangler.toml many of the examples use the inline list syntax ¹ such as

[[r2_buckets]]
binding = 'MY_BUCKET' # <~ valid JavaScript variable name
bucket_name = '<YOUR_BUCKET_NAME>'

While writing workers with just a single top level environment this syntax is clear but once you add environments into the mix things become more confusing and less intuitive. For example

name = "default-environment"

[[r2_buckets]]
binding = 'MY_BUCKET' # <~ valid JavaScript variable name
bucket_name = '<YOUR_BUCKET_NAME>'

[env.staging]
name = "staging-environment"

[[r2_buckets]] # this actually doesn't exist under `env.staging` prefix!
binding = 'MY_BUCKET' # <~ valid JavaScript variable name
bucket_name = '<YOUR_BUCKET_NAME>'

We can see here that the intent it the second [[r2_buckets]] block is to be a part of the staging environment it actually ends up being a duplicated definition of the first [[r2_buckets]] block.

There is a toml syntax for we can adopt in the docs to help make things clearer but the solution still isn't perfect. It involves not using the inline tables syntax.

name = "default-environment"
r2_buckets = [
    { binding = 'MY_BUCKET' , bucket_name = '<YOUR_BUCKET_NAME>' }
]


[env.staging]
name = "staging-environment"
r2_buckets = [
    { binding = 'MY_BUCKET' , bucket_name = '<YOUR_BUCKET_NAME>' }
]

This confusion extends to the other developer platform products as well and not just R2. A non-exhaustive list is

• unsafe.bindings
• durable_objects
• kv_namespaces

Finally, I should note for those who prefer the inline syntax, the correct form is to use a fully-qualified keyname.

[env.staging]
name = "staging-environment"

[[env.staging.r2_buckets]]
binding = 'MY_BUCKET' # <~ valid JavaScript variable name
bucket_name = '<YOUR_BUCKET_NAME>'

Footnotes

1. https://developers.cloudflare.com/r2/data-access/workers-api/workers-api-usage/ ↩

[its-jman]

Though I’ve not thought through the details, and this is a very ambiguous proposal to make, I’d like to at least float the idea of a CDK-first approach (eg. AWS CDK or Terraform), not just for bindings but for the infra as a whole.

This is obviously a large conversation to try to tackle at once, but it seems like a very intriguing idea. Especially given AWS is heavily leaning towards infra as code (CDK) compared to static config, especially internally.

(But otherwise, as a short term solution, I definitely prefer JSON or point 4 from above, for all of the other reasons mentioned in the thread)

[penalosa]

Thanks everyone for the responses! It seems there's a general consensus to move to JSON(C/5)/YAML from TOML, with various other ideas on the horizon. Limiting the scope of this to just the syntax of configuration, here's a proposed plan for how we get to a place where we've replaced wrangler.toml:

• Codemod: Write a codemod to transform wrangler.toml files to wrangler.(jsonc|json5|yaml) files. This could be published as (for example) @cloudflare/convert-wrangler-config, and run on your project with npx @cloudflare/convert-wrangler-config wrangler.toml (which would generate the file). Since there are obviously a lot of wrangler.toml files in the wild, I think it's worth us testing this on a sample of open-source wrangler.toml files from GitHub repositories. Something that I think is quite important here: can we preserve config comments from the TOML files?
• Rollout: For a transition period, Wrangler should be able to detect either wrangler.toml or wrangler.(jsonc|json5|yaml) configuration files, and read from either (preferring wrangler.(jsonc|json5|yaml) if it's present). If wrangler.toml is found, Wrangler should print a warning (something like wrangler.toml is being depreciated, please run npx @cloudflare/convert-wrangler-config wrangler.toml to convert your configuration file to the new format), but still continue to work. Longer term, support for wrangler.toml should be removed, but the warning should remain if a wrangler.toml file is found.
• Documentation: We have a lot of mentions of wrangler.toml across our docs. They should all be updated to use wording like "Wrangler configuration file".

[jbergstroem]

I would suggest narrowing around a simpler format (built-in like JSON) and allow the more complex configuration languages just output said format (variables, templates, reuse from things like yaml, cue) should the use-case be needed. This reduces dependencies and increases portability.

[huw]

cosmiconfig is quite popular as a solution in the JS ecosystem. This allows users to choose whether they want JS, JSON, YAML, whatever (even in package.json). Putting it forward in case this helps resolve some of the “what format should we use” issues.

[JustinGrote]

Regardless of the formats supported, I think a json schema for wrangler config including parameter documentation is essential. It can be used for intellisense and validation in JSON, JSONC, YAML, and TOML, so even if wrangler sticks with TOML, tools like vscode can be used for easy config. It should also be hosted/updated at schemastore.org for easy retrieval

[Cherry]

I'm a little late to respond here, but I'd like to vote against YAML, please, at least as the recommended or default config.

Wrangler recently moved to hard tabs for indentation for accessibility reasons as per my issue opened at #1298. Many of Cloudflare's other projects are doing the same. Using a config language that's inherently inaccessible by not supporting tabs for indentation is a bad idea. There are of course hacks that could be done to allow tabs for indentation, but many editors and IDEs won't play nicely with this.

JSON support seems the most logical to me, with lots of tooling already using it, such as the package.json, prettier config, eslint configs, etc. I am personally a fan of TOML, but understand that some folks dislike its inheritance syntax.

[notpeelz]

The even-better-toml VSCode extension supports JSON schemas, using the #:schema http://path/to/schema.json directive: https://taplo.tamasfe.dev/configuration/directives.html#the-schema-directive

[JacobMGEvans]

We want to support JSON-C directly, but thank you for the follow up!

[JacobMGEvans]

With the recent Release https://github.com/cloudflare/wrangler2/releases/tag/wrangler%402.9.0 there is now a --experimental-json-config first steps towards improved configuration support.

[VernonGrant]

I'm wondering if anyone knows of a JSON schema for wrangler.toml files that I can use to validate our configurations?

[snilan]

Check out this, it may work:
https://gist.githubusercontent.com/o-az/d298a1776165407ba63ee763d32f7a89/raw/509659a55fc732c12a8047ffc92fb2e1a96aeebb/wrangler.json

[petebacondarwin]

No schema, as such. But if you run a Wrangler command in a directory containing the directory containing the wrangler.toml, Wrangler will fail if the file is invalid. It actually is better than a schema as the validation can check a few more complex constraints.

[JacobMGEvans]

With the Zod implementation me and Samuel want to do, it should allow for a validation AND schema, which provides autocomplete and other DX benefits.

Examples:
https://twitter.com/zg_dev/status/1633429660389318656
https://twitter.com/scastiel/status/1633480196304109568
https://twitter.com/mattpocockuk/status/1658431146500276226
https://twitter.com/alexdotjs/status/1658409815536812032

Library example doing it with Env:
https://github.com/t3-oss/t3-env

Cc: @penalosa

[penalosa]

One issue with Zod is that it seems to only work for hard-fail validation errors—I don't know how it'd support validation warnings

[JacobMGEvans]

I can talk to Colin and a few others to see if there are any patterns or implementations around this. I assume you are referring to when people have something in the config that is old but falls back to something newer config.

[petebacondarwin]

There are a number of places in the config validation where we trigger a warning and not an error.

[snilan]

Hi, if anyone's interested, I ended up defining a JSON Schema for Wrangler using https://github.com/sinclairzx81/typebox.
I'm sure I missed some things and I know things change often, but I tried to cover whatever I could find in the documentation

You can find the repo here:
https://github.com/snilan/wrangler-json-schema

And the generated JSON Schema file here:
https://github.com/snilan/wrangler-json-schema/blob/main/wrangler.schema.json

[snilan]

The Even Better TOML extension kept on crashing on me, not sure why, but VSCode does have built-in support for JSON Schemas.

So if you're using the wrangler.json experimental support, you can enable it by adding these to your settings.json file:

"json.schemaDownload.enable": true,
"json.schemas": [
   {
     "fileMatch": ["wrangler.json"],
     "url": "https://raw.githubusercontent.com/snilan/wrangler-json-schema/main/wrangler.schema.json"
   }
],

[Odame]

This json schema contains docs, in case anyone is interested.
Originally posted in this comment - #3166 (comment)

"json.schemaDownload.enable": true,
"json.schemas": [
   {
     "fileMatch": ["wrangler.json"],
     "url": "https://github.com/cloudflare/workers-sdk/files/12887590/wrangler.schema.json"
   }
],

[jbergstroem]

Is there a plan for when the json configuration will be marked as stable? I would like to start using it in production.

[theoephraim]

Personally, I would love to see wrangler config move towards using js/ts for config.

For most users, it would still just feel like glorified JSON - although less annoying with commas/quotes and would allow for comments. (sidenote: would love to see more JSONC/5 adoption but shockingly IDE tooling still sucks!). But you'd actually get full IDE support and intellisense, not just the half-baked version we get with json schema or other IDE plugins.

Plus for those situations that require it, you could easily use regular code to construct your config as you see fit. Plus it would provide a much cleaner way to start thinking about plugins/hooks/etc... Look at vite for inspiration.

[theoephraim]

A bunch of options are just meant to configure esbuild. Now of course you can use a custom build if you really need to, but with a js config file, it would be trivial to load actual esbuild plugins, avoiding the need for a custom build.