toml editor

[threepointone]

There are numerous occasions when we want to be able to 'automatically' edit the wrangler.toml file from withiin wrangler. Examples -

• removing a deprecated field like type
• renaming a deprecated field like build.upload.main to something like entry (todo)
• adding durable object migrations based on some interactive flow (todo)

In rust land, we'd use a format preserving editor like toml-edit, but nothing like it exists in javascript land (that I know of). So this issue tracks the implementation of the thing.

We need to take a toml parser that given a toml string, returns an AST. We then apply edits to the AST directly, and convert it back to a toml string.

Potential code to learn from - https://github.com/ota-meshi/toml-eslint-parser

Sequencing / prioritisation: I think when we should first start simply logging the changes we want people to make directly to the terminal (for backward compat, migrations, whatever.) Once we have that in place, we can start working on this issue to automate the suggestions.

[petebacondarwin]

Consider using https://taplo.tamasfe.dev/lib

[petebacondarwin]

Other options:

• https://www.npmjs.com/package/tree-sitter-toml (uses C-based tree-sitter tool). See https://tree-sitter.github.io/tree-sitter/playground
• https://www.npmjs.com/package/@toml-tools/parser (sadly needs a new maintainer)

[JacobMGEvans]

https://www.npmjs.com/package/@toml-tools/parser (sadly needs a new maintainer)

Seems like the library was mostly leveraging this which seems still highly maintained. The rest of the toml parser can be implemented using https://github.com/chevrotain/chevrotain and the parsing rules for TOML & change the API as needed.
Assuming we don't find something else that works well for our needs.

[bd82]

Hello.

I am the author of both Chevrotain and @toml-tools/parser.

Some questions:

• Do I understand correctly that you need to make small modifications to an existing toml file, while honoring the
  existing formatting / style (no full document re-write on change)?

• Do you have any future "true editor" requirements in regards to the wrangler.toml file? e.g:

  • A VSCode extension that supports some semantic validations and content assist for wrangler.toml files?

Cheers.
Shahar.

[threepointone]

Yes to both!

[bd82]

Yes to both!

Graceful automated editing

Chevrotain could be well suited as an infrastructure for this due to the CST (Concrete Syntax Tree) support.
Basically it can auto-magically output a (verbose) structure that includes all the syntax details beyond what normally exists in an AST, for example in JSON:

• exact locations of commas / brackets in a JSON
• exact location of properties as opposed to only their names.

See: JSON + CST example in the playground

• https://chevrotain.io/playground/

TOML Parser / Grammar

As far as I remember, TOML is a very simple language (it has obvious and minimal in the name...)

• Grammar
• Spec

So implementing a parser for it should not be too large of an undertaking and there is already prior work to build on:

• https://www.npmjs.com/package/@toml-tools/lexer
• https://www.npmjs.com/package/@toml-tools/parser

Full fledged Editor (Semantic Validations / Content Assist / Tool-tips)

This is a fair bit more complicated.

You would need to implement a Language Server for your wrangler.toml syntax/semantics.

• https://microsoft.github.io/language-server-protocol/

I have worked on a similar problem for SAP's UI5 XML webviews.

• VSCode marketplace link - has GIFs...

The relevant source code is in these two repos:

• https://github.com/SAP/xml-tools (the generic XML parts and infrastructure)
• https://github.com/SAP/ui5-language-assistant (builds on-top of xml-tools for the UI5 specific parts)

Language Server infrastructure to evaluate

Langium by Typefox is a new framework that assists in building language servers for DSLs.
I have not used it personally and it is at an early stage, however, it may be worth investigating as it could theoretically greatly simplify the full fledged editor problem.

• https://github.com/langium/langium
• https://tomassetti.me/quickly-create-dsls-with-langium/

@msujew may be able to elaborate more on this option?

[msujew]

Hey @bd82, thanks for the heads-up!

If it's about editor support, I think Langium would be a good choice. Looking at the initial requirements, especially removing deprecated fields should be as simple as writing a validation and executing an associated code action (as in: Executing a quick fix provided by VSCode for unused imports for example).

Langium's main use case is to build a language server, so its editing interfaces are tightly coupled to the LSP types. However, everything is also working in CLI/normal app server mode, so once you conform to the LSP types, stuff like executing code actions should also be working fine. Although, we mostly just use the parsing and validation capabilities of Langium when running CLIs. Thinking about it, you probably don't even need to go through the LSP interfaces and can just work on the AST/CST itself, kind of mimicking how editors execute code actions.

It might be a bit tricky to get the toml grammar to work correctly in Langium (mostly about lexing, parsing shouldn't be an issue, white space sensitive languages are always a bit difficult to deal with), but definitely possible, since it's using Chevrotain under the hood. However, after that's done, everything else should be straightforward, since semantic validations, content assist, tool-tips, etc. as mentioned by Shahar come out-of-the-box with minimal effort with Langium.

[penalosa]

Closing this, since we're moving Wrangler's config to a JSONC format, and the library we use for that supports editing files (preserving comments).