From 15926ab5e49613ce9a2550129fd3d30a67cc991e Mon Sep 17 00:00:00 2001 From: CobusT Date: Mon, 12 Aug 2024 13:30:09 -0700 Subject: [PATCH 01/22] convert documentation --- docs/markdown.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/docs/markdown.md b/docs/markdown.md index e3ee11923..8b53ff582 100644 --- a/docs/markdown.md +++ b/docs/markdown.md @@ -368,3 +368,49 @@ You may also want to add `noopener noreferrer` if linking to an untrusted origin ![A horse](./horse.jpg) ![A happy kitten](https://placekitten.com/200/300) ``` + +## Converting from a notebook + +To create a Markdown file from a public notebook, use the `observable convert` command: + +```sh echo +npm run observable convert +``` +Or with Yarn: +```sh echo +yarn observable convert +``` +Note that, due to [syntax differences](https://observablehq.com/framework/javascript) between Observable notebooks and +Observable Framework, converted notebooks may require further +changes to function correctly. For example, to convert the notebook at `https://observablehq.com/@d3/zoomable-sunburst` to Markdown, run the following command in your `docs` directory: + +```sh echo +npm run observable convert https://observablehq.com/@d3/zoomable-sunburst +``` + +This will create 2 files: `zoomable-sunburst.md` and `flare-2.json` (the file attachment in the notebook). You can then view the markdown file in your dev environment, but you will see a few errors that needs to be corrected: + +Change the `chart` cell definition to an arrow function: +```js run=false +const chart = () => { + // Specify the chart’s dimensions. + const width = 928; + const height = width; + ... +``` + +and the file attachment code to: +~~~js run=false +```js +const data = FileAttachment("flare-2.json").json() +``` +~~~ + +and then, finally, add a JavaScript code block to display the chart: +~~~js run=false +```js +display(chart()); +``` +~~~ + + From df251812aefb59f343ea37973935c140f8cf7c67 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Philippe=20Rivi=C3=A8re?= Date: Mon, 12 Aug 2024 17:50:47 -0400 Subject: [PATCH 02/22] edits --- docs/markdown.md | 38 +++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/docs/markdown.md b/docs/markdown.md index 8b53ff582..3065dabe9 100644 --- a/docs/markdown.md +++ b/docs/markdown.md @@ -369,28 +369,30 @@ You may also want to add `noopener noreferrer` if linking to an untrusted origin ![A happy kitten](https://placekitten.com/200/300) ``` -## Converting from a notebook +## Converting notebooks -To create a Markdown file from a public notebook, use the `observable convert` command: +To help you convert a public Observable notebook to Observable Markdown, Framework includes the `observable convert` utility. Just run: ```sh echo npm run observable convert ``` + Or with Yarn: + ```sh echo yarn observable convert ``` -Note that, due to [syntax differences](https://observablehq.com/framework/javascript) between Observable notebooks and -Observable Framework, converted notebooks may require further -changes to function correctly. For example, to convert the notebook at `https://observablehq.com/@d3/zoomable-sunburst` to Markdown, run the following command in your `docs` directory: - + +Note that, due to [syntax differences](./javascript) between notebooks and Framework, the resulting page may require further changes to function correctly. For example, to convert the notebook at `https://observablehq.com/@d3/zoomable-sunburst` to Observable Markdown, run the following command in the project directory: + ```sh echo npm run observable convert https://observablehq.com/@d3/zoomable-sunburst ``` -This will create 2 files: `zoomable-sunburst.md` and `flare-2.json` (the file attachment in the notebook). You can then view the markdown file in your dev environment, but you will see a few errors that needs to be corrected: +This creates two files: `zoomable-sunburst.md` (combining Markdown and JavaScript) and `flare-2.json` (data). Move them to your documents folder (_e.g._, `src/`), then [preview](./getting-started#test-live-preview) the page (typically at `http://127.0.0.1:3000/zoomable-sunburst`). You should see a few errors that need to be corrected: + +a) Change the `chart` cell definition to an arrow function: -Change the `chart` cell definition to an arrow function: ```js run=false const chart = () => { // Specify the chart’s dimensions. @@ -399,18 +401,20 @@ const chart = () => { ... ``` -and the file attachment code to: -~~~js run=false -```js -const data = FileAttachment("flare-2.json").json() +b) Edit the file attachment code block like so: + +````js run=false +```js +const data = FileAttachment("flare-2.json").json(); ``` -~~~ +```` -and then, finally, add a JavaScript code block to display the chart: -~~~js run=false +c) Add a JavaScript code block to display the chart: + +````js run=false ```js display(chart()); ``` -~~~ - +```` +Enjoy! 🚀 From 1c5a3bd82d2d194d4d6abd839a2053ee37db6429 Mon Sep 17 00:00:00 2001 From: CobusT Date: Mon, 12 Aug 2024 22:04:48 -0700 Subject: [PATCH 03/22] moved to a new page --- docs/converting-notebooks.md | 52 ++++++++++++++++++++++++++++++++++++ docs/markdown.md | 49 --------------------------------- observablehq.config.ts | 1 + 3 files changed, 53 insertions(+), 49 deletions(-) create mode 100644 docs/converting-notebooks.md diff --git a/docs/converting-notebooks.md b/docs/converting-notebooks.md new file mode 100644 index 000000000..11efff4d3 --- /dev/null +++ b/docs/converting-notebooks.md @@ -0,0 +1,52 @@ +# Converting notebooks + +To help you convert a public Observable notebook to Observable Markdown, Framework includes the `observable convert` utility. Just run: + +```sh echo +npm run observable convert +``` + +Or with Yarn: + +```sh echo +yarn observable convert +``` + +Note that, due to [syntax differences](./javascript) between notebooks and Framework, the resulting page may require further changes to function correctly. + +## Example + +For example, to convert the notebook at [`https://observablehq.com/@d3/zoomable-sunburst`](https://observablehq.com/@d3/zoomable-sunburst) to Observable Markdown, run the following command in the project directory: + +```sh echo +npm run observable convert https://observablehq.com/@d3/zoomable-sunburst +``` + +This creates two files: `zoomable-sunburst.md` (combining Markdown and JavaScript) and `flare-2.json` (data). Move them to your documents folder (_e.g._, `src/`), then [preview](./getting-started#test-live-preview) the page (typically at [`http://127.0.0.1:3000/zoomable-sunburst`](http://127.0.0.1:3000/zoomable-sunburst)). You should see a few errors that need to be corrected: + +a) Change the `chart` cell definition to an arrow function: + +```js run=false +const chart = () => { + // Specify the chart’s dimensions. + const width = 928; + const height = width; + ... +``` + +b) Edit the file attachment code block like so: + +````js run=false +```js +const data = FileAttachment("flare-2.json").json(); +``` +```` + +c) Add a JavaScript code block to display the chart: + +````js run=false +```js +display(chart()); +``` +```` + diff --git a/docs/markdown.md b/docs/markdown.md index 3065dabe9..8cbbe6ab0 100644 --- a/docs/markdown.md +++ b/docs/markdown.md @@ -369,52 +369,3 @@ You may also want to add `noopener noreferrer` if linking to an untrusted origin ![A happy kitten](https://placekitten.com/200/300) ``` -## Converting notebooks - -To help you convert a public Observable notebook to Observable Markdown, Framework includes the `observable convert` utility. Just run: - -```sh echo -npm run observable convert -``` - -Or with Yarn: - -```sh echo -yarn observable convert -``` - -Note that, due to [syntax differences](./javascript) between notebooks and Framework, the resulting page may require further changes to function correctly. For example, to convert the notebook at `https://observablehq.com/@d3/zoomable-sunburst` to Observable Markdown, run the following command in the project directory: - -```sh echo -npm run observable convert https://observablehq.com/@d3/zoomable-sunburst -``` - -This creates two files: `zoomable-sunburst.md` (combining Markdown and JavaScript) and `flare-2.json` (data). Move them to your documents folder (_e.g._, `src/`), then [preview](./getting-started#test-live-preview) the page (typically at `http://127.0.0.1:3000/zoomable-sunburst`). You should see a few errors that need to be corrected: - -a) Change the `chart` cell definition to an arrow function: - -```js run=false -const chart = () => { - // Specify the chart’s dimensions. - const width = 928; - const height = width; - ... -``` - -b) Edit the file attachment code block like so: - -````js run=false -```js -const data = FileAttachment("flare-2.json").json(); -``` -```` - -c) Add a JavaScript code block to display the chart: - -````js run=false -```js -display(chart()); -``` -```` - -Enjoy! 🚀 diff --git a/observablehq.config.ts b/observablehq.config.ts index ad7a1acff..258d76eff 100644 --- a/observablehq.config.ts +++ b/observablehq.config.ts @@ -30,6 +30,7 @@ export default { {name: "Themes", path: "/themes"}, {name: "Configuration", path: "/config"}, {name: "Deploying", path: "/deploying"}, + {name: "Converting notebooks", path: "/converting-notebooks"}, {name: "Examples", path: "https://github.com/observablehq/framework/tree/main/examples"}, { name: "Inputs", From 907cc43ce15c4991cf8ee9069776e1f27ff7a62d Mon Sep 17 00:00:00 2001 From: CobusT Date: Mon, 12 Aug 2024 22:05:29 -0700 Subject: [PATCH 04/22] removed newline from markdown.md --- docs/markdown.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/markdown.md b/docs/markdown.md index 8cbbe6ab0..e3ee11923 100644 --- a/docs/markdown.md +++ b/docs/markdown.md @@ -368,4 +368,3 @@ You may also want to add `noopener noreferrer` if linking to an untrusted origin ![A horse](./horse.jpg) ![A happy kitten](https://placekitten.com/200/300) ``` - From 831c35554ddec0493b5fc416476c1f1824274eef Mon Sep 17 00:00:00 2001 From: CobusT Date: Tue, 13 Aug 2024 16:32:29 -0700 Subject: [PATCH 05/22] added a section of 'things to note' - didn't want to call them limitations --- docs/converting-notebooks.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/converting-notebooks.md b/docs/converting-notebooks.md index 11efff4d3..bd2dd24dc 100644 --- a/docs/converting-notebooks.md +++ b/docs/converting-notebooks.md @@ -50,3 +50,10 @@ display(chart()); ``` ```` +## Things to note + +- The `convert` utility only works with **public notebooks** at this point. We may add support for private notebooks in the future. +- It doesn't support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. +- No support for **Observable JavaScript syntax**. You will need to adjust the code to work with [Framework's JavaScript syntax](/javascript). +- **Reactivity** works slightly differently in Framework. For example, instead of using `viewof` with the Inputs library, you use `view`. See the [Reactivity](/reactivity) section for more details. +- Framework uses a slightly different **Standard Library** than notebooks. You may need to adjust the code to work with Framework's Standard Library. From 5d26eede8c821b40aaf6186d78d84c1f8be7560f Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Tue, 13 Aug 2024 22:34:48 -0400 Subject: [PATCH 06/22] checkpoint edits --- docs/convert.md | 108 +++++++++++++++++++++++++++++++++++ docs/converting-notebooks.md | 59 ------------------- observablehq.config.ts | 2 +- 3 files changed, 109 insertions(+), 60 deletions(-) create mode 100644 docs/convert.md delete mode 100644 docs/converting-notebooks.md diff --git a/docs/convert.md b/docs/convert.md new file mode 100644 index 000000000..9f00d94eb --- /dev/null +++ b/docs/convert.md @@ -0,0 +1,108 @@ +# Converting notebooks + +Framework’s built-in `convert` command helps you convert an [Observable notebook](https://observablehq.com/documentation/notebooks/) to standard [Markdown](./markdown) for use with Observable Framework. To convert a notebook, you just need its URL; pass it to the `convert` command like so: + +```sh echo +npm run observable convert +``` + +
+ +The `convert` command currently only supports public notebooks. To convert a private notebook, you can (temporarily) make the notebook public unlisted by clicking **Share…** on the notebook and choosing **Can view (unlisted)** under **Public** access. Please upvote [#1578](https://github.com/observablehq/framework/issues/1578) if you are interested in support for converting private notebooks. + +
+ +For example, to convert D3’s [_Zoomable sunburst_](https://observablehq.com/@d3/zoomable-sunburst) example: + +```sh echo +npm run observable convert https://observablehq.com/@d3/zoomable-sunburst +``` + +This will output something like: + + + +
┌   observable convert 
+│
+◇  Downloaded zoomable-sunburst.md in 443ms
+│
+◇  Downloaded flare-2.json in 288ms
+│
+└  1 notebook converted; 2 files written
+ +The `convert` command generates files in the current working directory. The command above generates two files: zoomable-sunburst.md, a Markdown file representing the converted notebook; and flare-2.json, an attached JSON file. + +## Limitations + +Due to differences between Observable Framework and Observable notebooks, the `convert` command typically won’t produce a working Markdown page out of the box. You’ll need to make some further edits to the generated Markdown. + +Differences between Framework and notebooks fall into three categories: + +- JavaScript syntax, including imports +- the standard library +- recommended libraries + +We’ll describe each of these below with examples. + +### Syntax differences + +While Framework uses [vanilla JavaScript](./javascript), Observable notebooks do not; notebooks use [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript), which extends JavaScript syntax with a few critical differences. While these differences are often small, you will likely have to edit the converted code to make it conform to vanilla JavaScript syntax and work correctly in Framework. + +TK Example of a `chart` cell declaration. + +a) Change the `chart` cell definition to an arrow function: + +```js run=false +const chart = () => { + // Specify the chart’s dimensions. + const width = 928; + const height = width; + ... +``` + +b) Edit the file attachment code block like so: + +````js run=false +```js +const data = FileAttachment("flare-2.json").json(); +``` +```` + +c) Add a JavaScript code block to display the chart: + +````js run=false +```js +display(chart()); +``` +```` + +- It doesn't support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. + +### Standard library differences + +TK + +### Recommended library differences + +TK + +## Command-line flags + +``` +--output +``` + +``` +--force +``` diff --git a/docs/converting-notebooks.md b/docs/converting-notebooks.md deleted file mode 100644 index bd2dd24dc..000000000 --- a/docs/converting-notebooks.md +++ /dev/null @@ -1,59 +0,0 @@ -# Converting notebooks - -To help you convert a public Observable notebook to Observable Markdown, Framework includes the `observable convert` utility. Just run: - -```sh echo -npm run observable convert -``` - -Or with Yarn: - -```sh echo -yarn observable convert -``` - -Note that, due to [syntax differences](./javascript) between notebooks and Framework, the resulting page may require further changes to function correctly. - -## Example - -For example, to convert the notebook at [`https://observablehq.com/@d3/zoomable-sunburst`](https://observablehq.com/@d3/zoomable-sunburst) to Observable Markdown, run the following command in the project directory: - -```sh echo -npm run observable convert https://observablehq.com/@d3/zoomable-sunburst -``` - -This creates two files: `zoomable-sunburst.md` (combining Markdown and JavaScript) and `flare-2.json` (data). Move them to your documents folder (_e.g._, `src/`), then [preview](./getting-started#test-live-preview) the page (typically at [`http://127.0.0.1:3000/zoomable-sunburst`](http://127.0.0.1:3000/zoomable-sunburst)). You should see a few errors that need to be corrected: - -a) Change the `chart` cell definition to an arrow function: - -```js run=false -const chart = () => { - // Specify the chart’s dimensions. - const width = 928; - const height = width; - ... -``` - -b) Edit the file attachment code block like so: - -````js run=false -```js -const data = FileAttachment("flare-2.json").json(); -``` -```` - -c) Add a JavaScript code block to display the chart: - -````js run=false -```js -display(chart()); -``` -```` - -## Things to note - -- The `convert` utility only works with **public notebooks** at this point. We may add support for private notebooks in the future. -- It doesn't support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. -- No support for **Observable JavaScript syntax**. You will need to adjust the code to work with [Framework's JavaScript syntax](/javascript). -- **Reactivity** works slightly differently in Framework. For example, instead of using `viewof` with the Inputs library, you use `view`. See the [Reactivity](/reactivity) section for more details. -- Framework uses a slightly different **Standard Library** than notebooks. You may need to adjust the code to work with Framework's Standard Library. diff --git a/observablehq.config.ts b/observablehq.config.ts index 258d76eff..8d581c356 100644 --- a/observablehq.config.ts +++ b/observablehq.config.ts @@ -30,7 +30,6 @@ export default { {name: "Themes", path: "/themes"}, {name: "Configuration", path: "/config"}, {name: "Deploying", path: "/deploying"}, - {name: "Converting notebooks", path: "/converting-notebooks"}, {name: "Examples", path: "https://github.com/observablehq/framework/tree/main/examples"}, { name: "Inputs", @@ -83,6 +82,7 @@ export default { {name: "ZIP", path: "/lib/zip"} ] }, + {name: "Converting notebooks", path: "/convert"}, {name: "Contributing", path: "/contributing", pager: false} ], base: "/framework", From cc35274735e65be5db5fba74dcbf136cee5f6aa1 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Tue, 13 Aug 2024 22:43:21 -0400 Subject: [PATCH 07/22] multi-convert --- docs/convert.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/convert.md b/docs/convert.md index 9f00d94eb..15570de11 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -12,6 +12,12 @@ The `convert` command currently only supports public notebooks. To convert a pri +
+ +You can pass multiple URLs to convert many notebooks simultaneously. + +
+ For example, to convert D3’s [_Zoomable sunburst_](https://observablehq.com/@d3/zoomable-sunburst) example: ```sh echo From 97af917979bb3817b8d8d438cd9a6353925bf7d7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Philippe=20Rivi=C3=A8re?= Date: Wed, 14 Aug 2024 14:18:45 +0200 Subject: [PATCH 08/22] more docs --- docs/convert.md | 87 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 76 insertions(+), 11 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 15570de11..ed62966e9 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -18,7 +18,7 @@ You can pass multiple URLs to convert many notebooks simultaneously. -For example, to convert D3’s [_Zoomable sunburst_](https://observablehq.com/@d3/zoomable-sunburst) example: +For example, to convert D3’s [_Zoomable sunburst_](https://observablehq.com/@d3/zoomable-sunburst): ```sh echo npm run observable convert https://observablehq.com/@d3/zoomable-sunburst @@ -65,44 +65,109 @@ We’ll describe each of these below with examples. While Framework uses [vanilla JavaScript](./javascript), Observable notebooks do not; notebooks use [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript), which extends JavaScript syntax with a few critical differences. While these differences are often small, you will likely have to edit the converted code to make it conform to vanilla JavaScript syntax and work correctly in Framework. -TK Example of a `chart` cell declaration. +For instance, let’s see how we fix the page converted from the [Zoomable sunburst](https://observablehq.com/@d3/zoomable-sunburst) notebook. At the bottom of the page we see that the `data` cell was transformed into: -a) Change the `chart` cell definition to an arrow function: +````js run=false +```js echo +data = FileAttachment("flare-2.json").json() +``` +```` -```js run=false -const chart = () => { +Fix this with the `const` keyword: + +````js run=false +```js echo +const data = FileAttachment("flare-2.json").json(); +``` +```` + +The largest code block at the top, named `chart`, contains the following: + +````js run=false +```js +chart = { // Specify the chart’s dimensions. const width = 928; const height = width; ... + return svg.node(); +} ``` +```` -b) Edit the file attachment code block like so: +There are various ways to make this into vanilla JavaScript. One possibility is to remove the main curly braces, like so: ````js run=false ```js -const data = FileAttachment("flare-2.json").json(); +// Specify the chart’s dimensions. +const width = 928; +const height = width; +... +const chart = svg.node(); ``` ```` -c) Add a JavaScript code block to display the chart: +Furthermore, we’ll need to [explicitly display](./javascript#explicit-display) the `chart` variable: ````js run=false ```js -display(chart()); +// Specify the chart’s dimensions. +const width = 928; +const height = width; +... +const chart = display(svg.node()); ``` ```` -- It doesn't support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. +(An alternative transformation would be to create a function called `chart`, and invoke `chart()` as an inline expression where we want to display the output.) + +Observable Markdown doesn’t support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. If you import functions and other helpers, it could be useful to add them to a [local module](./imports#local-imports). ### Standard library differences -TK +The `md` template literal is not available in Observable Markdown; instead, write Markdown directly (or import the `markdown-it` library from npm for advanced usage). + +The `require` and `resolve` functions are not available in Observable Markdown; instead, use `import` and `import.meta.resolve`. + +The `DOM.*`, `Files.*`, `Generators.*` and `Promises.*` methods are not available in Observable Markdown. Instead, use the appropriate vanilla JavaScript code — which you can grab from [observablehq/stdlib](https://github.com/observablehq/stdlib/). For example, to create an image with a [2D context](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can copy the code from [context2d.js](https://github.com/observablehq/stdlib/blob/main/src/dom/context2d.js): + +```js run=false +function context2d(width, height, dpi) { + if (dpi == null) dpi = devicePixelRatio; + var canvas = document.createElement("canvas"); + canvas.width = width * dpi; + canvas.height = height * dpi; + canvas.style.width = width + "px"; + var context = canvas.getContext("2d"); + context.scale(dpi, dpi); + return context; +} +``` + +Or, to create a Promise that resolves to a given `value` after a given [delay](https://github.com/observablehq/stdlib/blob/main/src/promises/delay.js): + +```js run=false +function delay(duration, value) { + return new Promise(function(resolve) { + setTimeout(function() { + resolve(value); + }, duration); + }); +} +``` + +If you use a specific function often, you can save it to a local module. ### Recommended library differences TK +### Other differences + +Some cell types cannot be converted to Observable Markdown. Data table cells can be replaced by `Inputs.table` (see [issue #23](https://github.com/observablehq/framework/issues/23) for future enhancements), and chart cells can be replaced by Observable Plot’s [auto mark](https://observablehq.com/plot/marks/auto). + +Database connectors can be replaced by [data loaders](./loaders). We recommend using the `.env` file to store your secrets (such as database passwords and API keys) in a central place outside of your checked-in code; see [Google Analytics](https://observablehq.observablehq.cloud/framework-example-google-analytics/) for an example. + ## Command-line flags ``` From cc150e7fc08420ef0a4bb968cafc26abae34d276 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Philippe=20Rivi=C3=A8re?= Date: Wed, 14 Aug 2024 14:34:26 +0200 Subject: [PATCH 09/22] recommended libs and command-line flags --- docs/convert.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index ed62966e9..33efccda5 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -160,7 +160,7 @@ If you use a specific function often, you can save it to a local module. ### Recommended library differences -TK +In Observable Framework, the recommended [libraries](./imports#implicit-imports) are generally not pinned to a given version — instead you get the latest version that was published on npm (you can still request any version explicitly by using an explicit `import … from "npm:module@version"` statement). Some of them, such as [graphviz](./lib/dot), have been slightly adapted in support of dark mode. For details, see the documentation for each library. ### Other differences @@ -170,10 +170,7 @@ Database connectors can be replaced by [data loaders](./loaders). We recommend u ## Command-line flags -``` ---output -``` +The `convert` command supports the following command-line flags: -``` ---force -``` +- `--output` - the path to the output directory; defaults to `.` for the current working directory. +- `--force` - if true, always download and overwrite existing resources; by default, the script will ask for user input when a file already exists in the output directory. From c73260c158f5b73fd6bee132d4b6a3b751f93cfd Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 10:00:40 -0400 Subject: [PATCH 10/22] edits --- docs/convert.md | 228 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 189 insertions(+), 39 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 15570de11..e7ac4f666 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -1,20 +1,30 @@ # Converting notebooks -Framework’s built-in `convert` command helps you convert an [Observable notebook](https://observablehq.com/documentation/notebooks/) to standard [Markdown](./markdown) for use with Observable Framework. To convert a notebook, you just need its URL; pass it to the `convert` command like so: +Framework’s built-in `convert` command helps you convert an [Observable notebook](https://observablehq.com/documentation/notebooks/) to standard [Markdown](./markdown) for use with Observable Framework. To convert a notebook, you need its URL; pass it to the `convert` command like so: ```sh echo npm run observable convert ``` -
+
-The `convert` command currently only supports public notebooks. To convert a private notebook, you can (temporarily) make the notebook public unlisted by clicking **Share…** on the notebook and choosing **Can view (unlisted)** under **Public** access. Please upvote [#1578](https://github.com/observablehq/framework/issues/1578) if you are interested in support for converting private notebooks. +The above command assumes you’re running `convert` within an existing project. Outside of a project, you can use npx: + +
npx "@observablehq/framework@latest" convert <notebook-url>
-You can pass multiple URLs to convert many notebooks simultaneously. +You can convert multiple notebooks by passing multiple URLs: + +
npm run observable convert <url1> <url2> <url3>
+ +
+ +
+ +The `convert` command currently only supports public notebooks. To convert a private notebook, you can (temporarily) make the notebook public unlisted by clicking **Share…** on the notebook and choosing **Can view (unlisted)** under **Public** access. Please upvote [#1578](https://github.com/observablehq/framework/issues/1578) if you are interested in support for converting private notebooks.
@@ -47,68 +57,208 @@ This will output something like: │ └ 1 notebook converted; 2 files written -The `convert` command generates files in the current working directory. The command above generates two files: zoomable-sunburst.md, a Markdown file representing the converted notebook; and flare-2.json, an attached JSON file. +The `convert` command generates files in the current working directory. The command above generates two files: zoomable-sunburst.md, a Markdown file representing the converted notebook; and flare-2.json, an attached JSON file. You can change the output directory using the --output command-line flag. -## Limitations +Due to differences between Observable Framework and Observable notebooks, the `convert` command typically won’t produce a working Markdown page out of the box; you’ll often need to make further edits to the generated Markdown. We describe these differences below, along with suggestions of how to make the remaining conversions manually. -Due to differences between Observable Framework and Observable notebooks, the `convert` command typically won’t produce a working Markdown page out of the box. You’ll need to make some further edits to the generated Markdown. +
-Differences between Framework and notebooks fall into three categories: +The `convert` command has minimal “magic” so that its behavior is easy to understand and because converting notebook code into standard Markdown and JavaScript requires interpretation. Still, we’re considering making `convert` smarter; let us know if you’re interested. -- JavaScript syntax, including imports -- the standard library -- recommended libraries +
-We’ll describe each of these below with examples. +## JavaScript syntax -### Syntax differences +While Framework uses vanilla [JavaScript syntax](./javascript), notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a standard JavaScript program (_i.e._, a sequence of statements), but a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). -While Framework uses [vanilla JavaScript](./javascript), Observable notebooks do not; notebooks use [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript), which extends JavaScript syntax with a few critical differences. While these differences are often small, you will likely have to edit the converted code to make it conform to vanilla JavaScript syntax and work correctly in Framework. +### Expression cells -TK Example of a `chart` cell declaration. +Named expression cells in notebooks should be converted into standard variable declarations, typically using `const`. -a) Change the `chart` cell definition to an arrow function: +Before: ```js run=false -const chart = () => { - // Specify the chart’s dimensions. - const width = 928; - const height = width; - ... +foo = 42 ``` -b) Edit the file attachment code block like so: +After: -````js run=false -```js -const data = FileAttachment("flare-2.json").json(); +```js run=false +const foo = 42; ``` -```` -c) Add a JavaScript code block to display the chart: +
+ +Variable declarations in Framework don’t implicitly display. To inspect the value of a variable (such as `foo` above), call `display` explicitly. + +
+ +
+ +Framework allows multiple variable declarations in the same code block, so you can often coalesce multiple JavaScript cells from a notebook into a single JavaScript code block in Framework. Though note that there’s no [implicit `await`](./reactivity#promises) when referring to a variable declared in the same code block, so beware of promises. + +
+ +Unnamed expression cells become expression code blocks in Framework, and they work the same way, so you shouldn’t have to make any changes. -````js run=false -```js -display(chart()); +```js run=false +1 + 2 ``` -```` -- It doesn't support **notebook imports**. If your notebook imports cells from other notebooks, you could manually copy the code from those notebooks into your converted markdown file. +
-### Standard library differences +While a notebook is limited to a linear sequence of cells, Framework allows you to interpolate dynamic values anywhere on the page: consider using an [inline expression](./javascript#inline-expressions) instead of a fenced code block. -TK +
-### Recommended library differences +### Block cells -TK +Consider a more elaborate block cell, here an abridged example of the typical D3 chart pattern (adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2) example): -## Command-line flags +Before: + +```js run=false +chart = { + const width = 960; + const height = 500; + const svg = d3.create("svg") + .attr("width", width) + .attr("height", height); + + return svg.node(); +} ``` ---output + +Block cells are characterized by curl braces (`{…}`) and a return statement that indicates the (displayed) value of the cell. + +To convert a block cell to a JavaScript fenced code block: remove the surrounding curly braces, and replace the return statement with a variable declaration and (if desired) a call to `display`. + +After: + +```js run=false +const width = 960; +const height = 500; + +const svg = d3.create("svg") + .attr("width", width) + .attr("height", height); + +const chart = display(svg.node()); ``` +## Import + +Notebook imports _vs._ JavaScript imports _vs._ require. + +Convert doesn’t convert imported notebooks. If your notebook imports cells from other notebooks, you should convert those notebooks, too, and then extract the desired JavaScript code into a standard JavaScript module that you can import. + +Convert doesn’t support [`import with`](https://observablehq.com/documentation/notebooks/imports#import-with). You’ll need to redesign your code if you want to use this feature. + +Dynamic imports should be converted into static imports. + +Imports from a CDN should be converted into self-hosted `npm:` imports. + +## Require + +Framework doesn’t support `require`. You should use a static `npm:` import instead. + +## Yield + +Notebooks allow you to use the `yield` operator to turn any cell into a generator. As vanilla JavaScript, Framework only allows the `yield` operator within generator functions. Therefore you’ll need to wrap a generator cell with an immediately-invoked generator function expression (IIGFE). + +Before: + +```js run=false +foo = { + for (let i = 0; i < 10; ++i) { + yield i; + } +} ``` ---force + +After: + +```js run=false +const foo = (function* () { + for (let i = 0; i < 10; ++i) { + yield i; + } +})(); ``` + +Framework doesn’t allow the `yield` operator outside of generators + +## Viewof + +… + +## Mutable + +… + +## Standard library + +Framework’s standard library is slightly different than the standard library in notebooks. + +Removals: + +- `md` +- `__query` +- `require` +- `resolve` +- `Promises` +- `DOM` +- `Files` +- `Generators.disposable` +- `Generators.filter` +- `Generators.map` +- `Generators.range` +- `Generators.valueAt` +- `Generators.worker` + +Additions: + +- `dark` +- `resize` +- `display` +- `duckdb` +- `echarts` +- `mapboxgl` +- `React` +- `ReactDOM` +- `sql` +- `vg` +- `Generators.dark` +- `Generators.now` +- `Generators.width` + +Changes: + +- `width` uses `ResizeObserver` instead of window _resize_ events +- `FileAttachment` is slightly different +- `Mutable` is unique to Framework +- `html` uses htl +- `svg` uses htl +- `Generators.input` is now an async generator +- `Generators.observe` is now an async generator +- `Generators.queue` is now an async generator + +See also the recommended libraries section next. + +## Recommended libraries + +Self-hosted. You control the version. They default to the latest versions. + +## Sample datasets + +Self-hosted. + +## Cell modes + +Framework doesn’t support non-code cell modes, so these features can’t be converted: + +- Data table cells +- Chart cells + +Framework’s SQL cell is very different from notebooks. From 951a5a790f8ff52e617e543cbb8c51504ea68c9d Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 10:24:36 -0400 Subject: [PATCH 11/22] more edits --- docs/convert.md | 34 +++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 0512a22cb..c49b34709 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -59,21 +59,21 @@ This will output something like: The `convert` command generates files in the current working directory. The command above generates two files: zoomable-sunburst.md, a Markdown file representing the converted notebook; and flare-2.json, an attached JSON file. You can change the output directory using the --output command-line flag. -Due to differences between Observable Framework and Observable notebooks, the `convert` command typically won’t produce a working Markdown page out of the box; you’ll often need to make further edits to the generated Markdown. We describe these differences below, along with suggestions of how to make the remaining conversions manually. +Due to differences between Observable Framework and Observable notebooks, the `convert` command typically won’t produce a working Markdown page out of the box; you’ll often need to make further edits to the generated Markdown. We describe these differences below, along with examples of manual conversion.
-The `convert` command has minimal “magic” so that its behavior is easy to understand and because converting notebook code into standard Markdown and JavaScript requires interpretation. Still, we’re considering making `convert` smarter; let us know if you’re interested. +The `convert` command has minimal “magic” so that its behavior is easier to understand and because converting notebook code into standard Markdown and JavaScript requires human interpretation. Still, we’re considering making `convert` smarter; let us know if you’re interested.
## JavaScript syntax -While Framework uses vanilla [JavaScript syntax](./javascript), notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a standard JavaScript program (_i.e._, a sequence of statements), but a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). +Framework uses vanilla [JavaScript syntax](./javascript), while notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a JavaScript program (_i.e._, a sequence of statements) but rather a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). ### Expression cells -Named expression cells in notebooks should be converted into standard variable declarations, typically using `const`. +Named expression cells in notebooks can be converted into standard variable declarations, typically using `const`. Before: @@ -99,7 +99,7 @@ Framework allows multiple variable declarations in the same code block, so you c
-Unnamed expression cells become expression code blocks in Framework, and they work the same way, so you shouldn’t have to make any changes. +Anonymous expression cells become expression code blocks in Framework, which work the same, so you shouldn’t have to make any changes. ```js run=false 1 + 2 @@ -113,9 +113,7 @@ While a notebook is limited to a linear sequence of cells, Framework allows you ### Block cells -Consider a more elaborate block cell, here an abridged example of the typical D3 chart pattern (adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2) example): - -Before: +Block cells are typically used for more elaborate definitions. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged example of the typical D3 chart pattern (adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2)): ```js run=false chart = { @@ -130,11 +128,8 @@ chart = { } ``` -Block cells are characterized by curl braces (`{…}`) and a return statement that indicates the (displayed) value of the cell. -To convert a block cell to a JavaScript fenced code block: remove the surrounding curly braces, and replace the return statement with a variable declaration and (if desired) an explicit call to [`display`](./javascript#explicit-display). - -After: +To convert a block cell: delete the cell name (`chart`), assignment operator (`=`), and surrounding curly braces (`{` and `}`); then replace the return statement with a variable declaration and a call to [`display`](./javascript#explicit-display) as desired. ```js run=false const width = 960; @@ -149,7 +144,20 @@ const chart = display(svg.node());
-An alternative transformation would be to create a function called `chart`, and invoke `chart()` as an inline expression where we want to display the output. +If you prefer, you can instead convert a block cell into a function such as: + +
function chart() {
+  const width = 960;
+  const height = 500;
+
+  const svg = d3.create("svg")
+      .attr("width", width)
+      .attr("height", height);
+
+  return svg.node();
+}
+ +Then call the function from an [inline expression](./javascript#inline-expressions) (_e.g._, `${chart()}`) to display its output anywhere on the page. This technique is also useful for importing a chart definition into multiple pages.
From 14942b42c2595efa443c7ad4a8df4e471371cb7d Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 10:33:20 -0400 Subject: [PATCH 12/22] more edits --- docs/convert.md | 25 ++++++------------------- 1 file changed, 6 insertions(+), 19 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index c49b34709..cc4148ec9 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -69,19 +69,17 @@ The `convert` command has minimal “magic” so that its behavior is easier to ## JavaScript syntax -Framework uses vanilla [JavaScript syntax](./javascript), while notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a JavaScript program (_i.e._, a sequence of statements) but rather a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). +Framework uses vanilla [JavaScript syntax](./javascript) while notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a JavaScript program (_i.e._, a sequence of statements) but rather a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). ### Expression cells -Named expression cells in notebooks can be converted into standard variable declarations, typically using `const`. - -Before: +Named expression cells in notebooks can be converted into standard variable declarations, typically using `const`. So this: ```js run=false foo = 42 ``` -After: +Becomes this: ```js run=false const foo = 42; @@ -173,9 +171,7 @@ Dynamic imports should be converted into static imports. Imports from a CDN should be converted into self-hosted `npm:` imports. -## Require - -Framework doesn’t support `require`. You should use a static `npm:` import instead. +Framework doesn’t include built-in support for `require` because the asynchronous module definition (AMD) convention has been superseded by standard [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). You should use a static `npm:` import instead. That said, you can import [d3-require](https://github.com/d3/d3-require) if you want to a `require` implementation; we just don’t recommend it. And instead of `resolve`, use `import.meta.resolve`. ## Yield @@ -205,11 +201,11 @@ Framework doesn’t allow the `yield` operator outside of generators ## Viewof -… +[./reactivity#inputs](./reactivity#inputs) ## Mutable -… +[./reactivity#mutables](./reactivity#mutables) ## Standard library @@ -262,8 +258,6 @@ See also the recommended libraries section next. The `md` template literal is not available in Observable Markdown; instead, write Markdown directly (or import the `markdown-it` library from npm for advanced usage). -The `require` and `resolve` functions are not available in Observable Markdown; instead, use `import` and `import.meta.resolve`. - The `DOM.*`, `Files.*`, `Generators.*` and `Promises.*` methods are not available in Observable Markdown. Instead, use the appropriate vanilla JavaScript code — which you can grab from [observablehq/stdlib](https://github.com/observablehq/stdlib/). For example, to create an image with a [2D context](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can copy the code from [context2d.js](https://github.com/observablehq/stdlib/blob/main/src/dom/context2d.js): ```js run=false @@ -321,10 +315,3 @@ Database connectors can be replaced by [data loaders](./loaders). ## Secrets We recommend using the `.env` file to store your secrets (such as database passwords and API keys) in a central place outside of your checked-in code; see [Google Analytics](https://observablehq.observablehq.cloud/framework-example-google-analytics/) for an example. - -## Command-line flags - -The `convert` command supports the following command-line flags: - -- `--output` - the path to the output directory; defaults to `.` for the current working directory. -- `--force` - if true, always download and overwrite existing resources; by default, the script will ask for user input when a file already exists in the output directory. From 3e36882eb1203b713623a9a43f23c24eaa7def17 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 10:54:50 -0400 Subject: [PATCH 13/22] more edits --- docs/convert.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index cc4148ec9..ef2f2d970 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -111,7 +111,7 @@ While a notebook is limited to a linear sequence of cells, Framework allows you ### Block cells -Block cells are typically used for more elaborate definitions. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged example of the typical D3 chart pattern (adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2)): +Block cells are typically used for more elaborate definitions. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2): ```js run=false chart = { @@ -127,7 +127,7 @@ chart = { ``` -To convert a block cell: delete the cell name (`chart`), assignment operator (`=`), and surrounding curly braces (`{` and `}`); then replace the return statement with a variable declaration and a call to [`display`](./javascript#explicit-display) as desired. +To convert a named block cell: delete the cell name (`chart`), assignment operator (`=`), and surrounding curly braces (`{` and `}`); then replace the return statement with a variable declaration and a call to [`display`](./javascript#explicit-display) as desired. ```js run=false const width = 960; @@ -140,6 +140,8 @@ const svg = d3.create("svg") const chart = display(svg.node()); ``` +For an anonymous block cell, omit the variable declaration. To display nothing, omit the call to `display`; you can use an [inline expression](./javascript#inline-expressions) (_e.g._, `${chart}`) to display the chart elsewhere. +
If you prefer, you can instead convert a block cell into a function such as: @@ -155,7 +157,7 @@ If you prefer, you can instead convert a block cell into a function such as: return svg.node(); } -Then call the function from an [inline expression](./javascript#inline-expressions) (_e.g._, `${chart()}`) to display its output anywhere on the page. This technique is also useful for importing a chart definition into multiple pages. +Then call the function from an inline expression (_e.g._, `${chart()}`) to display its output anywhere on the page. This technique is also useful for importing a chart definition into multiple pages.
@@ -287,6 +289,22 @@ function delay(duration, value) { If you use a specific function often, you can save it to a local module. +## File attachments + +Additions: + +- `file.href` +- `file.lastModified` +- `file.arquero` +- `file.parquet` + +Changes: + +- `file.csv`, `file.tsv,` and `file.dsv` don’t support `typed: "auto"` (only `typed: true`) +- `file.text` supports *encoding* option +- `file.arrow` doesn’t take a version and instead whatever `npm:apache-arrow` is +- `file.mimeType` is always defined + ## Recommended libraries Self-hosted. You control the version. They default to the latest versions. From 3d1ce93a95fc9bb87cb79a3a3a6e64dabcc15838 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 11:00:16 -0400 Subject: [PATCH 14/22] more edits --- docs/convert.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index ef2f2d970..cab3d2dce 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -260,29 +260,26 @@ See also the recommended libraries section next. The `md` template literal is not available in Observable Markdown; instead, write Markdown directly (or import the `markdown-it` library from npm for advanced usage). -The `DOM.*`, `Files.*`, `Generators.*` and `Promises.*` methods are not available in Observable Markdown. Instead, use the appropriate vanilla JavaScript code — which you can grab from [observablehq/stdlib](https://github.com/observablehq/stdlib/). For example, to create an image with a [2D context](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can copy the code from [context2d.js](https://github.com/observablehq/stdlib/blob/main/src/dom/context2d.js): +For removed methods, you’ll need to provide your own implementation in vanilla JavaScript. If you like, you can refer to the source code from the notebook implementations in the [@observablehq/stdlib repo](https://github.com/observablehq/stdlib/). For example, to create an image with a [2D context](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can copy the code from [context2d.js](https://github.com/observablehq/stdlib/blob/main/src/dom/context2d.js): ```js run=false -function context2d(width, height, dpi) { - if (dpi == null) dpi = devicePixelRatio; - var canvas = document.createElement("canvas"); +function context2d(width, height, dpi = devicePixelRatio) { + const canvas = document.createElement("canvas"); canvas.width = width * dpi; canvas.height = height * dpi; - canvas.style.width = width + "px"; - var context = canvas.getContext("2d"); + canvas.style = `width: ${width}px;`; + const context = canvas.getContext("2d"); context.scale(dpi, dpi); return context; } ``` -Or, to create a Promise that resolves to a given `value` after a given [delay](https://github.com/observablehq/stdlib/blob/main/src/promises/delay.js): +To create a Promise that resolves after a given delay ([`Promises.delay`](https://github.com/observablehq/stdlib/blob/main/src/promises/delay.js)): ```js run=false function delay(duration, value) { - return new Promise(function(resolve) { - setTimeout(function() { - resolve(value); - }, duration); + return new Promise((resolve) => { + setTimeout(() => resolve(value), duration); }); } ``` @@ -300,8 +297,8 @@ Additions: Changes: -- `file.csv`, `file.tsv,` and `file.dsv` don’t support `typed: "auto"` (only `typed: true`) -- `file.text` supports *encoding* option +- `file.csv` _etc._ don’t support `typed: "auto"` (only `typed: true`) +- `file.text` supports `encoding` option - `file.arrow` doesn’t take a version and instead whatever `npm:apache-arrow` is - `file.mimeType` is always defined From d36299a4a6762641eeff7dbdb121f9951c93d806 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 11:03:01 -0400 Subject: [PATCH 15/22] more edits --- docs/convert.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index cab3d2dce..def5c597e 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -165,9 +165,9 @@ Then call the function from an inline expression (_e.g._, `${chart()}`) to displ Notebook imports _vs._ JavaScript imports _vs._ require. -Convert doesn’t convert imported notebooks. If your notebook imports cells from other notebooks, you should convert those notebooks, too, and then extract the desired JavaScript code into a standard [JavaScript module](./imports#local-imports) that you can import. +Convert doesn’t convert imported notebooks. If your notebook imports cells from other notebooks, you should convert those notebooks too, and then extract the desired JavaScript code into standard [JavaScript modules](./imports#local-imports) that you can import. -Convert doesn’t support [`import with`](https://observablehq.com/documentation/notebooks/imports#import-with). You’ll need to redesign your code if you want to use this feature. +Convert doesn’t support [`import with`](https://observablehq.com/documentation/notebooks/imports#import-with). You’ll need to redesign code that uses this feature. Dynamic imports should be converted into static imports. @@ -321,7 +321,7 @@ Framework doesn’t support non-code cell modes, so these features can’t be co Framework’s SQL cell is very different from notebooks. -Some cell types cannot be converted to Observable Markdown. Data table cells can be replaced by `Inputs.table` (see [issue #23](https://github.com/observablehq/framework/issues/23) for future enhancements), and chart cells can be replaced by Observable Plot’s [auto mark](https://observablehq.com/plot/marks/auto). +Some cell types cannot be converted to Observable Markdown. Data table cells can be replaced by `Inputs.table` (see [#23](https://github.com/observablehq/framework/issues/23) for future enhancements), and chart cells can be replaced by Observable Plot’s [auto mark](https://observablehq.com/plot/marks/auto). ## Databases From f0aca54d7382575c40c3a537560cbaa84b54bf7b Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 11:36:37 -0400 Subject: [PATCH 16/22] more edits --- docs/convert.md | 70 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 53 insertions(+), 17 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index def5c597e..4c44ae8a8 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -30,9 +30,7 @@ The `convert` command currently only supports public notebooks. To convert a pri For example, to convert D3’s [_Zoomable sunburst_](https://observablehq.com/@d3/zoomable-sunburst): -```sh echo -npm run observable convert https://observablehq.com/@d3/zoomable-sunburst -``` +
npm run observable convert "https://observablehq.com/@d3/zoomable-sunburst
This will output something like: @@ -69,7 +67,7 @@ The `convert` command has minimal “magic” so that its behavior is easier to ## JavaScript syntax -Framework uses vanilla [JavaScript syntax](./javascript) while notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in an notebook is not a JavaScript program (_i.e._, a sequence of statements) but rather a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). +Framework uses vanilla [JavaScript syntax](./javascript) while notebooks use a nonstandard dialect called [Observable JavaScript](https://observablehq.com/documentation/cells/observable-javascript). A JavaScript cell in a notebook is technically not a JavaScript program (_i.e._, a sequence of statements) but rather a _cell declaration_; it can be either an _expression cell_ consisting of a single JavaScript expression (such as `1 + 2`) or a _block cell_ consisting of any number of JavaScript statements (such as `console.log("hello");`) surrounded by curly braces. These two forms of cell require slightly different treatment. The `convert` command converts both into JavaScript [fenced code blocks](./javascript#fenced-code-blocks). ### Expression cells @@ -111,7 +109,7 @@ While a notebook is limited to a linear sequence of cells, Framework allows you ### Block cells -Block cells are typically used for more elaborate definitions. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2): +Block cells are typically used for more elaborate definitions in notebooks. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2): ```js run=false chart = { @@ -127,7 +125,7 @@ chart = { ``` -To convert a named block cell: delete the cell name (`chart`), assignment operator (`=`), and surrounding curly braces (`{` and `}`); then replace the return statement with a variable declaration and a call to [`display`](./javascript#explicit-display) as desired. +To convert a named block cell to vanilla JavaScript: delete the cell name (`chart`), assignment operator (`=`), and surrounding curly braces (`{` and `}`); then replace the return statement with a variable declaration and a call to [`display`](./javascript#explicit-display) as desired. ```js run=false const width = 960; @@ -140,7 +138,7 @@ const svg = d3.create("svg") const chart = display(svg.node()); ``` -For an anonymous block cell, omit the variable declaration. To display nothing, omit the call to `display`; you can use an [inline expression](./javascript#inline-expressions) (_e.g._, `${chart}`) to display the chart elsewhere. +For an anonymous block cell, omit the variable declaration. To hide the display, omit the call to `display`; you can use an [inline expression](./javascript#inline-expressions) (_e.g._, `${chart}`) to display the chart elsewhere.
@@ -161,21 +159,59 @@ Then call the function from an inline expression (_e.g._, `${chart()}`) to displ
-## Import +## Imports -Notebook imports _vs._ JavaScript imports _vs._ require. +Notebooks often import other notebooks from Observable, or open-source libraries from npm. Imports (and requires) require additional manual conversion. -Convert doesn’t convert imported notebooks. If your notebook imports cells from other notebooks, you should convert those notebooks too, and then extract the desired JavaScript code into standard [JavaScript modules](./imports#local-imports) that you can import. +If the converted notebook [imports other notebooks](https://observablehq.com/documentation/notebooks/imports), you should convert the imported notebooks, too. Extract the desired JavaScript code from the imported notebooks into standard [JavaScript modules](./imports#local-imports) which you can then import in Framework. -Convert doesn’t support [`import with`](https://observablehq.com/documentation/notebooks/imports#import-with). You’ll need to redesign code that uses this feature. +
-Dynamic imports should be converted into static imports. +In Framework, reactivity only applies to [top-level variables](./reactivity#top-level-variables) declared in fenced code blocks. If the imported code depends on reactivity or uses [`import-with`](https://observablehq.com/documentation/notebooks/imports#import-with), you will likely need to do some additional refactoring, say converting JavaScript cells into functions that take options. + +
+ +Many notebooks use [`require`](https://observablehq.com/documentation/cells/require) to load open-source libraries from npm. Framework discourages the use of `require` and does not include built-in support for it because the asynchronous module definition (AMD) convention has been superseded by standard [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). Also, Framework preloads transitive dependencies using static analysis to improve performance, and self-hosts [npm imports](./imports#npm-imports) to eliminate a runtime dependency on external servers to improve security and give you control over library versioning. So this: + +```js run=false +regl = require("regl") +``` + +Should be converted to a static `npm:` import: + +```js run=false +import regl from "npm:regl"; +``` + +
+ +The code above imports the default export from [regl](https://github.com/regl-project/regl). For other libraries, such as D3, you should use a namespace import instead: -Imports from a CDN should be converted into self-hosted `npm:` imports. +
import * as d3 from "npm:d3";
-Framework doesn’t include built-in support for `require` because the asynchronous module definition (AMD) convention has been superseded by standard [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). You should use a static `npm:` import instead. That said, you can import [d3-require](https://github.com/d3/d3-require) if you want to a `require` implementation; we just don’t recommend it. And instead of `resolve`, use `import.meta.resolve`. +
+ +
+ +You can import [d3-require](https://github.com/d3/d3-require) if you really want to a `require` implementation; we just don’t recommend it. + +
+ +Likewise, instead of `resolve` or `require.resolve`, use [`import.meta.resolve`](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import.meta/resolve). So this: + +```js run=false +require.resolve("regl") +``` + +Should be converted to: + +```js run=false +import.meta.resolve("npm:regl") +``` + +Dynamic imports should be converted into static imports. -## Yield +## Generators Notebooks allow you to use the `yield` operator to turn any cell into a generator. As vanilla JavaScript, Framework only allows the `yield` operator within generator functions. Therefore you’ll need to wrap a generator cell with an immediately-invoked generator function expression (IIGFE). @@ -201,11 +237,11 @@ const foo = (function* () { Framework doesn’t allow the `yield` operator outside of generators -## Viewof +## Views [./reactivity#inputs](./reactivity#inputs) -## Mutable +## Mutables [./reactivity#mutables](./reactivity#mutables) From e1b0fca11be4e956e85a215446cae0cd91554e5a Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 11:42:51 -0400 Subject: [PATCH 17/22] more edits --- docs/convert.md | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/docs/convert.md b/docs/convert.md index 4c44ae8a8..9e215598b 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -209,7 +209,25 @@ Should be converted to: import.meta.resolve("npm:regl") ``` -Dynamic imports should be converted into static imports. +Notebooks also support dynamic import, so you might also see libraries being loaded from CDNs such as [jsDelivr](https://www.jsdelivr.com/esm) or [esm.sh](https://esm.sh/). While you can use dynamic imports in Framework, for security and performance, we recommend also converting these into static `npm:` imports. So this: + +```js run=false +isoformat = import("https://esm.sh/isoformat") +``` + +Should be converted to: + +```js run=false +import * as isoformat from "https://esm.sh/isoformat"; +``` + +
+ +If you expressly do not want to self-host the import, say because you want the latest version of the library to update without having to rebuild your app, you can load it from an external server by providing an absolute URL. + +
import * as isoformat from "https://esm.sh/isoformat";
+ +
## Generators From 882f38cf7e13cce0b1f0fb9141cb9989481c9cc1 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 11:47:20 -0400 Subject: [PATCH 18/22] more edits --- docs/convert.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 9e215598b..6f809cc7c 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -231,9 +231,7 @@ If you expressly do not want to self-host the import, say because you want the l ## Generators -Notebooks allow you to use the `yield` operator to turn any cell into a generator. As vanilla JavaScript, Framework only allows the `yield` operator within generator functions. Therefore you’ll need to wrap a generator cell with an immediately-invoked generator function expression (IIGFE). - -Before: +Notebooks allow you to use the `yield` operator to turn any cell [into a generator](https://observablehq.com/documentation/cells/observable-javascript#cells-implicitly-iterate-over-generators). In vanilla JavaScript, the `yield` operator is only allowed within generator functions. Therefore in Framework you’ll need to wrap a generator cell declaration with an immediately-invoked generator function expression (IIGFE). So this: ```js run=false foo = { @@ -243,7 +241,7 @@ foo = { } ``` -After: +Can be converted to: ```js run=false const foo = (function* () { @@ -253,7 +251,13 @@ const foo = (function* () { })(); ``` -Framework doesn’t allow the `yield` operator outside of generators +
+ +Since variables are evaluated lazily, the generator `foo` will only run above if it is referenced by another code block. If you’re using a generator to perform asynchronous side effects, consider using an animation loop and the [invalidation promise](./reactivity#invalidation) instead of a generator. + +
+ +If you need to use `await` with the generator, too, then use `async function*` to declare an async generator function instead. ## Views From bdcc2d8e352b230223206e4b0dbd530b04adb408 Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 12:40:46 -0400 Subject: [PATCH 19/22] more edits --- docs/convert.md | 134 +++++++++++++++++++++++++----------------------- 1 file changed, 69 insertions(+), 65 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 6f809cc7c..1a280790b 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -91,13 +91,13 @@ Variable declarations in Framework don’t implicitly display. To inspect the va
-Framework allows multiple variable declarations in the same code block, so you can often coalesce multiple JavaScript cells from a notebook into a single JavaScript code block in Framework. Though note that there’s no [implicit `await`](./reactivity#promises) when referring to a variable declared in the same code block, so beware of promises. +Framework allows multiple variable declarations in the same code block, so you can coalesce multiple JavaScript cells from a notebook into a single JavaScript code block in Framework. Though note that there’s no [implicit `await`](./reactivity#promises) when referring to a variable declared in the same code block, so beware of promises.
Anonymous expression cells become expression code blocks in Framework, which work the same, so you shouldn’t have to make any changes. -```js run=false +```js echo 1 + 2 ``` @@ -109,7 +109,7 @@ While a notebook is limited to a linear sequence of cells, Framework allows you ### Block cells -Block cells are typically used for more elaborate definitions in notebooks. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2): +Block cells are used in notebooks for more elaborate definitions. They are characterized by curly braces (`{…}`) and a return statement to indicate the cell’s value. Here is an abridged typical example adapted from D3’s [_Bar chart_](https://observablehq.com/@d3/bar-chart/2): ```js run=false chart = { @@ -161,7 +161,7 @@ Then call the function from an inline expression (_e.g._, `${chart()}`) to displ ## Imports -Notebooks often import other notebooks from Observable, or open-source libraries from npm. Imports (and requires) require additional manual conversion. +Notebooks often import other notebooks from Observable or open-source libraries from npm. Imports require additional manual conversion. If the converted notebook [imports other notebooks](https://observablehq.com/documentation/notebooks/imports), you should convert the imported notebooks, too. Extract the desired JavaScript code from the imported notebooks into standard [JavaScript modules](./imports#local-imports) which you can then import in Framework. @@ -171,13 +171,13 @@ In Framework, reactivity only applies to [top-level variables](./reactivity#top- -Many notebooks use [`require`](https://observablehq.com/documentation/cells/require) to load open-source libraries from npm. Framework discourages the use of `require` and does not include built-in support for it because the asynchronous module definition (AMD) convention has been superseded by standard [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). Also, Framework preloads transitive dependencies using static analysis to improve performance, and self-hosts [npm imports](./imports#npm-imports) to eliminate a runtime dependency on external servers to improve security and give you control over library versioning. So this: +Some notebooks use [`require`](https://observablehq.com/documentation/cells/require) to load libraries from npm. Framework discourages the use of `require` and does not include built-in support for it because the asynchronous module definition (AMD) convention has been superseded by standard [JavaScript modules](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules). Also, Framework preloads transitive dependencies using static analysis to improve performance, and self-hosts imports to eliminate a runtime dependency on external servers to improve security and give you control over library versioning. So this: ```js run=false regl = require("regl") ``` -Should be converted to a static `npm:` import: +Should be converted to a static [npm import](./imports#npm-imports): ```js run=false import regl from "npm:regl"; @@ -209,7 +209,7 @@ Should be converted to: import.meta.resolve("npm:regl") ``` -Notebooks also support dynamic import, so you might also see libraries being loaded from CDNs such as [jsDelivr](https://www.jsdelivr.com/esm) or [esm.sh](https://esm.sh/). While you can use dynamic imports in Framework, for security and performance, we recommend also converting these into static `npm:` imports. So this: +Since notebooks also support [dynamic import](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/import), you might also see libraries being loaded from CDNs such as [jsDelivr](https://www.jsdelivr.com/esm) or [esm.sh](https://esm.sh/). While you can use dynamic imports in Framework, for security and performance, we recommend also converting these into static imports. So this: ```js run=false isoformat = import("https://esm.sh/isoformat") @@ -218,12 +218,12 @@ isoformat = import("https://esm.sh/isoformat") Should be converted to: ```js run=false -import * as isoformat from "https://esm.sh/isoformat"; +import * as isoformat from "npm:isoformat"; ```
-If you expressly do not want to self-host the import, say because you want the latest version of the library to update without having to rebuild your app, you can load it from an external server by providing an absolute URL. +If you do not want to self-host an import, say because you want the latest version of the library to update without having to rebuild your app, you can load it from an external server by providing an absolute URL: 
import * as isoformat from "https://esm.sh/isoformat";
@@ -231,7 +231,7 @@ If you expressly do not want to self-host the import, say because you want the l ## Generators -Notebooks allow you to use the `yield` operator to turn any cell [into a generator](https://observablehq.com/documentation/cells/observable-javascript#cells-implicitly-iterate-over-generators). In vanilla JavaScript, the `yield` operator is only allowed within generator functions. Therefore in Framework you’ll need to wrap a generator cell declaration with an immediately-invoked generator function expression (IIGFE). So this: +In notebooks, the `yield` operator turns any cell [into a generator](https://observablehq.com/documentation/cells/observable-javascript#cells-implicitly-iterate-over-generators). In vanilla JavaScript, the `yield` operator is only allowed within generator functions. Therefore in Framework you’ll need to wrap a generator cell declaration with an immediately-invoked generator function expression (IIGFE). So this: ```js run=false foo = { @@ -253,7 +253,7 @@ const foo = (function* () {
-Since variables are evaluated lazily, the generator `foo` will only run above if it is referenced by another code block. If you’re using a generator to perform asynchronous side effects, consider using an animation loop and the [invalidation promise](./reactivity#invalidation) instead of a generator. +Since variables are evaluated lazily, the generator `foo` will only run above if it is referenced by another code block. If you want to perform asynchronous side effects, consider using an animation loop and the [invalidation promise](./reactivity#invalidation) instead of a generator.
@@ -261,64 +261,55 @@ If you need to use `await` with the generator, too, then use `async function*` t ## Views -[./reactivity#inputs](./reactivity#inputs) +In notebooks, the nonstandard [`viewof` operator](https://observablehq.com/@observablehq/views) is used to declare a reactive value that is controlled by a user interface element such as a range input. In Framework, the [`view` function](./reactivity#inputs) performs the equivalent task with vanilla syntax. So this: -## Mutables +```js run=false +viewof gain = Inputs.range([0, 11], {value: 5, step: 0.1, label: "Gain"}) +``` -[./reactivity#mutables](./reactivity#mutables) +Can be converted to: -## Standard library +```js run=false +const gain = view(Inputs.range([0, 11], {value: 5, step: 0.1, label: "Gain"})); +``` -Framework’s standard library is slightly different than the standard library in notebooks. +In other words: replace `viewof` with `const`, and then wrap the input declaration with a call to `view`. The `view` function both displays the given input and returns the corresponding value generator so you can define a top-level reactive value. -Removals: +## Mutables -- `md` -- `__query` -- `require` -- `resolve` -- `Promises` -- `DOM` -- `Files` -- `Generators.disposable` -- `Generators.filter` -- `Generators.map` -- `Generators.range` -- `Generators.valueAt` -- `Generators.worker` +In notebooks, the nonstandard [`mutable` operator](https://observablehq.com/@observablehq/mutable) is used to declare a reactive value that can be assigned from another cell. In Framework, the [`Mutable` function](./reactivity#mutables) performs the equivalent task with vanilla syntax. So this: -Additions: +```js run=false +mutable foo = 42 +``` -- `dark` -- `resize` -- `display` -- `duckdb` -- `echarts` -- `mapboxgl` -- `React` -- `ReactDOM` -- `sql` -- `vg` -- `Generators.dark` -- `Generators.now` -- `Generators.width` +Can be converted to: -Changes: +```js run=false +const foo = Mutable(42); +const setFoo = (x) => (foo.value = x); +``` -- `width` uses `ResizeObserver` instead of window _resize_ events -- `FileAttachment` is slightly different -- `Mutable` is unique to Framework -- `html` uses htl -- `svg` uses htl -- `Generators.input` is now an async generator -- `Generators.observe` is now an async generator -- `Generators.queue` is now an async generator +Then replace any assignments to `mutable foo` with calls to `setFoo`. -See also the recommended libraries section next. +## Standard library + +As part of our modernization efforts with Framework, we’ve pruned deprecated methods from the standard library used in notebooks. The following built-ins are not available in Framework: -The `md` template literal is not available in Observable Markdown; instead, write Markdown directly (or import the `markdown-it` library from npm for advanced usage). +- [`DOM`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/dom/index.js) +- [`Files`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/files/index.js) +- [`Generators.disposable`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/disposable.js) +- [`Generators.filter`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/filter.js) +- [`Generators.map`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/map.js) +- [`Generators.range`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/range.js) +- [`Generators.valueAt`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/valueAt.js) +- [`Generators.worker`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/generators/worker.js) +- [`Promises`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/promises/index.js) +- [`md`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/md.js) +- [`require`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) +- [`resolve`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) -For removed methods, you’ll need to provide your own implementation in vanilla JavaScript. If you like, you can refer to the source code from the notebook implementations in the [@observablehq/stdlib repo](https://github.com/observablehq/stdlib/). For example, to create an image with a [2D context](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can copy the code from [context2d.js](https://github.com/observablehq/stdlib/blob/main/src/dom/context2d.js): +For convenience, we’ve linked above to the implementations so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, to create a [2D canvas](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you replace `DOM.context2d` with: ```js run=false function context2d(width, height, dpi = devicePixelRatio) { @@ -332,17 +323,18 @@ function context2d(width, height, dpi = devicePixelRatio) { } ``` -To create a Promise that resolves after a given delay ([`Promises.delay`](https://github.com/observablehq/stdlib/blob/main/src/promises/delay.js)): +For `md`, we recommend writing literal Markdown. To parse dynamic Markdown, you can also import your preferred parser such as [markdown-it](https://github.com/markdown-it/markdown-it) from npm. -```js run=false -function delay(duration, value) { - return new Promise((resolve) => { - setTimeout(() => resolve(value), duration); - }); -} -``` +In addition to the above removals, a few of the built-in methods have changed: + +- `FileAttachment` (see [below](#file-attachments)) +- `Generators.input` is now an async generator +- `Generators.observe` is now an async generator +- `Generators.queue` is now an async generator +- `Mutable` (see [above](#mutables)) +- `width` uses [`ResizeObserver`](https://developer.mozilla.org/en-US/docs/Web/API/ResizeObserver) instead of window _resize_ events -If you use a specific function often, you can save it to a local module. +The Framework standard library also includes several new methods that are not available in notebooks. These are covered elsewhere: [`Generators.dark`](./lib/generators#dark) and [`dark`](./lib/generators#dark); [`Generators.now`](./lib/generators#now); [`Generators.width`](./lib/generators#width-element) and [`resize`](./javascript#resize-render); [`display`](./javascript#display-value); and [`sql`](./sql#sql-literals). ## File attachments @@ -364,6 +356,18 @@ Changes: Self-hosted. You control the version. They default to the latest versions. +Changes: +- `html` uses `htl.html` +- `svg` uses `htl.svg` + +Additions: +- `ReactDOM` +- `React` +- `duckdb` +- `echarts` +- `mapboxgl` +- `vg` + In Observable Framework, the recommended [libraries](./imports#implicit-imports) are generally not pinned to a given version — instead you get the latest version that was published on npm (you can still request any version explicitly by using an explicit `import … from "npm:module@version"` statement). Some of them, such as [graphviz](./lib/dot), have been slightly adapted in support of dark mode. For details, see the documentation for each library. ## Sample datasets From 0220ae1ca4625046b1525535a89a0d0f22334f3b Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 13:02:40 -0400 Subject: [PATCH 20/22] more edits --- docs/convert.md | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 1a280790b..2160f7e2a 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -309,7 +309,7 @@ As part of our modernization efforts with Framework, we’ve pruned deprecated m - [`require`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) - [`resolve`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) -For convenience, we’ve linked above to the implementations so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, to create a [2D canvas](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you replace `DOM.context2d` with: +For convenience, we’ve linked above to the implementations so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, for a [2D canvas](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can replace `DOM.context2d` with: ```js run=false function context2d(width, height, dpi = devicePixelRatio) { @@ -338,23 +338,38 @@ The Framework standard library also includes several new methods that are not av ## File attachments -Additions: +Framework’s [`FileAttachment`](./files) includes a few new features: - `file.href` - `file.lastModified` -- `file.arquero` -- `file.parquet` +- `file.mimeType` is always defined +- `file.text` now supports an `encoding` option +- [`file.arquero`](./lib/arquero) +- [`file.parquet`](./lib/arrow#apache-parquet) -Changes: +And two removals: -- `file.csv` _etc._ don’t support `typed: "auto"` (only `typed: true`) -- `file.text` supports `encoding` option -- `file.arrow` doesn’t take a version and instead whatever `npm:apache-arrow` is -- `file.mimeType` is always defined +- `file.csv` _etc._ treats the `typed: "auto"` option as `typed: true` +- `file.arrow` doesn’t take a `version` option + +For the latter, `file.arrow` now imports `npm:apache-arrow` internally, and thus uses the same version of Arrow as if you imported Arrow directly. ## Recommended libraries -Self-hosted. You control the version. They default to the latest versions. +One big (but subtle) change: you can now control the version of recommended libraries, and recommended libraries are self-hosted (along with all other npm imports), and default to the latest versions. + +Upgrades (as of current writing): +- `@duckdb/duckdb-wasm` from 1.24.0 to 1.28.0 +- `apache-arrow` from 4.0.1 to 17.0.0 +- `arquero` from 4.8.8 to 6.0.1 +- `dot` from `viz.js` 2.0.0 to `@viz-js/viz` at 3.7.0 +- `exceljs` from 4.3.0 to 4.4.0 +- `katex` from 0.11.0 to 0.16.11 +- `leaflet` from 1.9.3 to 1.9.4 +- `mermaid` from 9.2.2 to 10.9.1 +- `vega-lite-api` from 5.0.0 to 5.6.0 +- `vega-lite` from 5.6.0 to 5.20.1 +- `vega` from 5.22.1 to 5.30.0 Changes: - `html` uses `htl.html` From 921784824c6d5063425b95038c9fcb889e3ade4d Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 13:40:45 -0400 Subject: [PATCH 21/22] more edits --- docs/convert.md | 43 +++++++++++++++++++++---------------------- 1 file changed, 21 insertions(+), 22 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index 2160f7e2a..baf3b080a 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -356,34 +356,33 @@ For the latter, `file.arrow` now imports `npm:apache-arrow` internally, and thus ## Recommended libraries -One big (but subtle) change: you can now control the version of recommended libraries, and recommended libraries are self-hosted (along with all other npm imports), and default to the latest versions. +One big (but subtle) change: you can now control the version of recommended libraries, and recommended libraries are self-hosted (along with all other npm imports), and default to the latest versions. (you can still request any version explicitly by using an explicit `import … from "npm:module@version"` statement) Upgrades (as of current writing): -- `@duckdb/duckdb-wasm` from 1.24.0 to 1.28.0 -- `apache-arrow` from 4.0.1 to 17.0.0 -- `arquero` from 4.8.8 to 6.0.1 -- `dot` from `viz.js` 2.0.0 to `@viz-js/viz` at 3.7.0 -- `exceljs` from 4.3.0 to 4.4.0 -- `katex` from 0.11.0 to 0.16.11 -- `leaflet` from 1.9.3 to 1.9.4 -- `mermaid` from 9.2.2 to 10.9.1 -- `vega-lite-api` from 5.0.0 to 5.6.0 -- `vega-lite` from 5.6.0 to 5.20.1 -- `vega` from 5.22.1 to 5.30.0 +- [`@duckdb/duckdb-wasm`](./lib/duckdb) from 1.24.0 to 1.28.0 +- [`apache-arrow`](./lib/arrow) from 4.0.1 to 17.0.0 +- [`arquero`](./lib/arquero) from 4.8.8 to 6.0.1 +- [`dot`](./lib/dot) from `viz.js` 2.0.0 to `@viz-js/viz` at 3.7.0 +- [`exceljs`](./lib/xlsx) from 4.3.0 to 4.4.0 +- [`katex`](./lib/tex) from 0.11.0 to 0.16.11 +- [`leaflet`](./lib/leaflet) from 1.9.3 to 1.9.4 +- [`mermaid`](./lib/mermaid) from 9.2.2 to 10.9.1 +- [`vega`](./lib/vega-lite) from 5.22.1 to 5.30.0 +- [`vega-lite`](./lib/vega-lite) from 5.6.0 to 5.20.1 +- [`vega-lite-api`](./lib/vega-lite) from 5.0.0 to 5.6.0 Changes: -- `html` uses `htl.html` -- `svg` uses `htl.svg` +- [`html`](./lib/htl) uses `htl.html` +- [`svg`](./lib/htl) uses `htl.svg` +- [`dot`](./lib/dot) implements responsive dark mode & styling Additions: -- `ReactDOM` -- `React` -- `duckdb` -- `echarts` -- `mapboxgl` -- `vg` - -In Observable Framework, the recommended [libraries](./imports#implicit-imports) are generally not pinned to a given version — instead you get the latest version that was published on npm (you can still request any version explicitly by using an explicit `import … from "npm:module@version"` statement). Some of them, such as [graphviz](./lib/dot), have been slightly adapted in support of dark mode. For details, see the documentation for each library. +- [`ReactDOM`](./jsx) +- [`React`](./jsx) +- [`duckdb`](./lib/duckdb) +- [`echarts`](./lib/echarts) +- [`mapboxgl`](./lib/mapbox-gl) +- [`vg`](./lib/mosaic) ## Sample datasets From 2fc6b2f55e62b75fac2d25eeced3582e963bb9cb Mon Sep 17 00:00:00 2001 From: Mike Bostock Date: Wed, 14 Aug 2024 15:35:10 -0400 Subject: [PATCH 22/22] more edits --- docs/convert.md | 34 ++++++++-------------------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/docs/convert.md b/docs/convert.md index baf3b080a..11ec83a5d 100644 --- a/docs/convert.md +++ b/docs/convert.md @@ -294,7 +294,7 @@ Then replace any assignments to `mutable foo` with calls to `setFoo`. ## Standard library -As part of our modernization efforts with Framework, we’ve pruned deprecated methods from the standard library used in notebooks. The following built-ins are not available in Framework: +As part of our modernization efforts with Framework, we’ve pruned deprecated methods from the standard library used in notebooks. The following notebook built-ins are not available in Framework: - [`DOM`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/dom/index.js) - [`Files`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/files/index.js) @@ -309,7 +309,7 @@ As part of our modernization efforts with Framework, we’ve pruned deprecated m - [`require`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) - [`resolve`](https://github.com/observablehq/stdlib/blob/493bf210f5fcd9360cf87a961403aa963ba08c96/src/require.js) -For convenience, we’ve linked above to the implementations so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, for a [2D canvas](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can replace `DOM.context2d` with: +For convenience, we’ve linked to the implementations above so that you can see how they work, and if desired, copy the code into your own Framework app as vanilla JavaScript. For example, for a [2D canvas](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D), you can replace `DOM.context2d` with: ```js run=false function context2d(width, height, dpi = devicePixelRatio) { @@ -356,9 +356,9 @@ For the latter, `file.arrow` now imports `npm:apache-arrow` internally, and thus ## Recommended libraries -One big (but subtle) change: you can now control the version of recommended libraries, and recommended libraries are self-hosted (along with all other npm imports), and default to the latest versions. (you can still request any version explicitly by using an explicit `import … from "npm:module@version"` statement) +In Framework, implicit imports of recommended libraries are normal [npm imports](./imports#npm-imports), and thus are self-hosted, giving you control over versioning. If a requested library is not in your [npm cache](./imports#self-hosting-of-npm-imports), then by default the latest version will be downloaded. You can request a more specific version either by seeding the npm cache or by including a semver range in the import specifier (_e.g._, `import * as d3 from "npm:d3@6"`). -Upgrades (as of current writing): +Because Framework defaults to the latest version of recommended libraries, you will typically get a more recent version than what is available in notebooks. As of August 2024, here is a comparison of recommended library versions between notebooks and Framework: - [`@duckdb/duckdb-wasm`](./lib/duckdb) from 1.24.0 to 1.28.0 - [`apache-arrow`](./lib/arrow) from 4.0.1 to 17.0.0 - [`arquero`](./lib/arquero) from 4.8.8 to 6.0.1 @@ -371,33 +371,15 @@ Upgrades (as of current writing): - [`vega-lite`](./lib/vega-lite) from 5.6.0 to 5.20.1 - [`vega-lite-api`](./lib/vega-lite) from 5.0.0 to 5.6.0 -Changes: -- [`html`](./lib/htl) uses `htl.html` -- [`svg`](./lib/htl) uses `htl.svg` -- [`dot`](./lib/dot) implements responsive dark mode & styling - -Additions: -- [`ReactDOM`](./jsx) -- [`React`](./jsx) -- [`duckdb`](./lib/duckdb) -- [`echarts`](./lib/echarts) -- [`mapboxgl`](./lib/mapbox-gl) -- [`vg`](./lib/mosaic) +In Framework, the [`html`](./lib/htl) and [`svg`](./lib/htl) built-in template literals are implemented with [Hypertext Literal](./lib/htl) which automatically escapes interpolated values. The [`dot`](./lib/dot) template literal implements responsive dark mode & better styling. And Framework has several additional recommended libraries that are not available in notebooks: [`ReactDOM`](./jsx), [`React`](./jsx), [`duckdb`](./lib/duckdb), [`echarts`](./lib/echarts), [`mapboxgl`](./lib/mapbox-gl), and [`vg`](./lib/mosaic). ## Sample datasets -Self-hosted. +Like recommended libraries, Framework’s built-in sample datasets (_e.g._, `aapl` and `penguins`) are backed by npm imports that are self-hosted. ## Cell modes -Framework doesn’t support non-code cell modes, so these features can’t be converted: - -- Data table cells -- Chart cells - -Framework’s SQL cell is very different from notebooks. - -Some cell types cannot be converted to Observable Markdown. Data table cells can be replaced by `Inputs.table` (see [#23](https://github.com/observablehq/framework/issues/23) for future enhancements), and chart cells can be replaced by Observable Plot’s [auto mark](https://observablehq.com/plot/marks/auto). +The `convert` command only supports code cell modes: Markdown, JavaScript, HTML, TeX, and SQL. It does not support non-code cell modes: data table and chart. You can use the “Convert to SQL” or “Convert to JavaScript” feature to convert data table cells and chart cells to their code equivalents prior to conversion. Alternatively, you can manually replace data table cells with `Inputs.table` (see [#23](https://github.com/observablehq/framework/issues/23) for future enhancements), and chart cells with Observable Plot’s [auto mark](https://observablehq.com/plot/marks/auto). ## Databases @@ -405,4 +387,4 @@ Database connectors can be replaced by [data loaders](./loaders). ## Secrets -We recommend using the `.env` file to store your secrets (such as database passwords and API keys) in a central place outside of your checked-in code; see [Google Analytics](https://observablehq.observablehq.cloud/framework-example-google-analytics/) for an example. +We recommend using a `.env` file with [dotenv](https://github.com/motdotla/dotenv) to store your secrets (such as database passwords and API keys) in a central place outside of your checked-in code; see our [Google Analytics dashboard](https://github.com/observablehq/framework/tree/main/examples/google-analytics/) example.