NodeJS - Using Plot and Puppeteer to save charts

[nshiab]

Hello!

I already use Plot a lot in my observable notebooks, but I often need to do some work with NodeJS. I want to use Plot with NodeJS and save the charts as images.

Here's how I intend to use a function I named savePlotChart that takes Plot options as a parameter:

const data = [
    { age: 35, salary: 75000, department: "Engineering" },
    { age: 42, salary: 82000, department: "Sales" },
    { age: 28, salary: 60000, department: "Marketing" },
    { age: 55, salary: 90000, department: "HR" },
    { age: 49, salary: 72000, department: "Finance" },
    { age: 31, salary: 55000, department: "Engineering" },
    { age: 40, salary: 68000, department: "Sales" },
    { age: 23, salary: 48000, department: "HR" },
    { age: 37, salary: 77000, department: "Marketing" },
    { age: 45, salary: 88000, department: "Engineering" },
]

await savePlotChart(
    {
        color: { legend: true },
        marks: [
            Plot.dot(data, {
                x: "age",
                y: "salary",
                stroke: "salary",
            }),
        ],
    },
    "./my-chart.png"
)

I am trying to use puppeteer to render the chart and save it. Here's the function savePlotChart:

import puppeteer from "puppeteer"
import * as Plot from "@observablehq/plot"

export default async function savePlotChart(
    plotOptions: Plot.PlotOptions,
    path: string
) {
    const browser = await puppeteer.launch({ headless: "new" })
    const page = await browser.newPage()

    // We inject the d3 and plot code.
    await page.addScriptTag({
        path: "node_modules/d3/dist/d3.js",
    })
    await page.addScriptTag({
        path: "node_modules/@observablehq/plot/dist/plot.umd.js",
    })

    // We create an empty div.
    await page.setContent("<div id='dataviz'></div>")

    // We generate the chart and append it to our div.
    await page.evaluate((plotOptions) => {
        const plot = Plot.plot(plotOptions)
        const div = document.querySelector("#dataviz")
        if (!div) {
            throw new Error("No div with id dataviz")
        }
        div.append(plot)
    }, plotOptions)

    // We select the div and save a screenshot of it.
    const dataviz = await page.$("#dataviz")
    if (!dataviz) {
        throw new Error("No dataviz element.")
    }
    await dataviz.screenshot({ path })
    await browser.close()
}

But when I run this, I get this error:
TypeError: invalid mark; missing render function new Render (node_modules/@observablehq/plot/dist/plot.umd.js:7649:45)

And here's what's at node_modules/@observablehq/plot/dist/plot.umd.js:7649:45:

class Render extends Mark {
  constructor(render) {
    if (typeof render !== "function") throw new TypeError("invalid mark; missing render function");
    super();
    this.render = render;
  }
  render() {}
}

Do you have any ideas or suggestions on how to make it work?

Thanks! :)

[Fil]

Not sure what's happening here; it looks like the Plot.dot mark is not preserved between the two contexts?

Note: I tend to use linkedom to generate SVG in node (see for example https://www.val.town/v/fil/beckerBarley). (I realize this comment is probably not helpful.)

[mbostock]

I don’t think you can pass Plot options through the page.evaluate boundary (because Mark instances won’t be serializable). You could pass in a string and evaluate that, maybe. We use node-canvas in our tests to rasterize plots.

[nshiab]

Thanks for the suggestions! I tweaked the code to pass a string to page.evaluate. It worked great except with dates, of course. So, I also convert dates to their ms value before stringifying. And then I convert them back once in the browser.

And it works! 🥳

We pass the data, a function wrapping the Plot code, and a path.

const data = [
    { salary: 75000, hireDate: new Date("2022-12-15") },
    { salary: 82000, hireDate: new Date("2022-11-20") },
    { salary: 60000, hireDate: new Date("2022-10-25") },
    { salary: 90000, hireDate: new Date("2022-09-30") },
    { salary: 72000, hireDate: new Date("2022-08-15") },
    { salary: 55000, hireDate: new Date("2022-07-20") },
    { salary: 68000, hireDate: new Date("2022-06-25") },
    { salary: 48000, hireDate: new Date("2022-05-30") },
    { salary: 77000, hireDate: new Date("2022-04-15") },
    { salary: 88000, hireDate: new Date("2022-03-20") },
]

await savePlotChart(
    data,
    (data) =>
        Plot.plot({
            marginLeft: 75,
            color: { legend: true },
            marks: [
                Plot.dot(data, {
                    x: "hireDate",
                    y: "salary",
                    stroke: "salary",
                }),
            ],
        }),
    "./my-chart.png"
)

And this is what savePlotChart does.

import puppeteer from "puppeteer"

export default async function savePlotChart(
    data: { [key: string]: unknown }[],
    makeChart: (
        data: { [key: string]: unknown }[]
    ) => SVGSVGElement | HTMLElement,
    path: string
) {
    // We check which keys hold dates, based on the first data item.
    const keysWithDates = Object.keys(data[0]).filter(
        (key) => data[0][key] instanceof Date
    )
    // And we convert the dates to ms in the data. They will be easier to convert back to dates this way.
    if (keysWithDates.length > 0) {
        for (const d of data) {
            for (const key of keysWithDates) {
                d[key] = (d[key] as Date).getTime()
            }
        }
    }

    const browser = await puppeteer.launch({ headless: "new" })
    const page = await browser.newPage()
    // For better screenshot resolution
    await page.setViewport({ width: 1000, height: 1000, deviceScaleFactor: 2 })

    // We inject the d3 and plot code.
    await page.addScriptTag({
        path: "node_modules/d3/dist/d3.js",
    })
    await page.addScriptTag({
        path: "node_modules/@observablehq/plot/dist/plot.umd.js",
    })

    // We create an empty div with a display flex to avoid blank space in the screenshot.
    await page.setContent("<div id='dataviz' style='display:flex;'></div>")

    // We convert back dates, generate the chart and append it to our div.
    await page.evaluate(`
        const data = ${JSON.stringify(data)}
        const keysWithDates = ${JSON.stringify(keysWithDates)}
        for (const key of keysWithDates) {
            for (const d of data) {
                d[key] = new Date(d[key])
            }
        }
        const makeChart = ${makeChart.toString()}
        const chart = makeChart(data)
        const div = document.querySelector("#dataviz")
        if (!div) {
            throw new Error("No div with id dataviz")
        }
        div.append(chart)`)

    // We select the generated figure or svg and save a screenshot of it.
    const dataviz = await page.$("#dataviz > figure, #dataviz > svg")
    if (!dataviz) {
        throw new Error("No dataviz element.")
    }
    await dataviz.screenshot({ path })
    await browser.close()
}

Here's the output!

I published the code in the journalism library. I intend to use it with the simple-data-analysis library.

Again, thanks for the help. :)

Cheers