Running puppeteer gatsby netlify function

[kdichev]

Hi I am trying to use gatsby netlify function with puppeteer but I am running in quite a few issues when my app is deployed to netlify. I’ve found a few topics with the issue but haven’t found the solution for this yet. Hence I am creating this issue.

when I try to trigger the puppeteer function I get this error:
Error when executing function “create-pdf.ts”: “Could not find Chrome (ver. 112.0.5615.121). This can occur if either 1. you did not perform an installation before running the script (e.g. npm install) or 2. your cache path is incorrectly configured (which is: /home/sbx_user1051/.cache/puppeteer). For (2), check out our guide on configuring puppeteer at https://pptr.dev/guides/configuration.”

I’ve added @sparticuz/chromium and suggested fixes from this repo but then I get this erorr:

Error when executing function “create-pdf.ts”: “The input directory “/var/task/.cache/bin” does not exist.”

I've made a simple reproduction repo: https://github.com/kdichev/gatsby-netlify-puppeteer-function/pull/1/files

But I've still haven't managed to make it work. I've also made a netlify support thread with all my findings
https://answers.netlify.com/t/running-puppeteer-gatsby-netlify-function/91453/1

I know that this could not be issue with sparticuz/chromium but I want to get more feedback if possible. If you feel that this is not for here feel free to close. Thanks a lot!

[Sparticuz]

That error means that your bin folder, with the chromium.br file, is somewhere else. I've not used netlify, so I'm unsure how the file system looks. Other packaging systems require something like externalDependencies: "@sparticuz/chromium" or something similar so it knows to keep this package. You could also use @sparticuz/chromium-min and download the pack separately.

[kdichev]

That error means that your bin folder, with the chromium.br file, is somewhere else. I've not used netlify,

This is also my findings as I've done some debugging and the default paths used are not there at all so the bin folder is somewhere else indeed. I've found some "fixes" in this repo for netlify that ensures that the external package is kept but seems like it's not working anymore or I've got some issue with my implementation.

#24 (comment)

I will continue the research and will probably make a doc page here when I find the solution , hopefully with the help of netlify support.

[kdichev]

I tried with @sparticuz/chromium-min and I am getting a different error:

Error: Failed to launch the browser process!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

changed the node runtime to 18 without change

import { GatsbyFunctionRequest, GatsbyFunctionResponse } from "gatsby";
const puppeteer = require("puppeteer-core");
const chromium = require("@sparticuz/chromium-min");

export default async function createPdf(
  _: GatsbyFunctionRequest,
  res: GatsbyFunctionResponse
) {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(
      "https://github.com/Sparticuz/chromium/releases/download/v113.0.0/chromium-v113.0.0-pack.tar"
    ),
    headless: chromium.headless,
    ignoreHTTPSErrors: true,
  });
  const page = await browser.newPage();
  await page.goto("https://www.example.com", {
    waitUntil: "networkidle0",
  });
  const pdf = await page.pdf({ format: "A4" });
  console.log(pdf);
  await browser.close();
  res.status(200).json({ success: true });
}

here is how the temp folder looks

INFO   chromium
INFO   chromium-pack
INFO   libEGL.so
INFO   libGLESv2.so
INFO   libvk_swiftshader.so
INFO   libvulkan.so.1
INFO   vk_swiftshader_icd.json

[kdichev]

ok so I probably managed to find the place for chromium.br file is by checking netlify docs for external node modules, i had to update netlify.toml like so:

[[plugins]]
    package = "@netlify/plugin-gatsby"
[functions]
    node_bundler = "esbuild"
    external_node_modules = ["@sparticuz/chromium"]

had to add node_bundler to esbuild as stated by netlify docs in order to use external_node_modules
then needed to update the code with the proper path to the file

  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(
      "/var/task/node_modules/@sparticuz/chromium/bin"
    ),
    headless: chromium.headless,
  });

correct path is /var/task/node_modules/@sparticuz/chromium/bin

although this beringed me closer to the solution I still got an error:

Error when executing function "create-pdf.ts": "Failed to launch the browser process!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

which is the same as using @sparticuz/chromium-min with the url https://github.com/Sparticuz/chromium/releases/download/v113.0.0/chromium-v113.0.0-pack.tar

so I guess this is working but still have issue with the libnss3.so

wondering now what's the next step heh

[Sparticuz]

I tried with @sparticuz/chromium-min and I am getting a different error:

Error: Failed to launch the browser process!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

That means that the AWS package isn't being extracted. Can you get me the output of the AWS_EXECUTION_ENV environment variable?

[kdichev]

I tried with @sparticuz/chromium-min and I am getting a different error:

Error: Failed to launch the browser process!
/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

That means that the AWS package isn't being extracted. Can you get me the output of the AWS_EXECUTION_ENV environment variable?

in netlify the variable for the env is different (AWS_LAMBDA_JS_RUNTIME) but it's setup to nodejs:16.v13

[Sparticuz]

I published 113.0.1 that will look for that env var and extract the aws pack as well.

[kdichev]

@Sparticuz I've tested both of my branches @sparticuz/chromium and @sparticuz/chromium-min with the latest version but I still get the same error

https://github.com/kdichev/gatsby-netlify-puppeteer-function/pull/1/files
https://github.com/kdichev/gatsby-netlify-puppeteer-function/pull/3/files

[Sparticuz]

It checks that the AWS_LAMBDA_JS_RUNTIME env var starts with nodejs https://github.com/Sparticuz/chromium/pull/98/files#diff-0f2cb7ff7dfccabe7dbdb4b008cdfad1ab1a9003144226d414e5e817715dbacb. A look at process.env would help.

[kdichev]

unfortunately is still failing with the same error:
https://deploy-preview-1--gatsby-netlify-puppeteer-function.netlify.app/ chromium
https://deploy-preview-3--gatsby-netlify-puppeteer-function.netlify.app/ chromium-min

logging this is truly:

console.log(
    process.env["AWS_LAMBDA_JS_RUNTIME"] &&
      /^nodejs/.test(process.env["AWS_LAMBDA_JS_RUNTIME"]) === true
 );

here is my env:
AWS_LAMBDA_JS_RUNTIME=nodejs16.x

[kdichev]

@Sparticuz you have an idea what else could be wrong?

[Sparticuz]

The libnss error in the past, has meant that the aws package isn't being extracted. The code I added checks for that netlify variable and extracts it. https://github.com/Sparticuz/chromium/blob/master/source/index.ts#L308 You can try to see if that promise is being pushed by adding in a console log inside that if block.

[Astevens521]

Hi kdichev, I struggled with the same situation and was successful simple action, hope this can help you.
* still failing page and browser close, I want to close this page and open a new page.

    "@netlify/functions": "^1.4.0",
    "puppeteer-core": "^20.1.1",
    "@sparticuz/chromium-min": "^113.0.1",

learned from kdichev
netlify.toml

[functions]
  node_bundler = "esbuild"
  external_node_modules = ["@sparticuz/chromium-min"]

.../netlify/functions/chromium_test.ts

import { Handler } from "@netlify/functions";
import puppeteer from "puppeteer-core";
import chromium from "@sparticuz/chromium-min";

const handler: Handler = async () => {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath("https://github.com/Sparticuz/chromium/releases/download/v113.0.1/chromium-v113.0.1-pack.tar"),
    headless: chromium.headless,
  });

  try {
    const page = await browser.newPage();
    await page.goto("https://example.com");
    const pageTitle = await page.title();
    console.log("page title: ", pageTitle);
  } catch (err) {
    console.error("error", err.message);
  } finally {
    // some reason: dont return "await Page.close" & "await Browser.close" only Production
    browser.close();
  }

  return { statusCode: 200, body: "" };
};

export { handler };

[nikme]

Hi kdichev, I struggled with the same situation and was successful simple action, hope this can help you. * still failing page and browser close, I want to close this page and open a new page.

    "@netlify/functions": "^1.4.0",
    "puppeteer-core": "^20.1.1",
    "@sparticuz/chromium-min": "^113.0.1",

learned from kdichev netlify.toml

[functions]
  node_bundler = "esbuild"
  external_node_modules = ["@sparticuz/chromium-min"]

.../netlify/functions/chromium_test.ts

import { Handler } from "@netlify/functions";
import puppeteer from "puppeteer-core";
import chromium from "@sparticuz/chromium-min";

const handler: Handler = async () => {
  const browser = await puppeteer.launch({
    args: chromium.args,
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath("https://github.com/Sparticuz/chromium/releases/download/v113.0.1/chromium-v113.0.1-pack.tar"),
    headless: chromium.headless,
  });

  try {
    const page = await browser.newPage();
    await page.goto("https://example.com");
    const pageTitle = await page.title();
    console.log("page title: ", pageTitle);
  } catch (err) {
    console.error("error", err.message);
  } finally {
    // some reason: dont return "await Page.close" & "await Browser.close" only Production
    browser.close();
  }

  return { statusCode: 200, body: "" };
};

export { handler };

So, I managed to get it fixed.

First thing was to disable WebGL with chromium.setGraphicsMode = false;
and second was to close all pages before closing browser otherwise it hangs on const pages = await browser.pages(); await Promise.all(pages.map((p) => p.close()));
And of course, the key to getting it running even was adding executablePath: await chromium.executablePath( "https://github.com/Sparticuz/chromium/releases/download/v119.0.0/chromium-v119.0.0-pack.tar" ),

I hope this will save someone' 2 weeks of getting a PDF of a page =D

p.s. if you are using nextjs, you need to update next config with
const nextConfig = { experimental: { serverComponentsExternalPackages: ['puppeteer-core'], } }

e.q. creating pdf

import { NextRequest, NextResponse } from "next/server";
import puppeteer, { Browser } from "puppeteer-core";
import chromium from "@sparticuz/chromium-min";
const exePath = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome";

chromium.setHeadlessMode = true;
// STEP 1 - setting browser not to use webGL
chromium.setGraphicsMode = false; 

....

async function getOptions(isDevelopment?: boolean) {
   if (isDevelopment) {
       return {
           args: [],
           executablePath: exePath,
           headless: true,
       };
   }
   return {
       args: [...chromium.args, "--hide-scrollbars", "--disable-web-security"],
       defaultViewport: chromium.defaultViewport,
       executablePath: await chromium.executablePath(
           "https://github.com/Sparticuz/chromium/releases/download/v119.0.0/chromium-v119.0.0-pack.tar"
       ),

       ignoreHTTPSErrors: true,
       headless: chromium.headless,
   };
} 

...

let browser: Browser;
   try {
       const options = await getOptions(isDevelopment);
       browser = await puppeteer.launch(options);
       const page = await browser.newPage();
       await page.goto(url);

       const pdf = await page.pdf({ format: "A4", printBackground: true, scale: 0.8 });
       // set headers for binary response

       const init = {
           headers: {
               "Content-Type": "application/pdf",
               "Content-Disposition": `attachment; filename="name-${id}.pdf"`,
           },
       };
      // STEP 2 - close ALL pages, for some reason, there are 2 pages!
       const pages = await browser.pages();
       await Promise.all(pages.map((p) => p.close()));
       await browser.close();

       return new Response(pdf, init);
   } catch (error) {
       console.error({ error });
       return NextResponse.json({ error: "Something went wrong" }, { status: 500 });
   } 

Cudos to all guys here and external comments
#85 (comment)
puppeteer/puppeteer#11052 (comment)

[samlogan]

Hey, @nikme, how did you resolve the following error?

/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

I am stuck here with the Gatsby Netlify function I am working on.

[nikme]

Hey, @nikme, how did you resolve the following error?

/tmp/chromium: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

I am stuck here with the Gatsby Netlify function I am working on.

@samlogan I didn't get this error =(

[ayushrana182]

@samlogan were you able to get past that issue? I'm stuck there too! please share what worked for you if you did, much appreciated!