[Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 by tsullivan · Pull Request #156766 · elastic/kibana

tsullivan · 2023-05-04T23:45:21Z

Summary

Upgrades the dependency for capturing screenshots with Chromium to Puppeteer v20.1.0

We're using publicly available Chromium downloads for Windows and Mac. For Linux, we create a custom build that doesn't have dependencies on X11 shared libraries.

Versions:

Linux: 113.0.5672.63
Windows: 113.0.5672.0
Darwin x64: 113.0.5672.0
Darwin arm64: 113.0.5672.0

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
Verified in Cloud deployment

For maintainers

This was checked for breaking API changes and was labeled appropriately

tsullivan · 2023-05-04T23:47:11Z

src/dev/chromium_version.ts

This script had to be reworked as the Chromium team has changed the APIs to look up Chromium versions and commits. See https://groups.google.com/a/chromium.org/g/chromium-dev/c/uH-nFrOLWtE/m/HW1BoGNPAgAJ

…d browser checksum

…fix'

tsullivan · 2023-05-05T16:48:36Z

src/dev/chromium_version.ts

-  log.info(`Found Chromium commit ${commit} from revision ${revision}.`);
-  log.info(`Mac x64 download:     ${baseUrl}/Mac/${revision}/chrome-mac.zip`);
-  log.info(`Mac ARM download:     ${baseUrl}/Mac_Arm/${revision}/chrome-mac.zip`);
-  log.info(`Windows x64 download: ${baseUrl}/Win/${revision}/chrome-win.zip`);


Removed these lines since the download URLs are properly tested in the integration test

tsullivan · 2023-05-05T16:51:21Z

x-pack/plugins/screenshotting/server/browsers/chromium/integration_tests/paths.test.ts

+  console.error(message);
+});
+
+describe('Chromium Archive Paths', () => {


This test touches on #155753, but I think we need to keep that issue open. I still think we need a way to see the version of Chromium from a running instance.

tsullivan · 2023-05-05T16:52:23Z

x-pack/plugins/screenshotting/server/browsers/chromium/paths.ts

-      binaryChecksum: 'ef21a88efa18f000e6da6d9c51ee2fd7',
+      archiveFilename: 'chromium-38c3182-locales-linux_arm64.zip',
+      archiveChecksum: '96dca82cccea6ae82aaf0bc46104a501',
+      binaryChecksum: '753e07c59b6f269b2f06091155d53f4b',


The arm64/x64 linux binaries were built using the scripts in x-pack/build_chromium without any incidents.

tsullivan · 2023-05-05T16:53:30Z

src/dev/build/tasks/install_chromium.ts

 * Side Public License, v 1.
 */

+import type { Logger } from '@kbn/core/server';


Converted this file to TypeScript so that looking up references of our install function would work.

Context on this? Is the build logger not sufficient for install anymore?

Not blocking, sounds like it'll get imported either way so not worried.

elasticmachine · 2023-05-05T16:54:50Z

Pinging @elastic/appex-sharedux (Team:SharedUX)

tsullivan · 2023-05-05T17:36:05Z

x-pack/plugins/screenshotting/server/browsers/chromium/paths.ts

+      archiveChecksum: '100c8a63c32a17d8b5816ef09164dadf',
+      binaryChecksum: '9960dd00ab27b4e9ee1455692bb65701',
      binaryRelativePath: 'chrome-mac/Chromium.app/Contents/MacOS/Chromium',
+      revision: 1121473,


The revision number is what we use to find the commit for the custom build, and locate the download bucket for the default builds.

To look up the "version" string from the given revision, I use https://omahaproxy.appspot.com/. This is what I used to look up the versions for each OS in the PR description. Note: this tool is getting deprecated: https://github.com/elastic/kibana/pull/156766/files#r1185605869

Dosant

Great job!
I've just tested it locally on arm mac

in case you missed it, there is a warning when starting:

 Puppeteer old Headless deprecation warning:
    In the near feature `headless: true` will default to the new Headless mode
    for Chrome instead of the old Headless implementation. For more
    information, please see https://developer.chrome.com/articles/new-headless/.
    Consider opting in early by passing `headless: "new"` to `puppeteer.launch()`
    If you encounter any bugs, please report them to https://github.com/puppeteer/puppeteer/issues/new/choose.

Dosant · 2023-05-08T08:04:17Z

x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.ts

+  // As a workaround, we force hardware GL drivers on mac: v111 and older versions should work with --use-angle.
+  // The best way to do this when the issue is resolved will be to pass --enable-gpu,
  if (os.arch() === 'arm64' && process.platform === 'darwin') {
    flags.push('--use-angle');


Weird :(
tried '--enable-gpu' and it didn't work.
I guess have to leave --use-angle for now

Dosant · 2023-05-08T08:07:48Z

@elasticmachine merge upstream

tsullivan · 2023-05-08T19:02:46Z

in case you missed it, there is a warning when starting:

Looking now, to see if the suggested fix will suppress the warning.

Update: pushed 373496f. I highly recommend reading the article that was linked in the deprecation warning. It could be relevant to our process for updating Linux: https://developer.chrome.com/articles/new-headless/

kibana-ci · 2023-05-08T20:05:57Z

💛 Build succeeded, but was flaky

Failed CI Steps

FTR Configs #56

Test Failures

[job] [logs] FTR Configs #56 / ObservabilityApp Observability alerts / Add to case > When user has all privileges for cases opens a modal when Add to existing case is clicked

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`screenshotting`	3	4	+1

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	19	21	+2
`securitySolution`	399	403	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	20	22	+2
`screenshotting`	9	10	+1
`securitySolution`	479	483	+4
total			+7

History

💚 Build #126106 succeeded 0e7d86a
💚 Build #126031 succeeded dcd3671

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @tsullivan

…lastic#156766) ## Summary Upgrades the dependency for capturing screenshots with Chromium to [Puppeteer v20.1.0](https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-v20.1.0) We're using publicly available Chromium downloads for Windows and Mac. For Linux, we create a custom build that doesn't have dependencies on X11 shared libraries. **Versions:** * Linux: **113.0.5672.63** * Windows: **113.0.5672.0** * Darwin x64: **113.0.5672.0** * Darwin arm64: **113.0.5672.0** ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Verified in Cloud deployment ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 89a8ab8) # Conflicts: # src/dev/chromium_version/index.ts

kibanamachine · 2023-05-08T21:21:31Z

💔 All backports failed

Status	Branch	Result
❌	7.17	Backport failed because of merge conflicts
❌	8.6	Backport failed because of merge conflicts
❌	8.7	Backport failed because of merge conflicts
❌	8.8	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 156766

Questions ?

Please refer to the Backport tool documentation

…lastic#156766) ## Summary Upgrades the dependency for capturing screenshots with Chromium to [Puppeteer v20.1.0](https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-v20.1.0) We're using publicly available Chromium downloads for Windows and Mac. For Linux, we create a custom build that doesn't have dependencies on X11 shared libraries. **Versions:** * Linux: **113.0.5672.63** * Windows: **113.0.5672.0** * Darwin x64: **113.0.5672.0** * Darwin arm64: **113.0.5672.0** ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Verified in Cloud deployment ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 89a8ab8) # Conflicts: # package.json # src/dev/build/tasks/install_chromium.ts # src/dev/chromium_version/index.ts # src/dev/tsconfig.json # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.test.ts # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.ts # x-pack/plugins/screenshotting/server/browsers/chromium/paths.ts # yarn.lock

…lastic#156766) ## Summary Upgrades the dependency for capturing screenshots with Chromium to [Puppeteer v20.1.0](https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-v20.1.0) We're using publicly available Chromium downloads for Windows and Mac. For Linux, we create a custom build that doesn't have dependencies on X11 shared libraries. **Versions:** * Linux: **113.0.5672.63** * Windows: **113.0.5672.0** * Darwin x64: **113.0.5672.0** * Darwin arm64: **113.0.5672.0** ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Verified in Cloud deployment ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 89a8ab8) # Conflicts: # package.json # src/dev/build/tasks/install_chromium.ts # src/dev/chromium_version/index.ts # src/dev/tsconfig.json # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.test.ts # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.ts # x-pack/plugins/screenshotting/server/browsers/chromium/paths.ts # x-pack/plugins/screenshotting/tsconfig.json # yarn.lock

tsullivan · 2023-05-08T22:15:41Z

💚 All backports created successfully

Status	Branch	Result
✅	8.6
✅	7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…1.0 (#156766) (#157088) # Backport This will backport the following commits from `main` to `8.8`: - [[Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766)](#156766)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

…lastic#156766) ## Summary Upgrades the dependency for capturing screenshots with Chromium to [Puppeteer v20.1.0](https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-v20.1.0) We're using publicly available Chromium downloads for Windows and Mac. For Linux, we create a custom build that doesn't have dependencies on X11 shared libraries. **Versions:** * Linux: **113.0.5672.63** * Windows: **113.0.5672.0** * Darwin x64: **113.0.5672.0** * Darwin arm64: **113.0.5672.0** ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Verified in Cloud deployment ### For maintainers - [x] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> # Conflicts: # package.json # src/dev/build/tasks/install_chromium.ts # src/dev/chromium_version/index.ts # src/dev/tsconfig.json # x-pack/build_chromium/linux/args.gn # x-pack/plugins/reporting/server/browsers/chromium/driver_factory/args.ts # x-pack/plugins/reporting/server/browsers/chromium/paths.ts # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/args.test.ts # x-pack/plugins/screenshotting/server/browsers/chromium/driver_factory/index.ts # x-pack/plugins/screenshotting/server/browsers/download/index.ts # x-pack/plugins/screenshotting/server/browsers/install.ts # x-pack/plugins/screenshotting/tsconfig.json # yarn.lock

…1.0 (#156766) (#157091) # Backport This will backport the following commits from `main` to `8.7`: - [[Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766)](#156766)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

….1.0 (#156766) (#157111) # Backport This will backport the following commits from `main` to `7.17`: - [[Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766)](#156766) Depends on #157436  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

tsullivan added 4 commits May 4, 2023 13:58

update puppeteer package version to 20.1.0

0535b9a

fix chromium_version script again

c7ad1e4

add integration test to cover hardcoded checksums

917acbf

update checksums

62d1654

tsullivan commented May 4, 2023

View reviewed changes

test verification of the downloaded archive checksum and the extracte…

3432ae8

…d browser checksum

tsullivan force-pushed the puppeteer-v20.1.0 branch 2 times, most recently from 63da6ad to 3432ae8 Compare May 5, 2023 02:55

kibanamachine and others added 5 commits May 5, 2023 03:01

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

5a114f3

…fix'

revert signature change

a910c2e

Merge remote-tracking branch 'elastic/main' into puppeteer-v20.1.0

96ad529

convert install chromium build task to ts

080b5ae

update args test

a574e59

tsullivan commented May 5, 2023

View reviewed changes

tsullivan marked this pull request as ready for review May 5, 2023 16:54

tsullivan requested review from a team as code owners May 5, 2023 16:54

tsullivan added release_note:fix Team:SharedUX Platform AppEx-SharedUX (formerly Global Experience) t// labels May 5, 2023

tsullivan marked this pull request as draft May 5, 2023 17:02

fix all versions to 113

e1304a6

tsullivan marked this pull request as ready for review May 5, 2023 17:32

tsullivan commented May 5, 2023

View reviewed changes

tsullivan added ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment labels May 5, 2023

Merge branch 'main' into puppeteer-v20.1.0

f8e651d

tsullivan removed the v8.5.4 label May 6, 2023

tsullivan requested a review from a team May 6, 2023 01:21

Dosant approved these changes May 8, 2023

View reviewed changes

Merge branch 'main' into puppeteer-v20.1.0

0e7d86a

jbudz approved these changes May 8, 2023

View reviewed changes

tsullivan self-assigned this May 8, 2023

tsullivan added 2 commits May 8, 2023 12:17

Merge remote-tracking branch 'elastic/main' into puppeteer-v20.1.0

8b97e91

add new flag to suppress deprecation warning

373496f

tsullivan merged commit 89a8ab8 into elastic:main May 8, 2023

tsullivan mentioned this pull request May 8, 2023

[8.8] [Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766) #157088

Merged

tsullivan mentioned this pull request May 8, 2023

[8.7] [Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766) #157091

Merged

tsullivan mentioned this pull request May 8, 2023

[8.6] [Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766) #157106

Closed

tsullivan mentioned this pull request May 8, 2023

[7.17] [Reporting/Screenshotting] upgrade Puppeteer dependency to v20.1.0 (#156766) #157111

Merged

kibanamachine added the v8.8.0 label May 8, 2023

tsullivan deleted the puppeteer-v20.1.0 branch May 9, 2023 16:24

kibanamachine mentioned this pull request May 9, 2023

upgrade webdriver to 4.9.1 #157137

Merged

kibanamachine mentioned this pull request May 17, 2023

Upgrade xml2js (0.4.22 -> 0.5.0) #156737

Merged

Conversation

tsullivan commented May 4, 2023 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

For maintainers

Uh oh!

tsullivan May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsullivan May 5, 2023

Choose a reason for hiding this comment

Uh oh!

tsullivan May 5, 2023

Choose a reason for hiding this comment

Uh oh!

tsullivan May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tsullivan May 5, 2023

Choose a reason for hiding this comment

Uh oh!

jbudz May 8, 2023

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented May 5, 2023

Uh oh!

tsullivan May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dosant left a comment

Choose a reason for hiding this comment

Uh oh!

Dosant May 8, 2023

Choose a reason for hiding this comment

Uh oh!

Dosant commented May 8, 2023

Uh oh!

tsullivan commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kibana-ci commented May 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

ESLint disabled in files

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

kibanamachine commented May 8, 2023

💔 All backports failed

Manual backport

Questions ?

Uh oh!

tsullivan commented May 8, 2023

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tsullivan commented May 4, 2023 •

edited by kibanamachine

Loading

tsullivan May 4, 2023 •

edited

Loading

tsullivan May 5, 2023 •

edited

Loading

tsullivan May 5, 2023 •

edited

Loading

tsullivan commented May 8, 2023 •

edited

Loading

kibana-ci commented May 8, 2023 •

edited

Loading