feat: support sharding in HTML Reporter #19691

kevin940726 · 2022-12-24T17:56:29Z

This feature aims to bring official support to combine sharded reports in the HTML reporter. This is probably not the best solution, but I think it works with minimal changes.

How it works

We inline the shard config into a global variable along with other metadata for future usage called window.playwrightMetadata.
Instead of inlining the ZIP file into a base64 string, we write the ZIP into a report.zip file. If there's sharding, we instead write them into a report-{shardIndex}.zip file.
After each run, we typically upload the generated report directory somewhere to an artifact. Once every part is finished, we download and extract them to a single place. Everything except for the report-{shardIndex}.zip files should be overridden and replaced.
In the web app, instead of reading the ZIP from a base64 string, we read them from HTTP. If there's sharding, we read the reports one by one and merge them together.

Comparing to `playwright-merge-html-reports`

The mergeStats helper is largely inspired by playwright-merge-html-reports, but I think this approach has some advantages:

Official support.
No extra steps are needed, no need to run another script to combine the reports. This can be tedious for CI as they have to initialize an environment, npm install, download reports, run the scripts, then finally upload the report.
Works out of the box, no need for another API.

Example setup

GitHub Actions with artifacts

steps:
  playwright:
    name: 'Playwright Tests - ${{ matrix.project }} - Shard ${{ matrix.shardIndex }} of ${{ matrix.shardTotal }}'
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.29.0-focal
    strategy:
      fail-fast: false
      matrix:
        project: [chromium, webkit]
        shardIndex: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
        shardTotal: [10]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
      - name: Install dependencies
        run: npm ci
      - name: Run your tests
        run: npx playwright test --project=${{ matrix.project }} --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - uses: actions/upload-artifact@v3
        with:
          name: playwright-report
          path: playwright-report

WDYT? Once we're happy with the approach we can start writing some docs and tests.

kevin940726 · 2022-12-24T17:57:35Z

@microsoft-github-policy-service agree

pavelfeldman

Looks great! Sorry for the late review, just returned from the break.

pavelfeldman · 2023-01-10T16:57:39Z

packages/html-reporter/src/index.tsx

+      }
+      this._json = mergeReports(reports);
+    } else {
+      const zipReader = new zipjs.ZipReader(new zipjs.HttpReader('/report/report.zip'), { useWebWorkers: false }) as zip.ZipReader;


Having it all in one file is a strong requirement, let's keep it as is. Loading shards from http(s) is Ok though.

pavelfeldman · 2023-01-10T16:58:05Z

packages/html-reporter/index.html

  <body>
    <div id='root'></div>
    <script type='module' src='/src/index.tsx'></script>
+    <dialog id="fallback-error">


Let's remove this: we should be able to open it from file:// - it was a strong requirement from the community.

pavelfeldman · 2023-01-10T17:01:56Z

packages/html-reporter/src/index.tsx

+      const reports: HTMLReport[] = [];
+      for (let index = 1; index <= metadata.shard.total; index += 1) {
+        try {
+          const zipReader = new zipjs.ZipReader(new zipjs.HttpReader(`/report/report-${index}.zip`), { useWebWorkers: false }) as zip.ZipReader;


Let's keep those files siblings in order to be less intrusive, hard-code less names. So it'll be

new zipjs.HttpReader(`report-${index}.zip`)

pavelfeldman · 2023-01-10T17:02:24Z

packages/html-reporter/src/index.tsx

+          const currentJson = await this.entry(`report-${index}.json`) as HTMLReport;
+          reports.push(currentJson);
+        } catch (error) {
+          // Ignore not found error for viewing individual shard report.


Let's make sure these errors end up in devtools so that it was clear what is going on.

pavelfeldman · 2023-01-10T17:02:57Z

packages/html-reporter/src/index.tsx

+      for (let index = 1; index <= metadata.shard.total; index += 1) {
+        try {
+          const zipReader = new zipjs.ZipReader(new zipjs.HttpReader(`/report/report-${index}.zip`), { useWebWorkers: false }) as zip.ZipReader;
+          for (const entry of await zipReader.getEntries())


Looks like you could benefit from fetching them all concurrently instead.

pavelfeldman · 2023-01-10T17:06:04Z

packages/playwright-test/src/reporters/html.ts

+    // Inline metadata.
    const indexFile = path.join(this._reportFolder, 'index.html');
-    fs.appendFileSync(indexFile, '<script>\nwindow.playwrightReportBase64 = "data:application/zip;base64,');
+    fs.appendFileSync(indexFile, `<script>\nwindow.playwrightMetadata = ${JSON.stringify(metadata)}\n</script>`);


We would need both, the new data and the base64 placeholder.

dgozman · 2023-01-27T04:10:16Z

@kevin940726 Heads up: I will pick this up, create a new PR and attribute existing work in the description. This way it will be easier to manage and land the feature.

kevin940726 · 2023-01-27T04:27:07Z

@dgozman Sure! I'll be away until next week anyway (LNY), so feel free to do anything with this PR :). Also thank you @pavelfeldman for the review!

@kevin940726

This implementation is based on the [original PR](#19691) by @kevin940726. It makes the reporter produce single file when there is no sharding and multiple out-of-line report-x-of-y.zip reports which are automatically merged together when put in one folder. References #10437 Co-authored-by: Kai Hao <[email protected]>

mxschmitt · 2023-02-08T10:06:57Z

Looks like got implemented in #20700. Closing for now then and thanks for the contribution!

feat: supports sharding in HTML Reporter

5364c0a

kevin940726 force-pushed the feat/support-sharding-in-html-reporter branch from 7bf0899 to 5364c0a Compare December 24, 2022 18:05

fix: file:// protocol warning

8e29b22

kevin940726 changed the title ~~feat: supports sharding in HTML Reporter~~ feat: support sharding in HTML Reporter Dec 24, 2022

mxschmitt mentioned this pull request Jan 4, 2023

fix(network): Accommodate async route handler callbacks #19856

Closed

mxschmitt requested a review from dgozman January 4, 2023 18:27

pavelfeldman reviewed Jan 10, 2023

View reviewed changes

yury-s mentioned this pull request Feb 7, 2023

feat: sharded html report #20700

Merged

mxschmitt closed this Feb 8, 2023

kevin940726 deleted the feat/support-sharding-in-html-reporter branch February 8, 2023 10:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: support sharding in HTML Reporter #19691

feat: support sharding in HTML Reporter #19691

Uh oh!

kevin940726 commented Dec 24, 2022 •

edited

Loading

Uh oh!

kevin940726 commented Dec 24, 2022

Uh oh!

pavelfeldman left a comment

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

pavelfeldman Jan 10, 2023

Uh oh!

dgozman commented Jan 27, 2023

Uh oh!

kevin940726 commented Jan 27, 2023

Uh oh!

mxschmitt commented Feb 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat: support sharding in HTML Reporter #19691

feat: support sharding in HTML Reporter #19691

Uh oh!

Conversation

kevin940726 commented Dec 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Comparing to playwright-merge-html-reports

Example setup

Uh oh!

kevin940726 commented Dec 24, 2022

Uh oh!

pavelfeldman left a comment

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

pavelfeldman Jan 10, 2023

Choose a reason for hiding this comment

Uh oh!

dgozman commented Jan 27, 2023

Uh oh!

kevin940726 commented Jan 27, 2023

Uh oh!

mxschmitt commented Feb 8, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kevin940726 commented Dec 24, 2022 •

edited

Loading

Comparing to `playwright-merge-html-reports`