Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: .toMatchSnapshot() CLI output unusable with large text files #33692

Open
karlhorky opened this issue Nov 20, 2024 · 1 comment
Open

[Bug]: .toMatchSnapshot() CLI output unusable with large text files #33692

karlhorky opened this issue Nov 20, 2024 · 1 comment
Assignees
Labels

Comments

@karlhorky
Copy link
Contributor

karlhorky commented Nov 20, 2024

Version

1.48.2

Steps to reproduce

  1. Create a large text snapshot, eg. comparing all list items on a very long list (eg. (await page.locator('h2:text("Schedule") + div.grid a').allInnerTexts()).join('\n'))
  2. Make an intentional error in the text snapshot
  3. 💥 Run Playwright CLI and see very hard to parse output (see below) - especially hard in logs for GitHub Actions workflow failures
  1) playwright/schedulerCreatesCohort.spec.ts:115:1 › Scheduler creates cohort ────────────────────

    Error: Snapshot comparison failed:

      [
        {
          "title": "Intro to Course, Remote Collaboration",
          "startTimestamp": "2024-09-09T07:00:00.000Z",
          "endTimestamp": "2024-09-09T09:00:00.000Z"
        },

        ... many more lines here, potentially hundreds or thousands of lines ...

      ]

At some point, this visual diff using colored text via ANSI escape codes shows up in the output:

Image

This is really easy to miss, even if you quickly scroll through all lines.

Expected behavior

  1. Show a concise diff, with only a few lines of context on each side of each difference
  2. Maybe don't only show the changes via color, but also via text (accessibility)

Eg. the same type of diff as in .toEqual([ ... ])

Actual behavior

  1. Shows the full file content
  2. Only shows diff via hard-to-spot colors

Additional context

--

Environment

System:
    OS: macOS 15.1.1
    CPU: (8) arm64 Apple M1
    Memory: 81.94 MB / 16.00 GB
  Binaries:
    Node: 22.11.0 - /opt/homebrew/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.9.0 - /opt/homebrew/bin/npm
    bun: 1.1.30 - /opt/homebrew/bin/bun
  IDEs:
    VSCode: 1.95.3 - /opt/homebrew/bin/code
  Languages:
    Bash: 5.2.0 - /opt/homebrew/bin/bash
@karlhorky
Copy link
Contributor Author

karlhorky commented Nov 20, 2024

Workaround

Use toEqual() like this:

expect(await page.locator('h2:text("Schedule") + div.grid a').allInnerTexts()).toEqual([ ... ])

This will display a better, more usable, accessible diff with fewer lines of context:

·
  1) playwright/pernExtensiveImmersiveStudentBrowses.spec.ts:33:3 › PERN Extensive (Immersive) student browses › PERN Extensive (Immersive) student browses 
    Error: expect(received).toEqual(expected) // deep equality
    - Expected  - 1
    + Received  + 1
    @@ -64,11 +64,11 @@
      09:30 - 11:00",
        "Project Time
      11:00 - 17:00",
        "Retrospective and Review
      17:00 - 18:00",
    -   "🧑‍🏫 Intro to Cour, Remote Collaboration",
    +   "🧑‍🏫 Intro to Course, Remote Collaboration",
        "🧑‍🏫 Job Application Processes",
        "🧑‍🏫 LinkedIn and GitHub Profile Setup",
        "🧑‍🏫 CV Setup and Job Market",
        "Daily Scrum",
      326 |       expect(await allScheduleAnchors.count()).toBeGreaterThan(200);
      327 |     }).toPass();
    > 328 |     expect(await allScheduleAnchors.allInnerTexts()).toEqual([
          |                                                      ^
        at 

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants