[Bug]: VSCode extension not reloading test code in text editor and showing false errors

[jaktestowac]

Version

1.49.0

Steps to reproduce

Start new project on Windows 11 machine with:
npm init playwright@latest --yes '--' '--quiet' '--browser=chromium'

Make sure that you have Playwright Extension installed (tested on Playwright Test for VSCode v1.1.12)

Make changes to test script to make VSCode show some red underlines indicating that are problems with code.
Use Ctr (Cmd) + Z to udo some actions if bug output will not appear.
This is correct error - added const in line 4:

If incorrect code removed error persist:

With hoover on error:

The problem disappears when navigating to Testing | TEST EXPLORER | Refresh Tests

This issue was observed with already fixed one: #33531

Expected behavior

No errors displayed in code editor

Actual behavior

False error displayed, manual action required.

Additional context

All plugins except the Playwright Extension were turned off. All updates have been applied.

Sometimes several actions on the code need to be executed to trigger this error. It occurs on multiple machines, both Windows and macOS.

This bug appeared in previous extension releases, but recently it has been occurring frequently.

Use of W11 virtual machine can be needed: #33531 (comment)

Reproduction may include executing installation of any Playwright version and browsers since this is somehow connected. In my case issue appeared after fresh Playwright installation. After some time was not reproducible and after triggering latest installation it appeared again.

Environment

VSCode with Playwright Test for VSCode v1.1.12
System:
    OS: Windows 11
    CPU: (16) x64 11th Gen Intel(R) Core(TM)
    Memory: 43.39 GB / 63.71 GB
  Binaries:
    Node: 20.11.0 - \nodejs\node.EXE
    npm: 10.4.0 - \nodejs\npm.CMD
  IDEs:
    VSCode: 1.95.3 - \Microsoft VS Code\bin\code.CMD
  Browsers:
    Edge: Chromium (127.0.2651.74)
  npmPackages:
    @playwright/test: ^1.49.0

[TugaInVegas]

Having same issue and its a buzzkill

[LeggoTM]

+1 Having the same issue

[Makoehle]

Code snippet that causes trouble as well:

const USERNAME_TEST = process.env.USERNAME_TEST;
if (!USERNAME_TEST) {
  throw new Error("'USERNAME_TEST' is not set"); // <<< Throwing Error `USERNAME_TEST` not defined. Deactivating extension fixes issue.
}

[mxschmitt]

Could you confirm if the problems stay around after saving the file? For me when saving they disappear since it reloads the tests.

[Makoehle]

@mxschmitt For me the issue is the extension:

Version	1.1.12
Released on	17.8.2021, 15:06:11

The reproduction is actually fast and simple. I had this extension turned off since this issue and just copy pasted my snippet then turned on extension and a few seconds later 💥

[mxschmitt]

Do you have dotenv configured in your config? How do you set this env normally?

[Makoehle]

Hey, I didn't have it configured. So IMO this should be unrelated to each other. What if I like to set environment variables in my shell? Runtime and compiletime are beeing mixed here IMO.

I gave it a try. Indeed installing dotenv and import "dotenv/config resolves the issue IF an .env file contains the env vars. So thanks for supporting me here.

But this isn't intuitive or IMO not well documented and the error message if variables are missing is not helping either.

Also red underlining the error message of my own error constructor is misleading. I'm experienced enough to handle this on runtime on my own. For me personally not intuitive at all.

But now I understand it and I can handle that for the future. IMO this "feature" requires improvement especially in the error message or best case should be disabled. It confused me much more than it helped.

Finally I tried to reproduce the initial reporters issue and can't confirm that behaviour on my machine. Adding and removing incorrect code doesn't cause this kind of behaviour for me following your instructions.

[jaktestowac]

@mxschmitt maybe important information is Auto Save turned on in VSCode? For my tests it was not. Manual reloading or by shortcut helps.

As @Makoehle pointed out - extension is the problem here.

It is also somehow connected with installation of Playwright (when you change version) - just try to change it. Before installation I had struggle to reproduce on fresh Playwright project.

At the end remember that this issue was reproduced by @pavelfeldman here #33531 (comment)

If you need dedicated virtual machine for it I can prepare but let's try what your team already have.

[lgkula]

I have the same issue. VS Code extension “Playwright Test for VSCode” v1.1.12 erroneously marks code with errors.
When a defective code is entered, the code is marked as containing a defect. After removing the code defect, the error information does not disappear and is shown all the time. It is necessary to reload the VS Code window or reload the tests in the extension.
Disabling the “Playwright Test for VSCode” extension solves the problem.