Replace Playwright MCP server with Playwright CLI installation by mitchdenny · Pull Request #14569 · dotnet/aspire

mitchdenny · 2026-02-19T05:25:59Z

Summary

Replaces the Playwright MCP server configuration in aspire agent init with a secure Playwright CLI installation workflow.

What changed

Instead of writing MCP server configuration JSON to each agent environment's config file (different formats for VS Code, Claude Code, Copilot CLI, and OpenCode), the new approach:

Resolves the @playwright/cli package version from npm registry
Downloads the package tarball via npm pack
Verifies supply chain integrity (SHA-512 SRI hash comparison)
Runs npm audit signatures for provenance verification
Installs globally from the verified tarball
Runs playwright-cli install --skills to generate skill files

New abstractions

INpmRunner/NpmRunner: npm CLI command runner (resolve, pack, audit, install)
IPlaywrightCliRunner/PlaywrightCliRunner: playwright-cli command runner
PlaywrightCliInstaller: orchestrates the full secure install flow with SRI hash verification

Impact

Removes ~400 lines of per-scanner MCP config writing code
Replaces 4 different JSON config formats with a single global CLI install
Adds supply chain security (integrity verification + signature audit)
Skill files are managed by the Playwright CLI itself

Fixes Replace playwright mcp with playwright cli? #14430
Related: [Feature] install --skills should also drop skill files in .agents directory microsoft/playwright#39317 (.agents/skills/ directory support in Playwright CLI)
Related: https://github.com/nicolo-ribaudo/tc39-proposal-structs/issues/11 (linked from the Playwright PR)

Testing

All 1282 existing CLI tests pass
Added 11 new unit tests for PlaywrightCliInstaller covering:
- npm resolve failure
- Already installed (same/newer version) skip logic
- Pack failure
- Integrity check pass/fail
- Global install failure
- Upgrade from older version
- SRI hash verification (matching, non-matching, wrong algorithm)

github-actions · 2026-02-19T05:26:07Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 14569

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 14569"

Copilot

Pull request overview

This pull request replaces the Playwright MCP server configuration approach with a secure Playwright CLI installation workflow in the aspire agent init command. Instead of writing MCP server configuration JSON to each agent environment's config file (different formats for VS Code, Claude Code, Copilot CLI, and OpenCode), the new approach installs the Playwright CLI globally with supply chain security verification.

Changes:

Introduces secure npm-based installation with SRI hash verification and signature auditing
Removes approximately 400 lines of MCP config writing code across 4 agent environment scanners
Adds new abstractions: INpmRunner/NpmRunner, IPlaywrightCliRunner/PlaywrightCliRunner, and PlaywrightCliInstaller

Reviewed changes

Copilot reviewed 30 out of 31 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`src/Aspire.Cli/Npm/INpmRunner.cs`	New interface for npm CLI operations (resolve, pack, audit, install)
`src/Aspire.Cli/Npm/NpmRunner.cs`	Implementation of npm CLI runner with process spawning and output parsing
`src/Aspire.Cli/Agents/Playwright/IPlaywrightCliRunner.cs`	New interface for playwright-cli operations (version check, skill installation)
`src/Aspire.Cli/Agents/Playwright/PlaywrightCliRunner.cs`	Implementation of playwright-cli runner
`src/Aspire.Cli/Agents/Playwright/PlaywrightCliInstaller.cs`	Orchestrator for secure CLI installation with SRI verification
`src/Aspire.Cli/Agents/VsCode/VsCodeAgentEnvironmentScanner.cs`	Removed Playwright MCP config writing, added PlaywrightCliInstaller dependency
`src/Aspire.Cli/Agents/CopilotCli/CopilotCliAgentEnvironmentScanner.cs`	Removed Playwright MCP config writing, added PlaywrightCliInstaller dependency
`src/Aspire.Cli/Agents/ClaudeCode/ClaudeCodeAgentEnvironmentScanner.cs`	Removed Playwright MCP config writing, added PlaywrightCliInstaller dependency
`src/Aspire.Cli/Agents/OpenCode/OpenCodeAgentEnvironmentScanner.cs`	Removed Playwright MCP config writing, added PlaywrightCliInstaller dependency
`src/Aspire.Cli/Agents/CommonAgentApplicators.cs`	Replaced callback-based Playwright config with direct CLI installer invocation
`src/Aspire.Cli/Agents/AgentEnvironmentScanContext.cs`	Removed Playwright callback list, simplified to single applicator tracking
`src/Aspire.Cli/Program.cs`	Registered new npm and Playwright CLI services
`src/Aspire.Cli/Resources/McpCommandStrings.resx`	Updated prompt text from "Pre-configure Playwright MCP server?" to "Install Playwright CLI for browser automation?"
`src/Aspire.Cli/Resources/xlf/*.xlf`	Marked translations as "needs-review-translation" for localization teams
`tests/Aspire.Cli.Tests/Agents/PlaywrightCliInstallerTests.cs`	Comprehensive test coverage (11 tests) for installer logic
`tests/Aspire.Cli.Tests/TestServices/FakePlaywrightServices.cs`	Test fakes for npm and Playwright CLI runners
`tests/Aspire.Cli.Tests/Agents/*ScannerTests.cs`	Updated scanner tests to reflect new CLI installer approach

Files not reviewed (1)

src/Aspire.Cli/Resources/McpCommandStrings.Designer.cs: Language not supported

github-actions · 2026-02-19T05:48:43Z

🎬 CLI E2E Test Recordings

The following terminal recordings are available for commit 31157ef:

Test	Recording
AddPackageInteractiveWhileAppHostRunningDetached	▶️ View Recording
AddPackageWhileAppHostRunningDetached	▶️ View Recording
AgentCommands_AllHelpOutputs_AreCorrect	▶️ View Recording
AgentInitCommand_MigratesDeprecatedConfig	▶️ View Recording
AgentInitCommand_WithMalformedMcpJson_ShowsErrorAndExitsNonZero	▶️ View Recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps	▶️ View Recording
Banner_DisplayedOnFirstRun	▶️ View Recording
Banner_DisplayedWithExplicitFlag	▶️ View Recording
CreateAndDeployToDockerCompose	▶️ View Recording
CreateAndDeployToDockerComposeInteractive	▶️ View Recording
CreateAndPublishToKubernetes	▶️ View Recording
CreateAndRunAspireStarterProject	▶️ View Recording
CreateAndRunAspireStarterProjectWithBundle	▶️ View Recording
CreateAndRunJsReactProject	▶️ View Recording
CreateAndRunPythonReactProject	▶️ View Recording
CreateAndRunTypeScriptStarterProject	▶️ View Recording
CreateEmptyAppHostProject	▶️ View Recording
CreateStartAndStopAspireProject	▶️ View Recording
CreateStartWaitAndStopAspireProject	▶️ View Recording
CreateTypeScriptAppHostWithViteApp	▶️ View Recording
DescribeCommandResolvesReplicaNames	▶️ View Recording
DescribeCommandShowsRunningResources	▶️ View Recording
DetachFormatJsonProducesValidJson	▶️ View Recording
DoctorCommand_DetectsDeprecatedAgentConfig	▶️ View Recording
DoctorCommand_WithSslCertDir_ShowsTrusted	▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	▶️ View Recording
LogsCommandShowsResourceLogs	▶️ View Recording
PsCommandListsRunningAppHost	▶️ View Recording
PsFormatJsonOutputsOnlyJsonToStdout	▶️ View Recording
SecretCrudOnDotNetAppHost	▶️ View Recording
SecretCrudOnTypeScriptAppHost	▶️ View Recording
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels	▶️ View Recording
StopAllAppHostsFromAppHostDirectory	▶️ View Recording
StopAllAppHostsFromUnrelatedDirectory	▶️ View Recording
StopNonInteractiveMultipleAppHostsShowsError	▶️ View Recording
StopNonInteractiveSingleAppHost	▶️ View Recording
StopWithNoRunningAppHostExitsSuccessfully	▶️ View Recording

_{📹 Recordings uploaded automatically from CI run #22607490916}

Replace the Playwright MCP server configuration in `aspire agent init` with a secure Playwright CLI installation workflow. Instead of writing MCP server configuration to each agent environment's config file, the new approach: - Resolves the @playwright/cli package version from npm registry - Downloads the package tarball via `npm pack` - Verifies supply chain integrity (SHA-512 SRI hash comparison) - Runs `npm audit signatures` for provenance verification - Installs globally from the verified tarball - Runs `playwright-cli install --skills` to generate skill files New abstractions: - INpmRunner/NpmRunner: npm CLI command runner (resolve, pack, audit, install) - IPlaywrightCliRunner/PlaywrightCliRunner: playwright-cli command runner - PlaywrightCliInstaller: orchestrates the secure install flow This removes ~400 lines of per-scanner MCP config writing code (different JSON formats for VS Code, Claude Code, Copilot CLI, and OpenCode) and replaces it with a single global CLI install. Fixes #14430 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mitchdenny · 2026-02-24T03:17:25Z

Just tested this out on my machine. I didn't have Playright CLi installed initially and it worked just fine. Chose to use the CLI skill (even though it is in the .claude folder.

src/Aspire.Cli/Agents/Playwright/PlaywrightCliInstaller.cs

- Add INpmProvenanceChecker/NpmProvenanceChecker for SLSA attestation verification - Return rich ProvenanceVerificationResult with gate-specific outcome enum - Fix AuditSignaturesAsync with temp-project approach for global tools - Add disablePlaywrightCliPackageValidation break-glass config option - Add security design document (docs/specs/safe-npm-tool-install.md) - Verify SRI integrity, Sigstore attestations, and source repository provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Change version range from exact 0.1.1 to >=0.1.1 for future versions - Add playwrightCliVersion config override for pinning specific versions - Verify workflow path (.github/workflows/publish.yml) in provenance - Verify SLSA build type (GitHub Actions) to confirm OIDC token issuer - Add BuildType to NpmProvenanceData, WorkflowMismatch and BuildTypeMismatch outcomes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Verifies the full lifecycle: project creation, aspire agent init with Claude Code environment, Playwright CLI installation with npm provenance verification, and skill file generation. Marked as OuterloopTest since it requires npm and network access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Wrap the installation work in IInteractionService.ShowStatusAsync to display a spinner with 'Installing Playwright CLI...' status text while the npm operations are in progress. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

After playwright-cli install --skills creates files in .claude/skills/, the installer now mirrors the playwright-cli skill directory to all other detected agent environment skill directories (.github/skills/, .opencode/skill/, etc.) so every configured environment has identical skill files. Stale files in target directories are also removed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

DamianEdwards · 2026-02-27T01:53:57Z

docs/specs/safe-npm-tool-install.md

+2. Find the attestation with `predicateType: "https://slsa.dev/provenance/v1"` (SLSA Build L3 provenance)
+3. Base64-decode the DSSE envelope payload to extract the in-toto statement
+4. Parse `predicate.buildDefinition.externalParameters.workflow.repository`
+5. Verify it equals `https://github.com/microsoft/playwright-cli`


@mitchdenny update this to include the other metadata we verify, e.g. tag-based release, workflow name, etc?

DamianEdwards

This looks really good to me! We should likely get a tick from @blowdart or @GrabYourPitchforks.

- Step 4 now documents all three provenance gates: source repository, workflow path, and build type (with OIDC issuer implication) - Added table of verified fields with expected values - Updated implementation constants to include new fields - Added configuration section documenting break-glass keys - Updated verification diagram with workflow/build type checks - Step 7 now documents skill file mirroring across environments - Future improvements reflects experimental Sigstore branch status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add Gate 6 to provenance verification: check that the workflow ref (git tag) in the SLSA attestation matches refs/tags/v{version}. This ensures the build was triggered from the expected release tag, not an arbitrary branch or commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

src/Aspire.Cli/Npm/NpmProvenanceChecker.cs

blowdart · 2026-02-27T03:10:43Z

As I've already gone through it, @GrabYourPitchforks should look at the Threat Model when he has a moment.

Make the workflow ref validation configurable per-package by accepting a Func<WorkflowRefInfo, bool> callback instead of hardcoding the refs/tags/v{version} format. The ref is parsed into a WorkflowRefInfo record (Raw, Kind, Name) and the caller decides what's valid. PlaywrightCliInstaller validates Kind=tags and Name=v{version}. Other packages can use different tag conventions without modifying the provenance checker. Addresses review feedback from DamianEdwards. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use the Sigstore and Tuf NuGet packages (0.2.0) to cryptographically verify npm attestation bundles in-process, replacing the previous approach of shelling out to 'npm audit signatures'. - Add SigstoreNpmProvenanceChecker implementing INpmProvenanceChecker using SigstoreVerifier with CertificateIdentity.ForGitHubActions - Remove the npm audit signatures step from PlaywrightCliInstaller - Keep existing NpmProvenanceChecker but no longer register in DI - Add optional sriIntegrity parameter to INpmProvenanceChecker for digest-based bundle verification - Update safe-npm-tool-install.md spec to reflect new verification flow - Temporarily add nuget.org to NuGet.config for Sigstore/Tuf packages Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Break the monolithic VerifyProvenanceAsync method into focused methods: - FetchAttestationJsonAsync: fetches attestation JSON from npm registry - ParseAttestation: parses JSON in a single pass, extracting both the Sigstore bundle and provenance data (eliminates duplicate JSON parsing) - ParseProvenanceFromStatement: extracts provenance fields from in-toto statement - VerifySigstoreBundleAsync: cryptographic Sigstore verification - VerifyProvenanceFields: field-level provenance checks (source repo, workflow, build type, workflow ref) Removes dependency on NpmProvenanceChecker.ParseProvenance() which was re-parsing the same JSON and iterating attestations a second time. Adds NpmAttestationParseResult type to carry both bundle and provenance data from a single parse pass. Adds comprehensive unit tests for ParseAttestation, ParseProvenanceFromStatement, and VerifyProvenanceFields covering success and failure cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

libsodium is a transitive dependency of NSec.Cryptography (used by Sigstore) and needs to be mapped to the nuget-org source for CI restore to succeed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mitchdenny · 2026-03-03T09:24:17Z

Merging so that we can layer some over things on top. If @GrabYourPitchforks wants us to revert this change over any concerns, then we can disable/remove the code and just ship a static SKILL.md file (and all its references).

The Sigstore and Tuf NuGet packages (and their transitive dependency NSec.Cryptography) were added to Aspire.Cli in #14569 but the corresponding FileSignInfo entries in eng/Signing.props were not added. This causes SIGN004 errors in the internal build because the Arcade SDK detects 3rd-party DLLs being signed with the Microsoft certificate. Add FileSignInfo entries with CertificateName="3PartySHA2" for: - NSec.Cryptography.dll (transitive dep, © Klaus Hartke) - Sigstore.dll (direct dep) - Tuf.dll (direct dep) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

#14915) The Sigstore and Tuf NuGet packages (and their transitive dependency NSec.Cryptography) were added to Aspire.Cli in #14569 but the corresponding FileSignInfo entries in eng/Signing.props were not added. This causes SIGN004 errors in the internal build because the Arcade SDK detects 3rd-party DLLs being signed with the Microsoft certificate. Add FileSignInfo entries with CertificateName="3PartySHA2" for: - NSec.Cryptography.dll (transitive dep, © Klaus Hartke) - Sigstore.dll (direct dep) - Tuf.dll (direct dep) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 19, 2026 05:26

Copilot started reviewing on behalf of mitchdenny February 19, 2026 05:26 View session

Copilot AI reviewed Feb 19, 2026

View reviewed changes

mitchdenny linked an issue Feb 19, 2026 that may be closed by this pull request

Replace playwright mcp with playwright cli? #14430

Closed

1 task

github-actions bot mentioned this pull request Feb 19, 2026

[13.2-burndown] Daily Burndown Report - February 19, 2026 📊 #14581

Closed

mitchdenny marked this pull request as draft February 20, 2026 05:04

This was referenced Feb 20, 2026

[13.2-burndown] Daily Burndown Report - February 20, 2026 📊 #14594

Closed

[13.2-burndown] Daily Burndown Report - February 21, 2026 📊 #14605

Closed

[13.2-burndown] Daily Burndown Report - February 22, 2026 #14608

Closed

mitchdenny force-pushed the playwright-cli-skill branch from e6abec9 to f87d6d8 Compare February 23, 2026 21:56

mitchdenny changed the base branch from release/13.2 to main February 23, 2026 22:02

mitchdenny changed the base branch from main to release/13.2 February 23, 2026 22:40

mitchdenny force-pushed the playwright-cli-skill branch 3 times, most recently from 4218037 to 1965253 Compare February 23, 2026 23:19

mitchdenny force-pushed the playwright-cli-skill branch from 1965253 to 930698a Compare February 24, 2026 00:07

Pin Playwright CLI version range to 0.1.1

6dffb0e

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mitchdenny marked this pull request as ready for review February 24, 2026 03:17

github-actions bot mentioned this pull request Feb 24, 2026

[pull requests] Review required #13834

Open

DamianEdwards reviewed Feb 24, 2026

View reviewed changes

src/Aspire.Cli/Agents/Playwright/PlaywrightCliInstaller.cs Show resolved Hide resolved

github-actions bot mentioned this pull request Feb 25, 2026

[13.2-burndown] Daily Burndown Report - February 25, 2026 #14682

Closed

Mitch Denny and others added 5 commits February 26, 2026 15:27

Fix markdownlint: add language to fenced code block

abc4b83

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add tests for version pinning and default version range

ce6c598

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions bot mentioned this pull request Feb 26, 2026

[13.2-burndown] Daily Burndown Report - February 26, 2026 #14747

Closed

DamianEdwards reviewed Feb 27, 2026

View reviewed changes

DamianEdwards approved these changes Feb 27, 2026

View reviewed changes

Mitch Denny and others added 2 commits February 27, 2026 13:22

DamianEdwards reviewed Feb 27, 2026

View reviewed changes

src/Aspire.Cli/Npm/NpmProvenanceChecker.cs Outdated Show resolved Hide resolved

blowdart requested a review from GrabYourPitchforks February 27, 2026 03:10

mitchdenny added this to the 13.2 milestone Mar 3, 2026

Mitch Denny and others added 3 commits March 3, 2026 14:13

mitchdenny merged commit 5a3c52a into release/13.2 Mar 3, 2026
347 checks passed

mitchdenny deleted the playwright-cli-skill branch March 3, 2026 09:24

dotnet-policy-service bot modified the milestone: 13.2 Mar 3, 2026

github-actions bot mentioned this pull request Mar 3, 2026

[13.2-burndown] Daily Burndown Report - March 3, 2026 #14879

Closed

radical mentioned this pull request Mar 3, 2026

Add 3rd-party signing entries for Sigstore, Tuf, and NSec.Cryptography #14915

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace Playwright MCP server with Playwright CLI installation#14569

Replace Playwright MCP server with Playwright CLI installation#14569
mitchdenny merged 15 commits intorelease/13.2from
playwright-cli-skill

mitchdenny commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions bot commented Feb 19, 2026 •

edited

Loading

Uh oh!

mitchdenny commented Feb 24, 2026

Uh oh!

Uh oh!

DamianEdwards Feb 27, 2026

Uh oh!

DamianEdwards left a comment

Uh oh!

Uh oh!

blowdart commented Feb 27, 2026

Uh oh!

mitchdenny commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mitchdenny commented Feb 19, 2026

Summary

What changed

New abstractions

Impact

Related

Testing

Uh oh!

github-actions bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions bot commented Feb 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎬 CLI E2E Test Recordings

Uh oh!

mitchdenny commented Feb 24, 2026

Uh oh!

Uh oh!

DamianEdwards Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

DamianEdwards left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blowdart commented Feb 27, 2026

Uh oh!

mitchdenny commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Feb 19, 2026 •

edited

Loading

github-actions bot commented Feb 19, 2026 •

edited

Loading