Skip to content

Add browser automation resource commands - #16706

Draft
David Fowler (davidfowl) wants to merge 1 commit into
mainfrom
davidfowl/browser-improvements
Draft

Add browser automation resource commands#16706
David Fowler (davidfowl) wants to merge 1 commit into
mainfrom
davidfowl/browser-improvements

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented May 3, 2026

Copy link
Copy Markdown
Collaborator

Description

This builds on the browser diagnostics work and turns the browser child resource into a browser automation resource that can be driven from the dashboard/resource-command API. The goal is to make local web apps inspectable and operable by agents and humans without leaving the Aspire app model: open a tracked browser, inspect the current page, run common interactions, wait for UI readiness, capture screenshots, and stream browser diagnostics back into resource logs.

The user-visible behavior is:

  • WithBrowserAutomation() adds a browser child resource for an HTTP/HTTPS resource.
  • The browser resource can open a tracked Chromium-based browser session for the parent resource endpoint.
  • Resource commands can inspect page state, query selectors, navigate, click, fill, type, press keys, scroll, wait for URL/text/load/element/function conditions, manage cookies/storage/dialogs/tabs/downloads/uploads, run CDP escape-hatch commands, and capture screenshots.
  • Browser console, exception, log, and network events are forwarded into the browser resource logs.
  • Browser, profile, and user data mode can be configured via AppHost API, Aspire:Hosting:Browser configuration, or the dashboard configure command.
  • TypeScript AppHost code generation recognizes the new browser automation export.

Implementation details:

  • Renames the previous browser logs resource surface to browser automation while preserving legacy configuration fallback for earlier Aspire:Hosting:BrowserAutomation settings.
  • Adds a fixed browser-side helper script plus small generated JavaScript command expressions evaluated through CDP Runtime.evaluate.
  • Adds browser host/session management for owned pipe-backed Chromium processes and validated WebSocket endpoint adoption.
  • Adds CDP protocol framing/parsing, resource-command input validation callbacks, localized command strings, playground usage, and browser-focused tests.

User-facing usage

C# AppHost:

var frontend = builder.AddProject<Projects.MyFrontend>("frontend")
    .WithExternalHttpEndpoints()
    .WithBrowserAutomation();

Optional explicit browser configuration:

builder.AddProject<Projects.WebFrontend>("web")
    .WithExternalHttpEndpoints()
    .WithBrowserAutomation(
        browser: "msedge",
        profile: "Default",
        userDataMode: BrowserUserDataMode.Shared);

Configuration can also set global or resource-specific defaults:

{
  "Aspire": {
    "Hosting": {
      "Browser": {
        "Browser": "msedge",
        "UserDataMode": "Shared",
        "Profile": "Default",
        "frontend": {
          "Browser": "chrome"
        }
      }
    }
  }
}

Examples of resource-command payloads exposed by the browser child resource:

{ "command": "inspect-browser", "arguments": { "maxElements": 80, "maxTextLength": 8000 } }
{ "command": "click-browser", "arguments": { "selector": "#submit", "snapshotAfter": true } }
{ "command": "wait", "arguments": { "urlContains": "/dashboard", "timeoutMilliseconds": 3000 } }
{ "command": "capture-screenshot", "arguments": { "format": "jpeg", "quality": 80, "fullPage": true } }

Security considerations

This change needs security review because it introduces browser automation capabilities that spawn local Chromium-based browser processes, create persistent Aspire-managed browser user data directories, evaluate generated JavaScript in the tracked page through CDP, accept command input used for selectors/URLs/CDP parameters/file paths, and expose upload/download/screenshot commands.

Important boundaries and mitigations in this PR:

  • The tracked browser does not use the user's normal browser profile root; shared and isolated modes use Aspire-managed browser data paths.
  • The default owned-browser path uses a private CDP pipe instead of opening a browser debug port.
  • WebSocket endpoint adoption is treated as a stale/untrusted hint and validates schema, browser identity, user data root, process liveness, endpoint reachability, and profile compatibility before use.
  • Resource-command arguments are validated before execution for ranges, required conditional inputs, absolute navigation URLs, JSON object payloads, upload array shape, wait conditions, and screenshot quality.
  • JavaScript command arguments are JSON-literal encoded before embedding into generated script source.

Validation:

  • ./dotnet.sh build src/Aspire.Hosting.Browsers/Aspire.Hosting.Browsers.csproj --no-restore /p:SkipNativeBuild=true --nologo -v:minimal
  • ./dotnet.sh test --project tests/Aspire.Hosting.Browsers.Tests/Aspire.Hosting.Browsers.Tests.csproj --no-launch-profile -- --filter-method "*.WithBrowserAutomation_BrowserCommandsForwardArgumentsAndReturnJson" --filter-method "*.WithBrowserAutomation_BrowserCommandValidationRejectsInvalidArgumentsBeforeExecution" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
  • ./dotnet.sh test --project tests/Aspire.Hosting.CodeGeneration.TypeScript.Tests/Aspire.Hosting.CodeGeneration.TypeScript.Tests.csproj --no-launch-profile -- --filter-method "*.Scanner_BrowsersAssembly_WithBrowserAutomationCapability" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
  • git diff --check

Fixes # (issue)

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

@github-actions

github-actions Bot commented May 3, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 16706

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 16706"

@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch from 8db7f6f to 2a0481b Compare May 3, 2026 16:45
@davidfowl
David Fowler (davidfowl) changed the base branch from main to davidfowl/resource-command-features May 3, 2026 16:45
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch 2 times, most recently from e90667e to 2e0e8e2 Compare May 3, 2026 17:19
@github-actions

github-actions Bot commented May 3, 2026

Copy link
Copy Markdown
Contributor

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch from 2e0e8e2 to 8a8577c Compare May 3, 2026 20:32
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/resource-command-features branch from 8d54b4b to a79e590 Compare May 8, 2026 04:42
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch from 8a8577c to 40e5301 Compare May 9, 2026 02:00
@davidfowl
David Fowler (davidfowl) changed the base branch from davidfowl/resource-command-features to main May 9, 2026 02:01
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch 7 times, most recently from 1b8feda to f94de82 Compare May 9, 2026 03:54
@davidfowl David Fowler (davidfowl) changed the title Add browser automation resource commands Add browser resource commands May 9, 2026
Comment thread .agents/skills/aspire/references/agent-workflows.md Outdated
@davidfowl David Fowler (davidfowl) changed the title Add browser resource commands Add browser automation resource commands May 9, 2026
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch 3 times, most recently from 183cf29 to f200479 Compare May 9, 2026 06:45
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/browser-improvements branch from f200479 to 10588c1 Compare May 17, 2026 07:16
@davidfowl
David Fowler (davidfowl) marked this pull request as ready for review May 17, 2026 07:21
Copilot AI review requested due to automatic review settings May 17, 2026 07:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@davidfowl
David Fowler (davidfowl) marked this pull request as draft May 17, 2026 07:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@github-actions

Copy link
Copy Markdown
Contributor

CLI E2E Tests unknown — 85 passed, 0 failed, 1 unknown (commit 10588c1)

View all recordings
Status Test Recording
AddPackageInteractiveWhileAppHostRunningDetached ▶️ View recording
AddPackageWhileAppHostRunningDetached ▶️ View recording
AgentCommands_AllHelpOutputs_AreCorrect ▶️ View recording
AgentInitCommand_DefaultSelection_InstallsSkillOnly ▶️ View recording
AgentInitCommand_MigratesDeprecatedConfig ▶️ View recording
AspireAddPackageVersionToDirectoryPackagesProps ▶️ View recording
AspireInitSingleFileAppHostRunsViaDotnetRunAppHost ▶️ View recording
AspireInitWithExistingAppHostDirRecreatesMissingNuGetConfigAndPreservesFiles ▶️ View recording
AspireInitWithSolutionFileGeneratesAppHostThatBuildsAgainstChannelHive ▶️ View recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps ▶️ View recording
AspireUpdateRemovesOrphanAppHostPackageVersionWhenSdkAlreadyCurrent ▶️ View recording
Banner_DisplayedOnFirstRun ▶️ View recording
Banner_DisplayedWithExplicitFlag ▶️ View recording
Banner_NotDisplayedWithNoLogoFlag ▶️ View recording
CertificatesClean_RemovesCertificates ▶️ View recording
CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate ▶️ View recording
CertificatesTrust_WithUntrustedCert_TrustsCertificate ▶️ View recording
ConfigSetGet_CreatesNestedJsonFormat ▶️ View recording
CreateAndRunAspireStarterProject ▶️ View recording
CreateAndRunAspireStarterProjectWithBundle ▶️ View recording
CreateAndRunEmptyAppHostProject ▶️ View recording
CreateAndRunJavaEmptyAppHostProject ▶️ View recording
CreateAndRunJsReactProject ▶️ View recording
CreateAndRunPythonReactProject ▶️ View recording
CreateAndRunTypeScriptEmptyAppHostProject ▶️ View recording
CreateAndRunTypeScriptStarterProject ▶️ View recording
CreateJavaAppHostWithViteApp ▶️ View recording
CreateTypeScriptAppHostWithViteApp_UsesConfiguredToolchain ▶️ View recording
DashboardRunWithOtelTracesReturnsNoTraces ▶️ View recording
DeployK8sBasicApiService ▶️ View recording
DeployK8sWithExternalHelmChart ▶️ View recording
DeployK8sWithGarnet ▶️ View recording
DeployK8sWithMongoDB ▶️ View recording
DeployK8sWithMySql ▶️ View recording
DeployK8sWithPostgres ▶️ View recording
DeployK8sWithRabbitMQ ▶️ View recording
DeployK8sWithRedis ▶️ View recording
DeployK8sWithSqlServer ▶️ View recording
DeployK8sWithValkey ▶️ View recording
DeployTypeScriptAppToKubernetes ▶️ View recording
DescribeCommandResolvesReplicaNames ▶️ View recording
DescribeCommandShowsRunningResources ▶️ View recording
DetachFormatJsonProducesValidJson ▶️ View recording
DetachFormatJsonProducesValidJsonWhenRestartingExistingInstance ▶️ View recording
DoListStepsShowsPipelineSteps ▶️ View recording
DocsCommand_RendersInteractiveMarkdownFromLocalSource ▶️ View recording
DoctorCommand_DetectsDeprecatedAgentConfig ▶️ View recording
DoctorCommand_TypeScriptAppHostReportsMissingConfiguredToolchain ▶️ View recording
DoctorCommand_WithSslCertDir_ShowsTrusted ▶️ View recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted ▶️ View recording
GlobalMigration_HandlesCommentsAndTrailingCommas ▶️ View recording
GlobalMigration_HandlesMalformedLegacyJson ▶️ View recording
GlobalMigration_PreservesAllValueTypes ▶️ View recording
GlobalMigration_SkipsWhenNewConfigExists ▶️ View recording
GlobalSettings_MigratedFromLegacyFormat ▶️ View recording
InitTypeScriptAppHost_AugmentsExistingViteRepoAtRoot ▶️ View recording
InteractiveCSharpInitCreatesExpectedFiles ▶️ View recording
InvalidAppHostPathWithComments_IsHealedOnRun ▶️ View recording
LatestCliCanStartStableChannelAppHost ▶️ View recording
LatestCliCanStartStableChannelTypeScriptAppHost ▶️ View recording
LegacySettingsMigration_AdjustsRelativeAppHostPath ▶️ View recording
LogLevelTrace_ProducesTraceEntriesInCliLogFile ▶️ View recording
LogsCommandShowsResourceLogs ▶️ View recording
OtelLogsReturnsStructuredLogsFromStarterApp ▶️ View recording
OtelLogsReturnsStructuredLogsFromStarterAppIsolated ▶️ View recording
PsCommandListsRunningAppHost ▶️ View recording
PsFormatJsonOutputsOnlyJsonToStdout ▶️ View recording
PublishWithConfigureEnvFileUpdatesEnvOutput ▶️ View recording
PublishWithDockerComposeServiceCallbackSucceeds ▶️ View recording
PublishWithoutOutputPathUsesAppHostDirectoryDefault ▶️ View recording
ResourceCommand_FailsWhenInteractionServiceIsRequired ▶️ View recording
ResourceCommand_SetAndDeleteParameterUpdatesDescribeOutput ▶️ View recording
RestoreGeneratesSdkFiles ▶️ View recording
RestoreGeneratesSdkFiles_WithConfiguredToolchain ▶️ View recording
RestoreRefreshesGeneratedSdkAfterAddingIntegration ▶️ View recording
RestoreSupportsConfigOnlyHelperPackageAndCrossPackageTypes ▶️ View recording
RunFromParentDirectory_UsesExistingConfigNearAppHost ▶️ View recording
SecretCrudOnDotNetAppHost ▶️ View recording
SecretCrudOnTypeScriptAppHost ▶️ View recording
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels ▶️ View recording
StartAndWaitForTypeScriptSqlServerAppHostWithNativeAssets ▶️ View recording
StopAllAppHostsFromAppHostDirectory ▶️ View recording
StopNonInteractiveSingleAppHost ▶️ View recording
StopWithNoRunningAppHostExitsSuccessfully ▶️ View recording
UnAwaitedChainsCompileWithAutoResolvePromises ▶️ View recording
UpdateProjectChannelToStable_TypeScript_PicksUpStablePackages ▶️ View recording

📹 Recordings uploaded automatically from CI run #25984420370

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants