Convert WaitCommandTests to Hex1bTerminalAutomator imperative API by mitchdenny · Pull Request #15206 · microsoft/aspire

mitchdenny · 2026-03-13T08:57:09Z

Summary

Demonstrates the new Hex1bTerminalAutomator imperative testing API by converting WaitCommandTests from the declarative Hex1bTerminalInputSequenceBuilder pattern.

Motivation

The Hex1bTerminalInputSequenceBuilder (build-then-apply) pattern makes it difficult to debug failures in long E2E sequences — the exception doesn't pinpoint which step failed. The new Hex1bTerminalAutomator executes each step immediately and records a breadcrumb trail. On failure, Hex1bAutomationException includes:

Full step history with timings and source locations
Terminal snapshot at the point of failure
Caller file/line via [CallerFilePath]/[CallerLineNumber]

Changes

Package Updates

NuGet.config: Added hex1b-gh-packages feed for PR packages
Directory.Packages.props: Updated Hex1b packages to 0.116.0-pr.243 (from mitchdenny/hex1b#243)

New Files

tests/Shared/Hex1bAutomatorTestHelpers.cs — Automator extension methods:
- WaitForSuccessPromptAsync, WaitForAnyPromptAsync, WaitForErrorPromptAsync
- WaitForSuccessPromptFailFastAsync, DeclineAgentInitPromptAsync
- AspireNewAsync (full template wizard automation)
- Each step includes descriptive names like success prompt [3 OK] $ for diagnostics
tests/.../Helpers/CliE2EAutomatorHelpers.cs — Docker-specific extensions:
- PrepareDockerEnvironmentAsync, InstallAspireCliInDockerAsync

Converted Test

WaitCommandTests.cs — Rewritten from builder pattern to imperative automator

Before / After

Before (declarative):

sequenceBuilder.Type("aspire start")
    .Enter()
    .WaitUntil(s => waitForStarted.Search(s).Count > 0, TimeSpan.FromMinutes(3))
    .WaitForSuccessPrompt(counter);
// ... 20 more chained calls ...
var sequence = sequenceBuilder.Build();
await sequence.ApplyAsync(terminal, ct);

After (imperative):

await auto.TypeAsync("aspire start");
await auto.EnterAsync();
await auto.WaitUntilAsync(
    s => new CellPatternSearcher().Find("AppHost started successfully.").Search(s).Count > 0,
    timeout: TimeSpan.FromMinutes(3),
    description: "AppHost started successfully");
await auto.WaitForSuccessPromptAsync(counter);

Each await executes immediately. If it fails, the exception shows exactly which step, with the full history of what succeeded before it.

github-actions · 2026-03-13T08:57:18Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15206

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15206"

Copilot

Pull request overview

This PR converts the Aspire CLI E2E WaitCommandTests from the declarative Hex1bTerminalInputSequenceBuilder approach to the new imperative Hex1bTerminalAutomator API, and adds helper extensions to support common Aspire/Docker terminal flows with improved step-by-step diagnostics.

Changes:

Add new Hex1bTerminalAutomator extension helpers (Aspire prompt detection + aspire new wizard automation).
Add Docker-specific Hex1bTerminalAutomator helpers for environment prep and CLI installation.
Rewrite WaitCommandTests to execute steps imperatively via the automator.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/Shared/Hex1bAutomatorTestHelpers.cs	New shared automator extensions for prompt handling, agent init decline, and `aspire new` interactive flow.
tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2EAutomatorHelpers.cs	New Docker-focused automator extensions mirroring existing builder helpers.
tests/Aspire.Cli.EndToEnd.Tests/WaitCommandTests.cs	Converts the test to the imperative automator API for better failure diagnostics.
tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj	Links the new shared automator helper into the test project build.
NuGet.config	Adds a Hex1b GitHub Packages feed + package source mapping (and changes file encoding).
Directory.Packages.props	Updates Hex1b package versions to a PR build.

You can also share your feedback on Copilot code review. Take the survey.

tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2EAutomatorHelpers.cs

+/// <summary>
+/// Extension methods for <see cref="Hex1bTerminalAutomator"/> providing Docker E2E test helpers.
+/// These parallel the <see cref="Hex1bTerminalInputSequenceBuilder"/>-based methods in <see cref="CliE2ETestHelpers"/>.
+/// </summary>


NuGet.config

    <!--  Begin: Package sources from dotnet-runtime -->
    <!--  End: Package sources from dotnet-runtime -->
    <!--End: Package sources managed by Dependency Flow automation. Do not edit the sources above.-->
+    <add key="hex1b-gh-packages" value="https://nuget.pkg.github.com/mitchdenny/index.json" />


NuGet.config

@@ -1,4 +1,4 @@
-<?xml version="1.0" encoding="utf-8"?>
+<?xml version="1.0" encoding="utf-8"?>


Directory.Packages.props

+    <PackageVersion Include="Hex1b" Version="0.116.0-pr.243.23043055060.555d192.1" />
+    <PackageVersion Include="Hex1b.McpServer" Version="0.116.0-pr.243.23043055060.555d192.1" />
+    <PackageVersion Include="Hex1b.Tool" Version="0.116.0-pr.243.23043055060.555d192.1" />


IEvangelist

This is really cool. I do enjoy the fluidity of hex1b (reminds me of writing playwright tests). To that end, when writing playwright tests, I'd often find myself writing helpers just like you've done here - usually though I'd also have a look up class with const values for well-known DOM selectors, classes, or ids. We might want something similar and I think we might be able to use the existing localized strings from the CLI itself. I could be wrong though.

IEvangelist · 2026-03-13T13:09:58Z

tests/Shared/Hex1bAutomatorTestHelpers.cs

+                await auto.DownAsync();
+                await auto.DownAsync();
+                await auto.DownAsync();
+                await auto.DownAsync();


Is there a better (more reliable) way to find options in a selection list? This is essentially hardcoded and fragile, as it will break if the order changes, or more templates are added.

Along those lines, does it make sense to try to use the localized string lookup so that we're not hardcoding those raw values too?

Good callout! The hardcoded DownAsync() navigation is definitely fragile. A more robust approach would be a helper that knows how to find the top of the list, traverse it looking for the desired item, and detect when the end of the list has been reached — essentially a "search list" abstraction.

For the string matching side, we could use the CLI's resource strings in some places to avoid hardcoding raw text, though that won't cover everything (e.g., list item content that comes from templates rather than localized resources).

The automator API makes these kinds of improvements much easier to approach. They were technically possible before with the sequence builder, but the automator is a more natural fit for building these higher-level helpers. We'll iterate on this as we convert more tests.

Demonstrate the new Hex1bTerminalAutomator imperative testing API by converting WaitCommandTests from the Hex1bTerminalInputSequenceBuilder declarative pattern to the automator's step-by-step async pattern. Changes: - Update NuGet.config: Add hex1b-gh-packages feed for PR packages - Update Directory.Packages.props: Hex1b packages to 0.116.0-pr.243 (from mitchdenny/hex1b#243 which adds the automator API) - Add tests/Shared/Hex1bAutomatorTestHelpers.cs: Automator extension methods (WaitForSuccessPromptAsync, DeclineAgentInitPromptAsync, AspireNewAsync, etc.) with descriptive step names for diagnostics - Add tests/.../Helpers/CliE2EAutomatorHelpers.cs: Docker-specific automator extensions (PrepareDockerEnvironmentAsync, InstallAspireCliInDockerAsync) - Convert WaitCommandTests.cs: Replace builder pattern with imperative automator calls. Each step executes immediately with rich error context on failure (step history, terminal snapshot, caller location). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-14T02:13:49Z

The transient CI rerun workflow requested reruns for the following jobs after analyzing the failed attempt.
GitHub's job rerun API also reruns dependent jobs, so the retry is being tracked in the rerun attempt.
The job links below point to the failed attempt that matched the retry-safe transient failure rules.

Tests / Hosting.Redis / Hosting.Redis (windows-latest) - Ignored failed step 'Build test project' matched the feed network failure override allowlist.

github-actions · 2026-03-14T02:14:35Z

🎬 CLI E2E Test Recordings

The following terminal recordings are available for commit dfddfff:

Test	Recording
AddPackageInteractiveWhileAppHostRunningDetached	▶️ View Recording
AddPackageWhileAppHostRunningDetached	▶️ View Recording
AgentCommands_AllHelpOutputs_AreCorrect	▶️ View Recording
AgentInitCommand_DefaultSelection_InstallsSkillOnly	▶️ View Recording
AgentInitCommand_MigratesDeprecatedConfig	▶️ View Recording
AspireAddPackageVersionToDirectoryPackagesProps	▶️ View Recording
AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps	▶️ View Recording
Banner_DisplayedOnFirstRun	▶️ View Recording
Banner_DisplayedWithExplicitFlag	▶️ View Recording
CertificatesClean_RemovesCertificates	▶️ View Recording
CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate	▶️ View Recording
CertificatesTrust_WithUntrustedCert_TrustsCertificate	▶️ View Recording
CreateAndDeployToDockerCompose	▶️ View Recording
CreateAndDeployToDockerComposeInteractive	▶️ View Recording
CreateAndPublishToKubernetes	▶️ View Recording
CreateAndRunAspireStarterProject	▶️ View Recording
CreateAndRunAspireStarterProjectWithBundle	▶️ View Recording
CreateAndRunJsReactProject	▶️ View Recording
CreateAndRunPythonReactProject	▶️ View Recording
CreateAndRunTypeScriptStarterProject	▶️ View Recording
CreateEmptyAppHostProject	▶️ View Recording
CreateStartAndStopAspireProject	▶️ View Recording
CreateTypeScriptAppHostWithViteApp	▶️ View Recording
DescribeCommandResolvesReplicaNames	▶️ View Recording
DescribeCommandShowsRunningResources	▶️ View Recording
DetachFormatJsonProducesValidJson	▶️ View Recording
DoctorCommand_DetectsDeprecatedAgentConfig	❌ Upload failed
DoctorCommand_WithSslCertDir_ShowsTrusted	▶️ View Recording
DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	▶️ View Recording
LogsCommandShowsResourceLogs	❌ Upload failed
PsCommandListsRunningAppHost	❌ Upload failed
PsFormatJsonOutputsOnlyJsonToStdout	▶️ View Recording
RestoreGeneratesSdkFiles	❌ Upload failed
RunWithMissingAwaitShowsHelpfulError	▶️ View Recording
SecretCrudOnDotNetAppHost	▶️ View Recording
SecretCrudOnTypeScriptAppHost	❌ Upload failed
StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels	▶️ View Recording
StopAllAppHostsFromAppHostDirectory	❌ Upload failed
StopAllAppHostsFromUnrelatedDirectory	▶️ View Recording
StopNonInteractiveMultipleAppHostsShowsError	▶️ View Recording
StopNonInteractiveSingleAppHost	▶️ View Recording
StopWithNoRunningAppHostExitsSuccessfully	▶️ View Recording
TypeScriptAppHostWithProjectReferenceIntegration	▶️ View Recording

_{📹 Recordings uploaded automatically from CI run #23078628589}

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…5206) * Convert WaitCommandTests to Hex1bTerminalAutomator API Demonstrate the new Hex1bTerminalAutomator imperative testing API by converting WaitCommandTests from the Hex1bTerminalInputSequenceBuilder declarative pattern to the automator's step-by-step async pattern. Changes: - Update NuGet.config: Add hex1b-gh-packages feed for PR packages - Update Directory.Packages.props: Hex1b packages to 0.116.0-pr.243 (from mitchdenny/hex1b#243 which adds the automator API) - Add tests/Shared/Hex1bAutomatorTestHelpers.cs: Automator extension methods (WaitForSuccessPromptAsync, DeclineAgentInitPromptAsync, AspireNewAsync, etc.) with descriptive step names for diagnostics - Add tests/.../Helpers/CliE2EAutomatorHelpers.cs: Docker-specific automator extensions (PrepareDockerEnvironmentAsync, InstallAspireCliInDockerAsync) - Convert WaitCommandTests.cs: Replace builder pattern with imperative automator calls. Each step executes immediately with rich error context on failure (step history, terminal snapshot, caller location). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix unresolved XML doc cref for Hex1bTerminalInputSequenceBuilder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Mitch Denny <mitch@mitchdeny.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mitchdenny requested review from eerhardt and radical as code owners March 13, 2026 08:57

Copilot AI review requested due to automatic review settings March 13, 2026 08:57

Copilot started reviewing on behalf of mitchdenny March 13, 2026 08:58 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

IEvangelist approved these changes Mar 13, 2026

View reviewed changes

mitchdenny force-pushed the hex1b-automator branch from 1e262f8 to 73d32e1 Compare March 14, 2026 01:54

Fix unresolved XML doc cref for Hex1bTerminalInputSequenceBuilder

dfddfff

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mitchdenny enabled auto-merge (squash) March 14, 2026 03:09

mitchdenny merged commit 73082ee into release/13.2 Mar 14, 2026
494 of 497 checks passed

mitchdenny deleted the hex1b-automator branch March 14, 2026 04:08

dotnet-policy-service bot added this to the 13.2 milestone Mar 14, 2026

This was referenced Mar 14, 2026

Migrate E2E tests to Hex1bTerminalAutomator API and build reusable automation helpers #15241

Open

Migrate CLI and Deployment E2E tests to Hex1bTerminalAutomator API #15244

Merged

github-actions bot mentioned this pull request Mar 14, 2026

[13.2-burndown] Daily Burndown Report - March 14, 2026 #15245

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert WaitCommandTests to Hex1bTerminalAutomator imperative API#15206

Convert WaitCommandTests to Hex1bTerminalAutomator imperative API#15206
mitchdenny merged 2 commits intorelease/13.2from
hex1b-automator

mitchdenny commented Mar 13, 2026

Uh oh!

github-actions bot commented Mar 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

IEvangelist left a comment

Uh oh!

IEvangelist Mar 13, 2026

Uh oh!

mitchdenny Mar 14, 2026

Uh oh!

github-actions bot commented Mar 14, 2026

Uh oh!

github-actions bot commented Mar 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1,4 +1,4 @@
		<?xml version="1.0" encoding="utf-8"?>
		<?xml version="1.0" encoding="utf-8"?>

Conversation

mitchdenny commented Mar 13, 2026

Summary

Motivation

Changes

Package Updates

New Files

Converted Test

Before / After

Uh oh!

github-actions bot commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

IEvangelist left a comment

Choose a reason for hiding this comment

Uh oh!

IEvangelist Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

mitchdenny Mar 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 14, 2026

Uh oh!

github-actions bot commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎬 CLI E2E Test Recordings

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Mar 13, 2026 •

edited

Loading

github-actions bot commented Mar 14, 2026 •

edited

Loading