test: fix three deterministically-failing Outerloop tests - #18250
Merged
Ankit Jain (radical) merged 3 commits intoJun 16, 2026
Conversation
…of positional keystrokes Co-authored-by: radical <1472+radical@users.noreply.github.com>
The BrowserTokenDashboardServerWithHttpAndHttpsFixture bound the dashboard
frontend to "https://localhost:0;http://localhost:0". Kestrel rejects
dynamic-port (":0") binding on the "localhost" hostname:
System.InvalidOperationException: Dynamic port binding is not supported
when binding to localhost. You must either bind to 127.0.0.1:0 or
[::1]:0, or both.
So the fixture threw in InitializeAsync and
BrowserToken_QueryStringToken_HttpsThenHttp_WebKit_Success failed every
scheduled Outerloop run. Bind to 127.0.0.1:0 instead, matching the base
DashboardServerFixture. The WebKit test navigates to the resolved address
with IgnoreHTTPSErrors, so the loopback IP is fine.
Broken since the fixture was introduced in microsoft#17368.
Refs microsoft#18223
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ProxylessContainerCanBeReferenced and WithEndpointProxySupportDisablesProxies
([OuterloopTest], Docker-required) failed every scheduled Outerloop run on the
Hosting (8-core-ubuntu-latest) job while passing on Windows:
Assert.IsType() Failure: Value is null
Expected: typeof(int)
Actual: null
at ...AssertAllocatedProxylessPort(Service service) line 2319
at ...ProxylessContainerCanBeReferenced() line 1835
Both assert on the redisNoPort proxyless service's Status.EffectivePort.
PR microsoft#17924 made Aspire pre-assign the proxyless host port synchronously
(Service.Spec.Port) and deliberately excluded proxyless services from the
startup address-wait in DcpExecutor, since connection strings use the
Aspire-assigned Spec.Port immediately. DCP still echoes the bound port back
asynchronously via Status.EffectivePort. On a fast Linux agent the test reads
the service list before DCP populates EffectivePort, so the assertion sees
null; on Windows the update lands first, so it passes. The unit-test fake sets
EffectivePort = Spec.Port synchronously, hiding the race.
Fix is test-side: wait for the proxyless service to report a non-null
EffectivePort (via KubernetesHelper.GetResourceByNameAsync, the existing
watch-based waiter) before asserting, instead of reading a once-fetched
service list. No product change - the orchestrator behavior is correct.
Fixes microsoft#8773
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Jun 16, 2026
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 18250Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 18250" |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes three deterministically-failing Outerloop tests that caused the scheduled Outerloop Tests workflow to fail 7/7 runs per week (as reported in #18223). It consolidates the fixes from three individual PRs (#18248, #18246, #18249) into a single review/CI cycle.
Changes:
- Replace fragile positional keystrokes in the CLI E2E agent-init test with explicit
--skill-locations/--skillsCLI flags, making skill selection deterministic regardless of bundle contents. - Fix Kestrel dynamic-port binding failure in the dashboard Playwright fixture by using
127.0.0.1:0instead oflocalhost:0. - Fix a race condition in two Hosting tests by waiting for DCP to populate
Status.EffectivePorton proxyless services instead of asserting on a potentially-null value from a one-time list fetch.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
tests/Aspire.Cli.EndToEnd.Tests/NewWithAgentInitTests.cs |
Passes --skill-locations claudecode --skills playwright-cli to bypass interactive prompts; removes ~23 lines of fragile keystroke navigation |
tests/Aspire.Dashboard.Tests/Integration/Playwright/BrowserTokenAuthenticationTests.cs |
Changes frontend URL binding from localhost:0 to 127.0.0.1:0 with an explanatory comment |
tests/Aspire.Hosting.Tests/DistributedApplicationTests.cs |
Replaces GetEndpointService with WaitForAllocatedProxylessServiceAsync that uses the watch-based KubernetesHelper.GetResourceByNameAsync to wait for EffectivePort to be populated |
Ankit Jain (radical)
marked this pull request as ready for review
June 16, 2026 20:06
Ankit Jain (radical)
requested a review
from Mitch Denny (mitchdenny)
as a code owner
June 16, 2026 20:06
Ankit Jain (radical)
added
the
area-engineering-systems
Ankit Jain (radical)
requested review from
David Negstad (danegsta),
David Fowler (davidfowl) and
Jose Perez Rodriguez (joperezr)
June 16, 2026 20:07
Contributor
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
David Negstad (danegsta)
approved these changes
Jun 16, 2026
Ankit Jain (radical)
merged commit Jun 16, 2026
003ca64
into
microsoft:main
679 of 684 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidates three small
[OuterloopTest]fixes for tests that fail every scheduled Outerloop run (flagged in the workflow-health report #18223). Supersedes #18248, #18246, #18249 — none were approved, so combining them into one review/CI cycle. Each fix is an independent test-only change to a different assembly.1.
AspireNew_WithAgentInit_InstallsPlaywrightWithoutErrors(CLI e2e)Symptom. The test drove the
aspire agent initskill-selection menu with hardcoded positionalspace/downkeystrokes. When the skill bundle gained a new skill, the keystrokes landed onaspire-orchestrationinstead ofplaywright-cli, soplaywright-cliwas never installed and the laterplaywright-cli --versioncheck exited 127.Fix. Pass
--skill-locations claudecode --skills playwright-clitoaspire new; the flags are forwarded to the chainedagent initand bypass the interactive prompts, making selection deterministic regardless of bundle contents. All verification steps are preserved.tests/Aspire.Cli.EndToEnd.Tests/NewWithAgentInitTests.cs2.
BrowserToken_QueryStringToken_HttpsThenHttp_WebKit_Success(dashboard)Symptom. The class fixture threw in
InitializeAsyncbefore any test body ran:Root cause. The fixture bound the frontend to
https://localhost:0;http://localhost:0. Kestrel doesn't support dynamic-port (:0) binding on thelocalhosthostname — the constraint is hostname-based, not scheme-based. The baseDashboardServerFixturealready binds127.0.0.1:0.Fix. Bind to
https://127.0.0.1:0;http://127.0.0.1:0. The WebKit test navigates to the resolved address withIgnoreHTTPSErrors, so the loopback IP is fine.tests/Aspire.Dashboard.Tests/Integration/Playwright/BrowserTokenAuthenticationTests.cs3.
ProxylessContainerCanBeReferenced+WithEndpointProxySupportDisablesProxies(hosting)Symptom. Both
[OuterloopTest]+ Docker-required tests fail deterministically on the LinuxHostingjob (pass on Windows). They assert on a proxyless service'sStatus.EffectivePortimmediately after startup.Root cause. #17924 made Aspire pre-assign the proxyless host port synchronously (
Service.Spec.Port) and excluded proxyless services from the DCP startup address-wait, soStatus.EffectivePortcan still be null right after startup — a race.Fix. Wait for the proxyless service's
EffectivePortto be populated (viaKubernetesHelper.GetResourceByNameAsync) before asserting.tests/Aspire.Hosting.Tests/DistributedApplicationTests.csValidation
Outerloop Tests workflow dispatched against the combined change; the net diff is identical to what's under review here.
Fixes #8773
Refs #18223