Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .agents/routine-last-run.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,61 @@ so the next run can resume from the same line of work without re-discovering it.

**Handoff:** none. No work that belongs in `qyl-unit-tests`,
`qyl-integration-tests`, or `qyl-e2e-tests` was deferred from this run.

## qyl-e2e-tests 2026-05-17 02:45

**Outcome:** bootstrap PR opened.

First-ever run of `qyl-e2e-tests` on a workstation with the full toolchain
present (dotnet 10.0.300, Docker 29.4.0). The previous run (2026-05-16)
exited as a no-op because the remote container lacked `dotnet`.

Per the skill's bootstrap section, this run produced the **infrastructure-only
PR** (project + topology fixture + Nuke target + central-package pins). **No
scenario tests** were added; the next routine run picks up from here and adds
the first scenario.

**Changes shipped:**
- `tests/qyl.e2e.tests/qyl.e2e.tests.csproj` — new test project, `ANcpLua.NET.Sdk.Test`.
- `tests/qyl.e2e.tests/E2ECollection.cs` — `[CollectionDefinition("E2E", DisableParallelization = true)]`.
- `tests/qyl.e2e.tests/Topology/QylTopologyOptions.cs` — image tags + startup timeout.
- `tests/qyl.e2e.tests/Topology/QylTopologyFixture.cs` — programmatic Testcontainers
topology: bridge network → `qyl-collector:latest` → `qyl-mcp:latest`; WireMock
process-local; containers reach it via `host.docker.internal`.
`WithImagePullPolicy(_ => false)` enforces local-image-only.
- `tests/qyl.e2e.tests/Bootstrap/WireMockLlmSeamTests.cs` — `Category=E2EBootstrap`
smoke (no Docker) — proves the WireMock seam roundtrips a scripted
`/v1/chat/completions` and shows up in `LogEntries`.
- `eng/build/BuildTest.cs` — new `E2ETests` target depending on
`IDocker.DockerImageBuild`, filters `Category=E2E`. Default `Test` excludes
`Category=E2E` (bootstrap tests stay in).
- `qyl.slnx` — registered the project.
- `Version.props` + `Directory.Packages.props` — added `WireMock.Net 2.6.0` and
`Testcontainers 4.11.0`. Split-pinned `OpenTelemetry.Instrumentation.AspNetCore`
to 1.15.2 (forced by WireMock.Net 2.6.0 transitive); Http + Runtime stay at
the umbrella 1.15.1 (no 1.15.2 release exists for them).

**Verification:**
- `dotnet build qyl.slnx` — 0 errors, 1454 warnings (baseline 1393; ~61 new
warnings are all `MultipleGlobalAnalyzerKeys` from the dual-`.globalconfig`
worktree setup — benign, present on every worktree build).
- `dotnet test tests/qyl.e2e.tests --filter-trait Category=E2EBootstrap` — 3
consecutive runs, all green (~1s each, 0 flakes).
- `nuke E2ETests` target wired up but **not executed** this run (would require
rebuilding all four qyl Docker images — out of scope for the bootstrap PR).

**Overlap with `Smoke`:** `eng/smoke/run.sh` (Nuke target `Smoke`) is the
PRD #173 quality gate using real Ollama + real qyl Compose stack. E2E uses
**WireMock** for deterministic LLM stubbing. The two don't overlap — Smoke
covers "does the stack work against a real model"; E2E covers "does the
stack route data correctly given a known-bad/redacted LLM response".

**Gaps for the next run:** add the first real scenario. Highest-value candidates:
1. Agent submits chat → trace arrives at downstream sink with credentials redacted.
2. MCP HTTP session reconnect after transient collector failure.
3. Cost rollup updates after a single chat completion.

Pick exactly one. Add a sink container (e.g. an OTel collector configured to
write to a file volume) to `QylTopologyFixture` for scenario 1.

**Handoff:** bootstrap PR opened; next cycle picks up from here.
9 changes: 7 additions & 2 deletions Directory.Packages.props
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,10 @@
<PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="$(OpenTelemetryVersion)"/>
<!-- Instrumentation packages ship on a slower cadence than the SDK — pin
them to the shared instrumentation version (1.15.1) rather than the
SDK version (1.15.3). -->
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="$(OpenTelemetryAspNetCoreInstrumentationVersion)"/>
SDK version (1.15.3). AspNetCore is bumped to 1.15.2 because
WireMock.Net 2.6.0 (consumed by tests/qyl.e2e.tests) transitively
requires it; Http and Runtime have no 1.15.2 release, hence the split. -->
<PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.15.2"/>
<PackageVersion Include="OpenTelemetry.Instrumentation.Http" Version="$(OpenTelemetryAspNetCoreInstrumentationVersion)"/>
<PackageVersion Include="OpenTelemetry.Instrumentation.Runtime" Version="$(OpenTelemetryAspNetCoreInstrumentationVersion)"/>
Comment on lines +23 to 28
<!-- Microsoft.Extensions -->
Expand Down Expand Up @@ -108,6 +110,9 @@
<PackageVersion Include="YamlDotNet" Version="$(YamlDotNetVersion)"/>
<!-- Terminal UI -->
<PackageVersion Include="Spectre.Console" Version="$(SpectreConsoleVersion)"/>
<!-- E2E test topology (tests/qyl.e2e.tests only): deterministic LLM stub + Docker topology -->
<PackageVersion Include="WireMock.Net" Version="$(WireMockNetVersion)"/>
<PackageVersion Include="Testcontainers" Version="$(TestcontainersVersion)"/>
</ItemGroup>
<ItemGroup>
<PackageDownload Include="GitVersion.Tool" Version="[$(GitVersionToolVersion)]"/>
Expand Down
6 changes: 5 additions & 1 deletion Version.props
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@

<OpenAIVersion>2.10.0</OpenAIVersion>
<SystemClientModelVersion>1.11.0</SystemClientModelVersion>
<ModelContextProtocolVersion>1.2.0</ModelContextProtocolVersion>
<ModelContextProtocolVersion>1.3.0</ModelContextProtocolVersion>
<MicrosoftIdentityModelVersion>8.17.0</MicrosoftIdentityModelVersion>
<NukeVersion>10.1.0</NukeVersion>
<NetArchTestVersion>1.3.2</NetArchTestVersion>
Expand All @@ -57,5 +57,9 @@
<ReportGeneratorVersion>5.5.1</ReportGeneratorVersion>
<VersionizeVersion>2.5.0</VersionizeVersion>
<AwesomeAssertionsVersion>9.4.0</AwesomeAssertionsVersion>
<!-- E2E test topology: WireMock for deterministic LLM stubbing, Testcontainers for
Docker-driven topology spin-up. Used only by tests/qyl.e2e.tests. -->
<WireMockNetVersion>2.6.0</WireMockNetVersion>
<TestcontainersVersion>4.11.0</TestcontainersVersion>
</PropertyGroup>
</Project>
49 changes: 46 additions & 3 deletions eng/build/BuildTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,12 @@ interface IQylTest : ITest, IHazSourcePaths
AssertAtLeastOneTestExecuted("Functional");
});

Target E2ETests => d => d
.Description("Run end-to-end tests (full Docker topology against freshly built images)")
.DependsOn<ICompile>(static x => x.Compile)
.DependsOn<IDocker>(static x => x.DockerImageBuild)
.Executes(() => RunFilteredE2ETests());

Target TestSummary => d => d
.Unlisted()
.Description("Generate Markdown test summary from MTP TRX reports")
Expand All @@ -198,10 +204,12 @@ interface IQylTest : ITest, IHazSourcePaths
.ReportTrx($"{project.Name}.trx")
.IgnoreExitCode(8);

// Heavy/opt-in tests (Category=regen — shell out to Weaver, Category=integration etc.)
// are excluded from the default Test run; pass --IQylTest.TestFilter to include them.
// Heavy/opt-in tests (Category=regen — shell out to Weaver; Category=E2E — full
// Docker topology via DockerImageBuild) are excluded from the default Test run;
// pass --IQylTest.TestFilter to include them, or run the dedicated sub-target
// (E2ETests). E2EBootstrap-traited tests (no Docker) intentionally stay in.
if (TestFilter is { Length: > 0 } f) mtp.FilterQuery(f);
else mtp.FilterNotTrait("Category", "regen");
else mtp.FilterNotTrait("Category", "regen").FilterNotTrait("Category", "E2E");

if (StopOnFail == true) mtp.StopOnFail();
if (LiveOutput == true || IsLocalBuild) mtp.ShowLiveOutput();
Expand Down Expand Up @@ -236,6 +244,41 @@ sealed void RunFilteredTests(string namespaceFilter, string trxSuffix, bool need
}), completeOnFailure: true);
}

sealed void RunFilteredE2ETests()
{
EnsureTestcontainersConfigured();

var e2eProjects = TestProjects
.Where(static p => p.Name.Equals("qyl.e2e.tests", StringComparison.OrdinalIgnoreCase))
.ToArray();

if (e2eProjects.Length is 0)
{
Log.Warning("E2ETests: no qyl.e2e.tests project found; skipping");
return;
}

DotNetTasks.DotNetTest(s => s
.SetNoBuild(true)
.SetNoRestore(true)
.SetResultsDirectory(TestResultsDirectory)
.CombineWith(e2eProjects, (ss, project) =>
{
var mtp = MtpExtensions.Mtp()
.ReportTrx($"{project.Name}.E2E.trx")
.IgnoreExitCode(8)
.FilterTrait("Category", "E2E");

Comment on lines +267 to +271
if (StopOnFail == true) mtp.StopOnFail();
if (LiveOutput == true || IsLocalBuild) mtp.ShowLiveOutput();

var projectPath = project.Path ??
throw new InvalidOperationException($"Project '{project.Name}' has no path");
string[] args = ["--project", projectPath.ToString(), .. mtp.BuildArgs().Prepend("--")];
return ss.SetProcessAdditionalArguments(args);
}), completeOnFailure: true);
}

sealed void EnsureTestcontainersConfigured()
{
if (IsServerBuild)
Expand Down
1 change: 1 addition & 0 deletions qyl.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
<Folder Name="/tests/">
<Project Path="tests/qyl.collector.integration.tests/qyl.collector.integration.tests.csproj" />
<Project Path="tests/qyl.collector.tests/qyl.collector.tests.csproj" />
<Project Path="tests/qyl.e2e.tests/qyl.e2e.tests.csproj" />
<Project Path="tests/qyl.mcp.tests/qyl.mcp.tests.csproj" />
</Folder>

Expand Down
9 changes: 9 additions & 0 deletions services/qyl.mcp/Formatting/ErrorFormatter.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,19 @@ public static string FormatForLlm(Exception error, McpTransportMode transport) =
TaskCanceledException { InnerException: TimeoutException or null } =>
"**Timeout:** The collector did not respond in time. Retry or check if qyl collector is running.",
OperationCanceledException opEx => FormatOperationCancelled(opEx),
// ModelContextProtocol 1.3 throws IOException (incl. ClientTransportClosedException)
// for transport connect/closure failures that 1.2 wrapped as InvalidOperationException.
// This arm must precede InvalidOperationException — order matters in C# pattern switches.
IOException ioEx => FormatTransportError(ioEx, transport),
InvalidOperationException configEx => FormatConfigError(configEx, transport),
_ => FormatUnknown(error, transport)
Comment on lines +14 to 19
};

private static string FormatTransportError(IOException ex, McpTransportMode transport) =>
transport is McpTransportMode.Stdio
? $"**Connection Error**\n\n{ex.Message}\n\nCheck if the MCP server process is running."
: $"**Connection Error**\n\n{ex.Message}\n\nCheck the endpoint URL and network reachability.";

private static string FormatHttpError(HttpRequestException ex, McpTransportMode transport)
{
var (category, hint) = ex.StatusCode switch
Expand Down
48 changes: 48 additions & 0 deletions tests/qyl.e2e.tests/Bootstrap/WireMockLlmSeamTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
using System.Net.Http.Json;
using System.Text.Json;
using WireMock.RequestBuilders;
using WireMock.ResponseBuilders;
using WireMock.Server;

namespace Qyl.E2E.Tests.Bootstrap;

[Trait("Category", "E2EBootstrap")]
public sealed class WireMockLlmSeamTests
{
[Fact]
public async Task ScriptedChatCompletion_RoundtripsAndIsRecordedInLogEntries()
{
using var llm = WireMockServer.Start();

llm.Given(Request.Create().WithPath("/v1/chat/completions").UsingPost())
.RespondWith(Response.Create()
.WithStatusCode(200)
.WithHeader("Content-Type", "application/json")
.WithBodyAsJson(new
{
id = "chatcmpl-bootstrap",
choices = new[]
{
new { message = new { role = "assistant", content = "Bearer secret-token-12345" } },
},
}));

var ct = TestContext.Current.CancellationToken;
using var client = new HttpClient { BaseAddress = new Uri(llm.Url!) };
var request = new
{
model = "gpt-4o-mini",
messages = new[] { new { role = "user", content = "summarize this" } },
};
using var response = await client.PostAsJsonAsync("/v1/chat/completions", request, ct);

response.IsSuccessStatusCode.Should().BeTrue(
"the WireMock stub must respond 200 for the configured route");

using var payload = await JsonDocument.ParseAsync(await response.Content.ReadAsStreamAsync(ct), cancellationToken: ct);
payload.RootElement.GetProperty("id").GetString().Should().Be("chatcmpl-bootstrap");

llm.LogEntries.Should().ContainSingle(
"WireMock must record the single POST /v1/chat/completions for assertion replay");
}
}
7 changes: 7 additions & 0 deletions tests/qyl.e2e.tests/E2ECollection.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
namespace Qyl.E2E.Tests;

[CollectionDefinition(Name, DisableParallelization = true)]
public sealed class E2ECollection : ICollectionFixture<Topology.QylTopologyFixture>
{
public const string Name = "E2E";
}
Loading
Loading