Skip to content

Add AOT smoke test to CI#5251

Merged
tig merged 8 commits into
developfrom
aot-smoke-test-ci
May 8, 2026
Merged

Add AOT smoke test to CI#5251
tig merged 8 commits into
developfrom
aot-smoke-test-ci

Conversation

@tig
Copy link
Copy Markdown
Member

@tig tig commented May 7, 2026

Summary

Adds an AOT runtime regression test to CI that actually publishes and runs the NativeAot example binary.

Changes

  1. **\Examples/NativeAot/Program.cs**: Added --smoke-test\ flag that:

    • Creates all 25+ views (same AOT-sensitive codepaths as interactive mode)
    • Exercises ConfigurationManager, deep cloning, JSON serialization
    • Exits with success message instead of entering interactive event loop
  2. **.github/workflows/build-validation.yml**: Added \AOT Smoke Test\ step that runs the published AOT binary with --smoke-test\ after the existing \dotnet publish\ step.

Why

The existing CI only does \dotnet publish\ which catches compile-time AOT issues (trimming warnings, IL linker errors). But some AOT bugs only manifest at runtime — e.g., reflection-dependent Dictionary constructors, missing type metadata, serialization failures. This smoke test catches those.

Testing

  • Built and verified locally
  • The smoke test creates all views, verifies no crashes, and prints success message

Adds --smoke-test flag to NativeAot example that creates all views,
verifies no AOT crashes, and exits. CI now runs the published AOT
binary after dotnet publish to catch runtime AOT regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@BDisp
Copy link
Copy Markdown
Collaborator

BDisp commented May 7, 2026

The real exceptions in AOT only manifest themselves at runtime with the release configuration, using the NuGet package from the local_packages folder.

@tig
Copy link
Copy Markdown
Member Author

tig commented May 7, 2026

The real exceptions in AOT only manifest themselves at runtime with the release configuration, using the NuGet package from the local_packages folder.

Can you make a specific recommendation on what should change here?

Adds Examples/NativeAot/SmokeTest — a standalone project that references
Terminal.Gui via PackageReference (not ProjectReference) and AOT-publishes
+ runs it. This catches AOT runtime errors that only manifest when
consuming the NuGet package, where trimming decisions differ from a
direct project reference.

CI flow: Pack → AOT publish (ProjectRef) → run → AOT publish (NuGet) → run

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@BDisp
Copy link
Copy Markdown
Collaborator

BDisp commented May 7, 2026

Can you make a specific recommendation on what should change here?

I remember when I created the NativeAot project, there was a Debug configuration that used a reference to the Terminal.Gui project for immediate debugging purposes, and a Release configuration that used the latest NuGet package in the local_packages folder, which caught all compilation errors that weren't detectable through the Debug configuration. It seems this has been changed now, which doesn't give any certainty as to whether AOT is actually working correctly or not.

@BDisp
Copy link
Copy Markdown
Collaborator

BDisp commented May 7, 2026

Currently, I don't see anything related to trimmers in the NativeAOT project.

SmokeTest was inside Examples/NativeAot/ so its Program.cs got
auto-included in NativeAot.csproj (SDK default glob), causing
'more than one entry point' error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tig
Copy link
Copy Markdown
Member Author

tig commented May 7, 2026

@BDisp see the updated PR description. This test should now test end-to-end and catch almost all aot issues.

tig and others added 2 commits May 7, 2026 17:25
SmokeTest.csproj uses PackageReference Version='*' to pick up the
freshly-packed local nupkg. Central Package Management (NU1008)
blocks inline Version attributes, so opt this CI-only project out.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…oke test

Instead of just creating views and exiting, smoke tests now use
RunAsync with a 5-second CancellationToken timeout to exercise the
full app lifecycle: Init → Run → layout → draw → event loop → stop.

DisableRealDriverIO=1 env var enables headless execution in CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@BDisp
Copy link
Copy Markdown
Collaborator

BDisp commented May 7, 2026

@BDisp see the updated PR description. This test should now test end-to-end and catch almost all aot issues.

I see. It looks like it might work. Be aware that sometimes it compiled fine in the Release configuration, successfully publishing the executable, but when run it still threw exceptions that weren't caught at compile time. I don't know if CI will also address these cases.

@tig
Copy link
Copy Markdown
Member Author

tig commented May 7, 2026

@BDisp see the updated PR description. This test should now test end-to-end and catch almost all aot issues.

I see. It looks like it might work. Be aware that sometimes it compiled fine in the Release configuration, successfully publishing the executable, but when run it still threw exceptions that weren't caught at compile time. I don't know if CI will also address these cases.

Part of why the test intentionally uses a bunch of View sub-classes and enables CM.

tig and others added 2 commits May 7, 2026 17:35
In CI, dotnet pack produces a prerelease version (e.g. 2.1.0-PullRequest5251.135).
NuGet's Version='*' only matches stable releases, so it was resolving
an old stable package from nuget.org that didn't have RunAsync.
Version='*-*' matches prerelease versions too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The NuGet-based SmokeTest added complexity (CPM opt-out, prerelease
version resolution, packageSourceMapping) without catching different
bugs. The ILC runs the same analysis regardless of reference type.

The ProjectReference-based NativeAot --smoke-test already:
- Produces a real AOT native binary via dotnet publish
- Runs it with DisableRealDriverIO for headless CI
- Exercises full app lifecycle via RunAsync with timeout

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@BDisp
Copy link
Copy Markdown
Collaborator

BDisp commented May 7, 2026

Ideally, you should test with the most recently generated NuGet package because it may even contain some fixes that will be needed in the most recent release. At least try running the published file manually, because the published file is different from the file created in the Release configuration.
Well, I'm going to rest now.

@harder
Copy link
Copy Markdown
Collaborator

harder commented May 7, 2026

I ran this through my local Copilot agent where I was testing the previous AOT fixes. Here is some feedback.

  1. A Windows NativeAOT publish-and-run smoke test would be valuable, if possible.
  1. A downstream-package consumer smoke test, not just the in-repo project-reference example
  • Examples/NativeAot/NativeAot.csproj still uses a ProjectReference to Terminal.Gui, not a PackageReference to the just-packed ./local_packages nupkg.
  • The closest match to the real failure mode is pack Terminal.Gui, restore a tiny temp app from local_packages, publish AOT, run it.
  • This is the exact harness shape I used in the session workspace, and it avoids assuming the in-solution example exercises the same downstream package-consumer path.
  • Concrete recommendation: keep Examples/NativeAot if you want, but add a second CI smoke app that references the freshly packed Terminal.Gui package from ./local_packages and runs after dotnet pack.
  1. Explicit dictionary-clone coverage in --smoke-test
  • Right now the smoke test relies on “full app lifecycle + all views” to indirectly hit the sensitive config paths.
  • That’s broad, but it doesn’t explicitly assert the exact dictionary surfaces that caused the recent failures.
  • Useful addition: in --smoke-test, directly call DeepCloner.DeepClone (...) on the current config dictionaries that matter:
  • Color.Colors16
  • Application.DefaultKeyBindings
  • View.DefaultKeyBindings
  • View.ViewKeyBindings
  • ThemeManager.Themes
  • SchemeManager.Schemes
  • This gives a more targeted failure if a future regression lands in CreateDictionaryInstance again.

@tig
Copy link
Copy Markdown
Member Author

tig commented May 8, 2026

@copilot see #5251 (comment)

Do not do #2, but do the #1 and #3. We will create another PR for #2.

Copy link
Copy Markdown
Contributor

Copilot AI commented May 8, 2026

@tig I've opened a new pull request, #5255, to work on those changes. Once the pull request is ready, I'll request review from you.

@tig
Copy link
Copy Markdown
Member Author

tig commented May 8, 2026

FWIW: Check out https://github.com/gui-cs/clet#native-binaries-install

I do believe this is the first instance of a full TG app published as native AOT that really, truly, works!

@tig tig merged commit 8242d12 into develop May 8, 2026
11 checks passed
@tig tig deleted the aot-smoke-test-ci branch May 8, 2026 02:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants