Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 195 additions & 0 deletions .claude/skills/aot-guru/skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -598,6 +598,201 @@ Location: `templates/aot-workaround.md`

## BDD Testing for AOT

### Automated AOT Test Suite

morphir-dotnet has a comprehensive BDD test suite for AOT and trimming validation located at:
- `tests/Morphir.E2E.Tests/Features/AOT/AssemblyTrimming.feature` (11 scenarios)
- `tests/Morphir.E2E.Tests/Features/AOT/NativeAOTCompilation.feature` (9 scenarios)

**Step Definitions:**
- `AssemblyTrimmingSteps.cs` - Implements all 11 trimming scenarios
- `NativeAOTCompilationSteps.cs` - Implements all 9 AOT compilation scenarios

**Documentation:**
- `tests/Morphir.E2E.Tests/Features/AOT/README.md` - Complete usage guide

### When to Run AOT Tests

**Run AOT tests when:**
1. **Before releasing** trimmed or AOT executables
2. **After dependency updates** that might affect AOT compatibility
3. **After significant CLI changes** that could impact build configuration
4. **When investigating** trimming warnings or size regressions
5. **To validate** new features work with trimming/AOT

**DO NOT run in regular CI** - These tests are long-running (45-90 minutes total) and should only be executed manually for release preparation.

### How to Run AOT Tests

#### Manual Workflow (Recommended)

The AOT tests run in a dedicated GitHub Actions workflow:

1. Go to **Actions** → **Manual AOT Testing**
2. Click **Run workflow**
3. Select inputs:
- **Configuration**: Release or Debug
- **Platform**: linux-x64, osx-arm64, win-x64, linux-arm64, osx-x64
- **Test Suite**: both, trimming, or aot-compilation
- **Test Version**: Version to use for executables (e.g., 0.0.0-test)
4. Click **Run workflow**

The workflow will:
- Build required executables (trimmed, untrimmed, AOT)
- Run selected test suite with platform-specific validations
- Upload artifacts on failure for debugging
- Complete in approximately 45-90 minutes

#### Local Execution

To run AOT tests locally:

```bash
# 1. Build executables first
./build.sh --target PublishSingleFile --rid linux-x64
./build.sh --target PublishSingleFileUntrimmed --rid linux-x64 # For baseline comparisons
./build.sh --target PublishExecutable --rid linux-x64 # For AOT tests

# 2. Run trimming tests
cd tests/Morphir.E2E.Tests
MORPHIR_EXECUTABLE_TYPE=trimmed dotnet run -- --treenode-filter "*/Trimming*"

# 3. Run AOT tests
MORPHIR_EXECUTABLE_TYPE=aot dotnet run -- --treenode-filter "*/AOT*"

# 4. Run both test suites
INCLUDE_MANUAL_TESTS=true dotnet run
```

### Test Scenarios Covered

#### Assembly Trimming (11 scenarios)

1. **Trimming with link mode** - Validates link mode trimming effectiveness
2. **Preserving types with DynamicDependency** - Ensures attributes preserve types
3. **Trimming warnings detection** - Validates trim analyzers detect issues
4. **JSON serialization preservation** - Tests source-generated serialization
5. **Embedded resources in trimmed build** - Validates resource preservation
6. **Trimmed build size comparison** - Compares trimmed vs untrimmed sizes
7. **Trimming with third-party dependencies** - Tests dependency compatibility
8. **Feature switches for size reduction** - Validates feature switch effectiveness
9. **Trimmer root descriptors** - Tests custom preservation rules
10. **Invariant globalization size savings** - Measures globalization impact
11. Additional trimming validation scenarios

#### Native AOT Compilation (9 scenarios)

1. **Successful AOT compilation** - Validates basic AOT build
2. **AOT with size optimizations** - Tests size optimization flags
3. **AOT executable runs correctly** - Validates runtime behavior
4. **All CLI commands work in AOT** - Tests command compatibility
5. **JSON output works in AOT** - Validates source-generated serialization
6. **Detecting reflection usage during build** - Checks IL2XXX warnings
7. **Size target for minimal CLI** - Validates minimal build size (5-8 MB)
8. **Size target for feature-rich CLI** - Validates full build size (8-12 MB)
9. **Cross-platform AOT builds** - Tests linux-x64, win-x64, osx-x64, ARM variants
10. **AOT build performance** - Measures startup time and memory usage

### Test Implementation Details

**Build Strategy:**
- Tests invoke `dotnet publish` with scenario-specific MSBuild properties
- Each scenario builds executables in isolated `artifacts/test-builds/{guid}` directories
- Native AOT tests reuse existing artifacts from `artifacts/executables/` when available
- Cross-platform RID detection handles platform-specific differences

**Validations:**
- Exit code checks for build success
- File size comparisons and range validations
- Build warning detection (IL2026, IL2060, IL2070, etc.)
- Runtime command execution (--version, --help, ir verify)
- JSON output validation using JsonDocument parsing
- Platform-specific size assertions

**Duration:**
- Assembly Trimming tests: ~15-30 minutes (builds trimmed + untrimmed executables)
- Native AOT Compilation tests: ~30-60 minutes (AOT compilation is slower)
- Total for both suites: ~45-90 minutes

### Recommending Additional Tests

When recommending new AOT tests or changes:

**Consider adding tests for:**
1. **New CLI commands** - Ensure they work with trimming/AOT
2. **New dependencies** - Validate AOT compatibility
3. **Size-impacting features** - Track size regressions
4. **Reflection-heavy code** - Validate preservation mechanisms
5. **Platform-specific behavior** - Test on all target platforms

**Test patterns to follow:**
- Use Given/When/Then Gherkin syntax
- Focus on build-time validation (step definitions build executables)
- Include size assertions for size-sensitive features
- Test both success and failure paths
- Validate platform-specific behavior

**Example new scenario:**
```gherkin
Scenario: New feature works with trimming
Given a morphir-dotnet CLI with new feature enabled
And PublishTrimmed is enabled
When I build the application
Then the build should succeed without warnings
And the new feature should work correctly
And the size should not increase by more than 500 KB
```

### Modifying Test Execution

**To modify test execution workflow:**
1. Update `.github/workflows/manual-aot-test.yml` for workflow changes
2. Update `scripts/run-e2e-tests.cs` for filtering logic
3. Update step definitions in `tests/Morphir.E2E.Tests/Features/AOT/*Steps.cs`
4. Update `tests/Morphir.E2E.Tests/Features/AOT/README.md` documentation

**To add platform support:**
1. Add platform to workflow inputs in `manual-aot-test.yml`
2. Update runs-on mapping for new platform
3. Test locally on the platform first
4. Document platform-specific size targets

**To add new scenarios:**
1. Add Gherkin scenario to appropriate `.feature` file
2. Implement step definitions in corresponding `*Steps.cs` file
3. Test locally with `dotnet run -- --treenode-filter "*/Scenario Name*"`
4. Update README with new scenario documentation

### Troubleshooting AOT Tests

**Common test failures:**

1. **"Executable not found"**
- Ensure build succeeded (check `BuildExitCode` in scenario context)
- Check artifacts directory structure
- Verify RID matches platform

2. **"Size exceeds threshold"**
- Review recent changes for size regressions
- Check if new dependencies were added
- Run size analysis: `ls -lh artifacts/*/morphir*`

3. **"IL2XXX warnings present"**
- Expected for reflection usage scenarios
- Validate warnings are documented
- Check if source generators are missing

4. **"Runtime command failed"**
- Check stderr output for errors
- Validate executable has correct permissions
- Test executable manually: `./artifacts/.../morphir --version`

**Debug techniques:**
- Check uploaded artifacts in failed workflow runs
- Run tests locally with verbose output
- Inspect scenario context values in step definitions
- Review build logs in `artifacts/test-builds/*/build.log`

### Feature: Native AOT Compilation

```gherkin
Expand Down
129 changes: 129 additions & 0 deletions .github/workflows/manual-aot-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
name: "Manual AOT Testing"

on:
workflow_dispatch:
inputs:
configuration:
description: "Configuration to build"
required: true
default: "Release"
type: choice
options:
- Release
- Debug
test-version:
description: "Test version for executables"
required: true
default: "0.0.0-test"
type: string
platform:
description: "Platform to test"
required: true
default: "linux-x64"
type: choice
options:
- linux-x64
- linux-arm64
- osx-arm64
- osx-x64
- win-x64
test-suite:
description: "AOT test suite to run"
required: true
default: "both"
type: choice
options:
- both
- trimming
- aot-compilation

env:
CONFIGURATION: ${{ github.event.inputs.configuration }}
TEST_VERSION: ${{ github.event.inputs.test-version }}

jobs:
aot-tests:
timeout-minutes: 90
runs-on: ${{ (github.event.inputs.platform == 'linux-x64' && 'ubuntu-latest') || (github.event.inputs.platform == 'linux-arm64' && 'ubuntu-24.04-arm') || (github.event.inputs.platform == 'osx-x64' && 'macos-15-intel') || (github.event.inputs.platform == 'osx-arm64' && 'macos-latest') || (github.event.inputs.platform == 'win-x64' && 'windows-latest') || 'ubuntu-latest' }}
steps:
- name: Fetch Source Code
uses: actions/checkout@v6.0.1
with:
fetch-depth: 0

- name: Setup .NET SDK
uses: actions/setup-dotnet@v5
with:
dotnet-version: "10.0.x"

- name: Cache Nuget Packages
uses: actions/cache@v4
with:
path: ~/.nuget/packages
key: ${{ runner.os }}-nuget-${{ hashFiles('**/packages.lock.json') }}
restore-keys: |
${{ runner.os }}-nuget

- name: Restore dependencies
shell: bash
run: ./build.sh --target Restore

- name: Build E2E test project
shell: bash
run: ./build.sh --target BuildE2ETests --configuration ${{ env.CONFIGURATION }}

- name: Build trimmed single-file executable
if: github.event.inputs.test-suite == 'both' || github.event.inputs.test-suite == 'trimming'
shell: bash
run: ./build.sh --target PublishSingleFile --rid ${{ github.event.inputs.platform }} --configuration ${{ env.CONFIGURATION }} --version ${{ env.TEST_VERSION }}

- name: Build untrimmed single-file executable (for baseline comparison)
if: github.event.inputs.test-suite == 'both' || github.event.inputs.test-suite == 'trimming'
shell: bash
run: ./build.sh --target PublishSingleFileUntrimmed --rid ${{ github.event.inputs.platform }} --configuration ${{ env.CONFIGURATION }} --version ${{ env.TEST_VERSION }}

- name: Build AOT executable
if: github.event.inputs.test-suite == 'both' || github.event.inputs.test-suite == 'aot-compilation'
shell: bash
run: ./build.sh --target PublishExecutable --rid ${{ github.event.inputs.platform }} --configuration ${{ env.CONFIGURATION }} --version ${{ env.TEST_VERSION }}

- name: Run Assembly Trimming Tests
if: github.event.inputs.test-suite == 'both' || github.event.inputs.test-suite == 'trimming'
shell: bash
env:
MORPHIR_EXECUTABLE_TYPE: trimmed
INCLUDE_MANUAL_TESTS: "true"
run: |
cd tests/Morphir.E2E.Tests
dotnet run --no-build --configuration ${{ env.CONFIGURATION }} -- --treenode-filter "*/Trimming*"

- name: Run Native AOT Compilation Tests
if: github.event.inputs.test-suite == 'both' || github.event.inputs.test-suite == 'aot-compilation'
shell: bash
env:
MORPHIR_EXECUTABLE_TYPE: aot
INCLUDE_MANUAL_TESTS: "true"
run: |
cd tests/Morphir.E2E.Tests
dotnet run --no-build --configuration ${{ env.CONFIGURATION }} -- --treenode-filter "*/AOT*"

- name: Upload test results (on failure)
if: failure()
uses: actions/upload-artifact@v4
with:
name: aot-test-results-${{ github.event.inputs.platform }}
path: |
**/TestResults/**/*
tests/**/bin/${{ env.CONFIGURATION }}/net10.0/*.dll
retention-days: 7

- name: Upload executables (on failure)
if: failure()
uses: actions/upload-artifact@v4
with:
name: aot-executables-${{ github.event.inputs.platform }}
path: |
./artifacts/executables/${{ github.event.inputs.platform }}/
./artifacts/single-file/${{ github.event.inputs.platform }}/
./artifacts/single-file-untrimmed/${{ github.event.inputs.platform }}/
retention-days: 7
21 changes: 19 additions & 2 deletions scripts/run-e2e-tests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,14 @@ static int RunCommandWithEnv(string command, Dictionary<string, string>? envVars
}
Console.WriteLine("✓ E2E test project is built");

// Check if we should exclude manual-only tests (AOT tests)
// These should only run in the manual-aot-test workflow
var includeManualTests = Environment.GetEnvironmentVariable("INCLUDE_MANUAL_TESTS") == "true";
if (!includeManualTests)
{
Console.WriteLine("ℹ Excluding @manual-only tests (use INCLUDE_MANUAL_TESTS=true to include them)");
}

// Track results for each executable type
var failedTypes = new List<string>();
var passedTypes = new List<string>();
Expand All @@ -232,8 +240,17 @@ static int RunCommandWithEnv(string command, Dictionary<string, string>? envVars
// Set environment variable for executable type
Environment.SetEnvironmentVariable("MORPHIR_EXECUTABLE_TYPE", type);

// Run tests
var exitCode = RunCommand("dotnet", "exec", testDllPath);
// Run tests (exclude manual-only tests unless explicitly included)
var testArgs = new List<string> { "exec", testDllPath };
if (!includeManualTests)
{
// Exclude AOT tests by filtering out tests that contain "Trimming" or "AOT" in their names
// These are the manual-only tests that should run in the dedicated workflow
testArgs.Add("--treenode-filter");
testArgs.Add("*/!(Trimming*|*AOT*)");
}

var exitCode = RunCommand("dotnet", testArgs.ToArray());

if (exitCode == 0)
{
Expand Down
8 changes: 4 additions & 4 deletions tests/Morphir.E2E.Tests/Features/AOT/AssemblyTrimming.feature
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
@ignore
@slow @manual-only @aot
Feature: Assembly Trimming
As a CLI developer
I want trimmed assemblies
So that I reduce deployment size

# TODO: Implement step definitions for these scenarios
# These tests were added in PR #229 but step definitions were never implemented
# See issue #246 for implementation tracking
# These tests run in the manual-aot-test workflow
# They build executables during test execution (1-5 min per scenario)
# Use for awareness/preparation before releasing trimmed executables

Background:
Given a morphir-dotnet CLI project
Expand Down
Loading
Loading