Skip to content

Comments

Implement AOT BDD test step definitions with dedicated manual workflow and AOT Guru skill integration#248

Merged
DamianReeves merged 7 commits intomainfrom
copilot/implement-aot-bdd-steps
Dec 19, 2025
Merged

Implement AOT BDD test step definitions with dedicated manual workflow and AOT Guru skill integration#248
DamianReeves merged 7 commits intomainfrom
copilot/implement-aot-bdd-steps

Conversation

Copy link
Contributor

Copilot AI commented Dec 19, 2025

PR #229 added BDD feature files for AOT testing but no step definitions, causing all 20 scenarios to fail with "No matching step definition found". These tests were disabled with @ignore tags in PR #245.

Changes

Step Definitions

  • AssemblyTrimmingSteps.cs - 11 scenarios covering trimming modes, size reduction, feature switches, resource preservation
  • NativeAOTCompilationSteps.cs - 9 scenarios covering AOT builds, cross-platform targets, runtime validation, performance metrics

Implementation Approach

  • Builds executables on-demand via dotnet publish with scenario-specific MSBuild properties
  • Reuses existing artifacts from artifacts/executables/ when available (AOT tests)
  • Isolates builds in artifacts/test-builds/{guid} per scenario
  • Cross-platform RID detection (linux-x64, win-x64, osx-x64, ARM variants)

Validations

  • Size comparisons (baseline vs trimmed, target ranges for minimal/full CLI)
  • Build warnings (IL2XXX detection for reflection/trimming issues)
  • Runtime command execution (--version, --help, ir verify with JSON output)
  • Platform-specific size assertions (< 12 MB linux-x64, < 15 MB win-x64)

Manual Workflow

  • Created manual-aot-test.yml - dedicated GitHub Actions workflow for AOT testing
  • Tests tagged with @manual-only to exclude from regular CI runs
  • Updated run-e2e-tests.cs to filter out manual-only tests automatically
  • Platform selection: linux-x64, linux-arm64, osx-x64, osx-arm64, win-x64
  • Test suite selection: both, trimming, or aot-compilation
  • 90-minute timeout appropriate for long-running tests (45-90 min total)

Documentation

  • Added README.md in AOT test directory with usage instructions and troubleshooting
  • Documented workflow usage and local execution commands

AOT Guru Skill Integration

  • Updated .claude/skills/aot-guru/skill.md with comprehensive BDD test documentation
  • Documented all 20 test scenarios with descriptions and purposes
  • Added guidance on when to run tests (release preparation, dependency updates, etc.)
  • Included how-to instructions for both manual workflow and local execution
  • Added troubleshooting guide for common test failures
  • Provided patterns for recommending and adding new test scenarios
  • Documented test modification procedures for workflow and step definition changes

The AOT Guru skill can now intelligently:

  • Recommend when to run AOT tests based on context
  • Suggest additional test scenarios for new features
  • Guide users on test execution options
  • Help troubleshoot test failures
  • Advise on test modifications and workflow changes

Rationale for Manual Workflow

AOT tests are moved to a separate manually triggered workflow because:

  • Long duration: 1-5 minutes per scenario, 45-90 minutes total for all tests
  • Purpose: Awareness/preparation for release validation, not blocking CI
  • Use case: Validate trimmed/AOT executables are safe to release
  • Resource intensive: Each test builds executables with different configurations

This approach keeps CI fast (~minutes) while providing comprehensive AOT validation on-demand for release preparation.

Usage

Manual Workflow (recommended):

  1. Go to Actions → Manual AOT Testing
  2. Select platform and test suite
  3. Click Run workflow

Local Execution:

./build.sh --target PublishSingleFile --rid linux-x64
cd tests/Morphir.E2E.Tests
MORPHIR_EXECUTABLE_TYPE=trimmed dotnet run -- --treenode-filter "*/Trimming*"

Example

[When("I build with PublishAot=true")]
public async Task WhenIBuildWithPublishAotTrue()
{
    _publishAot = true;
    await BuildWithSettings(); // Invokes dotnet publish with -p:PublishAot=true
}

[Then("the executable size should be less than 12 MB for linux-x64")]
public void ThenTheExecutableSizeShouldBeLessThan12MbForLinuxX64()
{
    if (_rid != "linux-x64") return;
    
    var sizeMB = new FileInfo(_executablePath!).Length / (1024.0 * 1024.0);
    sizeMB.Should().BeLessThan(12, $"actual: {sizeMB:F2} MB");
}

All 20 scenarios now discoverable (41 E2E tests total). Tests execute actual builds to validate trimming/AOT behavior but run only in dedicated manual workflow. The AOT Guru skill has full awareness of the test suite for intelligent guidance on testing strategy.

Original prompt

This section details on the original issue you should resolve

<issue_title>Implement AOT BDD test step definitions</issue_title>
<issue_description>## Background

PR #229 added comprehensive BDD feature files for AOT (Ahead-of-Time compilation) testing but only implemented the Gherkin scenarios, not the C# step definitions. This causes all 20 AOT test scenarios to fail with "No matching step definition found".

Current State

Feature files exist:

  • tests/Morphir.E2E.Tests/Features/AOT/AssemblyTrimming.feature (11 scenarios)
  • tests/Morphir.E2E.Tests/Features/AOT/NativeAOTCompilation.feature (9 scenarios)

Step definitions missing:

Tasks

1. Create Step Definition Files

Create the following step definition classes:

tests/Morphir.E2E.Tests/Features/AOT/
├── AssemblyTrimmingSteps.cs
└── NativeAOTCompilationSteps.cs

2. Implement AssemblyTrimming Steps

Implement step definitions for all 11 scenarios including:

  • Given a morphir-dotnet CLI project
  • Given a self-contained morphir-dotnet build
  • Given PublishTrimmed is enabled
  • Given TrimMode is set to link
  • When I publish the application
  • Then unused assemblies should be removed
  • Then unused types should be trimmed
  • Then the output size should be reduced compared to untrimmed

3. Implement NativeAOTCompilation Steps

Implement step definitions for all 9 scenarios including:

  • Given PublishAot is enabled in the project
  • When I build the project with PublishAot=true
  • Then the build should succeed without errors
  • Then the output should be a native executable
  • Then no IL2XXX warnings should be present

4. Integration with ExecutableRunner

The step definitions should:

  • Use ExecutableRunner to execute morphir CLI commands
  • Validate build outputs, sizes, and performance metrics
  • Check for AOT-specific warnings and errors
  • Verify trimming effectiveness

5. Remove @ignore Tags

Once step definitions are implemented:

  • Remove @ignore tags from both feature files
  • Remove TODO comments
  • Update tracking references to this issue

Definition of Done

  • All step definitions implemented for both feature files
  • All 20 AOT scenarios passing
  • @ignore tags removed
  • Tests passing in CI
  • Code review completed
  • PR merged

References

Priority

Medium - AOT testing is valuable for future optimization but not blocking current deployments since trimmed executables work well.

🤖 Generated with Claude Code</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits December 19, 2025 08:49
- Created AssemblyTrimmingSteps.cs with all step definitions for 11 trimming scenarios
- Created NativeAOTCompilationSteps.cs with all step definitions for 9 AOT scenarios
- Removed @ignore tags and TODO comments from both feature files
- Step definitions use ExecutableRunner and build system for validation

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
Changed BeGreaterOrEqualTo to BeGreaterThanOrEqualTo to match FluentAssertions API

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
- Added @slow tags to AOT feature files to indicate long-running tests
- Applied dotnet format to fix whitespace issues across E2E tests

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
Copilot AI changed the title [WIP] Add AOT BDD test step definitions Implement AOT BDD test step definitions for assembly trimming and native compilation Dec 19, 2025
Copilot AI requested a review from DamianReeves December 19, 2025 08:56
@DamianReeves
Copy link
Member

@copilot As these tests help us to know if we are okay to release trimmed or AOT executables they are more awareness or preparation tasks so they should probably run in a separate manually triggered workflow, especially given the length of the test runs.

- Created manual-aot-test.yml workflow for dedicated AOT testing
- Added @manual-only tag to AOT feature files
- Updated run-e2e-tests.cs to exclude manual-only tests from CI
- Added README documenting AOT test suite and workflow usage
- Tests are long-running (45-90 min) and for release preparation

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Dec 19, 2025

@copilot As these tests help us to know if we are okay to release trimmed or AOT executables they are more awareness or preparation tasks so they should probably run in a separate manually triggered ...

Agreed! I've moved the AOT tests to a separate manual workflow in commit bbd5b90.

Changes:

  • Created manual-aot-test.yml - dedicated workflow with platform/test suite selection
  • Tagged tests with @manual-only to exclude from regular CI
  • Updated E2E test runner to filter out manual-only tests automatically
  • Added README documenting the workflow and usage

The tests now run only when manually triggered (Actions → Manual AOT Testing), keeping CI fast while providing comprehensive AOT validation for release preparation.

Fixed multi-line expression in runs-on to single-line format for valid YAML syntax

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
Copilot AI changed the title Implement AOT BDD test step definitions for assembly trimming and native compilation Implement AOT BDD test step definitions with dedicated manual workflow Dec 19, 2025
@DamianReeves
Copy link
Member

@copilot We should make sure the AOT Guru skill is aware of these tests, how to run them, when to run them, and can recommend additional tests or changes to the tests or test execution

- Added comprehensive documentation about AOT/trimming BDD test suite
- Documented when to run tests (release preparation, not CI)
- Explained how to run tests via manual workflow and locally
- Listed all 20 test scenarios with descriptions
- Included troubleshooting guide and test modification instructions
- Added guidance for recommending additional tests

Co-authored-by: DamianReeves <957246+DamianReeves@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Dec 19, 2025

@copilot We should make sure the AOT Guru skill is aware of these tests, how to run them, when to run them, and can recommend additional tests or changes to the tests or test execution

Done in commit 4b0160b! The AOT Guru skill now includes comprehensive documentation about the BDD test suite.

What the skill knows:

  • All 20 test scenarios with descriptions
  • When to run tests (release preparation, after dependency updates, etc.)
  • How to run via manual workflow and locally
  • Test implementation details (build strategy, validations, duration)
  • Troubleshooting common test failures
  • How to recommend and add new test scenarios
  • How to modify test execution workflow

The skill can now intelligently recommend when to run tests, suggest additional scenarios, and guide on test execution based on context.

Copilot AI changed the title Implement AOT BDD test step definitions with dedicated manual workflow Implement AOT BDD test step definitions with dedicated manual workflow and AOT Guru skill integration Dec 19, 2025
@DamianReeves DamianReeves marked this pull request as ready for review December 19, 2025 09:52
@DamianReeves DamianReeves merged commit 87c22c0 into main Dec 19, 2025
16 checks passed
@DamianReeves DamianReeves deleted the copilot/implement-aot-bdd-steps branch December 19, 2025 09:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement AOT BDD test step definitions

2 participants