Add Microsoft.Maui.Essentials.AI — on-device AI for .NET MAUI by mattleibow · Pull Request #171 · dotnet/maui-labs

mattleibow · 2026-04-28T20:30:07Z

Summary

Adds Microsoft.Maui.Essentials.AI, a cross-platform .NET MAUI library that provides on-device AI capabilities via Microsoft.Extensions.AI abstractions. On Apple platforms, it wraps Apple Intelligence (Foundation Models) through a native Swift binding. The package ships as Microsoft.Maui.Essentials.AI targeting net10.0, Android, iOS, Mac Catalyst, macOS, and Windows.

What's in this PR

Library (`src/AI/`)

Microsoft.Maui.Essentials.AI — IChatClient implementation backed by Apple Intelligence on iOS/macOS/Mac Catalyst, with streaming JSON deserialization, tool calling support, and NL embedding generation
Native Swift bindings — Xcode project (AppleNative/EssentialsAI/) compiled via XcodeProject MSBuild items, producing .xcframework bundles packaged as .resources.zip
AppleBindings.targets — MSBuild targets that handle the macOS to Windows native artifact flow: on macOS CI, Swift compiles normally; on Windows CI, pre-built .resources.zip files from artifacts/bin/ are included with Pack=True
Streaming infrastructure — JsonStreamChunker, PlainTextStreamChunker, StreamingResponseHandler for progressive JSON deserialization of LLM responses

Sample app (`samples/Microsoft.Maui.Essentials.AI.Sample/`)

Multi-page MAUI app demonstrating trip planning with on-device AI
Itinerary workflow with 5 chained AI executors (travel planner, researcher, itinerary, translator, output)
Semantic search via NL embeddings, chat overlay, streaming UI updates

Tests (`tests/AI/`)

Unit tests — JsonStreamChunker, PlainTextStreamChunker, StreamingJsonDeserializer, JsonMerger, BufferedChatClient, StreamingResponseHandler, NonStreamingResponseHandler, tool call logging with file-based test data streams
Device tests — On-device tests for AppleIntelligenceChatClient and NLEmbeddingGenerator (iOS/macOS), plus OpenAI smoke tests
Benchmarks — StreamingJsonDeserializer performance benchmarks

CI/CD

GitHub Actions (ci-essentialsai.yml) — Sequential macOS then Windows build: macOS compiles Swift and uploads native artifacts, Windows downloads them via pre-build-artifact input and builds+packs the complete NuGet
_build.yml additions — xcode-version input for Xcode selection, pre-build-artifact input for downloading artifacts before build, native Apple artifact upload step
Azure DevOps (devflow-official.yml) — EssentialsAI_macOS job builds on macOS and publishes native artifacts; EssentialsAI Windows job depends on it, downloads artifacts, builds+packs+signs

Build infrastructure

Directory.Packages.props — Added Microsoft.Extensions.AI.Abstractions, Microsoft.Maui.Controls.Maps, and related package versions
eng/Versions.props — EssentialsAIPreviewVersionIteration for always-preview versioning
MauiLabs.slnx — All new projects added to solution
EssentialsAI.slnf — Solution filter for focused development

Design docs

docs/ai/json-stream-chunker-design.md — Architecture and design of the streaming JSON chunker

This commit completes the full migration of the AI product from D:\GitHub\maui\src\AI into the dotnet/maui-labs repository. ## What's included ### New library - src/AI/Microsoft.Maui.Essentials.AI/ — main library (.NET 10 + platform TFMs) - src/AI/AndroidNative/ — Gradle native Android project (verbatim copy) - src/AI/AppleNative/ — Xcode project + ObjC bindings (verbatim copy) ### New sample - samples/Microsoft.Maui.Essentials.AI.Sample/ — MAUI sample app All source files copied verbatim; project file migrated to NuGet packages. ### New tests - tests/AI/Microsoft.Maui.Essentials.AI.DeviceTests/ — device test app Uses OSS DeviceRunners instead of internal TestUtils.DeviceTests.Runners. - tests/AI/Microsoft.Maui.Essentials.AI.UnitTests/ — 333 unit tests (all pass) - tests/AI/Microsoft.Maui.Essentials.AI.Benchmarks/ — BenchmarkDotNet benchmarks ### New docs - docs/ai/json-stream-chunker-design.md ## Infrastructure changes - NuGet.config: added nuget.org feed (for DeviceRunners and AI packages) - eng/Versions.props: added EssentialsAI + DeviceRunners version properties - Directory.Packages.props: added PackageVersion entries for all new deps (Microsoft.Extensions.AI, DeviceRunners.*, Microsoft.Agents.AI.*, etc.) - MauiLabs.slnx: added all new projects to the solution - src/AI/EssentialsAI.slnf: new solution filter for AI product - .github/workflows/ci-essentialsai.yml: CI workflow (build + unit tests) ## Key decisions - All source .cs files are verbatim copies; only .csproj files were changed. - netstandard targets dropped; library targets net10.0 + platform TFMs. - MauiProgram.cs (DeviceTests) rewritten to use DeviceRunners API. - Platform/MaciOS/*.cs excluded from non-Apple TFMs via csproj condition. - Platform/*.cs top-level files included for all TFMs (pure .NET, no platform APIs). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

- New parameter: publishEssentialsAINuget (default false) - New build job: EssentialsAI with MAUI workloads (platform TFMs require them) Uses src\AI\EssentialsAI.slnf, mirrors DevFlow job pattern - New publish stage: publish_essentialsai_nuget Filters Microsoft.Maui.Essentials.AI.*.nupkg from PackageArtifacts Follows same 2-job (PrepareArtifacts + PublishNuGet) pattern as other products Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…ry attributes - Replace Microsoft.Maui.Maps with Microsoft.Maui.Controls.Maps (provides XAML namespace) - Remove UseMauiMaps property (handled by NuGet package) - Add AppDelegate.cs + Program.cs for iOS and MacCatalyst platforms - Add MainActivity.cs + MainApplication.cs for Android platform - Strip [Category(...)] attributes from DeviceTests (no CategoryAttribute in OSS setup) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix JsonStreamChunker.Escape() to handle ASCII control chars (U+0000-U+001F) - Remove AndroidNative/.gradle, .idea, local.properties from git tracking - Add AndroidNative/.gitignore to prevent machine-specific files from being committed - Fix AddUserSecrets target to only run in Debug builds (prevent API key leakage in Release) - Fix Directory.Packages.props to use correct version variables for DI.Abstractions and Hosting.Abstractions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

All packages (Microsoft.Agents.AI, DeviceRunners, CommunityToolkit.Mvvm, OpenAI, etc.) are available through the existing dnceng proxy feeds. Removes the policy violation flagged in PR review. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Add EnableWindowsTargeting=true to Directory.Build.props so macOS CI runners can restore projects with net10.0-windows TFMs (fixes CI failure NETSDK1100 on macos-latest) - Remove direct nuget.org feed - all packages resolve via dnceng proxies Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions

Expert Code Review: 8 findings posted inline (2 moderate, 6 minor). See the lean summary comment for methodology details.

Generated by Expert Code Review · ● 29.3M

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The EssentialsAI Swift native project uses Foundation Model APIs (iOS/macOS 26) which require Xcode 26. The macos-latest runner has Xcode 26.0.1 installed but defaults to Xcode 16.4. Add xcode-version input to _build.yml reusable workflow and set it to 26.0.1 in ci-essentialsai.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Only include net10.0-windows TFM on Windows (WinUI XAML compiler cannot run on macOS) - Remove duplicate MauiSplashScreen entry (Resizetizer errors in Release) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Apple Intelligence APIs require iOS/macOS 26+ but the library targets older OS versions with runtime guards. Arcade enables TreatWarningsAsErrors in Release builds, promoting these to errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Same rationale as the library — tests intentionally call Apple Intelligence APIs annotated with [SupportedOSPlatform("ios26.0")]. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove unused dotnet_bot.png — it was included as both MauiImage and MauiAsset, causing MT7156 in Release builds. The file isn't referenced anywhere in tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

github-actions

Expert Code Review: 9 findings posted inline (2 moderate, 7 minor). See the lean summary comment for full details including overflow findings.

Generated by Expert Code Review · ● 41.7M

Resolve conflict in _build.yml: main simplified the package job into inline validation+upload steps. Adopted main's pattern and layered apple-native changes on top. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Keep _build.yml unchanged (just add xcode-version input + native artifact upload step). EssentialsAI gets its own 2-job workflow: build-macos: reuses _build.yml with os=["macos-latest"] — builds Swift, uploads .resources.zip via _build.yml's native artifact step build-windows: needs build-macos, downloads native artifacts into artifacts/bin/, builds+packs with AppleBindings.targets Other products (DevFlow, CLI, GTK4) are completely unaffected — they still use _build.yml with their default parallel matrix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Added two inputs to _build.yml: - pre-build-artifact: downloads a named artifact into artifacts/bin/ before building (for injecting native Apple .resources.zip) - xcode-version: selects Xcode on macOS runners ci-essentialsai.yml is now just two _build.yml calls — zero inline steps: build-macos → builds Swift, uploads native-apple-essentialsai artifact build-windows → needs build-macos, passes pre-build-artifact to download native artifacts before building Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Changed pre-build-artifact (single string) to pre-build-artifacts (JSON array of artifact names). Uses a shell loop with gh CLI to download each artifact into artifacts/bin/ before building. Extensible for future products that need multiple pre-build artifacts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use a simple string input with actions/download-artifact instead of JSON array with jq/gh CLI loop. KISS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-30T14:20:37Z

Expert Code Review — PR #171

Methodology: 3 independent reviewers with adversarial consensus (batch-split mode: 328 changed files split across reviewers)

15 findings posted as inline comments (1 moderate, 14 minor)

All findings are annotated "low confidence — single reviewer (batch split)" because each reviewer examined a different file batch. No cross-reviewer agreement was possible for individual findings.

Severity Breakdown

Severity	Count	Areas
🟡 Moderate	1	NuGet feed configuration (packages may not resolve)
🟢 Minor	14	Version governance, async patterns, CI coupling, dead code

CI Status

Check	Status
build (ubuntu-24.04)	✅ Success
build (macos-latest)	✅ Success
build (windows-latest)	✅ Success
build-macos (macos-latest)	🔄 In progress
ci-essentialsai jobs	🔄 In progress

Test Coverage Assessment

The PR includes 333 unit tests (all passing per description), device tests, and BenchmarkDotNet benchmarks. Test coverage appears comprehensive for the migrated library. The unit test project and device test MauiProgram.cs were rewritten for the new repo context.

Key Observations

The migration is well-structured with proper platform file exclusions and sound async patterns overall
The "verbatim copy" approach for source files minimizes migration risk
Infrastructure changes (CI workflow, pipeline, package management) follow repo conventions
Most findings are version governance and defensive coding suggestions, not blocking issues

Generated by Expert Code Review · 3 independent reviewers with adversarial consensus

Generated by Expert Code Review · ● 24.5M · ◷

github-actions

Batch 2: src/AI Library Source Code — Review Findings

Reviewed 46 files covering the main library code (csproj, Apple bindings, Swift native code, C# platform implementations, public API surface).

Summary

3 moderate issues found, 2 minor. The migration is well-executed overall — project structure, binding definitions, and async patterns are sound. The primary concerns are around null-safety in streaming callbacks and a missing dispose guard.

Generated by Expert Code Review · ● 24.5M

Review fixes: - Fix README.md: correct source paths, remove hardcoded developer path - Fix Benchmarks/Models.cs: namespace Maui.Controls.Sample → Microsoft.Maui.Essentials.AI.Benchmarks - Fix Directory.Packages.props: use correct version vars for Configuration.Json and Logging.Console, add missing version properties - Fix sample: remove Debug-only secrets restriction, make OpenAI registration conditional (skip if API key not configured) - Fix AzDO: add Xcode 26.3 selection + ios/maccatalyst workloads to EssentialsAI_macOS job - Fix _build.yml: parameterize Android SDK project via android-project input (default: DevFlow, EssentialsAI overrides to its own csproj) Sample rename: - Rename samples/Microsoft.Maui.Essentials.AI.Sample → samples/EssentialsAISample - Rename csproj: EssentialsAISample.csproj - Update all namespaces: Maui.Controls.Sample* → EssentialsAISample* - Update solution files, CI paths, test project references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The benchmark's own Models.cs is in the Microsoft.Maui.Essentials.AI.Benchmarks namespace (same as the benchmark), so no using needed. The EssentialsAISample.Models using was left over from the namespace rename. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Set PackRepoRootReadme=false and pack the project's own README.md, following the Linux.Gtk4 pattern. Fixes NuGet warning about missing readme. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The README.md is now a getting-started guide for NuGet consumers: platform support table, install, register, use, streaming, embeddings. The previous README (sharpie binding generation instructions) moved to CONTRIBUTING.md for repo developers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

jfversluis · 2026-05-02T09:11:10Z

@copilot resolve the merge conflicts in this pull request

…on-plan # Conflicts: # MauiLabs.slnx Co-authored-by: jfversluis <939291+jfversluis@users.noreply.github.com>

Copilot · 2026-05-02T09:12:50Z

@copilot resolve the merge conflicts in this pull request

Resolved. The only conflict was in MauiLabs.slnx where both branches added new /src/ sub-folders — this branch added /src/AI/ while main added /src/Comet/ and /src/Go/. Merged both sets of folders/projects and committed (a31202b).

jfversluis · 2026-05-02T11:35:42Z

@mattleibow I had a failing test in an earlier run so seems like something might be flaky, FYI

src/AI/ now hosts two products (AI Attributes + Essentials.AI). - src/AI/README. product-family landing page linking to bothmd - src/AI/Microsoft.Maui.AI.Attributes/README. NuGet READMEmd (moved from family README, packed into the nupkg) - src/AI/Microsoft.Maui.Essentials.AI/README. already existedmd (from PR #171) Each product has its own NuGet README, CI workflow, and solution filter. The family README is for GitHub browsing only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…mple Update namespace in Windows-specific service files and device tests to match the new sample project name (PR #171 rename). - AppContentIndexerSearchService.cs: Maui.Controls.Sample.Services -> EssentialsAISample.Services - PhiSilicaToolsAndSchemaClient.cs: Maui.Controls.Sample.Services -> EssentialsAISample.Services - PhiSilicaChatClientTests.cs: using -> EssentialsAISample.Services - App.xaml.cs: Remove temporary arg-passing test code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 28, 2026 20:30

Copilot AI reviewed Apr 28, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

mattleibow and others added 2 commits April 28, 2026 23:21

mattleibow mentioned this pull request Apr 28, 2026

[AI] Windows Copilot Runtime (Phi Silica) support #178

Draft

mattleibow requested a review from Copilot April 29, 2026 07:13

Copilot AI reviewed Apr 29, 2026

View reviewed changes

mattleibow and others added 3 commits April 29, 2026 19:23

Merge branch 'main' into mattleibow/ai-migration-plan

c7a470b

This comment has been minimized.

Sign in to view

github-actions Bot reviewed Apr 29, 2026

View reviewed changes

mattleibow and others added 9 commits April 29, 2026 20:21

Retry CI - packages should be mirrored now

d39547c

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retry CI - Maps package now mirrored

998ad5f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Retry CI - Microsoft.Maui.Maps now mirrored

a161b10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Use Xcode 26.3 for EssentialsAI CI

0afb5e7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Suppress CA1416 in device tests project

663adc3

Same rationale as the library — tests intentionally call Apple Intelligence APIs annotated with [SupportedOSPlatform("ios26.0")]. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mattleibow force-pushed the mattleibow/ai-migration-plan branch from b75c456 to b7a0450 Compare April 29, 2026 21:41

mattleibow requested a review from Copilot April 29, 2026 22:12

Copilot AI reviewed Apr 29, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

github-actions Bot reviewed Apr 29, 2026

View reviewed changes

mattleibow and others added 7 commits April 30, 2026 14:32

Reset _build.yml to main

7696aa6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Simplify pre-build-artifact to plain string

a78e655

Use a simple string input with actions/download-artifact instead of JSON array with jq/gh CLI loop. KISS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Trigger CI workflows

def9d3a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

mattleibow changed the title ~~Migrate Microsoft.Maui.Essentials.AI from dotnet/maui into maui-labs~~ Add Microsoft.Maui.Essentials.AI — on-device AI for .NET MAUI Apr 30, 2026

github-actions Bot reviewed Apr 30, 2026

View reviewed changes

mattleibow and others added 5 commits April 30, 2026 17:40

Pack product README instead of repo-root README

cc3103a

Set PackRepoRootReadme=false and pack the project's own README.md, following the Linux.Gtk4 pattern. Fixes NuGet warning about missing readme. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' into mattleibow/ai-migration-plan

80ac06d

Copilot started work on behalf of jfversluis May 2, 2026 09:11 View session

Merge remote-tracking branch 'origin/main' into mattleibow/ai-migrati…

a31202b

…on-plan # Conflicts: # MauiLabs.slnx Co-authored-by: jfversluis <939291+jfversluis@users.noreply.github.com>

Copilot finished work on behalf of jfversluis May 2, 2026 09:13

Copilot AI requested a review from jfversluis May 2, 2026 09:13

jfversluis approved these changes May 2, 2026

View reviewed changes

jfversluis merged commit f50a311 into main May 2, 2026
11 of 13 checks passed

jfversluis deleted the mattleibow/ai-migration-plan branch May 2, 2026 11:35

jfversluis mentioned this pull request May 2, 2026

Add Essentials.AI to main README and create product README #211

Merged

jfversluis added this to the v0.1.0-preview.8 milestone May 6, 2026

Conversation

mattleibow commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in this PR

Library (src/AI/)

Sample app (samples/Microsoft.Maui.Essentials.AI.Sample/)

Tests (tests/AI/)

CI/CD

Build infrastructure

Design docs

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 30, 2026

Expert Code Review — PR #171

Severity Breakdown

CI Status

Test Coverage Assessment

Key Observations

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Batch 2: src/AI Library Source Code — Review Findings

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jfversluis commented May 2, 2026

Uh oh!

Copilot AI commented May 2, 2026

Uh oh!

jfversluis commented May 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattleibow commented Apr 28, 2026 •

edited

Loading

Library (`src/AI/`)

Sample app (`samples/Microsoft.Maui.Essentials.AI.Sample/`)

Tests (`tests/AI/`)