Add AI Resilience Capabilities

CrestApps.Core.slnx

@@ -39,6 +39,9 @@
     <Project Path="src/Primitives/CrestApps.Core.AI.PostgreSQL/CrestApps.Core.AI.PostgreSQL.csproj">
       <Build Solution="Debug|*" Project="false" />
     </Project>
+    <Project Path="src/Primitives/CrestApps.Core.AI.Resilience/CrestApps.Core.AI.Resilience.csproj">
+      <Build Solution="Debug|*" Project="false" />
+    </Project>
     <Project Path="src/Primitives/CrestApps.Core.AI/CrestApps.Core.AI.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Azure.AISearch/CrestApps.Core.Azure.AISearch.csproj" />
     <Project Path="src/Primitives/CrestApps.Core.Azure/CrestApps.Core.Azure.csproj" />

Directory.Packages.props

@@ -59,6 +59,7 @@
     <PackageVersion Include="Microsoft.Extensions.Localization.Abstractions" Version="10.0.9" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.9" />
     <PackageVersion Include="Microsoft.Extensions.Options" Version="10.0.9" />
+    <PackageVersion Include="Microsoft.Extensions.Resilience" Version="10.7.0" />
     <PackageVersion Include="Microsoft.ML.Tokenizers.Data.O200kBase" Version="2.0.0" />
     <PackageVersion Include="Microsoft.SourceLink.GitHub" Version="10.0.300" />
   </ItemGroup>

src/CrestApps.Core.Docs/docs/changelog/v1.0.0.md

@@ -75,6 +75,7 @@ description: Initial standalone release notes for the CrestApps.Core repository.
 - introduces `AIDeploymentPurpose` as the primary deployment terminology, keeps the legacy type surface for backward compatibility, adds `Vision` plus `DefaultVisionDeploymentName`, updates the MVC and Blazor deployment/settings UX to say purpose, and allows vision-capable chat interactions and chat sessions to upload supported image files as multimodal inputs
 - distinguishes uploaded vision images from searchable documents in the shared document-availability prompt so multimodal chat sessions analyze supported attached images directly instead of defaulting to document-tool or metadata-only responses
 - caps the total uploaded vision-image bytes loaded into a single multimodal request through `ChatDocumentsOptions.MaxVisionInputBytesPerRequest`, removes the extra `MemoryStream` copy when attaching those images, and documents how to resolve a vision-capable chat client for direct image-description requests
+- adds the standalone `CrestApps.Core.AI.Resilience` package with opt-in Microsoft.Extensions.AI builder resilience extensions for chat, embeddings, image generation, speech-to-text, and text-to-speech clients, including `UseDefaultResilience()` for provider `429 Too Many Requests` retries and `UseResilience(...)` for custom Polly/Microsoft resilience pipelines; the docs now include a dedicated AI Resilience page, the default retry schedule uses exponential backoff with jitter (about 1-2, 2-4, 4-8, 8-16, and 16-32 seconds across five retries), framework-owned completion clients and utility-deployment chat flows apply the default retry policy automatically, host-created clients remain opt-in, builder examples require `Build(serviceProvider)` instead of `Build(null)`, and Azure OpenAI exposes shared SDK retry settings through `CrestApps:AI:AzureClient` with matching five-retry exponential defaults
 - adds `CrestApps.Core.PostgreSQL` and `CrestApps.Core.AI.PostgreSQL` packages providing a lightweight PostgreSQL + pgvector vector search backend as an alternative to Elasticsearch and Azure AI Search, registers the same keyed services (`ISearchIndexManager`, `ISearchDocumentManager`, `IDataSourceContentManager`, `IDataSourceDocumentReader`, `IODataFilterTranslator`) under the `"PostgreSQL"` provider name, supports `AddAIDocuments()`, `AddAIDataSources()`, and `AddAIMemory()` builder extensions, and integrates into both MVC and Blazor sample hosts
 - fixes hosted document and data-source indexing flows so background workers create a scoped service provider before resolving scoped indexing services, preventing upload-triggered failures and similar nightly alignment lifetime issues
 - standardizes Azure AI Search configuration on top-level `AuthenticationType`, `ApiKey`, `IdentityClientId`, and `IndexPrefix` settings under `CrestApps:AzureAISearch`, and refreshes the sample host / docs examples to list the full supported option set in one place

src/CrestApps.Core.Docs/docs/core/ai-core.md

@@ -257,9 +257,9 @@ public sealed class ChatApiController : ControllerBase
 | Exception | When | How to Handle |
 |-----------|------|--------------|
 | `InvalidOperationException` | No deployment found, no provider connection configured | Check AI configuration — this is a setup error |
-| `HttpRequestException` | Provider API unreachable (network error, DNS failure) | Retry with exponential backoff, check network connectivity |
+| `HttpRequestException` | Provider API unreachable (network error, DNS failure) | Check network connectivity; framework-owned completion and utility chat paths already use the default retry policy, and host-created AI clients can opt in separately through the resilience builders |
 | `OperationCanceledException` | Request was cancelled (user navigated away, timeout) | Normal flow — let it propagate |
-| Provider-specific rate limit errors | Too many requests to the AI provider | Implement retry policies at the HTTP client level |
+| Provider-specific rate limit errors | Too many requests to the AI provider | Framework-owned completion and utility chat paths already use the default retry policy; for host-created AI clients, use `CrestApps.Core.AI.Resilience` with `.AsBuilder().UseDefaultResilience()` or a custom `UseResilience(...)` pipeline; see [AI Resilience](./ai-resilience.md) |
 | Provider-specific auth errors | Invalid API key or expired credentials | Check provider connection configuration |
 
 ### Handling Provider Failures
@@ -309,6 +309,8 @@ public sealed class ResilientCompletionService
 }
 ```
 
+When you finish a `ChatClientBuilder` pipeline, always call `Build(serviceProvider)` with the active service provider instead of `Build(null)`. Several framework chat middlewares resolve services from DI at execution time, especially tool-related components.
+
 :::warning
 Never swallow `OperationCanceledException` — always re-throw it. Catching and ignoring it breaks the cancellation token contract and can cause resource leaks.
 :::

src/CrestApps.Core.Docs/docs/core/ai-resilience.md

@@ -0,0 +1,193 @@
+---
+sidebar_position: 4
+title: AI Resilience
+description: Builder-based resilience middleware for Microsoft.Extensions.AI chat, embeddings, image, speech-to-text, and text-to-speech clients.
+---
+
+# AI Resilience
+
+> Add reusable retry middleware to Microsoft.Extensions.AI clients without forcing a global policy on every host-created client.
+
+`CrestApps.Core.AI.Resilience` is a standalone package that adds builder-based resilience extensions for:
+
+- `IChatClient`
+- `IEmbeddingGenerator<TInput, TEmbedding>`
+- `IImageGenerator`
+- `ISpeechToTextClient`
+- `ITextToSpeechClient`
+
+Framework-owned completion and utility chat paths in `CrestApps.Core` already use the default retry policy internally. This package is for host-created clients and for applications that want to opt into the same pattern explicitly.
+
+## Package
+
+```xml
+<PackageReference Include="CrestApps.Core.AI.Resilience" Version="*" />
+```
+
+The package depends on:
+
+- `Microsoft.Extensions.AI`
+- `Microsoft.Extensions.Resilience`
+
+## Builder Extensions
+
+Every supported client follows the same pattern:
+
+1. Resolve or create the AI client
+2. Convert it to the corresponding builder with `.AsBuilder()`
+3. Apply either `UseDefaultResilience()` or `UseResilience(...)`
+4. Finish with `Build(serviceProvider)`
+
+Always pass the active `IServiceProvider` to `Build(serviceProvider)`. Do not use `Build()` or `Build(null)`, because downstream middleware may need DI to resolve services such as tools and related runtime components.
+
+## Default Policy
+
+`UseDefaultResilience()` is intentionally narrow: it retries provider rate-limit failures such as HTTP `429 Too Many Requests`.
+
+Default settings:
+
+| Setting | Default |
+|---|---|
+| `MaxRateLimitRetries` | `5` |
+| `RateLimitRetryDelay` | `1 second` |
+| `BackoffType` | `Exponential` |
+| `UseJitter` | `true` |
+| `MaxRetryDelay` | `32 seconds` |
+
+That produces an approximate retry schedule like this:
+
+| Attempt | Delay |
+|---|---|
+| Initial | immediately |
+| Retry 1 | ~1-2 seconds |
+| Retry 2 | ~2-4 seconds |
+| Retry 3 | ~4-8 seconds |
+| Retry 4 | ~8-16 seconds |
+| Retry 5 | ~16-32 seconds |
+
+The exact delay varies because jitter is enabled by default.
+
+## Chat Example
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseDefaultResilience()
+    .Build(serviceProvider);
+```
+
+## Customizing the Default Settings
+
+Use the options callback when you want to keep the built-in rate-limit handling but tune the retry shape:
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseDefaultResilience(options =>
+    {
+        options.MaxRateLimitRetries = 3;
+        options.RateLimitRetryDelay = TimeSpan.FromSeconds(2);
+        options.BackoffType = DelayBackoffType.Exponential;
+        options.UseJitter = true;
+        options.MaxRetryDelay = TimeSpan.FromSeconds(20);
+    })
+    .Build(serviceProvider);
+```
+
+If you prefer the old fixed schedule, configure it explicitly:
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseDefaultResilience(options =>
+    {
+        options.MaxRateLimitRetries = 4;
+        options.RateLimitRetryDelay = TimeSpan.FromSeconds(5);
+        options.BackoffType = DelayBackoffType.Constant;
+        options.UseJitter = false;
+        options.MaxRetryDelay = TimeSpan.FromSeconds(5);
+    })
+    .Build(serviceProvider);
+```
+
+## Fully Custom Pipelines
+
+Use `UseResilience(...)` when you want full control over the Polly pipeline:
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseResilience(pipeline => pipeline.AddRetry(new RetryStrategyOptions
+    {
+        MaxRetryAttempts = 2,
+        Delay = TimeSpan.FromSeconds(1),
+        BackoffType = DelayBackoffType.Exponential,
+        UseJitter = true,
+        ShouldHandle = args => ValueTask.FromResult(
+            args.Outcome.Exception is HttpRequestException ex &&
+            ex.StatusCode == HttpStatusCode.TooManyRequests),
+    }))
+    .Build(serviceProvider);
+```
+
+You can also supply a prebuilt `ResiliencePipeline`.
+
+## Other Client Types
+
+The same extension methods are available on the other Microsoft.Extensions.AI builders:
+
+### Embeddings
+
+```csharp
+var resilientGenerator = embeddingGenerator
+    .AsBuilder()
+    .UseDefaultResilience()
+    .Build(serviceProvider);
+```
+
+### Image Generation
+
+```csharp
+var resilientGenerator = imageGenerator
+    .AsBuilder()
+    .UseDefaultResilience()
+    .Build(serviceProvider);
+```
+
+### Speech to Text
+
+```csharp
+var resilientClient = speechToTextClient
+    .AsBuilder()
+    .UseDefaultResilience()
+    .Build(serviceProvider);
+```
+
+### Text to Speech
+
+```csharp
+var resilientClient = textToSpeechClient
+    .AsBuilder()
+    .UseDefaultResilience()
+    .Build(serviceProvider);
+```
+
+## Streaming Notes
+
+- `ITextToSpeechClient` streaming retries are supported when the failure happens before the first streamed update is yielded.
+- `ISpeechToTextClient` non-streaming retries work for both seekable and non-seekable streams.
+- `ISpeechToTextClient` streaming retries require a seekable input stream so the audio can be replayed safely across retry attempts.
+
+## When to Use It
+
+Use `UseDefaultResilience()` when:
+
+- you want a safe default for provider throttling
+- you want framework-style retries on your own clients
+- you do not need a custom Polly pipeline yet
+
+Use `UseResilience(...)` when:
+
+- you need custom retry predicates
+- you want to add additional strategies yourself
+- you want one shared prebuilt pipeline across multiple clients

src/CrestApps.Core.Docs/docs/core/getting-started-aspnet.md

@@ -313,6 +313,39 @@ services.AddSingleton<IConfigureOptions<GeneralAIOptions>, SiteSettingsConfigure
 
 That keeps settings refresh host-agnostic and avoids custom accessor interfaces.
 
+`AddCoreAIServices()` keeps host-created AI clients opt-in for retries. Framework-owned completion clients and utility-deployment chat paths already use the default retry policy internally. If you want the same builder extensions for your own resolved clients outside the framework defaults, reference the standalone `CrestApps.Core.AI.Resilience` package and wrap the client through the corresponding Microsoft.Extensions.AI builder pipeline:
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseDefaultResilience(options =>
+    {
+        options.MaxRateLimitRetries = 5;
+        options.RateLimitRetryDelay = TimeSpan.FromSeconds(1);
+    })
+    .Build(serviceProvider);
+```
+
+The same `UseDefaultResilience()` and `UseResilience(...)` extensions are also available on `IEmbeddingGenerator<TInput, TEmbedding>`, `IImageGenerator`, `ISpeechToTextClient`, and `ITextToSpeechClient` through their `.AsBuilder()` adapters. For streaming speech-to-text, retries require a seekable input stream so the audio can be replayed safely. See [AI Resilience](./ai-resilience.md) for the full builder surface, default retry schedule, and customization options.
+
+For a custom policy, use the lower-level builder extension and configure the resilience pipeline yourself:
+
+```csharp
+var resilientClient = chatClient
+    .AsBuilder()
+    .UseResilience(pipeline => pipeline.AddRetry(new RetryStrategyOptions
+    {
+        MaxRetryAttempts = 2,
+        Delay = TimeSpan.FromSeconds(1),
+        ShouldHandle = args => ValueTask.FromResult(
+            args.Outcome.Exception is HttpRequestException ex &&
+            ex.StatusCode == HttpStatusCode.TooManyRequests),
+    }))
+    .Build(serviceProvider);
+```
+
+Always pass the active `IServiceProvider` into `Build(serviceProvider)` rather than `null`. That keeps downstream middleware such as tool invocation and other DI-backed chat components able to resolve their required services correctly.
+
 ## 7. Add features one layer at a time
 
 The intended progression is:

src/CrestApps.Core.Docs/docs/orchestration/index.md

@@ -403,7 +403,7 @@ If no deployment can be resolved (no profile-level, connection-level, or global
 
 ### Provider Errors
 
-Errors from AI providers (rate limits, authentication failures, server errors) propagate up to the caller. The orchestrator does not retry automatically — retry policies should be configured at the HTTP client level or in the provider.
+Errors from AI providers (rate limits, authentication failures, server errors) still propagate up to the caller after retries are exhausted. Framework-owned planning and other utility-model chat paths now apply the default `UseDefaultResilience()` policy automatically, while host-created `IChatClient` instances remain opt-in through `chatClient.AsBuilder().UseDefaultResilience()` or a custom `UseResilience(...)` pipeline.
 
 | Error Type | Behavior |
 |-----------|----------|

src/CrestApps.Core.Docs/docs/providers/azure-openai.md

@@ -56,13 +56,19 @@ Azure OpenAI requires an endpoint URL and either an API key or Azure AD credenti
 |----------|-------|
 | `AzureOpenAIConstants.ClientName` | `"Azure"` |
 
-Use `CrestApps:AI:AzureClient` for Azure SDK logging switches shared by all Azure OpenAI connections:
+Use `CrestApps:AI:AzureClient` for Azure SDK settings shared by all Azure OpenAI connections, including logging plus the default SDK retry policy used by the framework's Azure OpenAI completion path:
 
 ```json
 {
   "CrestApps": {
     "AI": {
       "AzureClient": {
+        "EnableDefaultRetryPolicy": true,
+        "MaxRetryAttempts": 5,
+        "RateLimitRetryDelay": "00:00:01",
+        "BackoffType": "Exponential",
+        "UseJitter": true,
+        "MaxRetryDelay": "00:00:32",
         "EnableLogging": false,
         "EnableMessageLogging": false,
         "EnableMessageContentLogging": false
@@ -72,6 +78,18 @@ Use `CrestApps:AI:AzureClient` for Azure SDK logging switches shared by all Azur
 }
 ```
 
+The Azure SDK retry defaults now match the framework resilience defaults:
+
+| Setting | Default |
+|---|---|
+| `MaxRetryAttempts` | `5` |
+| `RateLimitRetryDelay` | `00:00:01` |
+| `BackoffType` | `Exponential` |
+| `UseJitter` | `true` |
+| `MaxRetryDelay` | `00:00:32` |
+
+That produces an approximate schedule of `1-2`, `2-4`, `4-8`, `8-16`, and `16-32` seconds across retries.
+
 ## Azure-Specific Behavior
 
 The Azure provider includes `AzurePatchOpenAIDataSourceHandler` which automatically:
@@ -105,13 +123,19 @@ The deployment name in Azure OpenAI is what you pass as the `deploymentName` par
 
 ## Configuration
 
-Full `appsettings.json` configuration with endpoint, deployment, and optional global Azure SDK logging:
+Full `appsettings.json` configuration with endpoint, deployment, optional Azure SDK retry settings, and optional global logging:
 
 ```json
 {
   "CrestApps": {
     "AI": {
         "AzureClient": {
+          "EnableDefaultRetryPolicy": true,
+          "MaxRetryAttempts": 5,
+          "RateLimitRetryDelay": "00:00:01",
+          "BackoffType": "Exponential",
+          "UseJitter": true,
+          "MaxRetryDelay": "00:00:32",
           "EnableLogging": false,
           "EnableMessageLogging": false,
           "EnableMessageContentLogging": false