Fix Azure AI Search Embedding (#83)

Author: MikeAlhayek

src/CrestApps.Core.Docs/docs/changelog/v1.0.0.md

@@ -72,3 +72,5 @@ description: Initial standalone release notes for the CrestApps.Core repository.
 - distinguishes uploaded vision images from searchable documents in the shared document-availability prompt so multimodal chat sessions analyze supported attached images directly instead of defaulting to document-tool or metadata-only responses
 - caps the total uploaded vision-image bytes loaded into a single multimodal request through `ChatDocumentsOptions.MaxVisionInputBytesPerRequest`, removes the extra `MemoryStream` copy when attaching those images, and documents how to resolve a vision-capable chat client for direct image-description requests
 - adds `CrestApps.Core.PostgreSQL` and `CrestApps.Core.AI.PostgreSQL` packages providing a lightweight PostgreSQL + pgvector vector search backend as an alternative to Elasticsearch and Azure AI Search, registers the same keyed services (`ISearchIndexManager`, `ISearchDocumentManager`, `IDataSourceContentManager`, `IDataSourceDocumentReader`, `IODataFilterTranslator`) under the `"PostgreSQL"` provider name, supports `AddAIDocuments()`, `AddAIDataSources()`, and `AddAIMemory()` builder extensions, and integrates into both MVC and Blazor sample hosts
+- fixes hosted document and data-source indexing flows so background workers create a scoped service provider before resolving scoped indexing services, preventing upload-triggered failures and similar nightly alignment lifetime issues
+- standardizes Azure AI Search configuration on top-level `AuthenticationType`, `ApiKey`, `IdentityClientId`, and `IndexPrefix` settings under `CrestApps:AzureAISearch`, and refreshes the sample host / docs examples to list the full supported option set in one place

src/CrestApps.Core.Docs/docs/data-sources/azure-ai.md

@@ -29,11 +29,12 @@ builder.Services.AddCoreAzureAISearchServices();
 ```json
 {
   "CrestApps": {
-    "Search": {
-      "AzureAISearch": {
-        "Endpoint": "https://my-search.search.windows.net",
-        "ApiKey": "your-admin-api-key"
-      }
+    "AzureAISearch": {
+      "Endpoint": "https://my-search.search.windows.net",
+      "AuthenticationType": "Default",
+      "ApiKey": "",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
     }
   }
 }
@@ -44,7 +45,10 @@ builder.Services.AddCoreAzureAISearchServices();
 | Property | Type | Description |
 |----------|------|-------------|
 | `Endpoint` | `string` | Azure AI Search endpoint URL |
-| `ApiKey` | `string` | Admin API key. If empty, uses `DefaultAzureCredential` |
+| `AuthenticationType` | `string` | Authentication mode. Supported values: `Default`, `ApiKey`, `ManagedIdentity` |
+| `ApiKey` | `string` | Admin API key. Required when `AuthenticationType` is `ApiKey` |
+| `IdentityClientId` | `string` | Optional managed identity client ID used by `DefaultAzureCredential` or `ManagedIdentityCredential` |
+| `IndexPrefix` | `string` | Optional prefix applied to framework-managed Azure AI Search index names |
 
 ## Services Registered (Keyed by `"AzureAISearch"`)
 
@@ -74,8 +78,11 @@ Override `IAIDataSourceIndexingQueue` when you need a durable or distributed que
 
 ## Authentication
 
-- **API Key** — Provide the `ApiKey` property
-- **Azure AD** — Leave `ApiKey` empty and the service uses `DefaultAzureCredential` (Managed Identity, VS credentials, etc.)
+Set `AuthenticationType` to one of these values:
+
+- **`Default`** — Uses `DefaultAzureCredential`. This is the default when `AuthenticationType` is omitted or invalid.
+- **`ApiKey`** — Uses the admin API key from `ApiKey`.
+- **`ManagedIdentity`** — Uses `ManagedIdentityCredential`. Set `IdentityClientId` when you need a user-assigned managed identity.
 
 ## Azure Setup
 
@@ -104,11 +111,12 @@ The simplest approach — provide the admin API key directly:
 ```json title="appsettings.json"
 {
   "CrestApps": {
-    "Search": {
-      "AzureAISearch": {
-        "Endpoint": "https://myapp-search.search.windows.net",
-        "ApiKey": "your-admin-api-key"
-      }
+    "AzureAISearch": {
+      "Endpoint": "https://myapp-search.search.windows.net",
+      "AuthenticationType": "ApiKey",
+      "ApiKey": "your-admin-api-key",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
     }
   }
 }
@@ -125,10 +133,12 @@ Leave `ApiKey` empty and the service uses `DefaultAzureCredential`, which automa
 ```json title="appsettings.json"
 {
   "CrestApps": {
-    "Search": {
-      "AzureAISearch": {
-        "Endpoint": "https://myapp-search.search.windows.net"
-      }
+    "AzureAISearch": {
+      "Endpoint": "https://myapp-search.search.windows.net",
+      "AuthenticationType": "Default",
+      "ApiKey": "",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
     }
   }
 }
@@ -141,6 +151,28 @@ Leave `ApiKey` empty and the service uses `DefaultAzureCredential`, which automa
 3. **Visual Studio / VS Code credentials** (for local development)
 4. **Azure CLI** (`az login`)
 
+If you want to prefer a specific user-assigned managed identity while still using `DefaultAzureCredential`, set `IdentityClientId`.
+
+### Option 3: Managed Identity Only
+
+Set `AuthenticationType` to `ManagedIdentity` when you want Azure AI Search to authenticate only with managed identity credentials:
+
+```json title="appsettings.json"
+{
+  "CrestApps": {
+    "AzureAISearch": {
+      "Endpoint": "https://myapp-search.search.windows.net",
+      "AuthenticationType": "ManagedIdentity",
+      "ApiKey": "",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
+    }
+  }
+}
+```
+
+Leave `IdentityClientId` empty for a system-assigned managed identity, or set it to the client ID of a user-assigned managed identity.
+
 To use Managed Identity:
 1. Enable system-assigned managed identity on your Azure App Service.
 2. In the Azure AI Search resource, go to **Access Control (IAM)** → **Add role assignment**.
@@ -153,11 +185,12 @@ To use Managed Identity:
 ```json
 {
   "CrestApps": {
-    "Search": {
-      "AzureAISearch": {
-        "Endpoint": "https://myapp-search.search.windows.net",
-        "ApiKey": "your-admin-api-key"
-      }
+    "AzureAISearch": {
+      "Endpoint": "https://myapp-search.search.windows.net",
+      "AuthenticationType": "Default",
+      "ApiKey": "",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
     }
   }
 }
@@ -168,7 +201,10 @@ To use Managed Identity:
 | Property | Type | Required | Default | Description |
 |----------|------|----------|---------|-------------|
 | `Endpoint` | `string` | Yes | — | Azure AI Search endpoint URL. Format: `https://{service-name}.search.windows.net` |
-| `ApiKey` | `string` | No | — | Admin API key. When empty, `DefaultAzureCredential` is used for authentication. |
+| `AuthenticationType` | `string` | No | `Default` | Supported values: `Default`, `ApiKey`, `ManagedIdentity`. |
+| `ApiKey` | `string` | No | — | Admin API key. Required when `AuthenticationType` is `ApiKey`. |
+| `IdentityClientId` | `string` | No | — | Optional managed identity client ID used by `DefaultAzureCredential` or `ManagedIdentityCredential`. |
+| `IndexPrefix` | `string` | No | — | Optional prefix applied to framework-managed index names. |
 
 :::info
 When `Endpoint` is provided, the framework registers a `SearchIndexClient` singleton that all keyed services share. If `Endpoint` is empty or null, no client is registered and the data source is effectively disabled.
@@ -187,6 +223,10 @@ curl -H "api-key: your-admin-api-key" \
 
 A successful response returns a JSON list of indexes (possibly empty).
 
+:::info
+The framework writes embeddings into the `embedding` vector field for AI Documents, AI Memory, and AI Data Sources. In Azure Search Explorer, make sure the vector field is retrievable and clear **Hide vector values in search results** if you want the raw float array to appear in the portal response.
+:::
+
 ### 2. Verify from the Application
 
 Inject `ISearchIndexManager` (keyed by `"AzureAISearch"`) and check if the connection is live:

src/CrestApps.Core.Docs/docs/data-sources/index.md

@@ -433,21 +433,22 @@ All six services must be registered with the same `providerName` key. The framew
 
 ## Configuration Guide
 
-Data source backends are configured in `appsettings.json` under the `CrestApps:Search` section. Each backend has its own configuration section:
+Data source backends are configured in `appsettings.json` under the `CrestApps` section. Each backend has its own configuration subsection:
 
 ```json
 {
   "CrestApps": {
-    "Search": {
-      "Elasticsearch": {
-        "Url": "https://localhost:9200",
-        "Username": "elastic",
-        "Password": "your-password"
-      },
-      "AzureAISearch": {
-        "Endpoint": "https://my-search.search.windows.net",
-        "ApiKey": "your-admin-api-key"
-      }
+    "Elasticsearch": {
+      "Url": "https://localhost:9200",
+      "Username": "elastic",
+      "Password": "your-password"
+    },
+    "AzureAISearch": {
+      "Endpoint": "https://my-search.search.windows.net",
+      "AuthenticationType": "ApiKey",
+      "ApiKey": "your-admin-api-key",
+      "IdentityClientId": "",
+      "IndexPrefix": ""
     }
   }
 }

src/Primitives/CrestApps.Core.AI.Documents/Handlers/DocumentPreemptiveRagHandler.cs

@@ -260,15 +260,7 @@ context.Resource is not AIProfile ||
             return [];
         }
 
-        if (!indexProfile.TryGet(out DataSourceIndexProfileMetadata metadata))
-        {
-            if (_logger.IsEnabled(LogLevel.Debug))
-            {
-                _logger.LogDebug("Unable to retrieve embedding configuration from index profile '{IndexProfileName}'.", settings.IndexProfileName);
-            }
-
-            return [];
-        }
+        indexProfile.TryGet(out DataSourceIndexProfileMetadata metadata);
 
         var deploymentName = metadata?.EmbeddingDeploymentName ?? indexProfile.EmbeddingDeploymentName;
 

src/Primitives/CrestApps.Core.AI/Indexing/SearchIndexProfileProvisioningService.cs

@@ -100,7 +100,7 @@ public async Task<ValidationResultDetails> CreateAsync(SearchIndexProfile profil
                 profile.IndexFullName.SanitizeForLog(),
                 profile.ProviderName.SanitizeForLog());
 
-            return Fail($"Unable to validate whether the remote index '{profile.IndexFullName}' already exists.", nameof(SearchIndexProfile.IndexName));
+            return Fail(GetRemoteIndexValidationErrorMessage(profile, ex), nameof(SearchIndexProfile.IndexName));
         }
 
         try
@@ -141,4 +141,41 @@ private static ValidationResultDetails Fail(string message, params string[] memb
 
         return result;
     }
+
+    private static string GetRemoteIndexValidationErrorMessage(SearchIndexProfile profile, Exception ex)
+    {
+        if (TryGetRequestFailedStatusCode(ex, out var statusCode) && (statusCode == 401 || statusCode == 403))
+        {
+            return $"Unable to validate whether the remote index '{profile.IndexFullName}' already exists because the remote search provider rejected the configured credentials. Verify the endpoint and use credentials with index management permissions.";
+        }
+
+        return $"Unable to validate whether the remote index '{profile.IndexFullName}' already exists.";
+    }
+
+    private static bool TryGetRequestFailedStatusCode(Exception ex, out int statusCode)
+    {
+        statusCode = default;
+
+        var exceptionType = ex.GetType();
+        if (!string.Equals(exceptionType.FullName, "Azure.RequestFailedException", StringComparison.Ordinal))
+        {
+            return false;
+        }
+
+        var statusProperty = exceptionType.GetProperty("Status");
+        if (statusProperty?.PropertyType != typeof(int))
+        {
+            return false;
+        }
+
+        var value = statusProperty.GetValue(ex);
+        if (value is not int typedStatusCode)
+        {
+            return false;
+        }
+
+        statusCode = typedStatusCode;
+
+        return true;
+    }
 }

src/Primitives/CrestApps.Core.AI/Services/AIDataSourceAlignmentBackgroundService.cs

@@ -90,9 +90,9 @@ private bool ShouldRunAlignment(out DateOnly runDateUtc)
                     _lastRunDateUtc != runDateUtc;
     }
 
-    private async Task AlignDataSourcesAsync(IServiceProvider services, CancellationToken cancellationToken)
+    private async Task AlignDataSourcesAsync(IServiceProvider serviceProvider, CancellationToken cancellationToken)
     {
-        var dataSourceStore = services.GetService<IAIDataSourceStore>();
+        var dataSourceStore = serviceProvider.GetService<IAIDataSourceStore>();
         if (dataSourceStore == null)
         {
             if (_logger.IsEnabled(LogLevel.Trace))
@@ -112,7 +112,7 @@ private async Task AlignDataSourcesAsync(IServiceProvider services, Cancellation
             return;
         }
 
-        var indexingService = services.GetRequiredService<IAIDataSourceIndexingService>();
+        var indexingService = serviceProvider.GetRequiredService<IAIDataSourceIndexingService>();
 
         if (_logger.IsEnabled(LogLevel.Information))
         {

src/Primitives/CrestApps.Core.Azure.AISearch/AzureAISearchConnectionOptions.cs

@@ -1,3 +1,5 @@
+using System;
+
 namespace CrestApps.Core.Azure.AISearch;
 
 /// <summary>
@@ -6,19 +8,97 @@ namespace CrestApps.Core.Azure.AISearch;
 /// </summary>
 public sealed class AzureAISearchConnectionOptions
 {
+    /// <summary>
+    /// The default Azure AI Search authentication type value.
+    /// </summary>
+    public const string DefaultAuthenticationType = "Default";
+
+    /// <summary>
+    /// The API key Azure AI Search authentication type value.
+    /// </summary>
+    public const string ApiKeyAuthenticationType = "ApiKey";
+
+    /// <summary>
+    /// The managed identity Azure AI Search authentication type value.
+    /// </summary>
+    public const string ManagedIdentityAuthenticationType = "ManagedIdentity";
+
     /// <summary>
     /// The Azure AI Search service endpoint (e.g. "https://my-search.search.windows.net").
     /// </summary>
     public string Endpoint { get; set; }
 
     /// <summary>
     /// The admin API key used for authentication.
-    /// When empty, <c>DefaultAzureCredential</c> is used instead.
+    /// When empty, <c>DefaultAzureCredential</c> is used instead unless <see cref="AuthenticationType"/>
+    /// explicitly requires API key authentication.
     /// </summary>
     public string ApiKey { get; set; }
 
     /// <summary>
     /// Optional prefix applied to MVC-managed remote index names.
     /// </summary>
     public string IndexPrefix { get; set; }
+
+    /// <summary>
+    /// Optional authentication mode.
+    /// Supported values are <c>Default</c>, <c>ApiKey</c>, and <c>ManagedIdentity</c>.
+    /// </summary>
+    public string AuthenticationType { get; set; }
+
+    /// <summary>
+    /// Optional managed identity client ID used when <c>DefaultAzureCredential</c> authenticates with Azure.
+    /// </summary>
+    public string IdentityClientId { get; set; }
+
+    /// <summary>
+    /// Backward-compatible alias for <see cref="IndexPrefix"/>.
+    /// </summary>
+    public string IndexesPrefix { get; set; }
+
+    /// <summary>
+    /// Gets the configured index prefix, including backward-compatible aliases.
+    /// </summary>
+    public string GetResolvedIndexPrefix()
+    {
+        if (!string.IsNullOrWhiteSpace(IndexPrefix))
+        {
+            return IndexPrefix;
+        }
+
+        return IndexesPrefix;
+    }
+
+    /// <summary>
+    /// Gets a value indicating whether API key authentication was selected explicitly.
+    /// </summary>
+    public bool UsesApiKeyAuthentication()
+    {
+        return string.Equals(GetAuthenticationType(), ApiKeyAuthenticationType, StringComparison.OrdinalIgnoreCase);
+    }
+
+    /// <summary>
+    /// Gets the configured authentication type.
+    /// </summary>
+    public string GetAuthenticationType()
+    {
+        if (string.IsNullOrWhiteSpace(AuthenticationType))
+        {
+            return DefaultAuthenticationType;
+        }
+
+        var normalizedAuthenticationType = AuthenticationType.Trim();
+
+        if (string.Equals(normalizedAuthenticationType, ApiKeyAuthenticationType, StringComparison.OrdinalIgnoreCase))
+        {
+            return ApiKeyAuthenticationType;
+        }
+
+        if (string.Equals(normalizedAuthenticationType, ManagedIdentityAuthenticationType, StringComparison.OrdinalIgnoreCase))
+        {
+            return ManagedIdentityAuthenticationType;
+        }
+
+        return DefaultAuthenticationType;
+    }
 }