[Internal] Direct package: Adds msdata/direct update from master

.github/copilot-instructions.md

@@ -0,0 +1,58 @@
+<!-- .github/copilot-instructions.md -->
+# Copilot / AI assistant instructions — Microsoft.Azure.Cosmos
+
+Purpose: quick, actionable context so an AI coding assistant can be immediately productive in this repo.
+
+- **Big picture**: This repository implements the v3 .NET SDK for Azure Cosmos DB. Major components:
+  - `Microsoft.Azure.Cosmos/` — core SDK client, most production code.
+  - `Microsoft.Azure.Cosmos.Encryption/` and `Microsoft.Azure.Cosmos.Encryption.Custom/` — client-side encryption extensions.
+  - `Microsoft.Azure.Cosmos.Samples/` — runnable examples and usage patterns.
+  - `docs/`, `templates/`, and top-level Azure Pipelines YAMLs — CI, packaging and emulator setup.
+
+- **Why this structure**: the SDK core is separated from optional features (encryption, fault-injection, direct mode) so consumers can opt into smaller packages. Versioning and feature flags are centralized in `Directory.Build.props`.
+
+- **Build & test (most common workflows)**:
+  - Build solution: `dotnet build Microsoft.Azure.Cosmos.sln -c Release` (or simply `dotnet build`).
+  - Run unit/integration tests: `dotnet test --no-build` in the solution or specific test project folders under `**/tests/`.
+  - CI uses the YAML files in the repository root and `templates/` — see `templates/emulator-setup.yml` for the Windows emulator script used in CI.
+
+- **Local emulator and integration testing**:
+  - The codebase expects the Windows Cosmos DB Emulator in many integration tests. CI installs/starts it via `templates/emulator-setup.yml` (PowerShell scripts that download and launch the MSI and call `Start-Process CosmosDB.Emulator.exe`).
+  - If running tests locally on Windows, install the emulator and ensure exclusions and local state paths match what's in `templates/emulator-setup.yml`.
+
+- **Versioning & build flags**:
+  - `Directory.Build.props` (repo root and project-level overrides) contains the canonical package versions and MSBuild flags (e.g. `<ClientOfficialVersion>`, `<LangVersion>`, and `DefineConstants` that add `PREVIEW`/`ENCRYPTIONPREVIEW`).
+  - Feature/preview builds are gated by MSBuild properties like `IsPreview` or `IsNightly`; set these via `dotnet msbuild /p:IsPreview=true` when needed.
+
+- **Conventions & patterns** (project-specific)
+  - Avoid introducing new global build properties; add versions to `Directory.Build.props` where applicable.
+  - Tests use the emulator or mocks; integration tests that depend on emulator are usually under `tests/` and expect environment-based setup. Look for CI templates for exact start-up sequence.
+  - Strong-name signing keys exist at repo root (`35MSSharedLib1024.snk`, `testkey.snk`); builds may require signing configuration on CI.
+
+- **Integration points & external deps**:
+  - Azure Cosmos DB Emulator (Windows) — required for many integration tests.
+  - NuGet packaging and pipeline tooling — see `templates/nuget-pack.yml` and the many `azure-pipelines-*.yml` files for packaging/release behavior.
+
+- **Where to look for examples** (use these as source-of-truth snippets):
+  - `Directory.Build.props` — versioning and define-constants
+  - `templates/emulator-setup.yml` — exact emulator install/start PowerShell used in CI
+  - `Microsoft.Azure.Cosmos/` — core SDK patterns (public APIs, partitioning, feed iterator usage)
+  - `Microsoft.Azure.Cosmos.Samples/` — minimal runnable samples for usage patterns
+
+- **How AI should produce code/changes here**:
+  - **🚫 HARD RULE: NEVER push directly to `master` — NO EXCEPTIONS.** Always create a feature branch and submit a pull request. This rule cannot be overridden.
+  - Keep changes minimal and focused; prefer small, targeted edits and follow existing code style.
+  - When suggesting build/test changes, reference the relevant MSBuild property or pipeline YAML (point to `Directory.Build.props` or `templates/*`).
+  - Do not change version numbers or packaging settings without explicit instruction — these are centrally managed.
+  - If adding or modifying tests that require the emulator, include/update relevant CI/template steps and document required environment variables.
+
+- **Quick examples to reference in suggestions**:
+  - Use `FeedIterator<T>` patterns as in `Microsoft.Azure.Cosmos` when generating query examples.
+  - For emulator-driven tests, mirror the startup sequence from `templates/emulator-setup.yml`.
+
+- **Named Copilot Agents**:
+  - **IssueFixAgent** (`.github/agents/issue-fix-agent.agent.md`) — Comprehensive workflow for triaging and fixing GitHub issues. Use `@IssueFixAgent` in VS Code Copilot Chat or follow the plan manually.
+  - Includes: environment setup, issue investigation, fix implementation, testing requirements, PR workflow, and learnings capture.
+  - Start with: `@IssueFixAgent investigate issue #XXXX` or see Quick Start in the agent file.
+
+If anything here is unclear or you want the file to include additional examples (specific files, common refactor targets, or typical PR reviewers), tell me what to add and I will iterate.

.gitignore

@@ -326,5 +326,11 @@ ASALocalRun/
 # NVidia Nsight GPU debugger configuration file
 *.nvuser
 
-# MFractors (Xamarin productivity tool) working folder 
+# MFractors (Xamarin productivity tool) working folder
 .mfractor/
+
+# Local scratch/test projects (not part of the SDK)
+ChangeFeedWithFaultInjection/
+ConsoleApp1/
+ItemGenerator/
+Backup/

.vscode/extensions.json

@@ -0,0 +1,6 @@
+{
+  "recommendations": [
+    "ms-azuretools.vscode-azurefunctions",
+    "ms-dotnettools.csharp"
+  ]
+}

.vscode/launch.json

@@ -0,0 +1,11 @@
+{
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Attach to .NET Functions",
+            "type": "coreclr",
+            "request": "attach",
+            "processId": "${command:azureFunctions.pickProcess}"
+        }
+    ]
+}

.vscode/settings.json

@@ -0,0 +1,7 @@
+{
+    "azureFunctions.deploySubpath": "Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions/bin/Release/net6.0/publish",
+    "azureFunctions.projectLanguage": "C#",
+    "azureFunctions.projectRuntime": "~4",
+    "debug.internalConsoleOptions": "neverOpen",
+    "azureFunctions.preDeployTask": "publish (functions)"
+}

.vscode/tasks.json

@@ -0,0 +1,81 @@
+{
+	"version": "2.0.0",
+	"tasks": [
+		{
+			"label": "clean (functions)",
+			"command": "dotnet",
+			"args": [
+				"clean",
+				"/property:GenerateFullPaths=true",
+				"/consoleloggerparameters:NoSummary"
+			],
+			"type": "process",
+			"problemMatcher": "$msCompile",
+			"options": {
+				"cwd": "${workspaceFolder}/Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions"
+			}
+		},
+		{
+			"label": "build (functions)",
+			"command": "dotnet",
+			"args": [
+				"build",
+				"/property:GenerateFullPaths=true",
+				"/consoleloggerparameters:NoSummary"
+			],
+			"type": "process",
+			"dependsOn": "clean (functions)",
+			"group": {
+				"kind": "build",
+				"isDefault": true
+			},
+			"problemMatcher": "$msCompile",
+			"options": {
+				"cwd": "${workspaceFolder}/Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions"
+			}
+		},
+		{
+			"label": "clean release (functions)",
+			"command": "dotnet",
+			"args": [
+				"clean",
+				"--configuration",
+				"Release",
+				"/property:GenerateFullPaths=true",
+				"/consoleloggerparameters:NoSummary"
+			],
+			"type": "process",
+			"problemMatcher": "$msCompile",
+			"options": {
+				"cwd": "${workspaceFolder}/Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions"
+			}
+		},
+		{
+			"label": "publish (functions)",
+			"command": "dotnet",
+			"args": [
+				"publish",
+				"--configuration",
+				"Release",
+				"/property:GenerateFullPaths=true",
+				"/consoleloggerparameters:NoSummary"
+			],
+			"type": "process",
+			"dependsOn": "clean release (functions)",
+			"problemMatcher": "$msCompile",
+			"options": {
+				"cwd": "${workspaceFolder}/Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions"
+			}
+		},
+		{
+			"type": "func",
+			"dependsOn": "build (functions)",
+			"options": {
+				"cwd": "${workspaceFolder}/Microsoft.Azure.Cosmos.Samples\\Usage\\AzureFunctions/bin/Debug/net6.0"
+			},
+			"command": "host start",
+			"isBackground": true,
+			"problemMatcher": "$func-dotnet-watch"
+		}
+	]
+}

Directory.Build.props

@@ -1,9 +1,9 @@
 <Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
 	<PropertyGroup>
-		<ClientOfficialVersion>3.52.1</ClientOfficialVersion>
-		<ClientPreviewVersion>3.53.0</ClientPreviewVersion>
-		<ClientPreviewSuffixVersion>preview.1</ClientPreviewSuffixVersion>
-		<DirectVersion>3.39.1</DirectVersion>
+		<ClientOfficialVersion>3.57.0</ClientOfficialVersion>
+		<ClientPreviewVersion>3.58.0</ClientPreviewVersion>
+		<ClientPreviewSuffixVersion>preview.0</ClientPreviewSuffixVersion>
+		<DirectVersion>3.41.3</DirectVersion>
 		<FaultInjectionVersion>1.0.0</FaultInjectionVersion>
 		<FaultInjectionSuffixVersion>beta.0</FaultInjectionSuffixVersion>
 		<EncryptionOfficialVersion>2.0.5</EncryptionOfficialVersion>

Exceptions.md

@@ -33,3 +33,60 @@ Cosmos DB SDK on any IO failure will attempt to retry the failed operation if re
 ## Common error status codes and troubleshooting guide <a id="error-codes"></a>
 
 To see a list of common error code and issues please see [.NET SDK troubleshooting guide](https://docs.microsoft.com/azure/cosmos-db/troubleshoot-dot-net-sdk)
+
+### Disambiguating 404 (Not Found) errors using SubStatusCode <a id="substatus-codes"></a>
+
+When you receive a 404 (Not Found) status code from Cosmos DB, it can indicate two different scenarios:
+1. **Item not found**: The requested item doesn't exist in the container
+2. **Owner resource not found**: The parent resource (container or database) doesn't exist
+
+To distinguish between these cases, check the `SubStatusCode` property:
+
+- **SubStatusCode 0**: Regular item not found (the item doesn't exist in an existing container)
+- **SubStatusCode 1003**: Owner resource not found (the container or database doesn't exist)
+
+#### Example with Typed APIs (throws CosmosException):
+
+```csharp
+try
+{
+    ItemResponse<MyItem> response = await container.ReadItemAsync<MyItem>("itemId", new PartitionKey("partitionKey"));
+}
+catch (CosmosException ex) when (ex.StatusCode == HttpStatusCode.NotFound)
+{
+    if (ex.SubStatusCode == 1003)
+    {
+        // The container or database doesn't exist
+        Console.WriteLine("Owner resource (container/database) not found");
+    }
+    else
+    {
+        // The item doesn't exist in an existing container
+        Console.WriteLine("Item not found");
+    }
+}
+```
+
+#### Example with Stream APIs (returns ResponseMessage):
+
+```csharp
+ResponseMessage response = await container.ReadItemStreamAsync("itemId", new PartitionKey("partitionKey"));
+
+if (response.StatusCode == HttpStatusCode.NotFound)
+{
+    int subStatusCode = (int)response.Headers.SubStatusCode;
+
+    if (subStatusCode == 1003)
+    {
+        // The container or database doesn't exist
+        Console.WriteLine("Owner resource (container/database) not found");
+    }
+    else
+    {
+        // The item doesn't exist in an existing container
+        Console.WriteLine("Item not found");
+    }
+}
+```
+
+This distinction is particularly useful when implementing retry logic or error handling strategies, as you may want to handle these scenarios differently (e.g., creating the container if it doesn't exist vs. handling a missing item).

Microsoft.Azure.Cosmos.Encryption.Custom/src/AeadAes/SymmetricKey.cs

@@ -23,10 +23,11 @@ internal class SymmetricKey
         /// <param name="rootKey">root key</param>
         internal SymmetricKey(byte[] rootKey)
         {
-            // Key validation
-            if (rootKey == null || rootKey.Length == 0)
+            ArgumentValidation.ThrowIfNull(rootKey);
+
+            if (rootKey.Length == 0)
             {
-                throw new ArgumentNullException(nameof(rootKey));
+                throw new ArgumentException("The root key cannot be empty.", nameof(rootKey));
             }
 
             this.rootKey = rootKey;

Microsoft.Azure.Cosmos.Encryption.Custom/src/Common/CosmosDiagnosticsContext.cs

@@ -5,32 +5,84 @@
 namespace Microsoft.Azure.Cosmos.Encryption.Custom
 {
     using System;
+    using System.Collections.Generic;
+    using System.Diagnostics;
+    using System.Threading;
 
     /// <summary>
-    /// This is an empty implementation of CosmosDiagnosticsContext which has been plumbed through the DataEncryptionKeyProvider and EncryptionContainer.
-    /// This may help adding diagnostics more easily in future.
+    /// Lightweight diagnostics context for Custom Encryption extension.
+    /// Manages Activity creation for OpenTelemetry integration.
     /// </summary>
     internal class CosmosDiagnosticsContext
     {
-        private static readonly CosmosDiagnosticsContext UnusedSingleton = new ();
-        private static readonly IDisposable UnusedScopeSingleton = new Scope();
+        private static readonly ActivitySource ActivitySource = new ("Microsoft.Azure.Cosmos.Encryption.Custom");
 
+        /// <summary>
+        /// Scope name prefix for MDE (Microsoft.Data.Encryption) encrypt operations.
+        /// </summary>
+        internal const string ScopeEncryptModeSelectionPrefix = "EncryptionProcessor.Encrypt.Mde.";
+
+        /// <summary>
+        /// Scope name prefix for MDE (Microsoft.Data.Encryption) decrypt operations.
+        /// </summary>
+        internal const string ScopeDecryptModeSelectionPrefix = "EncryptionProcessor.Decrypt.Mde.";
+
+        internal CosmosDiagnosticsContext()
+        {
+        }
+
+        /// <summary>
+        /// Creates a new diagnostics context instance.
+        /// </summary>
         public static CosmosDiagnosticsContext Create(RequestOptions options)
         {
             _ = options;
-            return CosmosDiagnosticsContext.UnusedSingleton;
+            return new CosmosDiagnosticsContext();
         }
 
-        public IDisposable CreateScope(string scope)
+        /// <summary>
+        /// Creates a new diagnostic scope for Activity tracking.
+        /// </summary>
+        /// <param name="scope">The name of the scope.</param>
+        /// <returns>A <see cref="Scope"/> that manages an Activity lifecycle.</returns>
+        /// <exception cref="ArgumentNullException">Thrown when <paramref name="scope"/> is null.</exception>
+        /// <exception cref="ArgumentException">Thrown when <paramref name="scope"/> is empty or whitespace.</exception>
+        /// <remarks>
+        /// Use with a <c>using</c> statement to ensure proper disposal.
+        /// </remarks>
+        public Scope CreateScope(string scope)
         {
-            _ = scope;
-            return CosmosDiagnosticsContext.UnusedScopeSingleton;
+            ArgumentValidation.ThrowIfNullOrWhiteSpace(scope, nameof(scope));
+
+            Activity activity = ActivitySource.HasListeners() ? ActivitySource.StartActivity(scope, ActivityKind.Internal) : null;
+
+            return new Scope(activity);
         }
 
-        private class Scope : IDisposable
+        /// <summary>
+        /// Represents a diagnostic scope for Activity tracking.
+        /// Must be used with the 'using' pattern to ensure proper disposal.
+        /// </summary>
+        /// <remarks>
+        /// Dispose() is idempotent - calling it multiple times will only dispose the Activity once.
+        /// </remarks>
+        public sealed class Scope : IDisposable
         {
+            private readonly Activity activity;
+            private bool isDisposed;
+
+            internal Scope(Activity activity)
+            {
+                this.activity = activity;
+            }
+
             public void Dispose()
             {
+                if (!this.isDisposed)
+                {
+                    this.isDisposed = true;
+                    this.activity?.Dispose();
+                }
             }
         }
     }