From f6ae4c482f337a14cd1aa81c9ce25c21b1bf55a2 Mon Sep 17 00:00:00 2001 From: Debdatta Kunda <87335885+kundadebdatta@users.noreply.github.com> Date: Wed, 23 Oct 2024 16:14:27 -0700 Subject: [PATCH] [Internal] Binary Encoding: Adds Binary Encoding Support for Point Operations (#4652) # Pull Request Template # Description This PR introduces binary encoding support on request and responses for different Point operations. ## What is Binary Encoding? As the name suggests, binary encoding is a encoding mechanism through which the request payload will be encoded to binary first and sent to backend for processing. Decoding to Text will happen on the response path. The biggest benefit of binary encoding is to reduce cost on backend storage which helps to reduce the overall COGS. ## Scope The point operations that are currently in and out of scope for binary encoding are given below in tabular format: | Operations Currently in Scope| Operations Currently Out of Scope| Reason for Out of Scope| | --- | --- | --- | | `CreateItemAsync()` | `PatchItemAsync()` | Operation Currently not Supported in BE | `CreateItemStreamAsync()` | `PatchItemStreamAsync()` | Operation Currently not Supported in BE | `ReadItemAsync()` | `TransactionalBatches` | Operation Currently not Supported in BE | `ReadItemStreamAsync()` | `Bulk APIs` | Operation Currently not Supported in BE | `UpsertItemAsync()` | | | | `UpsertItemStreamAsync()` | | | | `RepalceItemAsync()` | | | | `ReplaceItemStreamAsync()` | | | | `DeleteItemAsync()` | | | | `DeleteItemStreamAsync()` | | | ## How to Enable Binary Encoding? This PR introduces a new environment variable `AZURE_COSMOS_BINARY_ENCODING_ENABLED` to opt-in or opt-out the binary encoding feature on demand. Setting this environment variable to `True` will enable Binary encoding support. ## How Binary Encoding has been Achieved? The binary encoding in the .NET SDK has been divided into two parts which are applicable differently for `ItemAsync()` and `ItemStreamAsync()` apis. The details are given below: - **`ItemAsync()` APIs:** Currently the `CosmosJsonDotNetSerializer` has been refactored to read and write the binary bits directly into the stream. This reduces any conversion of the text stream to binary and vice versa and makes the serialization and de-serialization process even faster. - **`ItemStreamAsync()` APIs:** For these APIs, there are literally no serializes involved and the stream is returned directly to the caller. Therefore, this flow converts a Text stream into Binary and does the opposite on the response path. Conversion is a little bit costlier operation, in comparison with directly writing the binary stream using the serializer. Note that, irrespective of the binary encoding feature enabled or disabled, the output stream will always be in Text format, unless otherwise requested explicitly. ## Are There Any Way to Request Binary Bits on Response? The answer is yes. We introduced a new internal request option: `EnableBinaryResponseOnPointOperations` in the `ItemRequestOptions`, and setting this flag to `True` will not do any Text conversation, and will return the raw binary bits to the caller. However, please note that this option is applicable only for the `ItemStreamAsync()` APIs and will be helpful for some of the internal teams. ## Flow Diagrams ## To understand the changes better, please take a look at the flow diagrams below for both `ItemAsync()` and `ItemStreamAsync()` APIs. **Flow Diagram for `ItemAsync()` APIs that are in Scope per the Above Table:** ```mermaid flowchart TD A[All 'ItemAsync' APIs in Scope] -->|SerializerCore.ToStream| B{Select
Serializer} B -->|One| C[CosmosJsonDotNetSerializer] B -->|Two| D[CosmosSystemTextJsonSerializer] B -->|Three| E[Any Custom
Serializer] C -->|Serialize to
Binary Stream| F[ContainerCore
.ProcessItemStreamAsync] D -->|Serialize to
Text Stream| F[ContainerCore
.ProcessItemStreamAsync] E -->|Stream may or
may not be
Serialized to Binary| F[ContainerCore
.ProcessItemStreamAsync] F --> G{Is Input
Stream in
Binary ?} G -->|True| I[ProcessResourceOperationStreamAsync] G -->|False| H[Convert Input Text
Stream to
Binary Stream] H --> I I --> |SendAsync| J[RequestInvokerHandler] J --> |Sets following headers to request response in binary format: x-ms-cosmos-supported-serialization-formats = CosmosBinary x-ms-documentdb-content-serialization-format = CosmosBinary| K[TransportHandler] K --> |Binary Response
Stream|L[ContainerCore
.ProcessItemStreamAsync] L --> |Note: No explicit conversion to binary stream happens because we let the serializer directly de-serialize the binary stream into text. SerializerCore.FromStream| M{Select
Serializer} M -->|One| N[CosmosJsonDotNetSerializer] M -->|Two| O[CosmosSystemTextJsonSerializer] M -->|Three| P[Any Custom
Serializer] N -->|De-Serialize to
Text Stream| Q[Container
.ItemAsync Response] O -->|De-Serialize to
Text Stream| Q[Container
.ItemAsync Response] P -->|Stream may or
may not be
De-Serialized to Text| Q[Container
.ItemAsync Response
in Text] ``` **Flow Diagram for `ItemStreamAsync()` APIs that are in Scope per the Above Table:** ```mermaid flowchart TD A[All 'ItemStreamAsync' APIs in Scope] A -->|Stream may or
may not be
Serialized to Binary| F[ContainerCore
.ProcessItemStreamAsync] F --> G{Is Input
Stream in
Binary ?} G -->|True| I[ProcessResourceOperationStreamAsync] G -->|False| H[Convert Input Text
Stream to
Binary Stream] H --> I I --> |SendAsync| J[RequestInvokerHandler] J --> |Sets following headers to get binary response: x-ms-cosmos-supported-serialization-formats = CosmosBinary x-ms-documentdb-content-serialization-format = CosmosBinary| K[TransportHandler] K --> |Binary Response
Stream|L[ContainerCore
.ProcessItemStreamAsync] L --> M{Is Response
Stream in
Binary ?} M -->|Yes| N[CosmosSerializationUtil] M -->|No| Q N -->|Convert Binary Stream to
Text Stream| Q[Container
.ItemAsync Response] ``` ## Performance Testing Below are the comparison results for the perf testing done on the master branch and the current feature branch with binary encoding disabled: ``` ini BenchmarkDotNet=v0.13.5, OS=ubuntu 20.04 Intel Xeon Platinum 8272CL CPU 2.60GHz, 1 CPU, 16 logical and 8 physical cores .NET SDK=6.0.427 [Host] : .NET 6.0.35 (6.0.3524.45918), X64 RyuJIT AVX2 LongRun : .NET 6.0.35 (6.0.3524.45918), X64 RyuJIT AVX2 Job=LongRun IterationCount=100 LaunchCount=3 RunStrategy=Throughput WarmupCount=15 ``` **Benchmark Results with No Binary Encoding on master branch:** ![image](https://github.com/user-attachments/assets/3fa3407d-1fbd-45f9-9be0-e0257fc055da) **Benchmark Results with Binary Encoding Disabled on feature branch:** ![image](https://github.com/user-attachments/assets/dbac9e9a-ba1f-48e7-b231-1fe318967ab5) Benchmark results comparison in terms of percentage between `master` and `feature` branch: ![image](https://github.com/user-attachments/assets/6112d018-0801-4384-bf11-4c093da0ef23) ## Type of change Please delete options that are not relevant. - [x] New feature (non-breaking change which adds functionality) ## Closing issues To automatically close an issue: closes #4644 --- .../src/Handler/RequestInvokerHandler.cs | 25 + .../Interop/CosmosDBToNewtonsoftReader.cs | 8 +- .../Interop/CosmosDBToNewtonsoftWriter.cs | 3 +- .../src/RequestOptions/ItemRequestOptions.cs | 19 + .../Resource/Container/ContainerCore.Items.cs | 55 +- .../Serializer/CosmosBufferedStreamWrapper.cs | 184 ++ .../Serializer/CosmosJsonDotNetSerializer.cs | 74 +- .../src/Serializer/CosmosSerializationUtil.cs | 75 + .../src/Serializer/CosmosSerializerCore.cs | 43 +- .../CosmosSystemTextJsonSerializer.cs | 42 +- .../src/Util/ConfigurationManager.cs | 43 +- .../CosmosItemTests.cs | 1955 ++++++++++------- .../CosmosBufferedStreamWrapperTests.cs | 230 ++ .../CosmosItemUnitTests.cs | 645 ++++-- .../CosmosSerializationUtilTests.cs | 145 ++ .../Patch/PatchOperationTests.cs | 166 ++ .../Utils/CosmosSerializerUtils.cs | 175 ++ 17 files changed, 2821 insertions(+), 1066 deletions(-) create mode 100644 Microsoft.Azure.Cosmos/src/Serializer/CosmosBufferedStreamWrapper.cs create mode 100644 Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosBufferedStreamWrapperTests.cs create mode 100644 Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosSerializationUtilTests.cs create mode 100644 Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Utils/CosmosSerializerUtils.cs diff --git a/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs b/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs index f903906902..3cb3e6c748 100644 --- a/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs +++ b/Microsoft.Azure.Cosmos/src/Handler/RequestInvokerHandler.cs @@ -24,6 +24,7 @@ namespace Microsoft.Azure.Cosmos.Handlers internal class RequestInvokerHandler : RequestHandler { private static readonly HttpMethod httpPatchMethod = new HttpMethod(HttpConstants.HttpMethods.Patch); + private static readonly string BinarySerializationFormat = SupportedSerializationFormats.CosmosBinary.ToString(); private static (bool, ResponseMessage) clientIsValid = (false, null); private readonly CosmosClient client; @@ -67,6 +68,12 @@ public override async Task SendAsync( request.Headers.Add(HttpConstants.HttpHeaders.Prefer, HttpConstants.HttpHeaderValues.PreferReturnMinimal); } + if (ConfigurationManager.IsBinaryEncodingEnabled() + && RequestInvokerHandler.IsPointOperationSupportedForBinaryEncoding(request)) + { + request.Headers.Add(HttpConstants.HttpHeaders.SupportedSerializationFormats, RequestInvokerHandler.BinarySerializationFormat); + } + await this.ValidateAndSetConsistencyLevelAsync(request); this.SetPriorityLevel(request); @@ -94,6 +101,14 @@ public override async Task SendAsync( ((CosmosTraceDiagnostics)response.Diagnostics).Value.AddOrUpdateDatum("ExcludedRegions", request.RequestOptions.ExcludeRegions); } + if (ConfigurationManager.IsBinaryEncodingEnabled() + && RequestInvokerHandler.IsPointOperationSupportedForBinaryEncoding(request) + && response.Content != null + && response.Content is not CloneableStream) + { + response.Content = await StreamExtension.AsClonableStreamAsync(response.Content, default); + } + return response; } @@ -547,6 +562,16 @@ private static bool IsItemNoRepsonseSet(bool enableContentResponseOnWrite, Opera operationType == OperationType.Patch); } + private static bool IsPointOperationSupportedForBinaryEncoding(RequestMessage request) + { + return request.ResourceType == ResourceType.Document + && (request.OperationType == OperationType.Create + || request.OperationType == OperationType.Replace + || request.OperationType == OperationType.Delete + || request.OperationType == OperationType.Read + || request.OperationType == OperationType.Upsert); + } + private static bool IsClientNoResponseSet(CosmosClientOptions clientOptions, OperationType operationType) { return clientOptions != null diff --git a/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftReader.cs b/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftReader.cs index c2a1a7c6ee..1c2d149f79 100644 --- a/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftReader.cs +++ b/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftReader.cs @@ -192,7 +192,7 @@ public override byte[] ReadAsBytes() public override DateTime? ReadAsDateTime() { this.Read(); - if (this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) + if (this.jsonReader.CurrentTokenType == JsonTokenType.Null || this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) { return null; } @@ -211,7 +211,7 @@ public override byte[] ReadAsBytes() public override DateTimeOffset? ReadAsDateTimeOffset() { this.Read(); - if (this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) + if (this.jsonReader.CurrentTokenType == JsonTokenType.Null || this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) { return null; } @@ -260,7 +260,7 @@ public override byte[] ReadAsBytes() public override string ReadAsString() { this.Read(); - if (this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) + if (this.jsonReader.CurrentTokenType == JsonTokenType.Null || this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) { return null; } @@ -278,7 +278,7 @@ public override string ReadAsString() private double? ReadNumberValue() { this.Read(); - if (this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) + if (this.jsonReader.CurrentTokenType == JsonTokenType.Null || this.jsonReader.CurrentTokenType == JsonTokenType.EndArray) { return null; } diff --git a/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftWriter.cs b/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftWriter.cs index a78ecda558..d81f505b12 100644 --- a/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftWriter.cs +++ b/Microsoft.Azure.Cosmos/src/Json/Interop/CosmosDBToNewtonsoftWriter.cs @@ -266,7 +266,8 @@ public override void WriteValue(bool value) /// The value to write. public override void WriteValue(short value) { - base.WriteValue((long)value); + base.WriteValue((Int16)value); + this.jsonWriter.WriteInt16Value(value); } /// diff --git a/Microsoft.Azure.Cosmos/src/RequestOptions/ItemRequestOptions.cs b/Microsoft.Azure.Cosmos/src/RequestOptions/ItemRequestOptions.cs index d807f42f57..dca9ad46ce 100644 --- a/Microsoft.Azure.Cosmos/src/RequestOptions/ItemRequestOptions.cs +++ b/Microsoft.Azure.Cosmos/src/RequestOptions/ItemRequestOptions.cs @@ -127,6 +127,25 @@ public ConsistencyLevel? ConsistencyLevel /// public DedicatedGatewayRequestOptions DedicatedGatewayRequestOptions { get; set; } + /// + /// Gets or sets the boolean to enable binary response for point operations like Create, Upsert, Read, Patch, and Replace. + /// Setting this option to true will cause the response to be in binary format. This request option will remain internal only + /// since the consumer of thie flag will be the internal components of the cosmos db ecosystem. + /// + /// + /// + /// + /// + /// + /// + /// This is optimal for workloads where the returned resource can be processed in binary format. + /// + internal bool EnableBinaryResponseOnPointOperations { get; set; } + /// /// Fill the CosmosRequestMessage headers with the set properties /// diff --git a/Microsoft.Azure.Cosmos/src/Resource/Container/ContainerCore.Items.cs b/Microsoft.Azure.Cosmos/src/Resource/Container/ContainerCore.Items.cs index 48567603f6..ca7565cd2f 100644 --- a/Microsoft.Azure.Cosmos/src/Resource/Container/ContainerCore.Items.cs +++ b/Microsoft.Azure.Cosmos/src/Resource/Container/ContainerCore.Items.cs @@ -59,7 +59,8 @@ public async Task CreateItemStreamAsync( streamPayload: streamPayload, operationType: OperationType.Create, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -100,7 +101,8 @@ public async Task ReadItemStreamAsync( streamPayload: null, operationType: OperationType.Read, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -117,7 +119,8 @@ public async Task> ReadItemAsync( streamPayload: null, operationType: OperationType.Read, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: default, cancellationToken: cancellationToken); return this.ClientContext.ResponseFactory.CreateItemResponse(response); @@ -136,7 +139,8 @@ public async Task UpsertItemStreamAsync( streamPayload: streamPayload, operationType: OperationType.Upsert, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -178,7 +182,8 @@ public async Task ReplaceItemStreamAsync( streamPayload: streamPayload, operationType: OperationType.Replace, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -225,7 +230,8 @@ public async Task DeleteItemStreamAsync( streamPayload: null, operationType: OperationType.Delete, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -242,7 +248,8 @@ public async Task> DeleteItemAsync( streamPayload: null, operationType: OperationType.Delete, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: default, cancellationToken: cancellationToken); return this.ClientContext.ResponseFactory.CreateItemResponse(response); @@ -853,7 +860,8 @@ private async Task ExtractPartitionKeyAndProcessItemStreamAsync itemStream, operationType, requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: default, cancellationToken: cancellationToken); } @@ -868,7 +876,8 @@ private async Task ExtractPartitionKeyAndProcessItemStreamAsync itemStream, operationType, requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: default, cancellationToken: cancellationToken); if (responseMessage.IsSuccessStatusCode) @@ -897,7 +906,8 @@ private async Task ProcessItemStreamAsync( Stream streamPayload, OperationType operationType, ItemRequestOptions requestOptions, - ITrace trace, + ITrace trace, + JsonSerializationFormat? targetResponseSerializationFormat, CancellationToken cancellationToken) { if (trace == null) @@ -912,6 +922,11 @@ private async Task ProcessItemStreamAsync( ContainerInternal.ValidatePartitionKey(partitionKey, requestOptions); string resourceUri = this.GetResourceUri(requestOptions, operationType, itemId); + + // Convert Text to Binary Stream. + streamPayload = CosmosSerializationUtil.TrySerializeStreamToTargetFormat( + targetSerializationFormat: ContainerCore.GetTargetRequestSerializationFormat(), + inputStream: streamPayload == null ? null : await StreamExtension.AsClonableStreamAsync(streamPayload)); ResponseMessage responseMessage = await this.ClientContext.ProcessResourceOperationStreamAsync( resourceUri: resourceUri, @@ -925,6 +940,16 @@ private async Task ProcessItemStreamAsync( requestEnricher: null, trace: trace, cancellationToken: cancellationToken); + + // Convert Binary Stream to Text. + if (targetResponseSerializationFormat.HasValue + && (requestOptions == null || !requestOptions.EnableBinaryResponseOnPointOperations) + && responseMessage?.Content is CloneableStream outputCloneableStream) + { + responseMessage.Content = CosmosSerializationUtil.TrySerializeStreamToTargetFormat( + targetSerializationFormat: targetResponseSerializationFormat.Value, + inputStream: outputCloneableStream); + } return responseMessage; } @@ -1217,7 +1242,8 @@ public Task PatchItemStreamAsync( streamPayload: streamPayload, operationType: OperationType.Patch, requestOptions: requestOptions, - trace: trace, + trace: trace, + targetResponseSerializationFormat: JsonSerializationFormat.Text, cancellationToken: cancellationToken); } @@ -1255,6 +1281,13 @@ private ChangeFeedProcessorBuilder GetChangeFeedProcessorBuilderPrivate( applyBuilderConfiguration: changeFeedProcessor.ApplyBuildConfiguration).WithChangeFeedMode(mode); } + private static JsonSerializationFormat GetTargetRequestSerializationFormat() + { + return ConfigurationManager.IsBinaryEncodingEnabled() + ? JsonSerializationFormat.Binary + : JsonSerializationFormat.Text; + } + /// /// This method is useful for determining if a smaller, more granular feed range (y) is fully contained within a broader feed range (x), which is a common operation in distributed systems to manage partitioned data. /// diff --git a/Microsoft.Azure.Cosmos/src/Serializer/CosmosBufferedStreamWrapper.cs b/Microsoft.Azure.Cosmos/src/Serializer/CosmosBufferedStreamWrapper.cs new file mode 100644 index 0000000000..f59f14931a --- /dev/null +++ b/Microsoft.Azure.Cosmos/src/Serializer/CosmosBufferedStreamWrapper.cs @@ -0,0 +1,184 @@ +//------------------------------------------------------------ +// Copyright (c) Microsoft Corporation. All rights reserved. +//------------------------------------------------------------ + +namespace Microsoft.Azure.Cosmos.Serializer +{ + using System; + using System.IO; + using System.Linq; + using Microsoft.Azure.Cosmos.Json; + using Microsoft.Azure.Documents; + + /// + /// A wrapper for a stream that buffers the first byte. + /// + internal class CosmosBufferedStreamWrapper : Stream + { + /// + /// The inner stream being wrapped. + /// + private readonly CloneableStream innerStream; + + /// + /// Indicates whether the inner stream should be disposed. + /// + private readonly bool shouldDisposeInnerStream; + + /// + /// Buffer to hold the first byte read from the stream. + /// + private readonly byte[] firstByteBuffer = new byte[1]; + + /// + /// Indicates whether the first byte has been read. + /// + private bool hasReadFirstByte; + + /// + /// Initializes a new instance of the class. + /// + /// The input stream to wrap. + /// Indicates whether the inner stream should be disposed. + public CosmosBufferedStreamWrapper( + CloneableStream inputStream, + bool shouldDisposeInnerStream) + { + this.innerStream = inputStream ?? throw new ArgumentNullException(nameof(inputStream)); + this.shouldDisposeInnerStream = shouldDisposeInnerStream; + } + + /// + public override bool CanRead => this.innerStream.CanRead; + + /// + public override bool CanSeek => this.innerStream.CanSeek; + + /// + public override bool CanWrite => this.innerStream.CanWrite; + + /// + public override long Length => this.innerStream.Length; + + /// + public override long Position + { + get => this.innerStream.Position; + set => this.innerStream.Position = value; + } + + /// + public override void Flush() + { + this.innerStream.Flush(); + } + + /// + public override long Seek(long offset, SeekOrigin origin) + { + return this.innerStream.Seek(offset, origin); + } + + /// + public override void SetLength(long value) + { + this.innerStream.SetLength(value); + } + + /// + public override int Read(byte[] buffer, int offset, int count) + { + if (buffer == null) + { + throw new ArgumentNullException(nameof(buffer)); + } + + return this.innerStream.Read(buffer, offset, count); + } + + /// + public override void Write(byte[] buffer, int offset, int count) + { + this.innerStream.Write(buffer, offset, count); + } + + /// + protected override void Dispose(bool disposing) + { + if (disposing) + { + this.Flush(); + if (this.shouldDisposeInnerStream) + { + this.innerStream.Dispose(); + } + else + { + this.ResetStreamPosition(); + } + } + + base.Dispose(disposing); + } + + /// + /// Reads all bytes from the current position to the end of the stream. + /// + /// + /// A byte array containing all the bytes read from the stream, or null if no bytes were read. + /// + public byte[] ReadAll() + { + ArraySegment byteSegment = this.innerStream.GetBuffer(); + + return byteSegment.Array.Length == byteSegment.Count + ? byteSegment.Array + : byteSegment.ToArray(); + } + + /// + /// Determines the JSON serialization format of the stream based on the first byte. + /// + /// + /// The of the stream, which can be Binary, HybridRow, or Text. + /// + public JsonSerializationFormat GetJsonSerializationFormat() + { + this.ReadFirstByteAndResetStream(); + + return this.firstByteBuffer[0] switch + { + (byte)JsonSerializationFormat.Binary => JsonSerializationFormat.Binary, + (byte)JsonSerializationFormat.HybridRow => JsonSerializationFormat.HybridRow, + _ => JsonSerializationFormat.Text, + }; + } + + /// + /// Reads the first byte from the inner stream and stores it in the buffer. It also resets the stream position to zero. + /// + /// + /// This method sets the flag to true if the first byte is successfully read. + /// + private void ReadFirstByteAndResetStream() + { + if (!this.hasReadFirstByte + && this.innerStream.Read(this.firstByteBuffer, 0, 1) > 0) + { + this.hasReadFirstByte = true; + this.ResetStreamPosition(); + } + } + + /// + /// Resets the inner stream position to zero. + /// + private void ResetStreamPosition() + { + if (this.innerStream.CanSeek) + { + this.innerStream.Position = 0; + } + } + } +} diff --git a/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs b/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs index df15f3b5c7..279bf95c71 100644 --- a/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs +++ b/Microsoft.Azure.Cosmos/src/Serializer/CosmosJsonDotNetSerializer.cs @@ -7,6 +7,8 @@ namespace Microsoft.Azure.Cosmos using System; using System.IO; using System.Text; + using Microsoft.Azure.Cosmos.Serializer; + using Microsoft.Azure.Documents; using Newtonsoft.Json; using Newtonsoft.Json.Serialization; @@ -17,6 +19,7 @@ internal sealed class CosmosJsonDotNetSerializer : CosmosSerializer { private static readonly Encoding DefaultEncoding = new UTF8Encoding(false, true); private readonly JsonSerializerSettings SerializerSettings; + private readonly bool BinaryEncodingEnabled; /// /// Create a serializer that uses the JSON.net serializer @@ -25,9 +28,11 @@ internal sealed class CosmosJsonDotNetSerializer : CosmosSerializer /// This is internal to reduce exposure of JSON.net types so /// it is easier to convert to System.Text.Json /// - internal CosmosJsonDotNetSerializer() + internal CosmosJsonDotNetSerializer( + bool binaryEncodingEnabled = false) { this.SerializerSettings = null; + this.BinaryEncodingEnabled = binaryEncodingEnabled; } /// @@ -37,7 +42,9 @@ internal CosmosJsonDotNetSerializer() /// This is internal to reduce exposure of JSON.net types so /// it is easier to convert to System.Text.Json /// - internal CosmosJsonDotNetSerializer(CosmosSerializationOptions cosmosSerializerOptions) + internal CosmosJsonDotNetSerializer( + CosmosSerializationOptions cosmosSerializerOptions, + bool binaryEncodingEnabled = false) { if (cosmosSerializerOptions == null) { @@ -56,6 +63,7 @@ internal CosmosJsonDotNetSerializer(CosmosSerializationOptions cosmosSerializerO }; this.SerializerSettings = jsonSerializerSettings; + this.BinaryEncodingEnabled = binaryEncodingEnabled; } /// @@ -65,9 +73,12 @@ internal CosmosJsonDotNetSerializer(CosmosSerializationOptions cosmosSerializerO /// This is internal to reduce exposure of JSON.net types so /// it is easier to convert to System.Text.Json /// - internal CosmosJsonDotNetSerializer(JsonSerializerSettings jsonSerializerSettings) + internal CosmosJsonDotNetSerializer( + JsonSerializerSettings jsonSerializerSettings, + bool binaryEncodingEnabled = false) { this.SerializerSettings = jsonSerializerSettings ?? throw new ArgumentNullException(nameof(jsonSerializerSettings)); + this.BinaryEncodingEnabled = binaryEncodingEnabled; } /// @@ -85,11 +96,30 @@ public override T FromStream(Stream stream) return (T)(object)stream; } - using (StreamReader sr = new StreamReader(stream)) + JsonSerializer jsonSerializer = this.GetSerializer(); + + if (stream is CloneableStream cloneableStream) + { + using (CosmosBufferedStreamWrapper bufferedStream = new (cloneableStream, shouldDisposeInnerStream: false)) + { + if (bufferedStream.GetJsonSerializationFormat() == Json.JsonSerializationFormat.Binary) + { + byte[] content = bufferedStream.ReadAll(); + + using Json.Interop.CosmosDBToNewtonsoftReader reader = new ( + jsonReader: Json.JsonReader.Create( + jsonSerializationFormat: Json.JsonSerializationFormat.Binary, + buffer: content)); + + return jsonSerializer.Deserialize(reader); + } + } + } + + using (StreamReader sr = new (stream)) { - using (JsonTextReader jsonTextReader = new JsonTextReader(sr)) + using (JsonTextReader jsonTextReader = new (sr)) { - JsonSerializer jsonSerializer = this.GetSerializer(); return jsonSerializer.Deserialize(jsonTextReader); } } @@ -104,16 +134,34 @@ public override T FromStream(Stream stream) /// An open readable stream containing the JSON of the serialized object public override Stream ToStream(T input) { - MemoryStream streamPayload = new MemoryStream(); - using (StreamWriter streamWriter = new StreamWriter(streamPayload, encoding: CosmosJsonDotNetSerializer.DefaultEncoding, bufferSize: 1024, leaveOpen: true)) + MemoryStream streamPayload; + JsonSerializer jsonSerializer = this.GetSerializer(); + + // Binary encoding is currently not supported for internal types, for e.g. + // container creation, database creation requests etc. + if (this.BinaryEncodingEnabled + && !CosmosSerializerCore.IsInputTypeInternal(typeof(T))) { - using (JsonWriter writer = new JsonTextWriter(streamWriter)) + using (Json.Interop.CosmosDBToNewtonsoftWriter writer = new ( + jsonSerializationFormat: Json.JsonSerializationFormat.Binary)) { - writer.Formatting = Newtonsoft.Json.Formatting.None; - JsonSerializer jsonSerializer = this.GetSerializer(); + writer.Formatting = Formatting.None; jsonSerializer.Serialize(writer, input); - writer.Flush(); - streamWriter.Flush(); + streamPayload = new MemoryStream(writer.GetResult().ToArray()); + } + } + else + { + streamPayload = new (); + using (StreamWriter streamWriter = new (streamPayload, encoding: CosmosJsonDotNetSerializer.DefaultEncoding, bufferSize: 1024, leaveOpen: true)) + { + using (JsonWriter writer = new JsonTextWriter(streamWriter)) + { + writer.Formatting = Formatting.None; + jsonSerializer.Serialize(writer, input); + writer.Flush(); + streamWriter.Flush(); + } } } diff --git a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializationUtil.cs b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializationUtil.cs index 8946d9ac7a..f316b1c18f 100644 --- a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializationUtil.cs +++ b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializationUtil.cs @@ -5,6 +5,17 @@ namespace Microsoft.Azure.Cosmos { using System; + using System.IO; + using System.Text; + using System.Threading.Tasks; + using Microsoft.Azure.Cosmos.ChangeFeed; + using Microsoft.Azure.Cosmos.CosmosElements; + using Microsoft.Azure.Cosmos.Json; + using Microsoft.Azure.Cosmos.Json.Interop; + using Microsoft.Azure.Cosmos.Query.Core.QueryPlan; + using Microsoft.Azure.Cosmos.Scripts; + using Microsoft.Azure.Cosmos.Serializer; + using Microsoft.Azure.Documents; using Newtonsoft.Json.Serialization; internal static class CosmosSerializationUtil @@ -30,5 +41,69 @@ internal static string GetStringWithPropertyNamingPolicy(CosmosPropertyNamingPol _ => throw new NotImplementedException("Unsupported CosmosPropertyNamingPolicy value"), }; } + + /// + /// Attempts to serialize the input stream to the specified target JSON serialization format. + /// + /// The desired JSON serialization format for the output stream. + /// The input stream containing the data to be serialized. + /// Returns true if the input stream is successfully serialized to the target format, otherwise false. + internal static Stream TrySerializeStreamToTargetFormat( + JsonSerializationFormat targetSerializationFormat, + CloneableStream inputStream) + { + if (inputStream == null) + { + return null; + } + + using (CosmosBufferedStreamWrapper bufferedStream = new ( + inputStream, + shouldDisposeInnerStream: false)) + { + JsonSerializationFormat sourceSerializationFormat = bufferedStream.GetJsonSerializationFormat(); + + if (sourceSerializationFormat != JsonSerializationFormat.HybridRow + && sourceSerializationFormat != targetSerializationFormat) + { + byte[] targetContent = bufferedStream.ReadAll(); + + if (targetContent != null && targetContent.Length > 0) + { + return CosmosSerializationUtil.ConvertToStreamUsingJsonSerializationFormat( + targetContent, + targetSerializationFormat); + } + } + } + + return inputStream; + } + + /// + /// Converts raw bytes to a stream using the specified JSON serialization format. + /// + /// The raw byte array to be converted. + /// The desired JSON serialization format. + /// Returns a stream containing the formatted JSON data. + internal static Stream ConvertToStreamUsingJsonSerializationFormat( + ReadOnlyMemory rawBytes, + JsonSerializationFormat format) + { + IJsonWriter writer = JsonWriter.Create(format); + if (CosmosObject.TryCreateFromBuffer(rawBytes, out CosmosObject cosmosObject)) + { + cosmosObject.WriteTo(writer); + } + else + { + IJsonReader desiredReader = JsonReader.Create(rawBytes); + desiredReader.WriteAll(writer); + } + + byte[] formattedBytes = writer.GetResult().ToArray(); + + return new MemoryStream(formattedBytes, index: 0, count: formattedBytes.Length, writable: true, publiclyVisible: true); + } } } diff --git a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializerCore.cs b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializerCore.cs index 703b2f2374..bc65c189ec 100644 --- a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializerCore.cs +++ b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSerializerCore.cs @@ -17,7 +17,9 @@ namespace Microsoft.Azure.Cosmos /// internal class CosmosSerializerCore { - private static readonly CosmosSerializer propertiesSerializer = new CosmosJsonSerializerWrapper(new CosmosJsonDotNetSerializer()); + private static readonly CosmosSerializer propertiesSerializer = new CosmosJsonSerializerWrapper( + new CosmosJsonDotNetSerializer( + ConfigurationManager.IsBinaryEncodingEnabled())); private readonly CosmosSerializer customSerializer; private readonly CosmosSerializer sqlQuerySpecSerializer; @@ -58,7 +60,10 @@ internal static CosmosSerializerCore Create( if (serializationOptions != null) { - customSerializer = new CosmosJsonSerializerWrapper(new CosmosJsonDotNetSerializer(serializationOptions)); + customSerializer = new CosmosJsonSerializerWrapper( + new CosmosJsonDotNetSerializer( + serializationOptions, + binaryEncodingEnabled: ConfigurationManager.IsBinaryEncodingEnabled())); } return new CosmosSerializerCore(customSerializer); @@ -133,20 +138,7 @@ private CosmosSerializer GetSerializer() return CosmosSerializerCore.propertiesSerializer; } - if (inputType == typeof(AccountProperties) || - inputType == typeof(DatabaseProperties) || - inputType == typeof(ContainerProperties) || - inputType == typeof(PermissionProperties) || - inputType == typeof(StoredProcedureProperties) || - inputType == typeof(TriggerProperties) || - inputType == typeof(UserDefinedFunctionProperties) || - inputType == typeof(UserProperties) || - inputType == typeof(ConflictProperties) || - inputType == typeof(ThroughputProperties) || - inputType == typeof(OfferV2) || - inputType == typeof(ClientEncryptionKeyProperties) || - inputType == typeof(PartitionedQueryExecutionInfo) || - inputType == typeof(ChangeFeedQuerySpec)) + if (CosmosSerializerCore.IsInputTypeInternal(inputType)) { return CosmosSerializerCore.propertiesSerializer; } @@ -172,5 +164,24 @@ private CosmosSerializer GetSerializer() return this.customSerializer; } + + internal static bool IsInputTypeInternal( + Type inputType) + { + return inputType == typeof(AccountProperties) + || inputType == typeof(DatabaseProperties) + || inputType == typeof(ContainerProperties) + || inputType == typeof(PermissionProperties) + || inputType == typeof(StoredProcedureProperties) + || inputType == typeof(TriggerProperties) + || inputType == typeof(UserDefinedFunctionProperties) + || inputType == typeof(UserProperties) + || inputType == typeof(ConflictProperties) + || inputType == typeof(ThroughputProperties) + || inputType == typeof(OfferV2) + || inputType == typeof(ClientEncryptionKeyProperties) + || inputType == typeof(PartitionedQueryExecutionInfo) + || inputType == typeof(ChangeFeedQuerySpec); + } } } diff --git a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSystemTextJsonSerializer.cs b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSystemTextJsonSerializer.cs index ac5550b128..6fcf5ee2c9 100644 --- a/Microsoft.Azure.Cosmos/src/Serializer/CosmosSystemTextJsonSerializer.cs +++ b/Microsoft.Azure.Cosmos/src/Serializer/CosmosSystemTextJsonSerializer.cs @@ -9,6 +9,9 @@ namespace Microsoft.Azure.Cosmos using System.Reflection; using System.Text.Json; using System.Text.Json.Serialization; + using Microsoft.Azure.Cosmos.CosmosElements; + using Microsoft.Azure.Cosmos.Json; + using Microsoft.Azure.Cosmos.Serializer; /// /// This class provides a default implementation of System.Text.Json Cosmos Linq Serializer. @@ -49,8 +52,28 @@ public override T FromStream(Stream stream) using (stream) { - using StreamReader reader = new (stream); - return JsonSerializer.Deserialize(reader.ReadToEnd(), this.jsonSerializerOptions); + if (stream is Documents.CloneableStream cloneableStream) + { + using (CosmosBufferedStreamWrapper bufferedStream = new (cloneableStream, shouldDisposeInnerStream: false)) + { + if (bufferedStream.GetJsonSerializationFormat() == JsonSerializationFormat.Binary) + { + byte[] content = bufferedStream.ReadAll(); + + if (CosmosObject.TryCreateFromBuffer(content, out CosmosObject cosmosObject)) + { + return System.Text.Json.JsonSerializer.Deserialize(cosmosObject.ToString(), this.jsonSerializerOptions); + } + else + { + using Stream textStream = CosmosSerializationUtil.ConvertToStreamUsingJsonSerializationFormat(content, JsonSerializationFormat.Text); + return this.DeserializeStream(textStream); + } + } + } + } + + return this.DeserializeStream(stream); } } @@ -60,7 +83,7 @@ public override Stream ToStream(T input) MemoryStream streamPayload = new (); using Utf8JsonWriter writer = new (streamPayload); - JsonSerializer.Serialize(writer, input, this.jsonSerializerOptions); + System.Text.Json.JsonSerializer.Serialize(writer, input, this.jsonSerializerOptions); streamPayload.Position = 0; return streamPayload; @@ -100,5 +123,18 @@ public override string SerializeMemberName(MemberInfo memberInfo) return memberInfo.Name; } + + /// + /// Deserializes the stream into the specified type using STJ Serializer. + /// + /// The desired type, the input stream to be deserialize into + /// An instance of containing th raw input stream. + /// The deserialized output of type . + private T DeserializeStream( + Stream stream) + { + using StreamReader reader = new (stream); + return System.Text.Json.JsonSerializer.Deserialize(reader.ReadToEnd(), this.jsonSerializerOptions); + } } } diff --git a/Microsoft.Azure.Cosmos/src/Util/ConfigurationManager.cs b/Microsoft.Azure.Cosmos/src/Util/ConfigurationManager.cs index bffb8b93f9..40f48b555d 100644 --- a/Microsoft.Azure.Cosmos/src/Util/ConfigurationManager.cs +++ b/Microsoft.Azure.Cosmos/src/Util/ConfigurationManager.cs @@ -25,18 +25,25 @@ internal static class ConfigurationManager /// /// Environment variable name for overriding optimistic direct execution of queries. /// - internal static readonly string OptimisticDirectExecutionEnabled = "AZURE_COSMOS_OPTIMISTIC_DIRECT_EXECUTION_ENABLED"; - + internal static readonly string OptimisticDirectExecutionEnabled = "AZURE_COSMOS_OPTIMISTIC_DIRECT_EXECUTION_ENABLED"; + /// /// Environment variable name to disable sending non streaming order by query feature flag to the gateway. /// - internal static readonly string NonStreamingOrderByQueryFeatureDisabled = "AZURE_COSMOS_NON_STREAMING_ORDER_BY_FLAG_DISABLED"; - + internal static readonly string NonStreamingOrderByQueryFeatureDisabled = "AZURE_COSMOS_NON_STREAMING_ORDER_BY_FLAG_DISABLED"; + /// /// Environment variable name to enable distributed query gateway mode. /// internal static readonly string DistributedQueryGatewayModeEnabled = "AZURE_COSMOS_DISTRIBUTED_QUERY_GATEWAY_ENABLED"; + /// + /// A read-only string containing the environment variable name for enabling binary encoding. This will eventually + /// be removed once binary encoding is enabled by default for both preview + /// and GA. + /// + internal static readonly string BinaryEncodingEnabled = "AZURE_COSMOS_BINARY_ENCODING_ENABLED"; + public static T GetEnvironmentVariable(string variable, T defaultValue) { string value = Environment.GetEnvironmentVariable(variable); @@ -98,10 +105,10 @@ public static bool IsOptimisticDirectExecutionEnabled( .GetEnvironmentVariable( variable: OptimisticDirectExecutionEnabled, defaultValue: defaultValue); - } - + } + /// - /// Gets the boolean value indicating whether the non streaming order by query feature flag should be sent to the gateway + /// Gets the boolean value indicating whether the non streaming order by query feature flag should be sent to the gateway /// based on the environment variable override. /// public static bool IsNonStreamingOrderByQueryFeatureDisabled( @@ -111,10 +118,10 @@ public static bool IsNonStreamingOrderByQueryFeatureDisabled( .GetEnvironmentVariable( variable: NonStreamingOrderByQueryFeatureDisabled, defaultValue: defaultValue); - } - + } + /// - /// Gets the boolean value indicating if distributed query gateway mode is enabled + /// Gets the boolean value indicating if distributed query gateway mode is enabled /// based on the environment variable override. /// public static bool IsDistributedQueryGatewayModeEnabled( @@ -125,5 +132,21 @@ public static bool IsDistributedQueryGatewayModeEnabled( variable: DistributedQueryGatewayModeEnabled, defaultValue: defaultValue); } + + /// + /// Gets the boolean value indicating if binary encoding is enabled based on the environment variable override. + /// Note that binary encoding is disabled by default for both preview and GA releases. The user can set the + /// respective environment variable 'AZURE_COSMOS_BINARY_ENCODING_ENABLED' to override the value for both preview and GA. + /// This method will eventually be removed once binary encoding is enabled by default for both preview and GA. + /// + /// A boolean flag indicating if binary encoding is enabled. + public static bool IsBinaryEncodingEnabled() + { + bool defaultValue = false; + return ConfigurationManager + .GetEnvironmentVariable( + variable: ConfigurationManager.BinaryEncodingEnabled, + defaultValue: defaultValue); + } } } diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/CosmosItemTests.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/CosmosItemTests.cs index 25b46d670f..66bf5a2559 100644 --- a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/CosmosItemTests.cs +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTests/CosmosItemTests.cs @@ -72,341 +72,481 @@ public void ParentResourceTest() Assert.AreEqual(this.GetClient(), this.Container.Database.Client); } - [TestMethod] - public async Task CreateDropItemWithInvalidIdCharactersTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - testItem.id = "Invalid#/\\?Id"; - await this.Container.CreateItemAsync(testItem, new Cosmos.PartitionKey(testItem.pk)); - - try - { - await this.Container.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); - Assert.Fail("Read item should fail because id has invalid characters"); - } - catch (CosmosException ce) when (ce.StatusCode == HttpStatusCode.NotFound) - { - string message = ce.ToString(); - Assert.IsNotNull(message); - CosmosItemTests.ValidateCosmosException(ce); - } - - // Get a container reference that use RID values - ContainerProperties containerProperties = await this.Container.ReadContainerAsync(); - string[] selfLinkSegments = containerProperties.SelfLink.Split('/'); - string databaseRid = selfLinkSegments[1]; - string containerRid = selfLinkSegments[3]; - Container containerByRid = this.GetClient().GetContainer(databaseRid, containerRid); - - // List of invalid characters are listed here. - //https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.resource.id?view=azure-dotnet#remarks - FeedIterator invalidItemsIterator = this.Container.GetItemQueryIterator( - @"select * from t where CONTAINS(t.id, ""/"") or CONTAINS(t.id, ""#"") or CONTAINS(t.id, ""?"") or CONTAINS(t.id, ""\\"") "); - while (invalidItemsIterator.HasMoreResults) - { - foreach (JObject itemWithInvalidId in await invalidItemsIterator.ReadNextAsync()) - { - // It recommend to chose a new id that does not contain special characters, but - // if that is not possible then it can be Base64 encoded to escape the special characters - byte[] plainTextBytes = Encoding.UTF8.GetBytes(itemWithInvalidId["id"].ToString()); - itemWithInvalidId["id"] = Convert.ToBase64String(plainTextBytes); - - // Update the item with the new id value using the rid based container reference - JObject item = await containerByRid.ReplaceItemAsync( - item: itemWithInvalidId, - id: itemWithInvalidId["_rid"].ToString(), - partitionKey: new Cosmos.PartitionKey(itemWithInvalidId["pk"].ToString())); - - // Validate the new id can be read using the original name based contianer reference - await this.Container.ReadItemAsync( - item["id"].ToString(), - new Cosmos.PartitionKey(item["pk"].ToString())); ; - } - } - } - - [TestMethod] - public async Task CreateDropItemTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - ItemResponse response = await this.Container.CreateItemAsync(item: testItem); - Assert.IsNotNull(response); - Assert.IsNotNull(response.Resource); - Assert.IsNotNull(response.Diagnostics); - CosmosTraceDiagnostics diagnostics = (CosmosTraceDiagnostics)response.Diagnostics; - Assert.IsFalse(diagnostics.IsGoneExceptionHit()); - string diagnosticString = response.Diagnostics.ToString(); - Assert.IsTrue(diagnosticString.Contains("Response Serialization")); - - Assert.IsFalse(string.IsNullOrEmpty(diagnostics.ToString())); - Assert.IsTrue(diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - Assert.AreEqual(0, response.Diagnostics.GetFailedRequestCount()); - Assert.IsNull(response.Diagnostics.GetQueryMetrics()); - - response = await this.Container.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); - Assert.IsNotNull(response); - Assert.IsNotNull(response.Resource); - Assert.IsNotNull(response.Diagnostics); - Assert.IsFalse(string.IsNullOrEmpty(response.Diagnostics.ToString())); - Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - Assert.AreEqual(0, response.Diagnostics.GetFailedRequestCount()); - Assert.IsNotNull(response.Diagnostics.GetStartTimeUtc()); - - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); - ItemResponse deleteResponse = await this.Container.DeleteItemAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id); - Assert.IsNotNull(deleteResponse); - Assert.IsNotNull(response.Diagnostics); - Assert.IsFalse(string.IsNullOrEmpty(response.Diagnostics.ToString())); - Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task CreateDropItemWithInvalidIdCharactersTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + testItem.id = "Invalid#/\\?Id"; + await this.Container.CreateItemAsync(testItem, new Cosmos.PartitionKey(testItem.pk)); + + try + { + await this.Container.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); + Assert.Fail("Read item should fail because id has invalid characters"); + } + catch (CosmosException ce) when (ce.StatusCode == HttpStatusCode.NotFound) + { + string message = ce.ToString(); + Assert.IsNotNull(message); + CosmosItemTests.ValidateCosmosException(ce); + } + + // Get a container reference that use RID values + ContainerProperties containerProperties = await this.Container.ReadContainerAsync(); + string[] selfLinkSegments = containerProperties.SelfLink.Split('/'); + string databaseRid = selfLinkSegments[1]; + string containerRid = selfLinkSegments[3]; + Container containerByRid = this.GetClient().GetContainer(databaseRid, containerRid); + + // List of invalid characters are listed here. + //https://docs.microsoft.com/dotnet/api/microsoft.azure.documents.resource.id?view=azure-dotnet#remarks + FeedIterator invalidItemsIterator = this.Container.GetItemQueryIterator( + @"select * from t where CONTAINS(t.id, ""/"") or CONTAINS(t.id, ""#"") or CONTAINS(t.id, ""?"") or CONTAINS(t.id, ""\\"") "); + while (invalidItemsIterator.HasMoreResults) + { + foreach (JObject itemWithInvalidId in await invalidItemsIterator.ReadNextAsync()) + { + // It recommend to chose a new id that does not contain special characters, but + // if that is not possible then it can be Base64 encoded to escape the special characters + byte[] plainTextBytes = Encoding.UTF8.GetBytes(itemWithInvalidId["id"].ToString()); + itemWithInvalidId["id"] = Convert.ToBase64String(plainTextBytes); + + // Update the item with the new id value using the rid based container reference + JObject item = await containerByRid.ReplaceItemAsync( + item: itemWithInvalidId, + id: itemWithInvalidId["_rid"].ToString(), + partitionKey: new Cosmos.PartitionKey(itemWithInvalidId["pk"].ToString())); + + // Validate the new id can be read using the original name based contianer reference + await this.Container.ReadItemAsync( + item["id"].ToString(), + new Cosmos.PartitionKey(item["pk"].ToString())); ; + } + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } - [TestMethod] - public async Task ClientConsistencyTestAsync() - { - List cosmosLevels = Enum.GetValues(typeof(Cosmos.ConsistencyLevel)).Cast().ToList(); - - foreach (Cosmos.ConsistencyLevel consistencyLevel in cosmosLevels) - { - RequestHandlerHelper handlerHelper = new RequestHandlerHelper(); - using CosmosClient cosmosClient = TestCommon.CreateCosmosClient(x => - x.WithConsistencyLevel(consistencyLevel).AddCustomHandlers(handlerHelper)); - Container consistencyContainer = cosmosClient.GetContainer(this.database.Id, this.Container.Id); - - int requestCount = 0; - handlerHelper.UpdateRequestMessage = (request) => - { - Assert.AreEqual(consistencyLevel.ToString(), request.Headers[HttpConstants.HttpHeaders.ConsistencyLevel]); - requestCount++; - }; - - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - ItemResponse response = await consistencyContainer.CreateItemAsync(item: testItem); - response = await consistencyContainer.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); - - Assert.AreEqual(2, requestCount); + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task CreateDropItemTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + ItemResponse response = await this.Container.CreateItemAsync(item: testItem); + Assert.IsNotNull(response); + Assert.IsNotNull(response.Resource); + Assert.IsNotNull(response.Diagnostics); + CosmosTraceDiagnostics diagnostics = (CosmosTraceDiagnostics)response.Diagnostics; + Assert.IsFalse(diagnostics.IsGoneExceptionHit()); + string diagnosticString = response.Diagnostics.ToString(); + Assert.IsTrue(diagnosticString.Contains("Response Serialization")); + + Assert.IsFalse(string.IsNullOrEmpty(diagnostics.ToString())); + Assert.IsTrue(diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + Assert.AreEqual(0, response.Diagnostics.GetFailedRequestCount()); + Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + + response = await this.Container.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); + Assert.IsNotNull(response); + Assert.IsNotNull(response.Resource); + Assert.IsNotNull(response.Diagnostics); + Assert.IsFalse(string.IsNullOrEmpty(response.Diagnostics.ToString())); + Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + Assert.AreEqual(0, response.Diagnostics.GetFailedRequestCount()); + Assert.IsNotNull(response.Diagnostics.GetStartTimeUtc()); + + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); + ItemResponse deleteResponse = await this.Container.DeleteItemAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id); + Assert.IsNotNull(deleteResponse); + Assert.IsNotNull(response.Diagnostics); + Assert.IsFalse(string.IsNullOrEmpty(response.Diagnostics.ToString())); + Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task NegativeCreateItemTest() - { - HttpClientHandlerHelper httpHandler = new HttpClientHandlerHelper(); - HttpClient httpClient = new HttpClient(httpHandler); - using CosmosClient client = TestCommon.CreateCosmosClient(x => x.WithHttpClientFactory(() => httpClient)); - - httpHandler.RequestCallBack = (request, cancellation) => - { - if (request.Method == HttpMethod.Get && - request.RequestUri.AbsolutePath == "//addresses/") - { - HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.Forbidden); - - // Add a substatus code that is not part of the enum. - // This ensures that if the backend adds a enum the status code is not lost. - result.Headers.Add(WFConstants.BackendHeaders.SubStatus, 999999.ToString(CultureInfo.InvariantCulture)); - string payload = JsonConvert.SerializeObject(new Error() { Message = "test message" }); - result.Content = new StringContent(payload, Encoding.UTF8, "application/json"); - return Task.FromResult(result); - } - - return null; - }; - - try - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - await client.GetContainer(this.database.Id, this.Container.Id).CreateItemAsync(item: testItem); - Assert.Fail("Request should throw exception."); - } - catch (CosmosException ce) when (ce.StatusCode == HttpStatusCode.Forbidden) - { - Assert.AreEqual(999999, ce.SubStatusCode); - string exception = ce.ToString(); - Assert.IsTrue(exception.StartsWith("Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 999999; ")); - string diagnostics = ce.Diagnostics.ToString(); - Assert.IsTrue(diagnostics.Contains("999999")); - CosmosItemTests.ValidateCosmosException(ce); + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task ClientConsistencyTestAsync(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + List cosmosLevels = Enum.GetValues(typeof(Cosmos.ConsistencyLevel)).Cast().ToList(); + + foreach (Cosmos.ConsistencyLevel consistencyLevel in cosmosLevels) + { + RequestHandlerHelper handlerHelper = new RequestHandlerHelper(); + using CosmosClient cosmosClient = TestCommon.CreateCosmosClient(x => + x.WithConsistencyLevel(consistencyLevel).AddCustomHandlers(handlerHelper)); + Container consistencyContainer = cosmosClient.GetContainer(this.database.Id, this.Container.Id); + + int requestCount = 0; + handlerHelper.UpdateRequestMessage = (request) => + { + Assert.AreEqual(consistencyLevel.ToString(), request.Headers[HttpConstants.HttpHeaders.ConsistencyLevel]); + requestCount++; + }; + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + ItemResponse response = await consistencyContainer.CreateItemAsync(item: testItem); + response = await consistencyContainer.ReadItemAsync(testItem.id, new Cosmos.PartitionKey(testItem.pk)); + + Assert.AreEqual(2, requestCount); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task NegativeCreateDropItemTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - ResponseMessage response = await this.Container.CreateItemStreamAsync(streamPayload: TestCommon.SerializerCore.ToStream(testItem), partitionKey: new Cosmos.PartitionKey("BadKey")); - Assert.IsNotNull(response); - Assert.IsNull(response.Content); - Assert.AreEqual(HttpStatusCode.BadRequest, response.StatusCode); - Assert.AreNotEqual(0, response.Diagnostics.GetFailedRequestCount()); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task NegativeCreateItemTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + HttpClientHandlerHelper httpHandler = new HttpClientHandlerHelper(); + HttpClient httpClient = new HttpClient(httpHandler); + using CosmosClient client = TestCommon.CreateCosmosClient(x => x.WithHttpClientFactory(() => httpClient)); + + httpHandler.RequestCallBack = (request, cancellation) => + { + if (request.Method == HttpMethod.Get && + request.RequestUri.AbsolutePath == "//addresses/") + { + HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.Forbidden); + + // Add a substatus code that is not part of the enum. + // This ensures that if the backend adds a enum the status code is not lost. + result.Headers.Add(WFConstants.BackendHeaders.SubStatus, 999999.ToString(CultureInfo.InvariantCulture)); + string payload = JsonConvert.SerializeObject(new Error() { Message = "test message" }); + result.Content = new StringContent(payload, Encoding.UTF8, "application/json"); + return Task.FromResult(result); + } + + return null; + }; + + try + { + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + await client.GetContainer(this.database.Id, this.Container.Id).CreateItemAsync(item: testItem); + Assert.Fail("Request should throw exception."); + } + catch (CosmosException ce) when (ce.StatusCode == HttpStatusCode.Forbidden) + { + Assert.AreEqual(999999, ce.SubStatusCode); + string exception = ce.ToString(); + Assert.IsTrue(exception.StartsWith("Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 999999; ")); + string diagnostics = ce.Diagnostics.ToString(); + Assert.IsTrue(diagnostics.Contains("999999")); + CosmosItemTests.ValidateCosmosException(ce); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } - [TestMethod] - public async Task MemoryStreamBufferIsAccessibleOnResponse() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - ResponseMessage response = await this.Container.CreateItemStreamAsync(streamPayload: TestCommon.SerializerCore.ToStream(testItem), partitionKey: new Cosmos.PartitionKey(testItem.pk)); - Assert.IsNotNull(response); - Assert.IsTrue((response.Content as MemoryStream).TryGetBuffer(out _)); - FeedIterator feedIteratorQuery = this.Container.GetItemQueryStreamIterator(queryText: "SELECT * FROM c"); - - while (feedIteratorQuery.HasMoreResults) - { - ResponseMessage feedResponseQuery = await feedIteratorQuery.ReadNextAsync(); - Assert.IsTrue((feedResponseQuery.Content as MemoryStream).TryGetBuffer(out _)); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task NegativeCreateDropItemTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + ResponseMessage response = await this.Container.CreateItemStreamAsync(streamPayload: TestCommon.SerializerCore.ToStream(testItem), partitionKey: new Cosmos.PartitionKey("BadKey")); + Assert.IsNotNull(response); + Assert.IsNull(response.Content); + Assert.AreEqual(HttpStatusCode.BadRequest, response.StatusCode); + Assert.AreNotEqual(0, response.Diagnostics.GetFailedRequestCount()); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } + } - FeedIterator feedIterator = this.Container.GetItemQueryStreamIterator(requestOptions: new QueryRequestOptions() - { - PartitionKey = new Cosmos.PartitionKey(testItem.pk) - }); - - while (feedIterator.HasMoreResults) - { - ResponseMessage feedResponse = await feedIterator.ReadNextAsync(); - Assert.IsTrue((feedResponse.Content as MemoryStream).TryGetBuffer(out _)); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task MemoryStreamBufferIsAccessibleOnResponse(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + ResponseMessage response = await this.Container.CreateItemStreamAsync(streamPayload: TestCommon.SerializerCore.ToStream(testItem), partitionKey: new Cosmos.PartitionKey(testItem.pk)); + Assert.IsNotNull(response); + Assert.IsTrue((response.Content as MemoryStream).TryGetBuffer(out _)); + FeedIterator feedIteratorQuery = this.Container.GetItemQueryStreamIterator(queryText: "SELECT * FROM c"); + + while (feedIteratorQuery.HasMoreResults) + { + ResponseMessage feedResponseQuery = await feedIteratorQuery.ReadNextAsync(); + Assert.IsTrue((feedResponseQuery.Content as MemoryStream).TryGetBuffer(out _)); + } + + FeedIterator feedIterator = this.Container.GetItemQueryStreamIterator(requestOptions: new QueryRequestOptions() + { + PartitionKey = new Cosmos.PartitionKey(testItem.pk) + }); + + while (feedIterator.HasMoreResults) + { + ResponseMessage feedResponse = await feedIterator.ReadNextAsync(); + Assert.IsTrue((feedResponse.Content as MemoryStream).TryGetBuffer(out _)); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task CustomSerilizerTest() - { - string id1 = "MyCustomSerilizerTestId1"; - string id2 = "MyCustomSerilizerTestId2"; - string pk = "MyTestPk"; - - // Delete the item to prevent create conflicts if test is run multiple times - using (await this.Container.DeleteItemStreamAsync(id1, new Cosmos.PartitionKey(pk))) - { } - using (await this.Container.DeleteItemStreamAsync(id2, new Cosmos.PartitionKey(pk))) - { } - - // Both items have null description - dynamic testItem = new { id = id1, status = pk, description = (string)null }; - dynamic testItem2 = new { id = id2, status = pk, description = (string)null }; - - // Create a client that ignore null - CosmosClientOptions clientOptions = new CosmosClientOptions() - { - Serializer = new CosmosJsonDotNetSerializer( - new JsonSerializerSettings() - { - NullValueHandling = NullValueHandling.Ignore - }) - }; - - CosmosClient ignoreNullClient = TestCommon.CreateCosmosClient(clientOptions); - Container ignoreContainer = ignoreNullClient.GetContainer(this.database.Id, this.Container.Id); - - ItemResponse ignoreNullResponse = await ignoreContainer.CreateItemAsync(item: testItem); - Assert.IsNotNull(ignoreNullResponse); - Assert.IsNotNull(ignoreNullResponse.Resource); - Assert.IsNull(ignoreNullResponse.Resource["description"]); - - ItemResponse keepNullResponse = await this.Container.CreateItemAsync(item: testItem2); - Assert.IsNotNull(keepNullResponse); - Assert.IsNotNull(keepNullResponse.Resource); - Assert.IsNotNull(keepNullResponse.Resource["description"]); - - using (await this.Container.DeleteItemStreamAsync(id1, new Cosmos.PartitionKey(pk))) - { } - using (await this.Container.DeleteItemStreamAsync(id2, new Cosmos.PartitionKey(pk))) - { } + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task CustomSerilizerTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + string id1 = "MyCustomSerilizerTestId1"; + string id2 = "MyCustomSerilizerTestId2"; + string pk = "MyTestPk"; + + // Delete the item to prevent create conflicts if test is run multiple times + using (await this.Container.DeleteItemStreamAsync(id1, new Cosmos.PartitionKey(pk))) + { } + using (await this.Container.DeleteItemStreamAsync(id2, new Cosmos.PartitionKey(pk))) + { } + + // Both items have null description + dynamic testItem = new { id = id1, status = pk, description = (string)null }; + dynamic testItem2 = new { id = id2, status = pk, description = (string)null }; + + // Create a client that ignore null + CosmosClientOptions clientOptions = new CosmosClientOptions() + { + Serializer = new CosmosJsonDotNetSerializer( + new JsonSerializerSettings() + { + NullValueHandling = NullValueHandling.Ignore + }) + }; + + CosmosClient ignoreNullClient = TestCommon.CreateCosmosClient(clientOptions); + Container ignoreContainer = ignoreNullClient.GetContainer(this.database.Id, this.Container.Id); + + ItemResponse ignoreNullResponse = await ignoreContainer.CreateItemAsync(item: testItem); + Assert.IsNotNull(ignoreNullResponse); + Assert.IsNotNull(ignoreNullResponse.Resource); + Assert.IsNull(ignoreNullResponse.Resource["description"]); + + ItemResponse keepNullResponse = await this.Container.CreateItemAsync(item: testItem2); + Assert.IsNotNull(keepNullResponse); + Assert.IsNotNull(keepNullResponse.Resource); + Assert.IsNotNull(keepNullResponse.Resource["description"]); + + using (await this.Container.DeleteItemStreamAsync(id1, new Cosmos.PartitionKey(pk))) + { } + using (await this.Container.DeleteItemStreamAsync(id2, new Cosmos.PartitionKey(pk))) + { } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } - [TestMethod] - public async Task CreateDropItemUndefinedPartitionKeyTest() - { - dynamic testItem = new - { - id = Guid.NewGuid().ToString() - }; - - ItemResponse response = await this.Container.CreateItemAsync(item: testItem, partitionKey: new Cosmos.PartitionKey(Undefined.Value)); - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); - - ItemResponse deleteResponse = await this.Container.DeleteItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey(Undefined.Value)); - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task CreateDropItemUndefinedPartitionKeyTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + dynamic testItem = new + { + id = Guid.NewGuid().ToString() + }; + + ItemResponse response = await this.Container.CreateItemAsync(item: testItem, partitionKey: new Cosmos.PartitionKey(Undefined.Value)); + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); + + ItemResponse deleteResponse = await this.Container.DeleteItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey(Undefined.Value)); + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } - [TestMethod] - public async Task CreateDropItemPartitionKeyNotInTypeTest() - { - dynamic testItem = new - { - id = Guid.NewGuid().ToString() - }; - - ItemResponse response = await this.Container.CreateItemAsync(item: testItem); - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); - Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); - - ItemResponse readResponse = await this.Container.ReadItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); - Assert.IsNotNull(readResponse); - Assert.AreEqual(HttpStatusCode.OK, readResponse.StatusCode); - - ItemResponse deleteResponse = await this.Container.DeleteItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); - - try - { - readResponse = await this.Container.ReadItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); - Assert.Fail("Should throw exception."); - } - catch (CosmosException ex) - { - Assert.AreEqual(HttpStatusCode.NotFound, ex.StatusCode); - CosmosItemTests.ValidateCosmosException(ex); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task CreateDropItemPartitionKeyNotInTypeTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + dynamic testItem = new + { + id = Guid.NewGuid().ToString() + }; + + ItemResponse response = await this.Container.CreateItemAsync(item: testItem); + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.MaxResourceQuota)); + Assert.IsNotNull(response.Headers.GetHeaderValue(Documents.HttpConstants.HttpHeaders.CurrentResourceQuotaUsage)); + + ItemResponse readResponse = await this.Container.ReadItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); + Assert.IsNotNull(readResponse); + Assert.AreEqual(HttpStatusCode.OK, readResponse.StatusCode); + + ItemResponse deleteResponse = await this.Container.DeleteItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); + + try + { + readResponse = await this.Container.ReadItemAsync(id: testItem.id, partitionKey: Cosmos.PartitionKey.None); + Assert.Fail("Should throw exception."); + } + catch (CosmosException ex) + { + Assert.AreEqual(HttpStatusCode.NotFound, ex.StatusCode); + CosmosItemTests.ValidateCosmosException(ex); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task CreateDropItemMultiPartPartitionKeyTest() - { - Container multiPartPkContainer = await this.database.CreateContainerAsync(Guid.NewGuid().ToString(), "/a/b/c"); - - dynamic testItem = new - { - id = Guid.NewGuid().ToString(), - a = new - { - b = new - { - c = "pk1", - } - } - }; - - ItemResponse response = await multiPartPkContainer.CreateItemAsync(item: testItem); - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsNull(response.Diagnostics.GetQueryMetrics()); - - ItemResponse readResponse = await multiPartPkContainer.ReadItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); - Assert.IsNotNull(readResponse); - Assert.AreEqual(HttpStatusCode.OK, readResponse.StatusCode); - - ItemResponse deleteResponse = await multiPartPkContainer.DeleteItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); - - try - { - readResponse = await multiPartPkContainer.ReadItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); - Assert.Fail("Should throw exception."); - } - catch (CosmosException ex) - { - Assert.AreEqual(HttpStatusCode.NotFound, ex.StatusCode); - CosmosItemTests.ValidateCosmosException(ex); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task CreateDropItemMultiPartPartitionKeyTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + Container multiPartPkContainer = await this.database.CreateContainerAsync(Guid.NewGuid().ToString(), "/a/b/c"); + + dynamic testItem = new + { + id = Guid.NewGuid().ToString(), + a = new + { + b = new + { + c = "pk1", + } + } + }; + + ItemResponse response = await multiPartPkContainer.CreateItemAsync(item: testItem); + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + + ItemResponse readResponse = await multiPartPkContainer.ReadItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); + Assert.IsNotNull(readResponse); + Assert.AreEqual(HttpStatusCode.OK, readResponse.StatusCode); + + ItemResponse deleteResponse = await multiPartPkContainer.DeleteItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); + + try + { + readResponse = await multiPartPkContainer.ReadItemAsync(id: testItem.id, partitionKey: new Cosmos.PartitionKey("pk1")); + Assert.Fail("Should throw exception."); + } + catch (CosmosException ex) + { + Assert.AreEqual(HttpStatusCode.NotFound, ex.StatusCode); + CosmosItemTests.ValidateCosmosException(ex); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } @@ -423,247 +563,360 @@ public async Task ReadCollectionNotExists() await CosmosItemTests.TestNonePKForNonExistingContainer(testContainer); } - [TestMethod] - public async Task NonPartitionKeyLookupCacheTest() - { - int count = 0; - using CosmosClient client = TestCommon.CreateCosmosClient(builder => - { - builder.WithConnectionModeDirect(); - builder.WithSendingRequestEventArgs((sender, e) => - { - if (e.DocumentServiceRequest != null) - { - System.Diagnostics.Trace.TraceInformation($"{e.DocumentServiceRequest.ToString()}"); - } - - if (e.HttpRequest != null) - { - System.Diagnostics.Trace.TraceInformation($"{e.HttpRequest.ToString()}"); - } - - if (e.IsHttpRequest() - && e.HttpRequest.RequestUri.AbsolutePath.Contains("/colls/")) - { - count++; - } - - if (e.IsHttpRequest() - && e.HttpRequest.RequestUri.AbsolutePath.Contains("/pkranges")) - { - Debugger.Break(); - } - }); - }, - validatePartitionKeyRangeCalls: false); - - string dbName = Guid.NewGuid().ToString(); - string containerName = Guid.NewGuid().ToString(); - ContainerInternal testContainer = (ContainerInlineCore)client.GetContainer(dbName, containerName); - - int loopCount = 2; - for (int i = 0; i < loopCount; i++) - { - try - { - await testContainer.GetNonePartitionKeyValueAsync(NoOpTrace.Singleton, default(CancellationToken)); - Assert.Fail(); - } - catch (CosmosException dce) when (dce.StatusCode == HttpStatusCode.NotFound) - { - } - } - - Assert.AreEqual(loopCount, count); - - // Create real container and address - Cosmos.Database db = await client.CreateDatabaseAsync(dbName); - Container container = await db.CreateContainerAsync(containerName, "/id"); - - // reset counter - count = 0; - for (int i = 0; i < loopCount; i++) - { - await testContainer.GetNonePartitionKeyValueAsync(NoOpTrace.Singleton, default); - } - - // expected once post create - Assert.AreEqual(1, count); - - // reset counter - count = 0; - for (int i = 0; i < loopCount; i++) - { - await testContainer.GetCachedRIDAsync(forceRefresh: false, NoOpTrace.Singleton, cancellationToken: default); - } - - // Already cached by GetNonePartitionKeyValueAsync before - Assert.AreEqual(0, count); - - // reset counter - count = 0; - int expected = 0; - for (int i = 0; i < loopCount; i++) - { - await testContainer.GetRoutingMapAsync(default); - expected = count; - } - - // OkRagnes should be fetched only once. - // Possible to make multiple calls for ranges - Assert.AreEqual(expected, count); - - await db.DeleteStreamAsync(); - } - - [TestMethod] - public async Task CreateDropItemStreamTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - using (ResponseMessage response = await this.Container.CreateItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsTrue(response.Headers.RequestCharge > 0); - Assert.IsNotNull(response.Headers.ActivityId); - Assert.IsNotNull(response.Headers.ETag); - Assert.IsNotNull(response.Diagnostics); - Assert.IsTrue(!string.IsNullOrEmpty(response.Diagnostics.ToString())); - Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - } - } - - using (ResponseMessage response = await this.Container.ReadItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); - Assert.IsTrue(response.Headers.RequestCharge > 0); - Assert.IsNotNull(response.Headers.ActivityId); - Assert.IsNotNull(response.Headers.ETag); - Assert.IsNotNull(response.Diagnostics); - Assert.IsTrue(!string.IsNullOrEmpty(response.Diagnostics.ToString())); - Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - } - - using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) - { - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); - Assert.IsTrue(deleteResponse.Headers.RequestCharge > 0); - Assert.IsNotNull(deleteResponse.Headers.ActivityId); - Assert.IsNotNull(deleteResponse.Diagnostics); - Assert.IsTrue(!string.IsNullOrEmpty(deleteResponse.Diagnostics.ToString())); - Assert.IsTrue(deleteResponse.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); - } - } - - [TestMethod] - public async Task UpsertItemStreamTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - //Create the object - using (ResponseMessage response = await this.Container.UpsertItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsNotNull(response.Headers.Session); - using (StreamReader str = new StreamReader(response.Content)) - { - string responseContentAsString = await str.ReadToEndAsync(); - } - } - } - - //Updated the taskNum field - testItem.taskNum = 9001; - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - using (ResponseMessage response = await this.Container.UpsertItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); - Assert.IsNotNull(response.Headers.Session); - } - } - using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) - { - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(deleteResponse.StatusCode, HttpStatusCode.NoContent); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task NonPartitionKeyLookupCacheTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + int count = 0; + using CosmosClient client = TestCommon.CreateCosmosClient(builder => + { + builder.WithConnectionModeDirect(); + builder.WithSendingRequestEventArgs((sender, e) => + { + if (e.DocumentServiceRequest != null) + { + System.Diagnostics.Trace.TraceInformation($"{e.DocumentServiceRequest.ToString()}"); + } + + if (e.HttpRequest != null) + { + System.Diagnostics.Trace.TraceInformation($"{e.HttpRequest.ToString()}"); + } + + if (e.IsHttpRequest() + && e.HttpRequest.RequestUri.AbsolutePath.Contains("/colls/")) + { + count++; + } + + if (e.IsHttpRequest() + && e.HttpRequest.RequestUri.AbsolutePath.Contains("/pkranges")) + { + Debugger.Break(); + } + }); + }, + validatePartitionKeyRangeCalls: false); + + string dbName = Guid.NewGuid().ToString(); + string containerName = Guid.NewGuid().ToString(); + ContainerInternal testContainer = (ContainerInlineCore)client.GetContainer(dbName, containerName); + + int loopCount = 2; + for (int i = 0; i < loopCount; i++) + { + try + { + await testContainer.GetNonePartitionKeyValueAsync(NoOpTrace.Singleton, default(CancellationToken)); + Assert.Fail(); + } + catch (CosmosException dce) when (dce.StatusCode == HttpStatusCode.NotFound) + { + } + } + + Assert.AreEqual(loopCount, count); + + // Create real container and address + Cosmos.Database db = await client.CreateDatabaseAsync(dbName); + Container container = await db.CreateContainerAsync(containerName, "/id"); + + // reset counter + count = 0; + for (int i = 0; i < loopCount; i++) + { + await testContainer.GetNonePartitionKeyValueAsync(NoOpTrace.Singleton, default); + } + + // expected once post create + Assert.AreEqual(1, count); + + // reset counter + count = 0; + for (int i = 0; i < loopCount; i++) + { + await testContainer.GetCachedRIDAsync(forceRefresh: false, NoOpTrace.Singleton, cancellationToken: default); + } + + // Already cached by GetNonePartitionKeyValueAsync before + Assert.AreEqual(0, count); + + // reset counter + count = 0; + int expected = 0; + for (int i = 0; i < loopCount; i++) + { + await testContainer.GetRoutingMapAsync(default); + expected = count; + } + + // OkRagnes should be fetched only once. + // Possible to make multiple calls for ranges + Assert.AreEqual(expected, count); + + await db.DeleteStreamAsync(); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task UpsertItemTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - - { - ItemResponse response = await this.Container.UpsertItemAsync(testItem, partitionKey: new Cosmos.PartitionKey(testItem.pk)); - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - Assert.IsNotNull(response.Headers.Session); - Assert.IsNull(response.Diagnostics.GetQueryMetrics()); - } - - { - //Updated the taskNum field - testItem.taskNum = 9001; - ItemResponse response = await this.Container.UpsertItemAsync(testItem, partitionKey: new Cosmos.PartitionKey(testItem.pk)); - - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); - Assert.IsNotNull(response.Headers.Session); - Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + [TestMethod] + [DataRow(true, true, DisplayName = "Test scenario when binary encoding is enabled at client level and expected stream response type is binary.")] + [DataRow(true, false, DisplayName = "Test scenario when binary encoding is enabled at client level and expected stream response type is text.")] + [DataRow(false, true, DisplayName = "Test scenario when binary encoding is disabled at client level and expected stream response type is binary.")] + [DataRow(false, false, DisplayName = "Test scenario when binary encoding is disabled at client level and expected stream response type is text.")] + public async Task CreateDropItemStreamTest(bool binaryEncodingEnabledInClient, bool shouldExpectBinaryOnResponse) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ItemRequestOptions requestOptions = new() + { + EnableBinaryResponseOnPointOperations = binaryEncodingEnabledInClient && shouldExpectBinaryOnResponse, + }; + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + using (ResponseMessage response = await this.Container.CreateItemStreamAsync( + streamPayload: stream, + partitionKey: new Cosmos.PartitionKey(testItem.pk), + requestOptions: requestOptions)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsTrue(response.Headers.RequestCharge > 0); + Assert.IsNotNull(response.Headers.ActivityId); + Assert.IsNotNull(response.Headers.ETag); + Assert.IsNotNull(response.Diagnostics); + Assert.IsTrue(!string.IsNullOrEmpty(response.Diagnostics.ToString())); + Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + + if (requestOptions.EnableBinaryResponseOnPointOperations) + { + AssertOnResponseSerializationBinaryType(response.Content); + } + else + { + AssertOnResponseSerializationTextType(response.Content); + } + } + } + + using (ResponseMessage response = await this.Container.ReadItemStreamAsync( + id: testItem.id, + partitionKey: new Cosmos.PartitionKey(testItem.pk), + requestOptions: requestOptions)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); + Assert.IsTrue(response.Headers.RequestCharge > 0); + Assert.IsNotNull(response.Headers.ActivityId); + Assert.IsNotNull(response.Headers.ETag); + Assert.IsNotNull(response.Diagnostics); + Assert.IsTrue(!string.IsNullOrEmpty(response.Diagnostics.ToString())); + Assert.IsTrue(response.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + + if (requestOptions.EnableBinaryResponseOnPointOperations) + { + AssertOnResponseSerializationBinaryType(response.Content); + } + else + { + AssertOnResponseSerializationTextType(response.Content); + } + } + + using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync( + id: testItem.id, + partitionKey: new Cosmos.PartitionKey(testItem.pk), + requestOptions: requestOptions)) + { + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(HttpStatusCode.NoContent, deleteResponse.StatusCode); + Assert.IsTrue(deleteResponse.Headers.RequestCharge > 0); + Assert.IsNotNull(deleteResponse.Headers.ActivityId); + Assert.IsNotNull(deleteResponse.Diagnostics); + Assert.IsTrue(!string.IsNullOrEmpty(deleteResponse.Diagnostics.ToString())); + Assert.IsTrue(deleteResponse.Diagnostics.GetClientElapsedTime() > TimeSpan.Zero); + + if (requestOptions.EnableBinaryResponseOnPointOperations) + { + AssertOnResponseSerializationBinaryType(deleteResponse.Content); + } + else + { + AssertOnResponseSerializationTextType(deleteResponse.Content); + } + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task ReplaceItemStreamTest() - { - ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - //Replace a non-existing item. It should fail, and not throw an exception. - using (ResponseMessage response = await this.Container.ReplaceItemStreamAsync( - partitionKey: new Cosmos.PartitionKey(testItem.pk), - id: testItem.id, - streamPayload: stream)) - { - Assert.IsFalse(response.IsSuccessStatusCode); - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.NotFound, response.StatusCode, response.ErrorMessage); - } + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task UpsertItemStreamTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + //Create the object + using (ResponseMessage response = await this.Container.UpsertItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsNotNull(response.Headers.Session); + using (StreamReader str = new StreamReader(response.Content)) + { + string responseContentAsString = await str.ReadToEndAsync(); + } + } + } + + //Updated the taskNum field + testItem.taskNum = 9001; + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + using (ResponseMessage response = await this.Container.UpsertItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); + Assert.IsNotNull(response.Headers.Session); + } + } + using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) + { + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(deleteResponse.StatusCode, HttpStatusCode.NoContent); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } + } - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - //Create the item - using (ResponseMessage response = await this.Container.CreateItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); - } + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task UpsertItemTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + + { + ItemResponse response = await this.Container.UpsertItemAsync(testItem, partitionKey: new Cosmos.PartitionKey(testItem.pk)); + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + Assert.IsNotNull(response.Headers.Session); + Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + } + + { + //Updated the taskNum field + testItem.taskNum = 9001; + ItemResponse response = await this.Container.UpsertItemAsync(testItem, partitionKey: new Cosmos.PartitionKey(testItem.pk)); + + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); + Assert.IsNotNull(response.Headers.Session); + Assert.IsNull(response.Diagnostics.GetQueryMetrics()); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } + } - //Updated the taskNum field - testItem.taskNum = 9001; - using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) - { - using (ResponseMessage response = await this.Container.ReplaceItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id, streamPayload: stream)) - { - Assert.IsNotNull(response); - Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); - } - - using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) - { - Assert.IsNotNull(deleteResponse); - Assert.AreEqual(deleteResponse.StatusCode, HttpStatusCode.NoContent); - } + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task ReplaceItemStreamTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity testItem = ToDoActivity.CreateRandomToDoActivity(); + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + //Replace a non-existing item. It should fail, and not throw an exception. + using (ResponseMessage response = await this.Container.ReplaceItemStreamAsync( + partitionKey: new Cosmos.PartitionKey(testItem.pk), + id: testItem.id, + streamPayload: stream)) + { + Assert.IsFalse(response.IsSuccessStatusCode); + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.NotFound, response.StatusCode, response.ErrorMessage); + } + } + + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + //Create the item + using (ResponseMessage response = await this.Container.CreateItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), streamPayload: stream)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.Created, response.StatusCode); + } + } + + //Updated the taskNum field + testItem.taskNum = 9001; + using (Stream stream = TestCommon.SerializerCore.ToStream(testItem)) + { + using (ResponseMessage response = await this.Container.ReplaceItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id, streamPayload: stream)) + { + Assert.IsNotNull(response); + Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); + } + + using (ResponseMessage deleteResponse = await this.Container.DeleteItemStreamAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id)) + { + Assert.IsNotNull(deleteResponse); + Assert.AreEqual(deleteResponse.StatusCode, HttpStatusCode.NoContent); + } + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } @@ -717,133 +970,161 @@ await feedIterator.ReadNextAsync(this.cancellationToken)) Assert.AreEqual(itemIds.Count, 0); } - [TestMethod] - public async Task PartitionKeyDeleteTest() - { - string pKString = "PK1"; - string pKString2 = "PK2"; - dynamic testItem1 = new - { - id = "item1", - pk = pKString - }; - - dynamic testItem2 = new - { - id = "item2", - pk = pKString - }; - - dynamic testItem3 = new - { - id = "item3", - pk = pKString2 - }; - - ContainerInternal containerInternal = (ContainerInternal)this.Container; - await this.Container.CreateItemAsync(testItem1); - await this.Container.CreateItemAsync(testItem2); - await this.Container.CreateItemAsync(testItem3); - Cosmos.PartitionKey partitionKey1 = new Cosmos.PartitionKey(pKString); - Cosmos.PartitionKey partitionKey2 = new Cosmos.PartitionKey(pKString2); - using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partitionKey1)) - { - Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.OK); - } - - using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item1", partitionKey1)) - { - Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); - Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); - } - - using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item2", partitionKey1)) - { - Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); - Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); - } - - //verify item with the other Partition Key is not deleted - using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item3", partitionKey2)) - { - Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.OK); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task PartitionKeyDeleteTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + string pKString = "PK1"; + string pKString2 = "PK2"; + dynamic testItem1 = new + { + id = "item1", + pk = pKString + }; + + dynamic testItem2 = new + { + id = "item2", + pk = pKString + }; + + dynamic testItem3 = new + { + id = "item3", + pk = pKString2 + }; + + ContainerInternal containerInternal = (ContainerInternal)this.Container; + await this.Container.CreateItemAsync(testItem1); + await this.Container.CreateItemAsync(testItem2); + await this.Container.CreateItemAsync(testItem3); + Cosmos.PartitionKey partitionKey1 = new Cosmos.PartitionKey(pKString); + Cosmos.PartitionKey partitionKey2 = new Cosmos.PartitionKey(pKString2); + using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partitionKey1)) + { + Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.OK); + } + + using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item1", partitionKey1)) + { + Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); + Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); + } + + using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item2", partitionKey1)) + { + Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); + Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); + } + + //verify item with the other Partition Key is not deleted + using (ResponseMessage readResponse = await this.Container.ReadItemStreamAsync("item3", partitionKey2)) + { + Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.OK); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task PartitionKeyDeleteTestForSubpartitionedContainer() - { - string currentVersion = HttpConstants.Versions.CurrentVersion; - HttpConstants.Versions.CurrentVersion = "2020-07-15"; - using CosmosClient client = TestCommon.CreateCosmosClient(true); - Cosmos.Database database = null; - try - { - database = await client.CreateDatabaseIfNotExistsAsync("mydb"); - - ContainerProperties containerProperties = new ContainerProperties("subpartitionedcontainer", new List { "/Country", "/City" }); - Container container = await database.CreateContainerAsync(containerProperties); - ContainerInternal containerInternal = (ContainerInternal)container; - - //Document create. - ItemResponse[] documents = new ItemResponse[5]; - Document doc1 = new Document { Id = "document1" }; - doc1.SetValue("Country", "USA"); - doc1.SetValue("City", "Redmond"); - documents[0] = await container.CreateItemAsync(doc1); - - doc1 = new Document { Id = "document2" }; - doc1.SetValue("Country", "USA"); - doc1.SetValue("City", "Pittsburgh"); - documents[1] = await container.CreateItemAsync(doc1); - - doc1 = new Document { Id = "document3" }; - doc1.SetValue("Country", "USA"); - doc1.SetValue("City", "Stonybrook"); - documents[2] = await container.CreateItemAsync(doc1); - - doc1 = new Document { Id = "document4" }; - doc1.SetValue("Country", "USA"); - doc1.SetValue("City", "Stonybrook"); - documents[3] = await container.CreateItemAsync(doc1); - - doc1 = new Document { Id = "document5" }; - doc1.SetValue("Country", "USA"); - doc1.SetValue("City", "Stonybrook"); - documents[4] = await container.CreateItemAsync(doc1); - - Cosmos.PartitionKey partitionKey1 = new PartitionKeyBuilder().Add("USA").Add("Stonybrook").Build(); - - using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partitionKey1)) - { - Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.OK); - } - using (ResponseMessage readResponse = await containerInternal.ReadItemStreamAsync("document5", partitionKey1)) - { - Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); - Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); - } - - Cosmos.PartitionKey partitionKey2 = new PartitionKeyBuilder().Add("USA").Add("Pittsburgh").Build(); - using (ResponseMessage readResponse = await containerInternal.ReadItemStreamAsync("document2", partitionKey2)) - { - Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.OK); - } - - - //Specifying a partial partition key should fail - Cosmos.PartitionKey partialPartitionKey = new PartitionKeyBuilder().Add("USA").Build(); - using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partialPartitionKey)) - { - Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.BadRequest); - Assert.AreEqual(pKDeleteResponse.CosmosException.SubStatusCode, (int)SubStatusCodes.PartitionKeyMismatch); - Assert.IsTrue(pKDeleteResponse.ErrorMessage.Contains("Partition key provided either doesn't correspond to definition in the collection or doesn't match partition key field values specified in the document.")); - } - } - finally - { - HttpConstants.Versions.CurrentVersion = currentVersion; - if (database != null) await database.DeleteAsync(); + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task PartitionKeyDeleteTestForSubpartitionedContainer(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + string currentVersion = HttpConstants.Versions.CurrentVersion; + HttpConstants.Versions.CurrentVersion = "2020-07-15"; + using CosmosClient client = TestCommon.CreateCosmosClient(true); + Cosmos.Database database = null; + try + { + database = await client.CreateDatabaseIfNotExistsAsync("mydb"); + + ContainerProperties containerProperties = new ContainerProperties("subpartitionedcontainer", new List { "/Country", "/City" }); + Container container = await database.CreateContainerAsync(containerProperties); + ContainerInternal containerInternal = (ContainerInternal)container; + + //Document create. + ItemResponse[] documents = new ItemResponse[5]; + Document doc1 = new Document { Id = "document1" }; + doc1.SetValue("Country", "USA"); + doc1.SetValue("City", "Redmond"); + documents[0] = await container.CreateItemAsync(doc1); + + doc1 = new Document { Id = "document2" }; + doc1.SetValue("Country", "USA"); + doc1.SetValue("City", "Pittsburgh"); + documents[1] = await container.CreateItemAsync(doc1); + + doc1 = new Document { Id = "document3" }; + doc1.SetValue("Country", "USA"); + doc1.SetValue("City", "Stonybrook"); + documents[2] = await container.CreateItemAsync(doc1); + + doc1 = new Document { Id = "document4" }; + doc1.SetValue("Country", "USA"); + doc1.SetValue("City", "Stonybrook"); + documents[3] = await container.CreateItemAsync(doc1); + + doc1 = new Document { Id = "document5" }; + doc1.SetValue("Country", "USA"); + doc1.SetValue("City", "Stonybrook"); + documents[4] = await container.CreateItemAsync(doc1); + + Cosmos.PartitionKey partitionKey1 = new PartitionKeyBuilder().Add("USA").Add("Stonybrook").Build(); + + using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partitionKey1)) + { + Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.OK); + } + using (ResponseMessage readResponse = await containerInternal.ReadItemStreamAsync("document5", partitionKey1)) + { + Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.NotFound); + Assert.AreEqual(readResponse.Headers.SubStatusCode, SubStatusCodes.Unknown); + } + + Cosmos.PartitionKey partitionKey2 = new PartitionKeyBuilder().Add("USA").Add("Pittsburgh").Build(); + using (ResponseMessage readResponse = await containerInternal.ReadItemStreamAsync("document2", partitionKey2)) + { + Assert.AreEqual(readResponse.StatusCode, HttpStatusCode.OK); + } + + + //Specifying a partial partition key should fail + Cosmos.PartitionKey partialPartitionKey = new PartitionKeyBuilder().Add("USA").Build(); + using (ResponseMessage pKDeleteResponse = await containerInternal.DeleteAllItemsByPartitionKeyStreamAsync(partialPartitionKey)) + { + Assert.AreEqual(pKDeleteResponse.StatusCode, HttpStatusCode.BadRequest); + Assert.AreEqual(pKDeleteResponse.CosmosException.SubStatusCode, (int)SubStatusCodes.PartitionKeyMismatch); + Assert.IsTrue(pKDeleteResponse.ErrorMessage.Contains("Partition key provided either doesn't correspond to definition in the collection or doesn't match partition key field values specified in the document.")); + } + } + finally + { + HttpConstants.Versions.CurrentVersion = currentVersion; + if (database != null) await database.DeleteAsync(); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } @@ -1914,86 +2195,114 @@ public async Task NegativeQueryTest() } } - [TestMethod] - public async Task ItemRequestOptionAccessConditionTest() - { - // Create an item - ToDoActivity testItem = (await ToDoActivity.CreateRandomItems(this.Container, 1, randomPartitionKey: true)).First(); - - ItemRequestOptions itemRequestOptions = new ItemRequestOptions() - { - IfMatchEtag = Guid.NewGuid().ToString(), - }; - - using (ResponseMessage responseMessage = await this.Container.UpsertItemStreamAsync( - streamPayload: TestCommon.SerializerCore.ToStream(testItem), - partitionKey: new Cosmos.PartitionKey(testItem.pk), - requestOptions: itemRequestOptions)) - { - Assert.IsNotNull(responseMessage); - Assert.IsNull(responseMessage.Content); - Assert.AreEqual(HttpStatusCode.PreconditionFailed, responseMessage.StatusCode, responseMessage.ErrorMessage); - Assert.AreNotEqual(responseMessage.Headers.ActivityId, Guid.Empty); - Assert.IsTrue(responseMessage.Headers.RequestCharge > 0); - Assert.IsFalse(string.IsNullOrEmpty(responseMessage.ErrorMessage)); - Assert.IsTrue(responseMessage.ErrorMessage.Contains("One of the specified pre-condition is not met")); - } - - try - { - ItemResponse response = await this.Container.UpsertItemAsync( - item: testItem, - requestOptions: itemRequestOptions); - Assert.Fail("Access condition should have failed"); - } - catch (CosmosException e) - { - Assert.IsNotNull(e); - Assert.AreEqual(HttpStatusCode.PreconditionFailed, e.StatusCode, e.Message); - Assert.AreNotEqual(e.ActivityId, Guid.Empty); - Assert.IsTrue(e.RequestCharge > 0); - string expectedResponseBody = $"{Environment.NewLine}Errors : [{Environment.NewLine} \"One of the specified pre-condition is not met. Learn more: https://aka.ms/CosmosDB/sql/errors/precondition-failed\"{Environment.NewLine}]{Environment.NewLine}"; - Assert.AreEqual(expectedResponseBody, e.ResponseBody); - string expectedMessage = $"Response status code does not indicate success: PreconditionFailed (412); Substatus: 0; ActivityId: {e.ActivityId}; Reason: ({expectedResponseBody});"; - Assert.AreEqual(expectedMessage, e.Message); - } - finally - { - ItemResponse deleteResponse = await this.Container.DeleteItemAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id); - Assert.IsNotNull(deleteResponse); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task ItemRequestOptionAccessConditionTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + // Create an item + ToDoActivity testItem = (await ToDoActivity.CreateRandomItems(this.Container, 1, randomPartitionKey: true)).First(); + + ItemRequestOptions itemRequestOptions = new ItemRequestOptions() + { + IfMatchEtag = Guid.NewGuid().ToString(), + }; + + using (ResponseMessage responseMessage = await this.Container.UpsertItemStreamAsync( + streamPayload: TestCommon.SerializerCore.ToStream(testItem), + partitionKey: new Cosmos.PartitionKey(testItem.pk), + requestOptions: itemRequestOptions)) + { + Assert.IsNotNull(responseMessage); + Assert.IsNull(responseMessage.Content); + Assert.AreEqual(HttpStatusCode.PreconditionFailed, responseMessage.StatusCode, responseMessage.ErrorMessage); + Assert.AreNotEqual(responseMessage.Headers.ActivityId, Guid.Empty); + Assert.IsTrue(responseMessage.Headers.RequestCharge > 0); + Assert.IsFalse(string.IsNullOrEmpty(responseMessage.ErrorMessage)); + Assert.IsTrue(responseMessage.ErrorMessage.Contains("One of the specified pre-condition is not met")); + } + + try + { + ItemResponse response = await this.Container.UpsertItemAsync( + item: testItem, + requestOptions: itemRequestOptions); + Assert.Fail("Access condition should have failed"); + } + catch (CosmosException e) + { + Assert.IsNotNull(e); + Assert.AreEqual(HttpStatusCode.PreconditionFailed, e.StatusCode, e.Message); + Assert.AreNotEqual(e.ActivityId, Guid.Empty); + Assert.IsTrue(e.RequestCharge > 0); + string expectedResponseBody = $"{Environment.NewLine}Errors : [{Environment.NewLine} \"One of the specified pre-condition is not met. Learn more: https://aka.ms/CosmosDB/sql/errors/precondition-failed\"{Environment.NewLine}]{Environment.NewLine}"; + Assert.AreEqual(expectedResponseBody, e.ResponseBody); + string expectedMessage = $"Response status code does not indicate success: PreconditionFailed (412); Substatus: 0; ActivityId: {e.ActivityId}; Reason: ({expectedResponseBody});"; + Assert.AreEqual(expectedMessage, e.Message); + } + finally + { + ItemResponse deleteResponse = await this.Container.DeleteItemAsync(partitionKey: new Cosmos.PartitionKey(testItem.pk), id: testItem.id); + Assert.IsNotNull(deleteResponse); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } - [TestMethod] - public async Task ItemReplaceAsyncTest() - { - // Create an item - ToDoActivity testItem = (await ToDoActivity.CreateRandomItems(this.Container, 1, randomPartitionKey: true)).First(); - - string originalId = testItem.id; - testItem.id = Guid.NewGuid().ToString(); - - ItemResponse response = await this.Container.ReplaceItemAsync( - id: originalId, - item: testItem); - - Assert.AreEqual(testItem.id, response.Resource.id); - Assert.AreNotEqual(originalId, response.Resource.id); - - string originalStatus = testItem.pk; - testItem.pk = Guid.NewGuid().ToString(); - - try - { - response = await this.Container.ReplaceItemAsync( - id: testItem.id, - partitionKey: new Cosmos.PartitionKey(originalStatus), - item: testItem); - Assert.Fail("Replace changing partition key is not supported."); - } - catch (CosmosException ce) - { - Assert.AreEqual((HttpStatusCode)400, ce.StatusCode); + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task ItemReplaceAsyncTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + // Create an item + ToDoActivity testItem = (await ToDoActivity.CreateRandomItems(this.Container, 1, randomPartitionKey: true)).First(); + + string originalId = testItem.id; + testItem.id = Guid.NewGuid().ToString(); + + ItemResponse response = await this.Container.ReplaceItemAsync( + id: originalId, + item: testItem); + + Assert.AreEqual(testItem.id, response.Resource.id); + Assert.AreNotEqual(originalId, response.Resource.id); + + string originalStatus = testItem.pk; + testItem.pk = Guid.NewGuid().ToString(); + + try + { + response = await this.Container.ReplaceItemAsync( + id: testItem.id, + partitionKey: new Cosmos.PartitionKey(originalStatus), + item: testItem); + Assert.Fail("Replace changing partition key is not supported."); + } + catch (CosmosException ce) + { + Assert.AreEqual((HttpStatusCode)400, ce.StatusCode); + } + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } } @@ -3049,59 +3358,87 @@ public async Task AutoGenerateIdPatternTest() Assert.AreEqual(itemWithoutId.pk, createdItem.pk); } - [TestMethod] - public async Task CustomPropertiesItemRequestOptionsTest() - { - string customHeaderName = "custom-header1"; - string customHeaderValue = "value1"; - - CosmosClient clientWithIntercepter = TestCommon.CreateCosmosClient( - builder => builder.WithTransportClientHandlerFactory(transportClient => new TransportClientHelper.TransportClientWrapper( - transportClient, - (uri, resourceOperation, request) => + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task CustomPropertiesItemRequestOptionsTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + string customHeaderName = "custom-header1"; + string customHeaderValue = "value1"; + + CosmosClient clientWithIntercepter = TestCommon.CreateCosmosClient( + builder => builder.WithTransportClientHandlerFactory(transportClient => new TransportClientHelper.TransportClientWrapper( + transportClient, + (uri, resourceOperation, request) => { - if (resourceOperation.resourceType == ResourceType.Document && - resourceOperation.operationType == OperationType.Create) + if (resourceOperation.resourceType == ResourceType.Document && + resourceOperation.operationType == OperationType.Create) { bool customHeaderExists = request.Properties.TryGetValue(customHeaderName, out object value); Assert.IsTrue(customHeaderExists); Assert.AreEqual(customHeaderValue, value); } - }))); - - Container container = clientWithIntercepter.GetContainer(this.database.Id, this.Container.Id); - - ToDoActivity temp = ToDoActivity.CreateRandomToDoActivity("TBD"); - - Dictionary properties = new Dictionary() + }))); + + Container container = clientWithIntercepter.GetContainer(this.database.Id, this.Container.Id); + + ToDoActivity temp = ToDoActivity.CreateRandomToDoActivity("TBD"); + + Dictionary properties = new Dictionary() { { customHeaderName, customHeaderValue}, - }; - - ItemRequestOptions ro = new ItemRequestOptions - { - Properties = properties - }; - - ItemResponse responseAstype = await container.CreateItemAsync( - partitionKey: new Cosmos.PartitionKey(temp.pk), - item: temp, - requestOptions: ro); - - Assert.AreEqual(HttpStatusCode.Created, responseAstype.StatusCode); + }; + + ItemRequestOptions ro = new ItemRequestOptions + { + Properties = properties + }; + + ItemResponse responseAstype = await container.CreateItemAsync( + partitionKey: new Cosmos.PartitionKey(temp.pk), + item: temp, + requestOptions: ro); + + Assert.AreEqual(HttpStatusCode.Created, responseAstype.StatusCode); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } - [TestMethod] - public async Task RegionsContactedTest() - { - ToDoActivity item = ToDoActivity.CreateRandomToDoActivity(); - ItemResponse response = await this.Container.CreateItemAsync(item, new Cosmos.PartitionKey(item.pk)); - Assert.IsNotNull(response.Diagnostics); - IReadOnlyList<(string region, Uri uri)> regionsContacted = response.Diagnostics.GetContactedRegions(); - Assert.AreEqual(regionsContacted.Count, 1); - Assert.AreEqual(regionsContacted[0].region, Regions.SouthCentralUS); - Assert.IsNotNull(regionsContacted[0].uri); + [TestMethod] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + public async Task RegionsContactedTest(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ToDoActivity item = ToDoActivity.CreateRandomToDoActivity(); + ItemResponse response = await this.Container.CreateItemAsync(item, new Cosmos.PartitionKey(item.pk)); + Assert.IsNotNull(response.Diagnostics); + IReadOnlyList<(string region, Uri uri)> regionsContacted = response.Diagnostics.GetContactedRegions(); + Assert.AreEqual(regionsContacted.Count, 1); + Assert.AreEqual(regionsContacted[0].region, Regions.SouthCentralUS); + Assert.IsNotNull(regionsContacted[0].uri); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } } [TestMethod] @@ -3627,6 +3964,50 @@ private static async Task TestNonePKForNonExistingContainer(Container container) { Assert.AreEqual(HttpStatusCode.NotFound, ex.StatusCode); } + } + + private static void AssertOnResponseSerializationBinaryType( + Stream inputStream) + { + if (inputStream != null) + { + MemoryStream binaryStream = new(); + inputStream.CopyTo(binaryStream); + byte[] content = binaryStream.ToArray(); + inputStream.Position = 0; + + Assert.IsTrue(content.Length > 0); + Assert.IsTrue(CosmosItemTests.IsBinaryFormat(content[0], JsonSerializationFormat.Binary)); + } + } + + private static void AssertOnResponseSerializationTextType( + Stream inputStream) + { + if (inputStream != null) + { + MemoryStream binaryStream = new(); + inputStream.CopyTo(binaryStream); + byte[] content = binaryStream.ToArray(); + inputStream.Position = 0; + + Assert.IsTrue(content.Length > 0); + Assert.IsTrue(CosmosItemTests.IsTextFormat(content[0], JsonSerializationFormat.Text)); + } + } + + private static bool IsBinaryFormat( + int firstByte, + JsonSerializationFormat desiredFormat) + { + return desiredFormat == JsonSerializationFormat.Binary && firstByte == (int)JsonSerializationFormat.Binary; + } + + private static bool IsTextFormat( + int firstByte, + JsonSerializationFormat desiredFormat) + { + return desiredFormat == JsonSerializationFormat.Text && firstByte < (int)JsonSerializationFormat.Binary; } } } diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosBufferedStreamWrapperTests.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosBufferedStreamWrapperTests.cs new file mode 100644 index 0000000000..cc55880a4a --- /dev/null +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosBufferedStreamWrapperTests.cs @@ -0,0 +1,230 @@ +//------------------------------------------------------------ +// Copyright (c) Microsoft Corporation. All rights reserved. +//------------------------------------------------------------ + +namespace Microsoft.Azure.Cosmos.Tests +{ + using System; + using System.IO; + using System.Text; + using System.Threading.Tasks; + using Microsoft.Azure.Cosmos.Serializer; + using Microsoft.Azure.Cosmos.Json; + using Microsoft.VisualStudio.TestTools.UnitTesting; + using Microsoft.Azure.Documents; + + [TestClass] + public class CosmosBufferedStreamWrapperTests + { + [TestMethod] + public async Task TestReadFirstByte() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + using (MemoryStream memoryStream = new(data)) + using (CosmosBufferedStreamWrapper bufferedStream = new (await StreamExtension.AsClonableStreamAsync(memoryStream), true)) + { + byte[] buffer = new byte[1]; + int bytesRead = bufferedStream.Read(buffer, 0, 1); + + Assert.AreEqual(1, bytesRead); + Assert.AreEqual((byte)'H', buffer[0]); + } + } + + [TestMethod] + public async Task TestReadAll() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + using (MemoryStream memoryStream = new (data)) + using (CosmosBufferedStreamWrapper bufferedStream = new (await StreamExtension.AsClonableStreamAsync(memoryStream), true)) + { + byte[] result = bufferedStream.ReadAll(); + + Assert.IsNotNull(result); + Assert.AreEqual(data.Length, result.Length); + CollectionAssert.AreEqual(data, result); + } + } + + [TestMethod] + public async Task TestReadAllAfterFirstByteRead() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + using (MemoryStream memoryStream = new(data)) + using (CosmosBufferedStreamWrapper bufferedStream = new(await StreamExtension.AsClonableStreamAsync(memoryStream), true)) + { + bufferedStream.GetJsonSerializationFormat(); // This will trigger the first byte read. + byte[] result = bufferedStream.ReadAll(); + + Assert.IsNotNull(result); + Assert.AreEqual(data.Length, result.Length); + CollectionAssert.AreEqual(data, result); + } + } + + [TestMethod] + public async Task TestGetJsonSerializationFormat() + { + byte[] data = new byte[] { (byte)JsonSerializationFormat.Binary }; + using (MemoryStream memoryStream = new (data)) + using (CosmosBufferedStreamWrapper bufferedStream = new (await StreamExtension.AsClonableStreamAsync(memoryStream), true)) + { + JsonSerializationFormat format = bufferedStream.GetJsonSerializationFormat(); + + Assert.AreEqual(JsonSerializationFormat.Binary, format); + } + } + + [TestMethod] + public async Task TestReadWithNonSeekableStream() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + + using NonSeekableMemoryStream memoryStream = new(data); + using CloneableStream clonableStream = await StreamExtension.AsClonableStreamAsync(memoryStream); + using CosmosBufferedStreamWrapper bufferedStream = new(clonableStream, true); + + Assert.IsTrue(bufferedStream.CanSeek); + JsonSerializationFormat format = bufferedStream.GetJsonSerializationFormat(); + + Assert.AreEqual(JsonSerializationFormat.Text, format); + + byte[] result = new byte[bufferedStream.Length]; + int bytes = bufferedStream.Read(result, 0, (int)bufferedStream.Length); + + Assert.IsNotNull(result); + Assert.AreEqual(bytes, result.Length); + Assert.AreEqual(data.Length, result.Length); + CollectionAssert.AreEqual(data, result); + } + + [TestMethod] + public async Task TestReadWithNonSeekableStreamAndSmallerOffset() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World! This is a sample test."); + + using NonSeekableMemoryStream memoryStream = new(data); + using CloneableStream clonableStream = await StreamExtension.AsClonableStreamAsync(memoryStream); + using CosmosBufferedStreamWrapper bufferedStream = new(clonableStream, true); + + Assert.IsTrue(bufferedStream.CanSeek); + JsonSerializationFormat format = bufferedStream.GetJsonSerializationFormat(); + + Assert.AreEqual(JsonSerializationFormat.Text, format); + + byte[] result = new byte[bufferedStream.Length]; + + int count = 0, chunk = 4, offset = 0, length = (int)bufferedStream.Length, totalBytes = 0; + while ((count = bufferedStream.Read(result, offset, Math.Min(chunk, (int)(length - bufferedStream.Position)))) > 0) + { + offset += count; + totalBytes += count; + } + + int count2 = 0, chunk2 = 3, offset2 = 0, length2 = (int)bufferedStream.Length-1, totalBytes2 = 0; + byte[] result2 = new byte[bufferedStream.Length]; + while ((count2 = bufferedStream.Read(result2, offset2, chunk2)) > 0) + { + offset2 += Math.Min(count2, length); + totalBytes2 += count2; + } + + Assert.IsNotNull(result); + Assert.AreEqual(totalBytes, result.Length); + Assert.AreEqual(data.Length, result.Length); + Assert.AreEqual(0, totalBytes2); + Assert.AreEqual(0, count2); + CollectionAssert.AreEqual(data, result); + } + + [TestMethod] + public async Task TestReadAllWithNonSeekableStream() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + + using NonSeekableMemoryStream memoryStream = new(data); + using CloneableStream clonableStream = await StreamExtension.AsClonableStreamAsync(memoryStream); + using CosmosBufferedStreamWrapper bufferedStream = new(clonableStream, true); + + Assert.IsTrue(bufferedStream.CanSeek); + JsonSerializationFormat format = bufferedStream.GetJsonSerializationFormat(); + + Assert.AreEqual(JsonSerializationFormat.Text, format); + + byte[] result = bufferedStream.ReadAll(); + + Assert.IsNotNull(result); + Assert.AreEqual(data.Length, result.Length); + CollectionAssert.AreEqual(data, result); + } + + [TestMethod] + public async Task TestWriteAndRead() + { + byte[] data = Encoding.UTF8.GetBytes("Hello, World!"); + using (MemoryStream memoryStream = new ()) + using (CosmosBufferedStreamWrapper bufferedStream = new (await StreamExtension.AsClonableStreamAsync(memoryStream), true)) + { + bufferedStream.Write(data, 0, data.Length); + bufferedStream.Position = 0; + + byte[] buffer = new byte[data.Length]; + int bytesRead = bufferedStream.Read(buffer, 0, buffer.Length); + + Assert.AreEqual(data.Length, bytesRead); + CollectionAssert.AreEqual(data, buffer); + } + } + + internal class NonSeekableMemoryStream : Stream + { + private readonly byte[] buffer; + private int position; + + public NonSeekableMemoryStream(byte[] data) + { + this.buffer = data; + } + + public override bool CanRead => true; + public override bool CanSeek => false; + public override bool CanWrite => false; + + public override long Length => this.buffer.Length; + + public override long Position + { + get => this.position; + set => throw new NotSupportedException("Seeking is not supported on this stream."); + } + + public override int Read(byte[] buffer, int offset, int count) + { + int bytesToRead = Math.Min(count, this.buffer.Length - this.position); + Array.Copy(this.buffer, this.position, buffer, offset, bytesToRead); + this.position += bytesToRead; + return bytesToRead; + } + + public override void Flush() + { + // No operation needed as this stream is read-only + } + + public override long Seek(long offset, SeekOrigin origin) + { + throw new NotSupportedException("Seeking is not supported on this stream."); + } + + public override void SetLength(long value) + { + throw new NotSupportedException("Setting the length is not supported on this stream."); + } + + public override void Write(byte[] buffer, int offset, int count) + { + throw new NotImplementedException(); + } + } + } +} diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosItemUnitTests.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosItemUnitTests.cs index 09766b2962..f9990e4b35 100644 --- a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosItemUnitTests.cs +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosItemUnitTests.cs @@ -15,52 +15,99 @@ namespace Microsoft.Azure.Cosmos.Tests using Microsoft.Azure.Documents; using Microsoft.VisualStudio.TestTools.UnitTesting; using Moq; + using Newtonsoft.Json; using Newtonsoft.Json.Linq; [TestClass] public class CosmosItemUnitTests { [TestMethod] - public async Task TestItemPartitionKeyTypes() + [DataRow(false, true, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is disabled at client level and enabled in container level.")] + [DataRow(true, true, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is enabled at client level and enabled in container level.")] + [DataRow(false, false, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is enabled at client level and disabled in container level.")] + [DataRow(false, true, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is disabled at client level and enabled in container level.")] + [DataRow(true, true, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is enabled at client level and enabled in container level.")] + [DataRow(false, false, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is enabled at client level and disabled in container level.")] + public async Task TestItemPartitionKeyTypes( + bool binaryEncodingEnabledInClient, + bool binaryEncodingEnabledInContainer, + bool useStjSerializer) { dynamic item = new { id = Guid.NewGuid().ToString(), pk = "FF627B77-568E-4541-A47E-041EAC10E46F", }; - await VerifyItemOperations(new Cosmos.PartitionKey(item.pk), "[\"FF627B77-568E-4541-A47E-041EAC10E46F\"]", item); + await VerifyItemOperations( + new Cosmos.PartitionKey(item.pk), + "[\"FF627B77-568E-4541-A47E-041EAC10E46F\"]", + item, + binaryEncodingEnabledInClient, + binaryEncodingEnabledInContainer, + useStjSerializer); item = new { id = Guid.NewGuid().ToString(), pk = 4567, }; - await VerifyItemOperations(new Cosmos.PartitionKey(item.pk), "[4567.0]", item); + await VerifyItemOperations( + new Cosmos.PartitionKey(item.pk), + "[4567.0]", + item, + binaryEncodingEnabledInClient, + binaryEncodingEnabledInContainer, + useStjSerializer); item = new { id = Guid.NewGuid().ToString(), pk = 4567.1234, }; - await VerifyItemOperations(new Cosmos.PartitionKey(item.pk), "[4567.1234]", item); + await VerifyItemOperations( + new Cosmos.PartitionKey(item.pk), + "[4567.1234]", + item, + binaryEncodingEnabledInClient, + binaryEncodingEnabledInContainer, + useStjSerializer); item = new { id = Guid.NewGuid().ToString(), pk = true, }; - await VerifyItemOperations(new Cosmos.PartitionKey(item.pk), "[true]", item); + await VerifyItemOperations( + new Cosmos.PartitionKey(item.pk), + "[true]", + item, + binaryEncodingEnabledInClient, + binaryEncodingEnabledInContainer, + useStjSerializer); } [TestMethod] - public async Task TestNullItemPartitionKeyFlag() + [DataRow(false, true, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is disabled at client level and enabled in container level.")] + [DataRow(true, true, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is enabled at client level and enabled in container level.")] + [DataRow(false, false, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, false, DisplayName = "Test scenario with CosmosJsonDotNetSerializer when binary encoding is enabled at client level and disabled in container level.")] + [DataRow(false, true, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is disabled at client level and enabled in container level.")] + [DataRow(true, true, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is enabled at client level and enabled in container level.")] + [DataRow(false, false, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, true, DisplayName = "Test scenario with CosmosSystemTextJsonSerializer when binary encoding is enabled at client level and disabled in container level.")] + public async Task TestNullItemPartitionKeyFlag( + bool binaryEncodingEnabledInClient, + bool binaryEncodingEnabledInContainer, + bool useStjSerializer) { dynamic testItem = new { id = Guid.NewGuid().ToString() }; - await VerifyItemOperations(new Cosmos.PartitionKey(Undefined.Value), "[{}]", testItem); + await VerifyItemOperations(new Cosmos.PartitionKey(Undefined.Value), "[{}]", testItem, binaryEncodingEnabledInClient, binaryEncodingEnabledInContainer, useStjSerializer); } [TestMethod] @@ -78,109 +125,222 @@ public async Task TestNullItemPartitionKeyBehavior() } [TestMethod] - public async Task TestGetPartitionKeyValueFromStreamAsync() + public async Task TestBinaryResponseOnItemStreamOperations() { - ContainerInternal mockContainer = (ContainerInternal)MockCosmosUtil.CreateMockCosmosClient().GetContainer("TestDb", "Test"); - Mock containerMock = new Mock(); - ContainerInternal container = containerMock.Object; - - containerMock.Setup(e => e.GetPartitionKeyPathTokensAsync(It.IsAny(), It.IsAny())) - .Returns(Task.FromResult((IReadOnlyList>)new List> { new List { "pk" } })); - containerMock - .Setup( - x => x.GetPartitionKeyValueFromStreamAsync( - It.IsAny(), - It.IsAny(), - It.IsAny())) - .Returns( - (stream, trace, cancellationToken) => mockContainer.GetPartitionKeyValueFromStreamAsync( - stream, - trace, - cancellationToken)); - - DateTime dateTime = new DateTime(2019, 05, 15, 12, 1, 2, 3, DateTimeKind.Utc); - Guid guid = Guid.NewGuid(); - - //Test supported types - List supportedTypesToTest = new List { - new { pk = true }, - new { pk = false }, - new { pk = byte.MaxValue }, - new { pk = sbyte.MaxValue }, - new { pk = short.MaxValue }, - new { pk = ushort.MaxValue }, - new { pk = int.MaxValue }, - new { pk = uint.MaxValue }, - new { pk = long.MaxValue }, - new { pk = ulong.MaxValue }, - new { pk = float.MaxValue }, - new { pk = double.MaxValue }, - new { pk = decimal.MaxValue }, - new { pk = char.MaxValue }, - new { pk = "test" }, - new { pk = dateTime }, - new { pk = guid }, + dynamic item = new + { + id = Guid.NewGuid().ToString(), + pk = "FF627B77-568E-4541-A47E-041EAC10E46F", }; - foreach (dynamic poco in supportedTypesToTest) + ItemRequestOptions requestOptions = new ItemRequestOptions() { - object pk = await container.GetPartitionKeyValueFromStreamAsync( - MockCosmosUtil.Serializer.ToStream(poco), - NoOpTrace.Singleton, - default(CancellationToken)); - if (pk is bool boolValue) + EnableContentResponseOnWrite = true, + EnableBinaryResponseOnPointOperations = true + }; + + await VerifyItemOperations( + new Cosmos.PartitionKey(item.pk), + "[\"FF627B77-568E-4541-A47E-041EAC10E46F\"]", + item, + true, + true, + false, + requestOptions); + } + + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task CreateItemAsync_WithNonSeekableStream_ShouldConvertToClonnableStream(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) { - Assert.AreEqual(poco.pk, boolValue); + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); } - else if (pk is double doubleValue) + + dynamic item = new { - if (poco.pk is float) - { - Assert.AreEqual(poco.pk, Convert.ToSingle(pk)); - } - else if (poco.pk is double) - { - Assert.AreEqual(poco.pk, Convert.ToDouble(pk)); - } - else if (poco.pk is decimal) + id = Guid.NewGuid().ToString(), + pk = "FF627B77-568E-4541-A47E-041EAC10E46F", + }; + + ItemRequestOptions options = new(); + + ResponseMessage response = null; + HttpStatusCode httpStatusCode = HttpStatusCode.OK; + int testHandlerHitCount = 0; + string itemResponseString = "{\r\n \\\"id\\\": \\\"60362d85-ce1e-4ceb-9af3-f2ddfebf4547\\\",\r\n \\\"pk\\\": \\\"pk\\\",\r\n \\\"name\\\": \\\"1856531480\\\",\r\n " + + "\\\"email\\\": \\\"dkunda@test.com\\\",\r\n \\\"body\\\": \\\"This document is intended for binary encoding test.\\\",\r\n \\\"_rid\\\": \\\"fIsUAKsjjj0BAAAAAAAAAA==\\\",\r\n " + + "\\\"_self\\\": \\\"dbs/fIsUAA==/colls/fIsUAKsjjj0=/docs/fIsUAKsjjj0BAAAAAAAAAA==/\\\",\r\n \\\"_etag\\\": \\\"\\\\\"510096bc-0000-0d00-0000-66ccf70b0000\\\\\"\\\",\r\n " + + "\\\"_attachments\\\": \\\"attachments/\\\",\r\n \\\"_ts\\\": 1724708619\r\n}"; + + TestHandler testHandler = new TestHandler((request, cancellationToken) => + { + Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer")); + Assert.AreEqual(options, request.RequestOptions); + Assert.AreEqual(ResourceType.Document, request.ResourceType); + Assert.IsNotNull(request.Headers.PartitionKey); + // Assert.AreEqual("\"[4567.1234]\"", request.Headers.PartitionKey); + testHandlerHitCount++; + + bool shouldReturnBinaryResponse = request.Headers[HttpConstants.HttpHeaders.SupportedSerializationFormats] != null + && request.Headers[HttpConstants.HttpHeaders.SupportedSerializationFormats].Equals(SupportedSerializationFormats.CosmosBinary.ToString()); + + response = new ResponseMessage(httpStatusCode, request, errorMessage: null) { - Assert.AreEqual(Convert.ToDouble(poco.pk), (double)pk); - } + Content = shouldReturnBinaryResponse + ? CosmosSerializerUtils.ConvertInputToNonSeekableBinaryStream( + itemResponseString, + JsonSerializer.Create()) + : CosmosSerializerUtils.ConvertInputToTextStream( + itemResponseString, + JsonSerializer.Create()) + }; + return Task.FromResult(response); + }); + + using CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( + (builder) => builder.AddCustomHandlers(testHandler)); + + Container container = client.GetDatabase("testdb") + .GetContainer("testcontainer"); + + ItemResponse itemResponse = await container.CreateItemAsync( + item: item, + requestOptions: options); + + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } + } + + [TestMethod] + [DataRow(true, DisplayName = "Test scenario when binary encoding is enabled at client level.")] + [DataRow(false, DisplayName = "Test scenario when binary encoding is disabled at client level.")] + public async Task TestGetPartitionKeyValueFromStreamAsync(bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); } - else if (pk is string stringValue) + + ContainerInternal mockContainer = (ContainerInternal)MockCosmosUtil.CreateMockCosmosClient().GetContainer("TestDb", "Test"); + Mock containerMock = new Mock(); + ContainerInternal container = containerMock.Object; + + containerMock.Setup(e => e.GetPartitionKeyPathTokensAsync(It.IsAny(), It.IsAny())) + .Returns(Task.FromResult((IReadOnlyList>)new List> { new List { "pk" } })); + containerMock + .Setup( + x => x.GetPartitionKeyValueFromStreamAsync( + It.IsAny(), + It.IsAny(), + It.IsAny())) + .Returns( + (stream, trace, cancellationToken) => mockContainer.GetPartitionKeyValueFromStreamAsync( + stream, + trace, + cancellationToken)); + + DateTime dateTime = new DateTime(2019, 05, 15, 12, 1, 2, 3, DateTimeKind.Utc); + Guid guid = Guid.NewGuid(); + + //Test supported types + List supportedTypesToTest = new List { + new { pk = true }, + new { pk = false }, + new { pk = byte.MaxValue }, + new { pk = sbyte.MaxValue }, + new { pk = short.MaxValue }, + new { pk = ushort.MaxValue }, + new { pk = int.MaxValue }, + new { pk = uint.MaxValue }, + new { pk = long.MaxValue }, + new { pk = ulong.MaxValue }, + new { pk = float.MaxValue }, + new { pk = double.MaxValue }, + new { pk = decimal.MaxValue }, + new { pk = char.MaxValue }, + new { pk = "test" }, + new { pk = dateTime }, + new { pk = guid }, + }; + + foreach (dynamic poco in supportedTypesToTest) { - if (poco.pk is DateTime) + Stream stream = MockCosmosUtil.Serializer.ToStream(poco); + object pk = await container.GetPartitionKeyValueFromStreamAsync( + stream, + NoOpTrace.Singleton, + default); + if (pk is bool boolValue) { - Assert.AreEqual(poco.pk.ToString("yyyy-MM-ddTHH:mm:ss.fffZ"), stringValue); + Assert.AreEqual(poco.pk, boolValue); } - else + else if (pk is double doubleValue) { - Assert.AreEqual(poco.pk.ToString(), (string)pk); + if (poco.pk is float) + { + Assert.AreEqual(poco.pk, Convert.ToSingle(pk)); + } + else if (poco.pk is double) + { + Assert.AreEqual(poco.pk, Convert.ToDouble(pk)); + } + else if (poco.pk is decimal) + { + Assert.AreEqual(Convert.ToDouble(poco.pk), (double)pk); + } + } + else if (pk is string stringValue) + { + if (poco.pk is DateTime) + { + Assert.AreEqual(poco.pk.ToString("yyyy-MM-ddTHH:mm:ss.fffZ"), stringValue); + } + else + { + Assert.AreEqual(poco.pk.ToString(), (string)pk); + } } } - } - //Unsupported types should throw - List unsupportedTypesToTest = new List { + //Unsupported types should throw + List unsupportedTypesToTest = new List { new { pk = new { test = "test" } }, new { pk = new int[]{ 1, 2, 3 } }, new { pk = new ArraySegment(new byte[]{ 0 }) }, }; - foreach (dynamic poco in unsupportedTypesToTest) - { - await Assert.ThrowsExceptionAsync(async () => await container.GetPartitionKeyValueFromStreamAsync( - MockCosmosUtil.Serializer.ToStream(poco), + foreach (dynamic poco in unsupportedTypesToTest) + { + await Assert.ThrowsExceptionAsync(async () => await container.GetPartitionKeyValueFromStreamAsync( + MockCosmosUtil.Serializer.ToStream(poco), + NoOpTrace.Singleton, + default(CancellationToken))); + } + + //null should return null + object pkValue = await container.GetPartitionKeyValueFromStreamAsync( + MockCosmosUtil.Serializer.ToStream(new { pk = (object)null }), NoOpTrace.Singleton, - default(CancellationToken))); + default); + Assert.AreEqual(Cosmos.PartitionKey.Null, pkValue); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); } - - //null should return null - object pkValue = await container.GetPartitionKeyValueFromStreamAsync( - MockCosmosUtil.Serializer.ToStream(new { pk = (object)null }), - NoOpTrace.Singleton, - default); - Assert.AreEqual(Cosmos.PartitionKey.Null, pkValue); } [TestMethod] @@ -455,47 +615,6 @@ public async Task PartitionKeyDeleteUnitTest() await this.VerifyPartitionKeyDeleteOperation(new Cosmos.PartitionKey(item.pk), "[\"FF627B77-568E-4541-A47E-041EAC10E46F\"]"); } - [TestMethod] - public async Task VerifyPatchItemStreamOperation() - { - ResponseMessage response = null; - ItemRequestOptions requestOptions = null; - Mock mockPayload = new Mock(); - HttpStatusCode httpStatusCode = HttpStatusCode.OK; - int testHandlerHitCount = 0; - TestHandler testHandler = new TestHandler((request, cancellationToken) => - { - Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer/docs/cdbBinaryIdRequest")); - Assert.AreEqual(Cosmos.PartitionKey.Null.ToJsonString(), request.Headers.PartitionKey); - Assert.AreEqual(ResourceType.Document, request.ResourceType); - Assert.AreEqual(OperationType.Patch, request.OperationType); - Assert.AreEqual(mockPayload.Object, request.Content); - Assert.AreEqual(requestOptions, request.RequestOptions); - testHandlerHitCount++; - response = new ResponseMessage(httpStatusCode, request, errorMessage: null) - { - Content = request.Content - }; - return Task.FromResult(response); - }); - - CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( - (builder) => builder.AddCustomHandlers(testHandler)); - - Container container = client.GetDatabase("testdb") - .GetContainer("testcontainer"); - - ContainerInternal containerInternal = (ContainerInternal)container; - ResponseMessage responseMessage = await containerInternal.PatchItemStreamAsync( - id: "cdbBinaryIdRequest", - partitionKey: Cosmos.PartitionKey.Null, - streamPayload: mockPayload.Object, - requestOptions: requestOptions); - Assert.IsNotNull(responseMessage); - Assert.AreEqual(httpStatusCode, responseMessage.StatusCode); - Assert.AreEqual(1, testHandlerHitCount, "The operation did not make it to the handler"); - } - [TestMethod] public async Task TestNestedPartitionKeyValueFromStreamAsync() { @@ -812,129 +931,213 @@ private async Task VerifyItemOperations( Cosmos.PartitionKey partitionKey, string partitionKeySerialized, dynamic testItem, + bool binaryEncodingEnabledInClient, + bool binaryEncodingEnabledInContainer, + bool useStjSerializer, ItemRequestOptions requestOptions = null) { - ResponseMessage response = null; - HttpStatusCode httpStatusCode = HttpStatusCode.OK; - int testHandlerHitCount = 0; - TestHandler testHandler = new TestHandler((request, cancellationToken) => + try { - Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer")); - Assert.AreEqual(requestOptions, request.RequestOptions); - Assert.AreEqual(ResourceType.Document, request.ResourceType); - Assert.IsNotNull(request.Headers.PartitionKey); - Assert.AreEqual(partitionKeySerialized, request.Headers.PartitionKey); - testHandlerHitCount++; - response = new ResponseMessage(httpStatusCode, request, errorMessage: null) + if (binaryEncodingEnabledInClient) { - Content = request.Content - }; - return Task.FromResult(response); - }); + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } - using CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( - (builder) => builder.AddCustomHandlers(testHandler)); + ResponseMessage response = null; + HttpStatusCode httpStatusCode = HttpStatusCode.OK; + int testHandlerHitCount = 0; + string itemResponseString = "{\r\n \\\"id\\\": \\\"60362d85-ce1e-4ceb-9af3-f2ddfebf4547\\\",\r\n \\\"pk\\\": \\\"pk\\\",\r\n \\\"name\\\": \\\"1856531480\\\",\r\n " + + "\\\"email\\\": \\\"dkunda@test.com\\\",\r\n \\\"body\\\": \\\"This document is intended for binary encoding test.\\\",\r\n \\\"_rid\\\": \\\"fIsUAKsjjj0BAAAAAAAAAA==\\\",\r\n " + + "\\\"_self\\\": \\\"dbs/fIsUAA==/colls/fIsUAKsjjj0=/docs/fIsUAKsjjj0BAAAAAAAAAA==/\\\",\r\n \\\"_etag\\\": \\\"\\\\\"510096bc-0000-0d00-0000-66ccf70b0000\\\\\"\\\",\r\n " + + "\\\"_attachments\\\": \\\"attachments/\\\",\r\n \\\"_ts\\\": 1724708619\r\n}"; - Container container = client.GetDatabase("testdb") - .GetContainer("testcontainer"); + TestHandler testHandler = new TestHandler((request, cancellationToken) => + { + Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer")); + Assert.AreEqual(requestOptions, request.RequestOptions); + Assert.AreEqual(ResourceType.Document, request.ResourceType); + Assert.IsNotNull(request.Headers.PartitionKey); + Assert.AreEqual(partitionKeySerialized, request.Headers.PartitionKey); + testHandlerHitCount++; + + bool shouldReturnBinaryResponse = binaryEncodingEnabledInContainer + && request.Headers[HttpConstants.HttpHeaders.SupportedSerializationFormats] != null + && request.Headers[HttpConstants.HttpHeaders.SupportedSerializationFormats].Equals(SupportedSerializationFormats.CosmosBinary.ToString()); + + response = new ResponseMessage(httpStatusCode, request, errorMessage: null) + { + Content = shouldReturnBinaryResponse + ? CosmosSerializerUtils.ConvertInputToBinaryStream( + itemResponseString, + JsonSerializer.Create()) + : CosmosSerializerUtils.ConvertInputToTextStream( + itemResponseString, + JsonSerializer.Create()) + }; + return Task.FromResult(response); + }); - ItemResponse itemResponse = await container.CreateItemAsync( - item: testItem, - requestOptions: requestOptions); - Assert.IsNotNull(itemResponse); - Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + using CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( + (builder) => + { + if (useStjSerializer) + { + builder.WithSystemTextJsonSerializerOptions(new System.Text.Json.JsonSerializerOptions()); + } - itemResponse = await container.ReadItemAsync( - partitionKey: partitionKey, - id: testItem.id, - requestOptions: requestOptions); - Assert.IsNotNull(itemResponse); - Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + builder.AddCustomHandlers(testHandler); + }); - itemResponse = await container.UpsertItemAsync( - item: testItem, - requestOptions: requestOptions); - Assert.IsNotNull(itemResponse); - Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Container container = client.GetDatabase("testdb") + .GetContainer("testcontainer"); - itemResponse = await container.ReplaceItemAsync( - id: testItem.id, - item: testItem, - requestOptions: requestOptions); - Assert.IsNotNull(itemResponse); - Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + ItemResponse itemResponse = await container.CreateItemAsync( + item: testItem, + requestOptions: requestOptions); + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); - itemResponse = await container.DeleteItemAsync( - partitionKey: partitionKey, - id: testItem.id, - requestOptions: requestOptions); - Assert.IsNotNull(itemResponse); - Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + itemResponse = await container.ReadItemAsync( + partitionKey: partitionKey, + id: testItem.id, + requestOptions: requestOptions); + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); - Assert.AreEqual(5, testHandlerHitCount, "An operation did not make it to the handler"); + itemResponse = await container.UpsertItemAsync( + item: testItem, + requestOptions: requestOptions); + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); - using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) - { - using (ResponseMessage streamResponse = await container.CreateItemStreamAsync( + itemResponse = await container.ReplaceItemAsync( + id: testItem.id, + item: testItem, + requestOptions: requestOptions); + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); + + itemResponse = await container.DeleteItemAsync( partitionKey: partitionKey, - streamPayload: itemStream, - requestOptions: requestOptions)) + id: testItem.id, + requestOptions: requestOptions); + Assert.IsNotNull(itemResponse); + Assert.AreEqual(httpStatusCode, itemResponse.StatusCode); + Assert.AreEqual(itemResponseString, itemResponse.Resource.ToString()); + + Assert.AreEqual(5, testHandlerHitCount, "An operation did not make it to the handler"); + + using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) { - Assert.IsNotNull(streamResponse); - Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + using (ResponseMessage streamResponse = await container.CreateItemStreamAsync( + partitionKey: partitionKey, + streamPayload: itemStream, + requestOptions: requestOptions)) + { + Assert.IsNotNull(streamResponse); + Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + CosmosItemUnitTests.AssertOnBinaryEncodedContent( + streamResponse, + shouldExpectBinaryResponse: requestOptions?.EnableBinaryResponseOnPointOperations); + } } - } - using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) - { - using (ResponseMessage streamResponse = await container.ReadItemStreamAsync( - partitionKey: partitionKey, - id: testItem.id, - requestOptions: requestOptions)) + using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) { - Assert.IsNotNull(streamResponse); - Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + using (ResponseMessage streamResponse = await container.ReadItemStreamAsync( + partitionKey: partitionKey, + id: testItem.id, + requestOptions: requestOptions)) + { + Assert.IsNotNull(streamResponse); + Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + CosmosItemUnitTests.AssertOnBinaryEncodedContent( + streamResponse, + shouldExpectBinaryResponse: requestOptions?.EnableBinaryResponseOnPointOperations); + } } - } - using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) - { - using (ResponseMessage streamResponse = await container.UpsertItemStreamAsync( - partitionKey: partitionKey, - streamPayload: itemStream, - requestOptions: requestOptions)) + using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) { - Assert.IsNotNull(streamResponse); - Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + using (ResponseMessage streamResponse = await container.UpsertItemStreamAsync( + partitionKey: partitionKey, + streamPayload: itemStream, + requestOptions: requestOptions)) + { + Assert.IsNotNull(streamResponse); + Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + CosmosItemUnitTests.AssertOnBinaryEncodedContent( + streamResponse, + shouldExpectBinaryResponse: requestOptions?.EnableBinaryResponseOnPointOperations); + } } - } - using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) - { - using (ResponseMessage streamResponse = await container.ReplaceItemStreamAsync( - partitionKey: partitionKey, - id: testItem.id, - streamPayload: itemStream, - requestOptions: requestOptions)) + using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) { - Assert.IsNotNull(streamResponse); - Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + using (ResponseMessage streamResponse = await container.ReplaceItemStreamAsync( + partitionKey: partitionKey, + id: testItem.id, + streamPayload: itemStream, + requestOptions: requestOptions)) + { + Assert.IsNotNull(streamResponse); + Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + CosmosItemUnitTests.AssertOnBinaryEncodedContent( + streamResponse, + shouldExpectBinaryResponse: requestOptions?.EnableBinaryResponseOnPointOperations); + } } - } - using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) - { - using (ResponseMessage streamResponse = await container.DeleteItemStreamAsync( - partitionKey: partitionKey, - id: testItem.id, - requestOptions: requestOptions)) + using (Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem)) { - Assert.IsNotNull(streamResponse); - Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + using (ResponseMessage streamResponse = await container.DeleteItemStreamAsync( + partitionKey: partitionKey, + id: testItem.id, + requestOptions: requestOptions)) + { + Assert.IsNotNull(streamResponse); + Assert.AreEqual(httpStatusCode, streamResponse.StatusCode); + CosmosItemUnitTests.AssertOnBinaryEncodedContent( + streamResponse, + shouldExpectBinaryResponse: requestOptions?.EnableBinaryResponseOnPointOperations); + } } + + Assert.AreEqual(10, testHandlerHitCount, "A stream operation did not make it to the handler"); } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } + } - Assert.AreEqual(10, testHandlerHitCount, "A stream operation did not make it to the handler"); + private static void AssertOnBinaryEncodedContent( + ResponseMessage streamResponse, + bool? shouldExpectBinaryResponse) + { + if (shouldExpectBinaryResponse != null && shouldExpectBinaryResponse.Value) + { + Assert.IsTrue( + CosmosSerializerUtils.CheckFirstBufferByte( + streamResponse.Content, + Cosmos.Json.JsonSerializationFormat.Binary, + out byte[] byteArray)); + Assert.IsNotNull(byteArray); + Assert.AreEqual(Cosmos.Json.JsonSerializationFormat.Binary, (Cosmos.Json.JsonSerializationFormat)byteArray[0]); + } + else + { + Assert.IsFalse( + CosmosSerializerUtils.CheckFirstBufferByte( + streamResponse.Content, + Cosmos.Json.JsonSerializationFormat.Binary, + out byte[] byteArray)); + Assert.IsNull(byteArray); + } } private async Task VerifyPartitionKeyDeleteOperation( diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosSerializationUtilTests.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosSerializationUtilTests.cs new file mode 100644 index 0000000000..38e327f8ca --- /dev/null +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/CosmosSerializationUtilTests.cs @@ -0,0 +1,145 @@ +//------------------------------------------------------------ +// Copyright (c) Microsoft Corporation. All rights reserved. +//------------------------------------------------------------ + +namespace Microsoft.Azure.Cosmos.Tests +{ + using System.IO; + using System.Threading.Tasks; + using Microsoft.Azure.Cosmos.Json; + using Microsoft.Azure.Documents; + using Microsoft.VisualStudio.TestTools.UnitTesting; + + /// + /// Test class for + /// + [TestClass] + public class CosmosSerializationUtilTests + { + [TestMethod] + public void GetStringWithPropertyNamingPolicy_CamelCase() + { + // Arrange + CosmosLinqSerializerOptions options = new() { PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase }; + string propertyName = "TestProperty"; + + // Act + string result = CosmosSerializationUtil.GetStringWithPropertyNamingPolicy(options, propertyName); + + // Assert + Assert.AreEqual("testProperty", result); + } + + [TestMethod] + public void GetStringWithPropertyNamingPolicy_Default() + { + // Arrange + CosmosLinqSerializerOptions options = new() { PropertyNamingPolicy = CosmosPropertyNamingPolicy.Default }; + string propertyName = "TestProperty"; + + // Act + string result = CosmosSerializationUtil.GetStringWithPropertyNamingPolicy(options, propertyName); + + // Assert + Assert.AreEqual("TestProperty", result); + } + + [TestMethod] + public void IsBinaryFormat_True() + { + // Arrange + int firstByte = (int)JsonSerializationFormat.Binary; + JsonSerializationFormat format = JsonSerializationFormat.Binary; + + // Act + bool result = CosmosSerializerUtils.IsBinaryFormat(firstByte, format); + + // Assert + Assert.IsTrue(result); + } + + [TestMethod] + public void IsBinaryFormat_False() + { + // Arrange + int firstByte = (int)JsonSerializationFormat.Text; + JsonSerializationFormat format = JsonSerializationFormat.Binary; + + // Act + bool result = CosmosSerializerUtils.IsBinaryFormat(firstByte, format); + + // Assert + Assert.IsFalse(result); + } + + [TestMethod] + public void IsTextFormat_True() + { + // Arrange + int firstByte = (int)JsonSerializationFormat.Text; + JsonSerializationFormat format = JsonSerializationFormat.Text; + + // Act + bool result = CosmosSerializerUtils.IsTextFormat(firstByte, format); + + // Assert + Assert.IsTrue(result); + } + + [TestMethod] + public void IsTextFormat_False() + { + // Arrange + int firstByte = (int)JsonSerializationFormat.Binary; + JsonSerializationFormat format = JsonSerializationFormat.Text; + + // Act + bool result = CosmosSerializerUtils.IsTextFormat(firstByte, format); + + // Assert + Assert.IsFalse(result); + } + + [TestMethod] + [DataRow("text", "binary", DisplayName = "Validate Text to Binary Conversation.")] + [DataRow("binary", "text", DisplayName = "Validate Binary to Text Conversation.")] + public async Task TrySerializeStreamToTargetFormat_Success(string expected, string target) + { + // Arrange + JsonSerializationFormat expectedFormat = expected.Equals("text") ? JsonSerializationFormat.Text : JsonSerializationFormat.Binary; + JsonSerializationFormat targetFormat = target.Equals("text") ? JsonSerializationFormat.Text : JsonSerializationFormat.Binary; + string json = "{\"name\":\"test\"}"; + + Stream inputStream = JsonSerializationFormat.Text.Equals(expectedFormat) + ? CosmosSerializerUtils.ConvertInputToTextStream(json, Newtonsoft.Json.JsonSerializer.Create()) + : CosmosSerializerUtils.ConvertInputToBinaryStream(json, Newtonsoft.Json.JsonSerializer.Create()); + + // Act + CloneableStream cloneableStream = await StreamExtension.AsClonableStreamAsync(inputStream); + Stream outputStream = CosmosSerializationUtil.TrySerializeStreamToTargetFormat(targetFormat, cloneableStream); + + // Assert + Assert.IsNotNull(outputStream); + Assert.IsTrue(CosmosSerializerUtils.CheckFirstBufferByte(outputStream, targetFormat, out byte[] binBytes)); + Assert.IsNotNull(binBytes); + Assert.IsTrue(binBytes.Length > 0); + } + + [TestMethod] + public async Task TrySerializeStreamToTargetFormat_Failure() + { + // Arrange + string json = "{\"name\":\"test\"}"; + Stream inputStream = CosmosSerializerUtils.ConvertInputToTextStream(json, Newtonsoft.Json.JsonSerializer.Create()); + JsonSerializationFormat targetFormat = JsonSerializationFormat.Text; + + // Act + CloneableStream cloneableStream = await StreamExtension.AsClonableStreamAsync(inputStream); + Stream outputStream = CosmosSerializationUtil.TrySerializeStreamToTargetFormat(targetFormat, cloneableStream); + + // Assert + Assert.IsNotNull(outputStream); + Assert.AreEqual(cloneableStream, outputStream); + } + } +} \ No newline at end of file diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Patch/PatchOperationTests.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Patch/PatchOperationTests.cs index 0ca67e60c5..339522b806 100644 --- a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Patch/PatchOperationTests.cs +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Patch/PatchOperationTests.cs @@ -5,7 +5,13 @@ namespace Microsoft.Azure.Cosmos.Tests { using System; + using System.Collections.Generic; using System.IO; + using System.Net; + using System.Net.Mail; + using System.Threading.Tasks; + using global::Azure.Core; + using global::Azure; using Microsoft.Azure.Cosmos; using Microsoft.VisualStudio.TestTools.UnitTesting; @@ -67,6 +73,140 @@ public void ConstructPatchOperationTest() PatchOperationTests.ValidateOperations(operation, PatchOperationType.Set, null); } + [TestMethod] + [DataRow(false, true, DisplayName = "Test scenario when binary encoding is disabled at client level and supported in container level.")] + [DataRow(true, true, DisplayName = "Test scenario when binary encoding is enabled at client level and supported in container level.")] + [DataRow(false, false, DisplayName = "Test scenario when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, DisplayName = "Test scenario when binary encoding is enabled at client level and disabled in container level.")] + public async Task VerifyPatchItemOperation( + bool binaryEncodingEnabledInContainer, + bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + ItemRequestOptions requestOptions = null; + HttpStatusCode httpStatusCode = HttpStatusCode.OK; + int testHandlerHitCount = 0; + TestHandler testHandler = new TestHandler((request, cancellationToken) => + { + Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer/docs/cdbBinaryIdRequest")); + Assert.AreEqual(new PartitionKey("FF627B77-568E-4541-A47E-041EAC10E46F").ToString(), request.Headers.PartitionKey); + Assert.AreEqual(Documents.ResourceType.Document, request.ResourceType); + Assert.AreEqual(Documents.OperationType.Patch, request.OperationType); + Assert.AreEqual(requestOptions, request.RequestOptions); + testHandlerHitCount++; + + return Task.FromResult( + PatchOperationTests.GetContainerItemResponse( + request, + httpStatusCode, + binaryEncodingEnabledInContainer)); + }); + + CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( + (builder) => builder.AddCustomHandlers(testHandler)); + + Container container = client.GetDatabase("testdb") + .GetContainer("testcontainer"); + + ContainerInternal containerInternal = (ContainerInternal)container; + + dynamic testItem = new + { + id = Guid.NewGuid().ToString(), + pk = "FF627B77-568E-4541-A47E-041EAC10E46F", + }; + + List patchOperations = new List() + { + PatchOperation.Add("/name", "replaced_name") + }; + + ItemResponse responseMessage = await containerInternal.PatchItemAsync( + id: "cdbBinaryIdRequest", + partitionKey: new Cosmos.PartitionKey(testItem.pk), + patchOperations: patchOperations); + + Assert.IsNotNull(responseMessage); + Assert.AreEqual(httpStatusCode, responseMessage.StatusCode); + Assert.AreEqual(1, testHandlerHitCount, "The operation did not make it to the handler"); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } + } + + [TestMethod] + [DataRow(false, true, DisplayName = "Test scenario when binary encoding is disabled at client level and supported in container level.")] + [DataRow(true, true, DisplayName = "Test scenario when binary encoding is enabled at client level and supported in container level.")] + [DataRow(false, false, DisplayName = "Test scenario when binary encoding iis disabled at client level and disabled in container level.")] + [DataRow(true, false, DisplayName = "Test scenario when binary encoding is enabled at client level and disabled in container level.")] + public async Task VerifyPatchItemStreamOperation( + bool binaryEncodingEnabledInContainer, + bool binaryEncodingEnabledInClient) + { + try + { + if (binaryEncodingEnabledInClient) + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True"); + } + + dynamic testItem = new + { + id = Guid.NewGuid().ToString(), + pk = "FF627B77-568E-4541-A47E-041EAC10E46F", + }; + + using Stream itemStream = MockCosmosUtil.Serializer.ToStream(testItem); + + ItemRequestOptions requestOptions = null; + HttpStatusCode httpStatusCode = HttpStatusCode.OK; + int testHandlerHitCount = 0; + TestHandler testHandler = new TestHandler((request, cancellationToken) => + { + Assert.IsTrue(request.RequestUri.OriginalString.StartsWith(@"dbs/testdb/colls/testcontainer/docs/cdbBinaryIdRequest")); + Assert.AreEqual(PartitionKey.Null.ToJsonString(), request.Headers.PartitionKey); + Assert.AreEqual(Documents.ResourceType.Document, request.ResourceType); + Assert.AreEqual(Documents.OperationType.Patch, request.OperationType); + Assert.AreEqual(requestOptions, request.RequestOptions); + testHandlerHitCount++; + + return Task.FromResult( + PatchOperationTests.GetContainerItemResponse( + request, + httpStatusCode, + binaryEncodingEnabledInContainer)); + }); + + CosmosClient client = MockCosmosUtil.CreateMockCosmosClient( + (builder) => builder.AddCustomHandlers(testHandler)); + + Container container = client.GetDatabase("testdb") + .GetContainer("testcontainer"); + + ContainerInternal containerInternal = (ContainerInternal)container; + ResponseMessage responseMessage = await containerInternal.PatchItemStreamAsync( + id: "cdbBinaryIdRequest", + partitionKey: Cosmos.PartitionKey.Null, + streamPayload: itemStream, + requestOptions: requestOptions); + Assert.IsNotNull(responseMessage); + Assert.AreEqual(httpStatusCode, responseMessage.StatusCode); + Assert.AreEqual(1, testHandlerHitCount, "The operation did not make it to the handler"); + } + finally + { + Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, null); + } + } + private static void ValidateOperations(PatchOperation patchOperation, PatchOperationType operationType, T value) { Assert.AreEqual(operationType, patchOperation.OperationType); @@ -99,6 +239,32 @@ private static void ValidateOperations(PatchOperation patchOperation, PatchOp } } + private static ResponseMessage GetContainerItemResponse( + RequestMessage request, + HttpStatusCode httpStatusCode, + bool binaryEncodingEnabledInContainer) + { + string itemResponseString = "{\r\n \\\"id\\\": \\\"60362d85-ce1e-4ceb-9af3-f2ddfebf4547\\\",\r\n \\\"pk\\\": \\\"pk\\\",\r\n \\\"name\\\": \\\"1856531480\\\",\r\n " + + "\\\"email\\\": \\\"dkunda@test.com\\\",\r\n \\\"body\\\": \\\"This document is intended for binary encoding test.\\\",\r\n \\\"_rid\\\": \\\"fIsUAKsjjj0BAAAAAAAAAA==\\\",\r\n " + + "\\\"_self\\\": \\\"dbs/fIsUAA==/colls/fIsUAKsjjj0=/docs/fIsUAKsjjj0BAAAAAAAAAA==/\\\",\r\n \\\"_etag\\\": \\\"\\\\\"510096bc-0000-0d00-0000-66ccf70b0000\\\\\"\\\",\r\n " + + "\\\"_attachments\\\": \\\"attachments/\\\",\r\n \\\"_ts\\\": 1724708619\r\n}"; + + bool shouldReturnBinaryResponse = binaryEncodingEnabledInContainer + && request.Headers[Documents.HttpConstants.HttpHeaders.SupportedSerializationFormats] != null + && request.Headers[Documents.HttpConstants.HttpHeaders.SupportedSerializationFormats].Equals(Documents.SupportedSerializationFormats.CosmosBinary.ToString()); + + return new ResponseMessage(httpStatusCode, request, errorMessage: null) + { + Content = shouldReturnBinaryResponse + ? CosmosSerializerUtils.ConvertInputToBinaryStream( + itemResponseString, + Newtonsoft.Json.JsonSerializer.Create()) + : CosmosSerializerUtils.ConvertInputToTextStream( + itemResponseString, + Newtonsoft.Json.JsonSerializer.Create()) + }; + } + private class CustomSerializer : CosmosSerializer { private readonly CosmosSerializer cosmosSerializer = new CosmosJsonDotNetSerializer(); diff --git a/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Utils/CosmosSerializerUtils.cs b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Utils/CosmosSerializerUtils.cs new file mode 100644 index 0000000000..6ea41d3d15 --- /dev/null +++ b/Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.Tests/Utils/CosmosSerializerUtils.cs @@ -0,0 +1,175 @@ +//------------------------------------------------------------ +// Copyright (c) Microsoft Corporation. All rights reserved. +//------------------------------------------------------------ + +namespace Microsoft.Azure.Cosmos.Tests +{ + using System.IO; + using Microsoft.Azure.Cosmos.Json.Interop; + using Microsoft.Azure.Cosmos.Json; + using System.Text; + using System; + + /// + /// Utility class for performing different operation on a given serializer. + /// + internal static class CosmosSerializerUtils + { + /// + /// Checks the first byte of the provided stream to determine if it matches the desired JSON serialization format. + /// + /// The stream containing the message content to be checked. + /// The desired JSON serialization format to check against. + /// The output byte array containing the stream content if the first byte matches the desired format. + /// Returns a boolean flag indicating if the first byte of the stream matches the desired format. + internal static bool CheckFirstBufferByte( + Stream messageContent, + JsonSerializationFormat desiredFormat, + out byte[] content) + { + content = default; + + if (messageContent == null || !messageContent.CanRead) + { + return false; + } + + // Use a buffer to read the first byte + byte[] buffer = new byte[1]; + int readCount = messageContent.Read(buffer, 0, 1); + + // Reset the stream position if it supports seeking + if (messageContent.CanSeek) + { + messageContent.Position = 0; + } + + // Check if the first byte matches the desired format + if (readCount > 0 + && (IsBinaryFormat(buffer[0], desiredFormat) || IsTextFormat(buffer[0], desiredFormat))) + { + // If the first byte matches with the desired format then + // copy the stream content to a byte array. + using (MemoryStream stream = new()) + { + messageContent.CopyTo(stream); + content = stream.ToArray(); + } + + // Reset the stream position again after copying + if (messageContent.CanSeek) + { + messageContent.Position = 0; + } + + return true; + } + + return false; + } + + /// + /// Determines if the first byte of a stream matches the binary JSON serialization format. + /// + /// The first byte of the stream to check. + /// The desired JSON serialization format. + /// Returns true if the first byte matches the binary format, otherwise false. + internal static bool IsBinaryFormat( + int firstByte, + JsonSerializationFormat desiredFormat) + { + return desiredFormat == JsonSerializationFormat.Binary && firstByte == (int)JsonSerializationFormat.Binary; + } + + /// + /// Determines if the first byte of a stream matches the text JSON serialization format. + /// + /// The first byte of the stream to check. + /// The desired JSON serialization format. + /// Returns true if the first byte matches the text format, otherwise false. + internal static bool IsTextFormat( + int firstByte, + JsonSerializationFormat desiredFormat) + { + return desiredFormat == JsonSerializationFormat.Text && firstByte < (int)JsonSerializationFormat.Binary; + } + + /// + /// Converts the given input object to a binary stream using the specified JSON serializer. + /// + /// The type of the input object. + /// The input object to be serialized. + /// The JSON serializer to use for serialization. + /// Returns a stream containing the binary serialized data of the input object. + internal static Stream ConvertInputToBinaryStream( + T input, + Newtonsoft.Json.JsonSerializer serializer) + { + MemoryStream streamPayload = new(); + using (CosmosDBToNewtonsoftWriter writer = new(JsonSerializationFormat.Binary)) + { + writer.Formatting = Newtonsoft.Json.Formatting.None; + serializer.Serialize(writer, input); + byte[] binBytes = writer.GetResult().ToArray(); + streamPayload.Write(binBytes, 0, binBytes.Length); + } + + streamPayload.Position = 0; + return streamPayload; + } + + /// + /// Converts the given input object to a binary stream using the specified JSON serializer. + /// + /// The type of the input object. + /// The input object to be serialized. + /// The JSON serializer to use for serialization. + /// Returns a stream containing the binary serialized data of the input object. + internal static Stream ConvertInputToNonSeekableBinaryStream( + T input, + Newtonsoft.Json.JsonSerializer serializer) + { + using (CosmosDBToNewtonsoftWriter writer = new(JsonSerializationFormat.Binary)) + { + writer.Formatting = Newtonsoft.Json.Formatting.None; + serializer.Serialize(writer, input); + byte[] binBytes = writer.GetResult().ToArray(); + + CosmosBufferedStreamWrapperTests.NonSeekableMemoryStream streamPayload = new(binBytes); + + if (streamPayload.CanSeek) + { + streamPayload.Position = 0; + } + + return streamPayload; + } + } + /// + /// Converts the given input object to a binary stream using the specified JSON serializer. + /// + /// The type of the input object. + /// The input object to be serialized. + /// The JSON serializer to use for serialization. + /// Returns a stream containing the binary serialized data of the input object. + internal static Stream ConvertInputToTextStream( + T input, + Newtonsoft.Json.JsonSerializer serializer) + { + MemoryStream streamPayload = new(); + using (StreamWriter streamWriter = new(streamPayload, encoding: new UTF8Encoding(false, true), bufferSize: 1024, leaveOpen: true)) + { + using (Newtonsoft.Json.JsonWriter writer = new Newtonsoft.Json.JsonTextWriter(streamWriter)) + { + writer.Formatting = Newtonsoft.Json.Formatting.None; + serializer.Serialize(writer, input); + writer.Flush(); + streamWriter.Flush(); + } + } + + streamPayload.Position = 0; + return streamPayload; + } + } +}