Binary Encoding - Analyze and Fix `DateTime` Parsing Issue with Trailing Zeros in the Milli-Seconds Precision with .NET SDK Item Operations

[kundadebdatta]

Background:

During some of the internal validation, it was identified that when the cosmos .net SDK is initialized with binary encoding, and a date time field is persisted into a database, the value is getting stored with trailing zeros. To understand this better, please see the following steps to repro the issue.

Goal is to identify the serialization gap here and fix it in the .NET SDK.

Steps to Repro:

• Cosmos Client Creation:

        Environment.SetEnvironmentVariable(ConfigurationManager.BinaryEncodingEnabled, "True");
        string databaseName = "binary-encoding-db";
        string containerName = "binary-encoding-container";
        CosmosClientOptions clientOptions = new()
        {
            ConnectionMode = ConnectionMode.Direct,
            RequestTimeout = TimeSpan.FromSeconds(5000),
        };

        CosmosClient client = new CosmosClient(connectionString,
            clientOptions);

• Item Creation:

      string[] dateStrings = {
          "12/25/2023",
          "2023-12-25",
          "12-25-2023",
          "25.12.2023",
          "25/12/2023",
          "Dec 25, 2023",
          "Dec 25 2023",
          "2023-12-25T10:00:00",
          "2023-12-25T10:00:00.123",
          "12/25/2023 10:00 AM",
          "12/25/2023 10:00:00 AM",
          "12/25/2023 10:00:00.123 AM"
      };

      // Define the supported formats
      string[] formats = {
          "MM/dd/yyyy",
          "yyyy-MM-dd",
          "MM-dd-yyyy",
          "dd.MM.yyyy",
          "dd/MM/yyyy",
          "MMM dd, yyyy",
          "MMM dd yyyy",
          "yyyy-MM-ddTHH:mm:ss",
          "yyyy-MM-ddTHH:mm:ss.fff",
          "MM/dd/yyyy hh:mm tt",
          "MM/dd/yyyy hh:mm:ss tt",
          "MM/dd/yyyy hh:mm:ss.fff tt"
      };

      int idx = 0;
      DateTime[] dateTimes = new DateTime[dateStrings.Length];
      // Try to parse each date string
      foreach (string dateString in dateStrings)
      {
          try
          {
              // Use TryParseExact to handle multiple formats
              DateTime parsedDate = DateTime.ParseExact(dateString, formats, CultureInfo.InvariantCulture, DateTimeStyles.None);
              dateTimes[idx++] = parsedDate;
              // Display the parsed date
          }
          catch (FormatException)
          {
              Console.WriteLine($"Failed to parse: {dateString}");
          }
      }


      Comment comment = new Comment(
          Guid.NewGuid().ToString(),
          "pk", random.Next().ToString(),
          "dkunda@test.com",
          "This document is intended for binary encoding demo.",
          createdAt: DateTime.UtcNow,
          modifiedAt: DateTime.Parse("2025-03-26T20:22:20Z"),
          datetimes: dateTimes);

      ItemResponse<Comment> writeResponse = await container.CreateItemAsync<Comment>(
          item: comment,
          partitionKey: new Cosmos.PartitionKey(comment.pk),
          requestOptions: requestOptions
      );

      Console.WriteLine(writeResponse .Diagnostics);

• Sample Data Saved in Container. Please see below the comparison results, when binary encoding is Disabled vs Enabled:

Analysis:

Initially, it appears that the current binary DateTime parsing logic, in the CosmosDBToNewtonsoftWriter uses the round-trip date/time pattern, which is responsible for adding the trailing zeros, if the milli seconds precision is missing:

    /// <summary>
    /// Writes a <see cref="DateTime"/> value.
    /// </summary>
    /// <param name="value">The <see cref="DateTime"/> value to write.</param>
    public override void WriteValue(DateTime value)
    {
        this.WriteValue(value.ToString("O"));
    }

Instead, we should use a hybrid approach during the parsing, where if the milli seconds are present, then we can use the round-trip date/time pattern ("O"), however, if the milli-seconds precision is missing we will need to use the Universal sortable date/time pattern.

Acceptance Criteria:

Goal is to fix the behavior so that the date time fields are persisted correctly.