CosmosDBOutput Unsupported PartitionKey value component '[]'

[Benjiiim]

In a .NET 8 Isolated function (migrating from .NET 6 In-Process), I'm trying to use CosmosDBOutput to write multiple documents without knowing their types but I'm getting the following error:

System.Private.CoreLib: Exception while executing function: Functions.Function1. Microsoft.Azure.Cosmos.Client: Unsupported PartitionKey value component '[]'. Numeric, string, bool, null, Undefined are the only supported types.

Here is a repo:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Newtonsoft.Json;

namespace TestCosmos
{
    public class Function1
    {
        [Function("Function1")]
        [CosmosDBOutput(databaseName: "Test", containerName: "Test", CreateIfNotExists = true, PartitionKey = "/email", Connection = "cosmosConnectionString")]
        public Object[] Run([HttpTrigger(AuthorizationLevel.Function, "get", "post")] HttpRequestData req)
        {
            var input = @"[
                        {
                            ""email"": ""hello@toto.com"",
                            ""event"": ""processed""
                        },
                        {
                            ""email"": ""hello@toto.com"",
                            ""event"": ""deferred"",
                            ""category"": ""cat facts""
                        }]";

            dynamic data = JsonConvert.DeserializeObject(input);

            List<Object> outputList = new List<Object>();

            foreach(var item in data)
            {
                item.id = System.Guid.NewGuid().ToString();

                outputList.Add(item);
            }

            return outputList.ToArray();
        }
    }
}

Am I doing anything wrong?

[kshyju]

@ealsur Could you help with this?

[ealsur]

@Benjiiim Are you sure the container is partitioned by /email?

This is the first time I see these kind of syntax, following the public documentation (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2-output?tabs=python-v2%2Cin-process%2Cnodejs-v4%2Cextensionv4&pivots=programming-language-csharp#queue-trigger-write-docs-using-iasynccollector) I would expect IAsyncCollector to be used?

@kshyju Is this something particular to the Worker packages?

[ealsur]

@Benjiiim Can you also post the full Stack Trace of the exception? This message is coming from the Cosmos DB SDK (not the Extension code). The Stack Trace would normally show where is it happening. My hunch is that the SDK is attempting to extract the Partition Key Value from the documents and cannot do so, maybe because of the type of objects.

Output bindings work in 2 ways:

• Using out/return variables -> In this case it supports 1 item.
• Using IAsyncCollector -> For cases when you want to store multiple items.

You seem to be using the first approach but attempting to save multiple items.

[Benjiiim]

Thanks @ealsur

Yes, I'm sure that the container is partitioned by /email.

From my understanding, IAsyncCollector<T> was used in In-process model. Isolated model must use T[].
I should be able to output multiple items with T[] with the isolated model.

https://learn.microsoft.com/en-us/azure/azure-functions/migrate-dotnet-to-isolated-model?tabs=net8

For output bindings, if the in-process model version used an IAsyncCollector<T>, you can replace this with binding to an array of the target type: T[].

On https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2-output, you should use the isolated tab and not the in-process tab.

I think the issue is coming from the use of dynamic but I have to use it as I don't know the properties of the Json objects.

The same output was working with IAsyncCollector when my function was In-process.

[Benjiiim]

Stack trace:

[2023-12-01T18:48:08.427Z] Executed 'Functions.Function1' (Failed, Id=6f6c0851-87fa-43c5-8b71-b51a9246551e, Duration=2672ms)
[2023-12-01T18:48:08.430Z] Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.Function1
[2023-12-01T18:48:08.431Z]  ---> System.ArgumentException: Unsupported PartitionKey value component '[]'. Numeric, string, bool, null, Undefined are the only supported types.
[2023-12-01T18:48:08.432Z]    at Microsoft.Azure.Cosmos.ContainerCore.CosmosElementToPartitionKeyObject(IReadOnlyList`1 cosmosElementList)
[2023-12-01T18:48:08.433Z]    at Microsoft.Azure.Cosmos.ContainerCore.GetPartitionKeyValueFromStreamAsync(Stream stream, ITrace trace, CancellationToken cancellation)
[2023-12-01T18:48:08.436Z]    at Microsoft.Azure.Cosmos.ContainerCore.ExtractPartitionKeyAndProcessItemStreamAsync[T](Nullable`1 partitionKey, String itemId, T item, OperationType operationType, ItemRequestOptions requestOptions, ITrace trace, CancellationToken cancellationToken)
[2023-12-01T18:48:08.437Z]    at Microsoft.Azure.Cosmos.ContainerCore.UpsertItemAsync[T](T item, ITrace trace, Nullable`1 partitionKey, ItemRequestOptions requestOptions, CancellationToken cancellationToken)
[2023-12-01T18:48:08.438Z]    at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Func`2 openTelemetry, String operationName, RequestOptions requestOptions)
[2023-12-01T18:48:08.439Z]    at Microsoft.Azure.Cosmos.ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func`2 task, Func`2 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel)
[2023-12-01T18:48:08.441Z]    at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBAsyncCollector`1.AddAsync(T item, CancellationToken cancellationToken) in D:\a\_work\1\s\src\WebJobs.Extensions.CosmosDB\Bindings\CosmosDBAsyncCollector.cs:line 26
[2023-12-01T18:48:08.442Z]    at Microsoft.Azure.WebJobs.Extensions.CosmosDB.CosmosDBAsyncCollector`1.AddAsync(T item, CancellationToken cancellationToken) in D:\a\_work\1\s\src\WebJobs.Extensions.CosmosDB\Bindings\CosmosDBAsyncCollector.cs:line 48
[2023-12-01T18:48:08.443Z]    at Microsoft.Azure.WebJobs.Script.Binding.FunctionBinding.BindAsyncCollectorAsync[T](BindingContext context) in /_/src/WebJobs.Script/Binding/FunctionBinding.cs:line 200
[2023-12-01T18:48:08.444Z]    at Microsoft.Azure.WebJobs.Script.Binding.ExtensionBinding.BindAsync(BindingContext context) in /_/src/WebJobs.Script/Binding/ExtensionBinding.cs:line 84
[2023-12-01T18:48:08.445Z]    at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.<>c__DisplayClass13_0.<<BindOutputsAsync>b__0>d.MoveNext() in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 182
[2023-12-01T18:48:08.446Z] --- End of stack trace from previous location ---
[2023-12-01T18:48:08.447Z]    at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.BindOutputsAsync(Object input, Binder binder, ScriptInvocationResult result) in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 186
[2023-12-01T18:48:08.448Z]    at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 109
[2023-12-01T18:48:08.449Z]    at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
[2023-12-01T18:48:08.450Z]    at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /_/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
[2023-12-01T18:48:08.451Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs:line 52
[2023-12-01T18:48:08.452Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 581
[2023-12-01T18:48:08.453Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 527
[2023-12-01T18:48:08.454Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 306
[2023-12-01T18:48:08.455Z]    --- End of inner exception stack trace ---
[2023-12-01T18:48:08.456Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 352
[2023-12-01T18:48:08.457Z]    at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:line 108

[ealsur]

@Benjiiim Yes, this confirm the case. The problem is you are using the return pattern, which is for 1 item as per the documentation, but you are returning an object which is an array.

Please use the IAsyncCollector approach for Output bindings meant to save multiple items (this applies to any output binding, not just Cosmos DB).

[Benjiiim]

Sorry but the doc says that IAsyncCollector should not be used in isolated mode.
According to the doc, in isolated mode, using an array of objects as the output is the way to save multiple objects.
This is working well with strongly typed objects in isolated mode but doesn't work with the object I'm passing in the code above (was working with IAsyncCollector in in-process mode).

[ealsur]

I have personally no knowledge how isolated mode works or why IAsyncCollector won't work there.

Are you saying that when you use a Strongly Typed Array, it works, but in this example (using Object) it does not? Why are you using Object and not, for example, JObject, or dynamic?

List<JObject> outputList = new List<JObject>();

I wonder if the problem is the underlying Cosmos DB SDK cannot work with Object.

[Benjiiim]

I don't want to be rude but why not involving someone who knows about the difference between how the CosmosDB output worked with in-process functions and how it is working now with isolated functions?
I've actually found a way to output an other type of objects while migrating to System.Text.Json and avoiding the use of dynamic.
However, understanding why the code above doesn't work while it was working with IAsyncCollector in an in-process function might be useful to either fix something or document the change?
Let me know if and how I can help.

[ealsur]

involving someone who knows about the difference between how the CosmosDB output worked with in-process functions and how it is working now with isolated functions

I sadly do not know. @kshyju, any ideas who knows how Isolated Functions work in terms of Output Bindings?

understanding why the code above doesn't work

The key error is this: Unsupported PartitionKey value component '[]', this is what is telling that the Cosmos DB SDK, when attempting to extract the Partition Key of the document, is finding a type of value that is an Array. Hence my first thought was, the problem is the Array is being sent and treated as 1 document instead of an array of documents.

I've actually found a way to output an other type of objects

You mean that you have a working code that still returns a List but using different type works? Can you share which Type?

[chocvanstraw]

I was able to output multiple documents to the CosmosDBOutput binding as follows. Perhaps it might be helpful to you. I use JsonNode since I only care about a few properties on the document.

        [CosmosDBOutput("%DatabaseId%", "Container2", Connection = "ConnectionString")]
        public IEnumerable<JsonNode> Run([CosmosDBTrigger(
            databaseName: "%DatabaseId%",
            containerName: "Container1",
            Connection = "ConnectionString",
            LeaseContainerName = "leases",
            LeaseContainerPrefix = "xxx_",
            CreateLeaseContainerIfNotExists = true)] IReadOnlyList<JsonNode> documents)
        {
            var outputDocs = new List<JsonNode>();

            foreach (var doc in documents)
            {
                doc["partitionKey"] = Guid.NewGuid().ToString(); //change the partition key to something else

                outputDocs.Add(doc);
            }

            return outputDocs;
        }

[ealsur]

Thanks @chocvanstraw for the example, JsonNode works if you are using System.Text.Json (are you customizing the serializer for the extension?). The default serialization for the Cosmos DB SDK uses Newtonsoft.Json, that is why I suggested to use JObject because the OP is using dynamic data = JsonConvert.DeserializeObject(input); (Newtonsoft.Json) for the example.

[chocvanstraw]

While newtonsoft is required when interacting with cosmos directly using the cosmos client I’ve found that the function output binding doesn’t seem to be as strict. I haven’t had any issues using system.text.json in this use case On Dec 8, 2023, at 5:07 PM, Matias Quaranta ***@***.***> wrote: Thanks @chocvanstraw for the example, JsonNode works if you are using System.Text.Json (are you customizing the serializer for the extension?). The default serialization for the Cosmos DB SDK uses Newtonsoft.Json, that is why I suggested to use JObject because the OP is using dynamic data = JsonConvert.DeserializeObject(input); (Newtonsoft.Json) for the example. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[ealsur]

Interesting. Seems like the Isolation Mode might be using System.Text.Json to serializer/communicate between processes? @kshyju is there anyone that works on the Worker/Isolation mode area that can confirm this?