[BUG] Serialization error with AI Agents: Unable to cast object of type 'd__2' to type

[karthiksimplist]

Library name and version

Azure.AI.Agents.Persistent 1.2.0-beta.4

Describe the bug

Message: Unable to cast object of type 'd__2' to type 'System.Collections.Generic.List`1[System.Text.Json.JsonElement]'.

Stack:
   at System.Text.Json.JsonSerializer.UnboxOnWrite[T](Object value)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsObject(Utf8JsonWriter writer, Object rootValue)
   at System.Text.Json.JsonSerializer.WriteStringAsObject(Object value, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Serialize(Object value, JsonTypeInfo jsonTypeInfo)
   at Azure.AI.Agents.Persistent.ThreadMessages.CreateMessageAsync(String threadId, MessageRole role, IEnumerable`1 contentBlocks, IEnumerable`1 attachments, IReadOnlyDictionary`2 metadata, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.AzureAI.Internal.AgentThreadActions.CreateMessageAsync(PersistentAgentsClient client, String threadId, ChatMessageContent message, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgentThread.OnNewMessageInternalAsync(ChatMessageContent newMessage, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.AgentThread.OnNewMessageAsync(ChatMessageContent newMessage, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.Agent.EnsureThreadExistsWithMessagesAsync[TThreadType](ICollection`1 messages, AgentThread thread, Func`1 constructThread, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync(ICollection`1 messages, AgentThread thread, AzureAIAgentInvokeOptions options, CancellationToken cancellationToken)+MoveNext()
   at Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync(ICollection`1 messages, AgentThread thread, AzureAIAgentInvokeOptions options, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource.GetResult()

Expected behavior

Throws this exception all the time when using Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync
which used to work on the same codebase of ours.

Actual behavior

Throws the above mentioned exception.

Reproduction Steps

Invoke
Microsoft.SemanticKernel.Agents.AzureAI.AzureAIAgent.InvokeStreamingAsync method with the following packages.

<PackageReference Include="Azure.AI.Agents.Persistent" Version="1.2.0-beta.4" />
<PackageReference Include="Azure.AI.OpenAI" Version="2.3.0-beta.1" />
<PackageReference Include="Azure.AI.Projects" Version="1.0.0-beta.11" />
<PackageReference Include="System.ClientModel" Version="1.6.1" />
<PackageReference Include="System.Linq.Async" Version="6.0.3" />
<PackageReference Include="Simplist.Framework.Services.Core" Version="1.0.0" />
<PackageReference Include="Simplist.Framework.Bootstrap" Version="1.0.0" />
<PackageReference Include="Microsoft.SemanticKernel" Version="1.64.0" />
<PackageReference Include="Microsoft.SemanticKernel.Core" Version="1.64.0" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.Core" Version="1.64.0" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.OpenAI" Version="1.64.0-preview" />
<PackageReference Include="Microsoft.SemanticKernel.Agents.AzureAI" Version="1.64.0-preview" />
<PackageReference Include="Microsoft.SemanticKernel.Plugins.Web" Version="1.64.0-alpha" />

Environment

.NET

[dargilco]

Thank you @karthiksimplist for opening this issue. Can you please share small sample code that shows the issue?

[github-actions]

Hi @karthiksimplist. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[stephentoub]

The bug is here:

azure-sdk-for-net/sdk/ai/Azure.AI.Agents.Persistent/src/Custom/ThreadMessages.cs

Line 252 in b3cdb73

var jsonString = JsonSerializer.Serialize(contentBlocks, JsonElementSerializer.Default.ListJsonElement);

It's trying to use a JsonTypeInfo for List<JsonElement> to serialize an unrelated IEnumerable<MessagesInputContentBlock>. Looks like that was changed in a9d0bb6.

The fix will be to either:

1. Change that line to use the right JsonTypeInfo and ensure that MessagesInputContentBlock is configured appropriately for polymorphic serialization (it's an abstract base class), or
2. Use the IJsonModel<MessageInputImageFileBlock> of each MessageInputImageFileBlock to serialize each to JSON, presumably as part of larger use of Utf8JsonWriter that handles creating the JSON array around each.

[dargilco]

@karthiksimplist we released a new version of Azure.AI.Agents.Persistant a few days ago. Can you please try with latest version (1.2.0-beta.5)?