Assistant Vector Store results are not retrieved

[Licantrop0]

Service

Azure OpenAI

Describe the bug

as described in the OpenAI documentation here, you can retrieve the search result by adding include[]=step_details.tool_calls[*].file_search.results[*].content query string to a GetRunStep call.

Steps to reproduce

add the following code for initialization of the httpClient:

var httpClient = new HttpClient();
httpClient.BaseAddress = OpenAIHelpers.AzureOAIEndpoint;
var token = await new DefaultAzureCredential().GetTokenAsync(new Azure.Core.TokenRequestContext(["https://cognitiveservices.azure.com/.default"]));
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token.Token);

this code to get the retreived chunks from the vector db:

private static async Task GetSearchInfo(RunStepUpdate runStepUpdate)
{
    if (runStepUpdate.UpdateKind != StreamingUpdateReason.RunStepCompleted
        || !runStepUpdate.Value.Details.ToolCalls.Any(t => t.ToolKind == RunStepToolCallKind.FileSearch))
        return; // skip if not completed or doesn't use file search

    var SearchInfoQueryString = "include[]=step_details.tool_calls[*].file_search.results[*].content";

    string endpoint = $"/openai/threads/{runStepUpdate.Value.ThreadId}/runs/{runStepUpdate.Value.RunId}/steps/{runStepUpdate.Value.Id}";
    var response = await httpClient.GetAsync($"{endpoint}?{SearchInfoQueryString}&api-version=2024-09-01-preview");
    if (response.IsSuccessStatusCode)
    {
        var json = await response.Content.ReadAsStringAsync();
        var rootNode = JsonNode.Parse(json);

        var searchResults = rootNode?["step_details"]?["tool_calls"]?[0]?["file_search"]?["results"]?
            .AsArray().Select(r => JsonSerializer.Deserialize<string>(r?["content"]?[0]?["text"]));

        Debug.WriteLine("Search Results:");
        Debug.WriteLine(string.Join("\n\n", searchResults ?? []));
    }
}

and this code to run the streaming response:

private static async Task OutputCompletionStreaming(AssistantThread thread, Assistant assistant, RunCreationOptions runOptions)
{
    var asyncUpdates = OpenAIHelpers.OpenAIAssistantClient.CreateRunStreamingAsync(thread.Id, assistant.Id, runOptions);

    ThreadRun? currentRun;
    do
    {
        currentRun = null;

        List<ToolOutput> outputsToSubmit = [];

        await foreach (var update in asyncUpdates ?? AsyncEnumerable.Empty<StreamingUpdate>())
        {
            switch (update)
            {
                case RequiredActionUpdate requiredActionUpdate:
                    // handle tool calls
                    break;

                case MessageStatusUpdate messageStatus:
                    //Debug.WriteLine($" {messageStatus.UpdateKind} {messageStatus.Value.Id}");
                    break;

                case MessageContentUpdate contentUpdate:
                    Console.Write(contentUpdate.Text);

                    if (contentUpdate.TextAnnotation != null)
                    {
                        // Reference to RAG results
                        Debug.WriteLine($"  --> From file: {contentUpdate.TextAnnotation.InputFileId}, replacement: {contentUpdate.TextAnnotation.TextToReplace}");
                    }
                    break;

                case RunUpdate runUpdate:
                    currentRun = runUpdate;
                    Debug.WriteLine($"{runUpdate.UpdateKind} {runUpdate.Value.Id}");
                    if (runUpdate.UpdateKind == StreamingUpdateReason.RunFailed)
                    {
                        Console.WriteLine($"{ConsoleColors.Red}Error: {runUpdate.Value.LastError.Message}");
                    }
                    break;

                case RunStepUpdate runStepUpdate:
                    Debug.WriteLine(
                        $" {runStepUpdate.UpdateKind} {runStepUpdate.Value.Id} " +
                        $"Tools: [ {runStepUpdate.Value.Details.ToolCalls.Select(t => t.ToolKind.ToString()).StringJoin("; ")} ] ");

                    await GetSearchInfo(runStepUpdate);
                    break;
            }
        }
    }
    while (currentRun?.Status.IsTerminal == false);
    Console.WriteLine();
}

You'll see that whenever the file search tool is called, the results are always empty.
The same exact code (except few changes in how httpClient is initialized) works with vanilla OpenAI SDK.

Code snippets

No response

OS

Windows

.NET version

8

Library version

2.1.0-beta.1

[hemnaathR]

Hello @joseharriaga , thanks for adding support to retrieve File Search result (#294) in the latest release. However, the Content property is coming back empty.

Using httpClient like @Licantrop0 has done, I can get the content.

       CollectionResult<RunStep> runSteps = _assistantClient.GetRunSteps(run.ThreadId, run.Id);
        List<RunStep> toolCallRunSteps = runSteps.Where(runStep => runStep.Kind == RunStepKind.ToolCall).ToList();

        foreach (RunStep toolCallRunStep in toolCallRunSteps)
        {

            var SearchInfoQueryString = "include[]=step_details.tool_calls[*].file_search.results[*].content";

            string endpoint = $"/openai/threads/{run.ThreadId}/runs/{run.Id}/steps/{toolCallRunStep.Id}";
            
            var response = await httpClient.GetAsync($"{endpoint}?{SearchInfoQueryString}&api-version=2024-09-01-preview");
            if (response.IsSuccessStatusCode)
            {
                var json = await response.Content.ReadAsStringAsync();
                var rootNode = JsonNode.Parse(json);

                var searchResults = rootNode?["step_details"]?["tool_calls"]?[0]?["file_search"]?["results"]?
                    .AsArray().Select(r => JsonSerializer.Deserialize<string>(r?["content"]?[0]?["text"]));

                Debug.WriteLine("Search Results:");
                Debug.WriteLine(string.Join("\n\n", searchResults ?? []));
            }

            foreach (RunStepToolCall toolCall in toolCallRunStep.Details.ToolCalls)
            {
                RunStepFileSearchResult fileSearchResult = toolCall.FileSearchResults[0];

                // content is empty. Count = 0
                RunStepFileSearchResultContent fileSearchResultContent = fileSearchResult.Content[0];
            }
        }