Unable to inference image using PromptFunction

[vicperdana]

Hi,
We're trying to inference an image using Prompt Function and its been unsuccessful. Making a request to OpenAI API directly yields the expected result but not via SK. We have turned trace logging on.

Is there any way to view the raw request (rendered prompt) sent to OAI in SK so we can compare?

Can you please assist?

Code snippets of SK call below.

Semantic Kernel

var builder = Kernel.CreateBuilder();
        builder.Services.AddSingleton(loggerFactory);
var kernel = builder.AddOpenAIChatCompletion("gpt-4o", "oursecretkey")
            .Build(); 
using StreamReader reader = new(Assembly.GetExecutingAssembly().GetManifestResourceStream("SomeNamespace.Prompts.TranscribeImage.prompt.yaml")!);
        KernelFunction getIntent = kernel.CreateFunctionFromPromptYaml(
            await reader.ReadToEndAsync(),
            promptTemplateFactory: new HandlebarsPromptTemplateFactory()
        );
        var result = await kernel.InvokeAsync(getIntent);

Output

[09:54:45 INF] Function transcribeScreenshot invoking. <s:transcribeScreenshot>
info: transcribeScreenshot[0]
      Function transcribeScreenshot invoking.
trce: transcribeScreenshot[0]
      Function arguments: {}
trce: Microsoft.SemanticKernel.KernelFunctionFactory[0]
      Rendered prompt: [{
         "role": "user",
         "content": [
           {
             "type": "text",
             "text": "What’s in this image?"
           },
           {
             "type": "image_url",
             "image_url": {
               "url": "
[https://i.imgur.com/1RP78PA.jpeg"](https://i.imgur.com/1RP78PA.jpeg%22)
             }
           }
         ]
       }]
trce: Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIChatCompletionService[0]
      ChatHistory: [{"Role":{"Label":"user"},"Items":[{"$type":"TextContent","Text":"[{\n   \u0022role\u0022: \u0022user\u0022,\n   \u0022content\u0022: [\n     {\n       \u0022type\u0022: \u0022text\u0022,\n       \u0022text\u0022: \u0022What\u2019s in this image?\u0022\n     },\n     {\n       \u0022type\u0022: \u0022image_url\u0022,\n       \u0022image_url\u0022: {\n         \u0022url\u0022: \u0022
[https://i.imgur.com/1RP78PA.jpeg\u0022\n](https://i.imgur.com/1RP78PA.jpeg/u0022/n)
       }\n     }\n   ]\n }]"}]}], Settings: {"temperature":1,"top_p":1,"presence_penalty":0,"frequency_penalty":0,"max_tokens":null,"stop_sequences":null,"results_per_prompt":1,"seed":null,"response_format":null,"chat_system_prompt":null,"token_selection_biases":null,"ToolCallBehavior":null,"User":null,"logprobs":null,"top_logprobs":null,"model_id":null}
[09:54:47 INF] Prompt tokens: 83. Completion tokens: 34. Total tokens: 117. <s:Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIChatCompletionService>
info: Microsoft.SemanticKernel.Connectors.OpenAI.OpenAIChatCompletionService[0]
      Prompt tokens: 83. Completion tokens: 34. Total tokens: 117.
[09:54:47 INF] Prompt tokens: 83. Completion tokens: 34. <s:Microsoft.SemanticKernel.KernelFunctionFactory>
info: Microsoft.SemanticKernel.KernelFunctionFactory[0]
      Prompt tokens: 83. Completion tokens: 34.
[09:54:47 INF] Function transcribeScreenshot succeeded. <s:transcribeScreenshot>
info: transcribeScreenshot[0]
      Function transcribeScreenshot succeeded.
trce: transcribeScreenshot[0]
      Function result: I'm currently unable to view images directly. However, you can describe the image to me and I can try to help with any questions or provide information based on your description.
[09:54:47 INF] Function completed. Duration: 1.3812798s <s:transcribeScreenshot>
info: transcribeScreenshot[0]
      Function completed. Duration: 1.3812798s
[09:54:47 DBG] **I'm currently unable to view images directly. However, you can describe the image to me and I can try to help with any questions or provide information based on your description**.

Running via OAI in Postman works

[dmytrostruk]

Hi @vicperdana !
In output that you presented you can find rendered prompt which is sent to OpenAI:

Rendered prompt: [{
         "role": "user",
         "content": [
           {
             "type": "text",
             "text": "What’s in this image?"
           },
           {
             "type": "image_url",
             "image_url": {
               "url": "
[https://i.imgur.com/1RP78PA.jpeg"](https://i.imgur.com/1RP78PA.jpeg%22)
             }
           }
         ]
       }]

It looks like url property with [https://i.imgur.com/1RP78PA.jpeg"](https://i.imgur.com/1RP78PA.jpeg%22) value is invalid. Based on your Postman screenshot, you are using https://i.imgur.com/1RP78PA.jpeg value there, while in YAML it looks like markdown link is used.

Is it possible to modify YAML file so the image will be in the same format as in Postman?

[vicperdana]

Sorry, that was a copy and paste error, here's the rendered prompt from debug.

trce: Microsoft.SemanticKernel.KernelFunctionFactory[0]
      Rendered prompt: [{
         "role": "user",
         "content": [
           {
             "type": "text",
             "text": "What's in this image?"
           },
           {
             "type": "image_url",
             "image_url": {
               "url": "https://i.imgur.com/1RP78PA.jpeg"
             }
           }
         ]
       }]

Our YAML template (hardcoded just for testing)

name: transcribeScreenshot
description: Extract text from image
template: |
  [{
     "role": "user",
     "content": [
       {
         "type": "text",
         "text": "What's in this image?"
       },
       {
         "type": "image_url",
         "image_url": {
           "url": "https://i.imgur.com/1RP78PA.jpeg"
         }
       }
     ]
   }]

  
template_format: handlebars

[vicperdana]

Fixed after updating the handlebar template.

name: transcribeScreenshot
description: Extract text from image
template: |
  <message role='user'>
   <text>what is this image</text>
   <image>https://i.imgur.com/1RP78PA.jpeg</image>
  </message>
template_format: handlebars