Does onnxruntime or onnxruntime_genai support tool_calling?

[jivaniyash]

I am looking to use phi-3 model (using ONNX runtime) to integrate function_calling / tool_calling? Is there any documentation/API to add tool_calling along the prompt? Also, how can I change the system prompt?

[baijumeswani]

Could you share more details on the tool_calling functionality? Any open-source document/code where this is done?

I will add this to our planning document, and we will discuss internally if this can be supported for the upcoming release.

[Ben-Epstein]

@baijumeswani Some examples of SoTA for open source

• https://huggingface.co/NousResearch/Hermes-2-Pro-Llama-3-8B#ask-for-a-structured-json-output
• https://docs.cohere.com/docs/tool-use

[MaxAkbar]

I was looking for this as well and found this Phi3Family. It states that function calling is not supported but I was able to get it to work just not reliably. If you change models it breaks. I was using the microsoft/Phi-3-mini-4k-instruct-onnx using c#.

This might work for you:

You are an AI assistant that can help the user with a variety of tasks. You have access to the functions provided by the schema below:
<|functions_schema|>[
    {
     "name": "Add",
     "description": "Given two numbers we add them and return the results",
     "parameters": 
     [
      {
        "name": "num1",
        "type": "System.Int32"
      },
      {
        "name": "num2",
        "type": "System.Int32"
      }
     ]
    },
    {
      "name": "Subtract",
      "description": "Given two numbers we subtract them and return the results",
      "parameters": 
      [
       {
         "name": "num1",
         "type": "System.Int32"
       },
       {
         "name": "num2",
         "type": "System.Int32"
       }
      ]
     }
]
   
When the user asks you a question, if you need to use functions, provide ONLY ALL OF THE function calls, ALL IN ONE PLACE, in the format:
<|function_calls|>
[
 { "name": "function_name", "parameters": {}, "returns": ["output_1"]},
 { "name": "function_name", "parameters": {"paramater_1": "value_1", "paramater_2": "value_2"}, "returns": ["output_1"]},
]
...
<|end_function_calls|>

If you need to modify strings ONLY USE THE AVAILABLE FUNCTIONS GIVEN TO YOU. If and only if you don't need to use functions, give your answer in between <|answer|> and <|end_answer|> blocks. For your thoughts and reasoning behind using or not using functions, place ALL OF THEM in between a SINGLE <|thoughts|> and <|end_thoughts|> block, before the <|function_calls|> and <|end_function_calls|> tags, like so:

<|thoughts|>
The user wants X, to do that, I should call the following functions:
1. function_name: Reasoning,
2. function_name_2: Reasoning2,
3. etc.
<|end_thoughts|>

Provide nothing else than the information in the <|function_calls|> & <|end_function_calls|>, <|answer|> & <|end_answer|> and <|thoughts|> & <|end_thoughts|> blocks.<|end|>
<|user|>
What is 2+2=?.<|end|>
<|assistant|>

Just submit this prompt and you will get a response like this:

<|thoughts|>
The user is asking for the result of a simple addition operation. Since the Add function is available, I will use it to calculate the sum of 2 and 2.
<|function_calls|>
[
 { "name": "Add", "parameters": {"num1": 2, "num2": 2}, "returns": ["4"]},
<|end_function_calls|>
<|end_thoughts|>

I used regex to extract the function and reflection to invoke it.

[elephantpanda]

Probably it would work better if you provide it with more examples rather than just instructions??? I found that phi-3 is not as good at following instructions as chatgpt, so you have to give it more examples. And also it seems to work better if you tell it to give the answers in JSON format. It likes that! I was using it to tell it to convert some english sentences into a domain specific language for my game.

[MaxAkbar]

I ended up using clustering to get an intent on what the user wanted to do then once I knew that I mapped it to the function after that I did use JSON to extract the entities I wanted if not then I knew to keep asking the user for the details. That worked way better.

My next step is to use an agent to get what the user wants and to get the entities I need for my functions.