Thoughts to improve the tool caller

[ElisonSherton]

TLDR

1. Arguments to the tools should be categorized or tagged as follows
   • MUST be provided by the user and cannot be assumed.
   • Can be inferred from the context
2. Tool Insights in MessageGenerator
   • At present, the message generator simply gets the tool result
   • We need to show the message generator why certain tools were not triggered in order for it to not infer or assume things that it doesn't know.

For First Point

Motivation

I am working on a bot which is supposed to do a task of credit card bill payment. You could find the code and the instructions to run the same here.

Here is how a conversation to make a payment to the credit card happened

Here are the guidelines for quick reference

Now, here look at the last message generated by the bot. If we actually inspect this message by going into the session logs, we find this

Further if we look at the tool caller inspection completion log, we find this

Now, the issue here is that the tool caller inferred the payment_date_provided_by_customer and payment_source on it's own without actually asking for these fields explicitly from the user. While this may be useful in some cases, we need a way to enforce the bot to ask for certain arguments before actually making a tool call because they are not to be inferred automatically from the context. Things like payment date and the sender's account etc. are really critical pieces which MUST come from the user. This is what motivated the need for the first topic in the TLDR; section above.

Now, just hypothetically saying, consider a scenario where a user is asking questions about credit cards and just mentions that he has a parlant-premium-card in the passing. Then he digresses and then he comes back and asks a very generic question like What is the rate of interest on credit cards nowadays? Now let's say we have a tool, fetch_roi_of_card which given a credit card type as a string input, outputs the rate of interest of that card by fetching it from the DB. Here, if the engine infers that the user talked about a parlant-premium-card earlier and uses this card in the tool to fetch it's rate of interest and presents it to the user, it is OK. Although the user might have 10 other cards but the fact that bot inferred the card type to be parlant-premium-card is not so bad. This could be an example of where inferring the tool arguments based on context could be considered alright.

Current Behaviour

In Tool calling, even if we have specified a particular argument of a given tool as required for that tool to get executed, and the user has not explicitly provided that argument, the tool caller sometimes assumes the value and proceeds with the tool calling.

Desired Behaviour

Any developer using parlant's tool calling service should be able to do the following

1. While defining a tool, have a way to define certain arguments which the LLM should explicitly ask the user.
2. While defining a tool, have a way to define certain arguments which the LLM may infer from the conversation context so far.

This will enable a developer to bring about determinism in their tool calling workflows rather than being at the mercy of the spurious generations that the LLM may come up with for certain arguments in the tool.

For Second Point

Now refer this code to reproduce the error below.

Consider the same task which I was trying to do above i.e. making a payment to a credit card. Here is the conversation

Here is the log of everythin that happened during the last message generation step

(parlant_rc16) ➜  card-bot git:(card-bot-tool-call-failed-demo) parlant session inspect --session-id $SESSION_ID --event-id oS8nh4SIB2
Session ID: 'CVmPCqCaGZ'
Event ID: 'oS8nh4SIB2'

╭────────────┬────────────────────────────┬─────────────────────────────┬──────────┬────────┬─────────┬──────────────────────────────────────────┬─────────╮
│ Event ID   │ Creation Date              │ Correlation ID              │ Source   │ Offset │ Kind    │ Data                                     │ Deleted │
├────────────┼────────────────────────────┼─────────────────────────────┼──────────┼────────┼─────────┼──────────────────────────────────────────┼─────────┤
│ oS8nh4SIB2 │ 2025-01-17 04:38:23 PM UTC │ RID(2qVaYUmjPf)::HhVsxU2sSw │ ai_agent │ 24     │ message │ {'message': "It seems there was an issue │ False   │
│            │                            │                             │          │        │         │ with the payment date being set in the   │         │
│            │                            │                             │          │        │         │ past. Let's try setting the payment date │         │
│            │                            │                             │          │        │         │ for tomorrow, November 7, 2023, to       │         │
│            │                            │                             │          │        │         │ ensure it's in the future. I'll attempt  │         │
│            │                            │                             │          │        │         │ to process the payment of $200 towards   │         │
│            │                            │                             │          │        │         │ your Visa card ending in 5432 again.",   │         │
│            │                            │                             │          │        │         │ 'participant': {'id': 'nJWt2OSiuT',      │         │
│            │                            │                             │          │        │         │ 'display_name': 'Bar'}}                  │         │
╰────────────┴────────────────────────────┴─────────────────────────────┴──────────┴────────┴─────────┴──────────────────────────────────────────┴─────────╯


Iteration #0:
  Guideline Propositions:
    Condition: user wants to pay credit card bill and has provided the card details
    Action: make the payment
    Relevance Score: 9/10
    Rationale: Condition Application: "The user has expressed a desire to pay their credit card bill and has specified the Visa card ending in 5432 for the 
payment. They have also provided the payment amount of $200."; Guideline Previously Applied: "The payment has not yet been made, as the user just provided 
the payment amount."

  Tool Calls:
    Tool Id: card_service:make_payment
    Arguments: {'card_id': '1', 'payment_amount': 200.0, 'payment_source': 'checking account', 'payment_date': '04-11-2023'}
    Result: data={'error': 'Something went wrong while validating your payment date. Error: Payment date must be in the future'} metadata={}

  Context Variables:
    (none)

  Glossary Terms:
    (none)

Iteration #1:
  Guideline Propositions:
    Condition: user wants to pay credit card bill and has provided the card details
    Action: make the payment
    Relevance Score: 9/10
    Rationale: Condition Application: "The user has expressed the desire to pay their credit card bill and has specified the Visa card ending in 5432, along
with the payment amount of $200."; Guideline Previously Applied: "The action to make the payment was attempted, but it failed due to an error with the 
payment date validation."

  Tool Calls:
    Tool Id: card_service:make_payment
    Arguments: {'card_id': '1', 'payment_amount': 200.0, 'payment_source': 'checking account', 'payment_date': '05-11-2023'}
    Result: data={'error': 'Something went wrong while validating your payment date. Error: Payment date must be in the future'} metadata={}

  Context Variables:
    (none)

  Glossary Terms:
    (none)

Iteration #2:
  Guideline Propositions:
    Condition: user wants to pay credit card bill and has provided the card details
    Action: make the payment
    Relevance Score: 8/10
    Rationale: Condition Application: "The user has expressed a desire to pay their credit card bill and has provided the card details by specifying the 
Visa card."; Guideline Previously Applied: "The action was attempted twice but failed due to an error with the payment date. Since the payment was not 
successfully made, the guideline was not fully applied."

  Tool Calls:
    Tool Id: card_service:make_payment
    Arguments: {'card_id': '1', 'payment_amount': 200.0, 'payment_source': 'checking account', 'payment_date': '06-11-2023'}
    Result: data={'error': 'Something went wrong while validating your payment date. Error: Payment date must be in the future'} metadata={}

  Context Variables:
    (none)

  Glossary Terms:
    (none)

Here is the log for Tool Calling and Message Generation for the last message in even more detail

(parlant_rc16) ➜  parlant-projects git:(card-bot-tool-call-failed-demo) ✗ parlant log -tma Completion
Streaming logs...
[DEBUG] [RID(2qVaYUmjPf)::HhVsxU2sSw] [ToolCaller][Inference][Completion]
{
  "last_customer_message": "200$",
  "most_recent_customer_inquiry_or_need": "Make a payment of $200 towards the Visa card ending in 5432.",
  "most_recent_customer_inquiry_or_need_was_already_resolved": false,
  "name": "card_service:make_payment",
  "subtleties_to_be_aware_of": "The customer has specified the amount ($200) and the card (Visa ending in 5432). 
However, the payment source and date are not explicitly provided and need to be clarified or assumed based on 
defaults.",
  "tool_calls_for_candidate_tool": [
    {
      "applicability_rationale": "The customer has provided the card details and the payment amount, so the tool is 
needed to process the payment.",
      "applicability_score": 9,
      "arguments": {
        "card_id": "1",
        "payment_amount": 200.0,
        "payment_source": "checking account",
        "payment_date": "04-11-2023"
      },
      "same_call_is_already_staged": false,
      "comparison_with_rejected_tools_including_references_to_subtleties": "No other tools are relevant for this task.",
      "relevant_subtleties": "The payment source is assumed to be 'checking account' and the payment date is assumed to 
be today (04-11-2023).",
      "a_rejected_tool_would_have_been_a_better_fit_if_it_werent_already_rejected": false,
      "potentially_better_rejected_tool_name": null,
      "potentially_better_rejected_tool_rationale": null,
      "should_run": true
    }
  ]
}
[DEBUG] [RID(2qVaYUmjPf)::HhVsxU2sSw] [ToolCaller][Inference][Completion]
{
  "last_customer_message": "200$",
  "most_recent_customer_inquiry_or_need": "Make a payment of $200 towards the Visa card ending in 5432.",
  "most_recent_customer_inquiry_or_need_was_already_resolved": false,
  "name": "card_service:make_payment",
  "subtleties_to_be_aware_of": "The previously staged tool call failed due to an invalid payment date (must be in the 
future). The payment date needs to be corrected.",
  "tool_calls_for_candidate_tool": [
    {
      "applicability_rationale": "The customer has requested to make a payment of $200 towards their Visa card ending in
5432. The previous attempt failed due to an invalid payment date, so the tool needs to be re-executed with a corrected 
date.",
      "applicability_score": 10,
      "arguments": {
        "card_id": "1",
        "payment_amount": 200.0,
        "payment_source": "checking account",
        "payment_date": "05-11-2023"
      },
      "same_call_is_already_staged": false,
      "comparison_with_rejected_tools_including_references_to_subtleties": "This tool is the only one relevant for 
making a payment to the credit card.",
      "relevant_subtleties": "The payment date must be corrected to a future date to avoid the same error.",
      "a_rejected_tool_would_have_been_a_better_fit_if_it_werent_already_rejected": false,
      "potentially_better_rejected_tool_name": null,
      "potentially_better_rejected_tool_rationale": null,
      "should_run": true
    }
  ]
}
[DEBUG] [RID(2qVaYUmjPf)::HhVsxU2sSw] [ToolCaller][Inference][Completion]
{
  "last_customer_message": "200$",
  "most_recent_customer_inquiry_or_need": "Make a payment of $200 towards the Visa card ending in 5432.",
  "most_recent_customer_inquiry_or_need_was_already_resolved": false,
  "name": "card_service:make_payment",
  "subtleties_to_be_aware_of": "Previous attempts to make the payment failed due to the payment date being invalid (not 
in the future). A valid future date must be provided.",
  "tool_calls_for_candidate_tool": [
    {
      "applicability_rationale": "The customer has specified the payment amount and card details, and the payment needs 
to be processed with a valid future date.",
      "applicability_score": 10,
      "arguments": {
        "card_id": "1",
        "payment_amount": 200.0,
        "payment_source": "checking account",
        "payment_date": "06-11-2023"
      },
      "same_call_is_already_staged": false,
      "comparison_with_rejected_tools_including_references_to_subtleties": "This tool is the only one relevant for 
processing the payment.",
      "relevant_subtleties": "Previous attempts failed due to invalid payment dates. A valid future date is now being 
used.",
      "a_rejected_tool_would_have_been_a_better_fit_if_it_werent_already_rejected": false,
      "potentially_better_rejected_tool_name": null,
      "potentially_better_rejected_tool_rationale": null,
      "should_run": true
    }
  ]
}
[DEBUG] [RID(2qVaYUmjPf)::HhVsxU2sSw] [MessageEventGenerator][Completion]
{
  "last_message_of_customer": "200$",
  "produced_reply": true,
  "produced_reply_rationale": "",
  "guidelines": [
    "When user wants to pay credit card bill and has provided the card details, then make the payment"
  ],
  "context_evaluation": {
    "most_recent_customer_inquiries_or_needs": "The customer wants to make a payment of $200 towards their Visa credit 
card ending in 5432.",
    "parts_of_the_context_i_have_here_if_any_with_specific_information_on_how_to_address_these_needs": "The customer has
chosen the Visa card and specified the payment amount. Previous attempts to make the payment failed due to an issue with
the payment date.",
    "topics_for_which_i_have_sufficient_information_and_can_therefore_help_with": "I have the card details and payment 
amount, but I need to ensure the payment date is set correctly.",
    "what_i_do_not_have_enough_information_to_help_with_with_based_on_the_provided_information_that_i_have": "I do not 
have information on how to resolve the payment date error.",
    "was_i_given_specific_information_here_on_how_to_address_some_of_these_specific_needs": true,
    "should_i_tell_the_customer_i_cannot_help_with_some_of_those_needs": true
  },
  "insights": [
    "The payment date needs to be in the future to successfully process the payment.",
    "The customer wants to pay $200 towards their Visa card.",
    "Previous payment attempts failed due to an incorrect payment date."
  ],
  "evaluation_for_each_instruction": [
    {
      "number": 1,
      "instruction": "make the payment",
      "evaluation": "I need to ensure the payment date is correctly set in the future to process the payment.",
      "data_available": "I have the card details and payment amount, but previous attempts failed due to a payment date 
issue."
    }
  ],
  "revisions": [
    {
      "revision_number": 1,
      "content": "It seems there was an issue with the payment date being set in the past. Let's try setting the payment
date for tomorrow, November 7, 2023, to ensure it's in the future. I'll attempt to process the payment of $200 towards 
your Visa card ending in 5432 again.",
      "instructions_followed": [
        "make the payment"
      ],
      "instructions_broken": [],
      "is_repeat_message": false,
      "followed_all_instructions": true,
      "instructions_broken_due_to_missing_data": false,
      "missing_data_rationale": null,
      "instructions_broken_only_due_to_prioritization": false,
      "prioritization_rationale": null
    }
  ]
}

As seen above, the tool caller failed because of the validation step present in card_service:make_payment tool where we assert that the date of payment should be in the future i.e. not today or in the past.

Firstly, the payment date is wrongly made up/assumed by the tool caller. That aside, now look at the next issue. We see that the tool caller gets called and it fails on a certain validation step. This happens for all the three iterations (we have set max engine iterations to 3).

Now, the message generator does not get the fact that the tool caller was called and that the validation for payment date has failed. Knowing this information may help guide the message generation step better is what I think. Any thoughts on this are welcome.

[kichanyurd]

@ElisonSherton Awesome detailed write-up, thank you!

Regarding the second issue, I have implemented a fix for this and it will be released in 1.6.0. It is actually already available for testing in a release-candidate, 1.6.0rc5 (installable by explicitly running pip install parlant==1.6.0rc5).

Regarding the first issue with ToolInsights, this is an extremely useful idea that I think needs to (and will) be a first-class feature in the engine.

I'll find time soon to expand this part into a new discussion with a design proposal for ToolInsights. I will also try to prioritize this feature as I think it's a game-changer that's perfectly aligned with our "guided tool use" paradigm.