Custom Plugin in Chat-Copilot Fetching Outdated Data Instead of Up-to-Date Results

[PBitukunti]

Describe the bug
When enabling my custom plugin through Chat-Copilot and running it, the plugin doesn't provide up-to-date results. Specifically, I've created a custom plugin named "Grocery App Plugin," allowing users to add, list, and remove items. While using Swagger to add items to the list, I can retrieve the complete list of items. However, when running the same plugin through Chat-Copilot, it seems to fetch outdated data, appearing as if the planner is providing historical details. I'm utilizing the "Action" Planner /"Sequential" Planner with the gpt-35-turbo model.

To Reproduce
Steps to reproduce the behavior:
1.Enable custom plugin -"Grocery App Plugin"
2.List the grocery items using "Grocery App Plugin."
3.Attempt to add an item using the plugin or Swagger
4.Run the plugin to list the items.
5.Compare the listed items with the actual up-to-date list.

Expected behavior
I expected the plugin to fetch and display the most recent data when using Chat-Copilot, similar to the behavior when interacting with Swagger directly. This would ensure that users receive accurate and current information from the plugin.

Screenshots
Please find the screenshots of the outdated results from Chat-Copilot and the expected up-to-date results from Swagger.

Platform

• OS: Windows
• IDE: VS Code
• Language: C#, Python
• Source: Main branch of repository: https://github.com/microsoft/chat-copilot.git

Additional context
I noticed that this issue is specific to Chat-Copilot's interaction with my custom plugin. Swagger integration seems to work correctly and provides real-time updates. This context could potentially help in identifying whether the problem lies within my plugin's implementation or Chat-Copilot's integration with both "Action"/ "Sequential" Planner along with the gpt-35-turbo model.

[crickman]

Thank you for reporting this. I'm happy to review chat-copilot's interaction with custom-plug-ins for result handling and if additional logging might be helpful.

I'm wondering if the model is cosolidating test, Test, and Test5 as a single logical item: Test. Is it possible to provide the prompt metrics (click the "information" icon in the chat bubble) and the short/long term memory values from the persona tab. This may help determine if actual values are provided and if the model then consolidates.

Another test might be to add distinct items such as Cheese, Celery, Catfood, etc...

I look forward to hearing back from you and will also investigate on this end.

[PBitukunti]

@crickman - Thank you for your prompt response. I appreciate your suggestion to explore whether the model is consolidating similar terms and to examine the prompt metrics and memory values.

I have conducted the tests as you've suggested. I created distinct items such as 'Cheese,' 'Celery,' and other 'Test' to ensure the items are unique. Despite this, the issue persists, and the plugin still fetches outdated data when interacting with Chat-Copilot.

I have attached the screenshots showing the error. The first screenshot demonstrates the outdated results from Chat-Copilot, while the second screenshot displays the prompt metrics and expected up-to-date results from Swagger for comparison. The third and fourth screenshots showcases the additional tests.

In addition, I've captured the short/long term memory values from the persona tab. I hope these details will be helpful in your investigation.

Please let me know if there's any other information or steps you'd like me to take to assist in resolving this issue. Your expertise is greatly valued, and I'm looking forward to your insights.

Thank you once again for your assistance.

[crickman]

Wonderful, thank you for these additional data-points. I aim to complete my analysis today.

[crickman]

I was able to reproduce your report with a function that adds a new guid to the list upon each invocation:

In my case, the "cached" value is being derived from the chat history. I suspect something similiar is occuring for you while the function result is being captured as expected. I am a little uncertain as to which prompt details you've shared...is that the planner prompt? I'd expect the prompt related to the plan/function/plug-in execution to contain the function results.

In anycase, let me raise this to the team for ideas.

[crickman]

I think we may have a work-around to try...unclear how effective it may be.