AI Agent failing to handle multi-paragraph response from Tools

[YuzuruT]

Bug Description

I am creating a RAG AI Agent and instruct it to respond as "I don't know" when the connected Vector Store Tool cannot provide a good answer. I noticed that some information that I surely am aware that are available in the Vector Store are not being reflected in the Agent's response, and the Agent is prone to respond "I don't know".

Below is my suspicion for WHY this is happening (and hence is the title).
On examining the logs, the Vector Store Tool provides good responses. However, the Tools' responses are occasionally multi-paragraphed containing many line breaks. This seems to cause an awkwardly formatted prompt for the final "Chat Model" step as the AI Agent adds "Human: xxxx, AI: [xxxx], Tool: xxxx. " to the bottom of the prompt to fetch the final answer as a RAG agent.

In my case, the Vector Store Tool responds an extremely rich multi-paragraph text which is directly inserted after the "Tool:" and sent to the Chat Model. Even for a human, it is difficult to understand where exactly is the last word provided by the "Tool" from this prompt. Sadly, in such cases the chat model resorts to responding "I don't know" completely discarding the Tool's findings.

The issue seems to not happen when the text that the Vector Store Tool supplies is small.

Below is my current set up.
[email protected]
Chat Model - Gemini 2.0 Flash 001

To Reproduce

It is difficult to reproduce without a correct prompt and set up. Below are 2.

To reproduce the Observed Issue

1. Set up a RAG Agent with vector data to retrieve. Instruct the AI agent to respond "I don't know" if Tool cannot provide information.
2. Prompt the AI Agent a question that should return a complex answer.
3. See AI Agent respond "I don't know"
4. Check the log and see that the Tool responded rich information, but all that information getting lost in translation at the final Chat Model prompt.

To reproduce the Suspected root cause.

1. Prompt a Chat Model as if the AI Agent has retrieved information from Vector Store, but was in multi-paragraph (rich-format) information. Below code-block is an example using pseudonym.
2. Observe that the Chat Model (likely) answers "I don't know" ignoring the text appended after Tool:.

System: You are a helpful tech support assistant designed to troubleshoot issues that clients are facing with a Honey-crisp operating system.

Retrieve relevant information from the tech-support documents and provide a concise, accurate, and informative answer to help solve the client's issues.

Use the tool "honey_crisp_techsupport_documents" to get the answer to the question.

If the tool cannot provide any information for the inquiry, respond with "I cannot find the answer in the available resources."

If the human is using a language other than English, please try to answer in the same language as the human.


Human: Hi! This is my first contacting you guys for tech support. But one of my PC is giving me a local host error, and the replacement PC, just won't start the program at all. It just sits on the desktop.

AI: [
{
"functionCall": {
"name": "honey_crisp_techsupport_documents",
"args": {
"input": "troubleshooting local host error and program not starting on desktop"
}
}
},
{
"type": "text",
"text": "\n"
},
{
"type": "text",
"text": ""
}
]
Tool: Okay, I can help you troubleshoot the local host error and the program not starting on the desktop.

Here's what we know from the conversation:

**Problem 1: Local Host Error**

* The error message is: "No connection could be made because the target machine actively refused it for local host at a port that was already being used."
* The port number displayed is 1111.
* The error is a .NET Framework error.
* The Honey-crisp software is running in the background when the error appears.
* Exiting out of the error message allows the pairing code to be visible.

**Problem 2: Program Not Starting on Replacement PC**

* The replacement PC boots to the Honey-crisp 4 desktop with the company logo background.
* The technician suspects it might be a server connection issue or an IP conflict.
* There's no video output initially, but restarting the PC doesn't resolve it.

**Troubleshooting Steps Taken**

* Restarting the PC with the local host error did not fix the issue.
* Unplugging the PC with the local host error and plugging it back in temporarily resolved the issue once before.
* The client exited out of the .Net framework error on the local host error, to reveal the pairing code.

Based on the information, here's what the technician suggested and what might be going on:

* **IP Address Conflict:** The technician mentioned the possibility of an IP address conflict, especially if the room isn't set with a static IP.
* **Server Connection:** The technician suspects the replacement PC issue might be related to a server connection problem.
* **Static IP Addresses:** The technician also mentions that the room clients should have static IP addresses.

Expected behavior

If the Tool provided a multi-paragraph answer with rich information, I would expect the chat model to respect that and either relay the exact message or summarize or contextualize it.

Operating System

n/a

n8n Version

1.84.1

Node.js Version

n/a

Database

SQLite (default)

Execution mode

main (default)

[Joffcom]

Hey @YuzuruT,

We have created an internal ticket to look into this which we will be tracking as "GHC-1356"

[Joffcom]

Hey @YuzuruT

Can you provide a workflow that reproduces this, It seems odd that it be an issue with multi paragraph responses as I have seen them work with tools recently.

Have you tried a different model or tweaking your prompt?

If you can share a workflow that reproduces this issue and some screenshots of the ai log we can take a proper look at this.

[YuzuruT]

I have not yet tried other chat models as it requires quite an overhaul to the system with embedding and all for the Vector Storage. I also may need additional licenses. Yet, I am planning to look into it soon.

That being said, it seems very odd that the rich information is lost from the at the last prompt casted by the AI Agent. Copy+Pasting the same prompt into Public Gemini prompt does not lose the clause/context provided by the Tool... So I am not sure what exactly might be causing this.

I also noticed that this might NOT be a issue with Multi-paragraph Tool response. I have been observing a >50% chance of the chat model completely ignoring the text provided from the Tool..

The N8N Workflow is

{
  "name": "Chatbot copy",
  "nodes": [
    {
      "parameters": {
        "options": {
          "systemMessage": "You are a helpful technical assistant designed to provide support for trouble shooting client's issues with the Karaoke System.\n\nUse the tool named \"techsupport_documents\" to retrieve relevant information from the Tech-support documents so that you can provide a concise, accurate, and informative answer to the client's inquiries.\n\nIf the answer cannot be found in the retrieved information, respond with \"I cannot find the answer in the available resources. Would you like one of our technicians to reach out to you?\"\n\n"
        }
      },
      "id": "e1ca7bdb-7c03-461f-b028-66b756ad6c99",
      "name": "AI Agent",
      "type": "@n8n/n8n-nodes-langchain.agent",
      "position": [
        520,
        1400
      ],
      "typeVersion": 1.7
    },
    {
      "parameters": {
        "name": "techsupport_documents",
        "description": "Call this tool to query the tech-support documents"
      },
      "id": "2ea0a555-b02d-4236-b2ae-77f1de860b87",
      "name": "Vector Store Tool",
      "type": "@n8n/n8n-nodes-langchain.toolVectorStore",
      "position": [
        640,
        1740
      ],
      "typeVersion": 1
    },
    {
      "parameters": {
        "pineconeIndex": {
          "__rl": true,
          "value": "techsupport-rag-source",
          "mode": "list",
          "cachedResultName": "techsupport-rag-source"
        },
        "options": {}
      },
      "id": "23946eff-71d1-4f2c-82f0-361c7bc6b3e5",
      "name": "Pinecone Vector Store (Retrieval)",
      "type": "@n8n/n8n-nodes-langchain.vectorStorePinecone",
      "position": [
        500,
        1900
      ],
      "typeVersion": 1,
      "credentials": {
        "pineconeApi": {
          "id": "24hC28ZcX5YWTuZA",
          "name": "PineconeApi account"
        }
      }
    },
    {
      "parameters": {
        "modelName": "models/text-embedding-004"
      },
      "id": "4e0bf795-fdc9-456b-a8a3-99877e4383ae",
      "name": "Embeddings Google Gemini (retrieval)",
      "type": "@n8n/n8n-nodes-langchain.embeddingsGoogleGemini",
      "position": [
        460,
        2060
      ],
      "typeVersion": 1,
      "credentials": {
        "googlePalmApi": {
          "id": "CAsCJH2GFi1bMU2e",
          "name": "Google Gemini(PaLM) Api account"
        }
      }
    },
    {
      "parameters": {},
      "id": "eeadadad-847c-458e-ae17-60e35465bf51",
      "name": "Window Buffer Memory",
      "type": "@n8n/n8n-nodes-langchain.memoryBufferWindow",
      "position": [
        480,
        1720
      ],
      "typeVersion": 1.3
    },
    {
      "parameters": {
        "public": true,
        "mode": "webhook",
        "options": {
          "allowedOrigins": "https://xxxxxxxxx.tech",
          "loadPreviousSession": "notSupported"
        }
      },
      "id": "f49ef83f-b34c-4fda-bc9a-a8c01df52221",
      "name": "When chat message received",
      "type": "@n8n/n8n-nodes-langchain.chatTrigger",
      "position": [
        140,
        1400
      ],
      "webhookId": "fc0d5c12-aea5-4ee7-9537-01faee13ef68",
      "typeVersion": 1.1
    },
    {
      "parameters": {
        "modelName": "models/gemini-2.0-flash-001",
        "options": {}
      },
      "id": "a704f1fb-f1ec-4052-9471-faa2441cbc0c",
      "name": "Google Gemini Chat Model",
      "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
      "position": [
        320,
        1720
      ],
      "typeVersion": 1,
      "credentials": {
        "googlePalmApi": {
          "id": "CAsCJH2GFi1bMU2e",
          "name": "Google Gemini(PaLM) Api account"
        }
      }
    },
    {
      "parameters": {
        "modelName": "models/gemini-2.0-flash-001",
        "options": {}
      },
      "id": "e3b53dd9-6cec-4fda-a378-7c5ad0fb6b84",
      "name": "Google Gemini Chat Model (retrieval)",
      "type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
      "position": [
        820,
        1900
      ],
      "typeVersion": 1,
      "credentials": {
        "googlePalmApi": {
          "id": "CAsCJH2GFi1bMU2e",
          "name": "Google Gemini(PaLM) Api account"
        }
      }
    }
  ],
  "pinData": {},
  "connections": {
    "AI Agent": {
      "main": [
        []
      ]
    },
    "Vector Store Tool": {
      "ai_tool": [
        [
          {
            "node": "AI Agent",
            "type": "ai_tool",
            "index": 0
          }
        ]
      ]
    },
    "Window Buffer Memory": {
      "ai_memory": [
        [
          {
            "node": "AI Agent",
            "type": "ai_memory",
            "index": 0
          }
        ]
      ]
    },
    "Google Gemini Chat Model": {
      "ai_languageModel": [
        [
          {
            "node": "AI Agent",
            "type": "ai_languageModel",
            "index": 0
          }
        ]
      ]
    },
    "When chat message received": {
      "main": [
        [
          {
            "node": "AI Agent",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Pinecone Vector Store (Retrieval)": {
      "ai_vectorStore": [
        [
          {
            "node": "Vector Store Tool",
            "type": "ai_vectorStore",
            "index": 0
          }
        ]
      ]
    },
    "Embeddings Google Gemini (retrieval)": {
      "ai_embedding": [
        [
          {
            "node": "Pinecone Vector Store (Retrieval)",
            "type": "ai_embedding",
            "index": 0
          }
        ]
      ]
    },
    "Google Gemini Chat Model (retrieval)": {
      "ai_languageModel": [
        [
          {
            "node": "Vector Store Tool",
            "type": "ai_languageModel",
            "index": 0
          }
        ]
      ]
    }
  },
  "active": false,
  "settings": {},
  "versionId": "842c88fc-1414-4a00-8637-6af5c748db6e",
  "meta": {
    "templateId": "2753",
    "templateCredsSetupCompleted": true,
    "instanceId": "1db3826a53944b70bd3663b3d3df73cf686a59b54e518109bc3737fab1dd588f"
  },
  "id": "xTZti9tpIPWEANt9",
}

The final prompt sent to the ChatModel is as follows in JSON format

{
  "messages": [
    "System: You are a helpful technical assistant designed to provide support for trouble shooting client's issues with the Karaoke System.\n\nUse the tool named \"techsupport_documents\" to retrieve relevant information from the Tech-support documents so that you can provide a concise, accurate, and informative answer to the client's inquiries.\n\nIf the answer cannot be found in the retrieved information, respond with \"I cannot find the answer in the available resources. Would you like one of our technicians to reach out to you?\"\n\n\nHuman:  Hi!  This is my first contacting you guys for tech support. But one of my Karaoke rooms that is, giving me a local post error, and the replacement PC, just won't start the program at all. It just sits on the desktop.\nAI: model, [\n  {\n    \"functionCall\": {\n      \"name\": \"techsupport_documents\",\n      \"args\": {\n        \"input\": \"troubleshooting local post error and program not starting on desktop\"\n      }\n    }\n  }\n]\nTool: Okay, so you're experiencing a \"local host error\" on one Karaoke room PC, and a replacement PC isn't starting the program at all.\n\nHere's a breakdown of the troubleshooting steps discussed, and the current status:\n\n**Problem 1: Local Host Error**\n\n*   The error message indicates \"no connection could be made because the target machine actively refused it\" for the local host at a specific port. The port number mentioned was 9999.\n*   This is a .NET framework error, appearing on top of the Karaoke software.\n*   The technician suggested that restarting sometimes fixes the issue, but isn't sure why.\n*   The client mentioned that the room may not have a static IP, causing it to change and potentially leading to the error.\n\n**Problem 2: Replacement PC Not Starting**\n\n*   The replacement PC boots to Windows 11 with the company background, but the Karaoke program won't start when clicking the shortcut.\n*   The program attempts to start (dll loads), but then closes silently.\n*   The technician suspects a server connection issue or an IP conflict.\n*   Currently, the replacement PC is only showing a black screen, but the TV is getting a signal.\n\n**Resolution:**\n\n*   The client was able to exit out of the error message on the PC with the local host error. After quitting out of the error, the client was able to see the pairing code."
  ],
  "estimatedTokens": 496,
  "options": {
    "google_api_key": {
      "lc": 1,
      "type": "secret",
      "id": [
        "GOOGLE_API_KEY"
      ]
    },
    "base_url": "https://generativelanguage.googleapis.com",
    "model_name": "gemini-2.0-flash-001"
  }
}

Below is a set of screenshots taken from the Logs panel of the AI agent node. Specifically where the context provided by the Tool being lost.

[YuzuruT]

To follow up with few of the unanswered questions.

Yes, I have tried various framing of prompts.

Another anecdotal observation is that the probability of failure to answer seems to increase as the interface moves away from the node. I.e., "Test step" works better than the n8n built-in chat widget, and the n8n built-in chat widget seems to perform better than prompts sent from the Web deployed UI. Here, "perform better" means the probability of opting to the "I don't know" messaging is lower. - Since the behavior or very indeterministic, please take this with a grain of salt. But if this tendency continues, I may look into pull some stats to open a different bug ticket.

[YuzuruT]

Update.
I was able to find this thread in the community (https://community.n8n.io/t/gemini-rag-workflow-returning-odd-response/66487/4), and switching the AI Agent from Tool Agent to Conversational Agent significantly improved the performance.

Although I am not completely convinced that this workaround should be the resolution, please feel free to close this case if the issue will not be prioritized. Thank you so much for all your work!! A huge fan of n8n!!

[Fank]

I have exactly the same issue, that the Tool Agent (1.85.4) returns following JSON instead of calling any tools:

{
  "output": [
    {
      "type": "text",
      "text": "Okay, ich werde die Details für die Setköpfe 1 und 2 im Angebot 161000 (ID: 101060) nachschlagen.\n\n"
    },
    {
      "functionCall": {
        "name": "get_offer_positions",
        "args": {
          "OfferPosititionNO": "1",
          "ProductNumber": "",
          "OfferID": "101060"
        }
      }
    },
    {
      "functionCall": {
        "name": "get_offer_positions",
        "args": {
          "ProductNumber": "",
          "OfferID": "101060",
          "OfferPosititionNO": "2"
        }
      }
    }
  ]
}

Which causes following issue using chat trigger:

Execution:

In my case switchting to Claude fixed my issue.

Maybe related to #14304 & #13807