AstraDB component Ingest all items twice

[thomasbachet]

As I run the flow to ingest data, it does it twice. So each item is captured twice in the AstraDB data collection.

Browser and Version

• Langflow version 1.0.6 running on local a virtual environment

To Reproduce

• use the pre-built flow 'Vector Store RAG'
• use the existing flow to ingest any content

Does anyone that experience the same?

[dosubot]

Hey @thomasbachet! I'm here to help you with any issues you're experiencing. I can assist with bugs, answer questions, and guide you on becoming a contributor. Let's work together to resolve any issues you may have.

The issue of the AstraDB component ingesting items twice could be related to the caching mechanism within the AstraDBVectorStoreComponent. The component's configuration includes fields with "cache": true, which might cause the same data to be ingested twice ^[1].

Additionally, the concurrency settings in the AstraDBVectorStoreComponent could potentially lead to duplicate entries if the same data is processed simultaneously by different threads or processes. The component handles concurrency through parameters like bulk_insert_batch_concurrency, bulk_insert_overwrite_concurrency, and bulk_delete_concurrency, but it does not explicitly address deduplication logic ^[2].

To mitigate this issue, you might want to review the caching settings and ensure that there is a mechanism to check for existing documents before adding new ones.

To continue talking to Dosu, mention @dosu.

[nicoloboschi]

Hey @thomasbachet thanks for reporting. Yes it's a bug. I've fixed it in this PR #2573
Meanwhile you could replace the entire astradb component with this content: https://raw.githubusercontent.com/langflow-ai/langflow/98beda5532b9029e016d966cfc81bca1722d4c5c/src/backend/base/langflow/components/vectorstores/AstraDB.py